Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

feat(router): add escalation_router profile with judge-latched strong/weak routing (#62)

Lin Jiab03b6a6fed08
diff --git a/.agents/skills/switchyard-lib-core/SKILL.md b/.agents/skills/switchyard-lib-core/SKILL.md
index 17f7409d4..c2812ebaf 100644
--- a/.agents/skills/switchyard-lib-core/SKILL.md
+++ b/.agents/skills/switchyard-lib-core/SKILL.md
@@ -39,7 +39,7 @@ right validation set. If the change is driven by a launcher need, also read
| An OpenAI-compatible provider target such as NVIDIA Inference Hub or OpenRouter | Use the existing OpenAI-compatible backend/profile with `base_url`, `api_key`, and model id wiring. Add a new backend only when the provider has a real wire-format, auth, retry, or health contract that cannot fit that path. |
| Direct Rust component bindings | Add concrete PyO3 classes under `crates/switchyard-py/src/component_bindings/`, keep config bindings near the component binding that consumes them, and expose them lazily from `switchyard_rust/components.py`. Do not keep growing `core_bindings.rs` or `switchyard_rust/core.py` with concrete component classes. |
| Route YAML / model dispatch | Use `switchyard/cli/route_bundle.py` and `switchyard/lib/route_table_builders.py`. They build `RouteTable` entries from profile-backed runtimes and keep launchers plus `switchyard serve --routing-profiles` on one path. |
-| Shared/persistent session-affinity pins across workers or pod churn | Configure the latency route with `session_affinity: true` + `affinity_store: redis` + `affinity_store_url` (optional `affinity_store_ttl_seconds`, `affinity_key_prefix`). `SessionAffinity` keeps the Rust `SessionCache` as L1 and reads/writes through the `AffinityPinStore` L2 (`switchyard/lib/redis_pin_store.py`), fail-open behind a 0.1s socket timeout and a 3-failure/10s-cooldown circuit breaker (`switchyard_affinity_l2_breaker_open` gauge). Requires the `switchyard[affinity-redis]` extra. |
+| Shared/persistent session-affinity pins across workers or pod churn | Configure the latency route with `session_affinity: true` + `affinity_store: redis` + `affinity_store_url` (optional `affinity_store_ttl_seconds`, `affinity_key_prefix`); the escalation_router route takes the same `affinity_store*` keys (no `session_affinity` flag — its latch is always on; default prefix `swyd:esc:`). `SessionAffinity` keeps the Rust `SessionCache` as L1 and reads/writes through the `AffinityPinStore` L2 (`switchyard/lib/redis_pin_store.py`), fail-open behind a 0.1s socket timeout and a 3-failure/10s-cooldown circuit breaker (`switchyard_affinity_l2_breaker_open` gauge). Requires the `switchyard[affinity-redis]` extra. |
| Stats / telemetry | Reuse `StatsRequestProcessor`, `StatsResponseProcessor`, `StatsLlmBackend`, and `StatsAccumulator`. A profile config should thread one accumulator through all three when stats are enabled. Do not write a parallel collector. |
| A fixed-path endpoint contributed by per-route components | Set `Endpoint.register_once = True`; `build_switchyard_app(...)` mounts the first instance while still running every component's lifecycle. Leave the default `False` for configurable endpoint classes that may mount distinct instances. |
| Per-endpoint attribution on `/metrics` for a Python backend that can't be wrapped by `StatsLlmBackend` | Set `ctx.selected_model = endpoint_id` before returning the response. Also set `ctx.backend_call_latency_ms = upstream_call_ms` so the response processor can compute routing overhead. `LatencyServiceLLMBackend.call` is the reference. |
@@ -62,9 +62,9 @@ Profile-owned construction keeps the full behavior in one reviewable module:
5. Use `ProfileSwitchyard(config.build())` only when serving through the
existing Python endpoint contract.
-Random routing, deterministic routing, plan-execute, passthrough, no-op,
-stage-router, latency-service, RouteLLM, and OSS-router profiles are the reference
-set under `switchyard/lib/profiles/`.
+Random routing, deterministic routing, escalation-router, plan-execute,
+passthrough, no-op, stage-router, latency-service, RouteLLM, and OSS-router
+profiles are the reference set under `switchyard/lib/profiles/`.
## Anti-Patterns
@@ -76,6 +76,7 @@ set under `switchyard/lib/profiles/`.
| A new role-shaped abstraction invented outside the backend role. | Keep pre-call logic in request-side profile code or plain request components, call logic in `LLMBackend`, post-call logic in response-side profile code or plain response components, and final wire conversion in `TranslationEngine`. |
| Adding an OpenRouter-specific backend or translator just to point at `https://openrouter.ai/api/v1`. | Route it through the OpenAI-compatible backend/profile and provider configuration. Keep provider-specific code for actual protocol differences. |
| Making route YAML declare arbitrary Python processors. | Route YAML is deployment config for supported profile route types. Runtime-only hooks such as intake are injected by the caller through the route-table builder kwargs. |
+| Hand-billing token costs from raw stats fields — charging `prompt_tokens` at the fresh-input rate and adding `cached_tokens` on top. | `prompt_tokens` in stats snapshots is **total** input: `uncached + cached_tokens + cache_creation_tokens` (Anthropic's three sibling usage counters are normalized into it). Fresh input = `prompt − cached − creation`. Use the built-in estimators — the snapshot's own `cost_estimate`, `estimate_model_cost()` in `cost_estimator.py`, or Rust `stats/cost.rs` — instead of ad-hoc math; ignoring the subtraction overstates cost ~5× on cache-heavy agentic traffic. |
## When Something Genuinely Doesn't Fit
diff --git a/crates/switchyard-components/src/stats/cost.rs b/crates/switchyard-components/src/stats/cost.rs
index 1e21a636c..2b8d89282 100644
--- a/crates/switchyard-components/src/stats/cost.rs
+++ b/crates/switchyard-components/src/stats/cost.rs
@@ -189,11 +189,28 @@ fn raw_model_price(model: &str) -> Option<ModelPrice> {
cached: 0.15,
cache_write: 1.50,
},
- "aws/anthropic/bedrock-claude-opus-4-7"
+ // Nemotron 3 Ultra (550B/55B MoE) — OpenRouter reference list
+ // price, June 2026. evals-N ids are parallel benchmarking-gateway
+ // deployments of the same model.
+ "nvidia/nvidia/nemotron-3-ultra"
+ | "openai/nvidia/nvidia/nemotron-3-ultra"
+ | "nvidia/nvidia/evals-nemotron-ultra"
+ | "nvidia/nvidia/evals-nemotron-ultra-2"
+ | "nvidia/nvidia/evals-nemotron-ultra-3"
+ | "nvidia/nvidia/evals-nemotron-ultra-4" => ModelPrice {
+ input: 0.50,
+ output: 2.20,
+ cached: 0.05,
+ cache_write: 0.50,
+ },
+ "aws/anthropic/bedrock-claude-opus-4-8"
+ | "aws/anthropic/bedrock-claude-opus-4-7"
| "aws/anthropic/bedrock-claude-opus-4-6"
| "aws/anthropic/bedrock-claude-opus-4-5"
+ | "azure/anthropic/claude-opus-4-8"
| "azure/anthropic/claude-opus-4-7"
| "azure/anthropic/claude-opus-4-6"
+ | "claude-opus-4-8"
| "claude-opus-4-7"
| "claude-opus-4-6"
| "claude-opus-4-5" => ModelPrice {
@@ -204,6 +221,7 @@ fn raw_model_price(model: &str) -> Option<ModelPrice> {
},
"aws/anthropic/bedrock-claude-sonnet-4-6"
| "aws/anthropic/bedrock-claude-sonnet-4-5"
+ | "azure/anthropic/claude-sonnet-4-5"
| "claude-sonnet-4-6"
| "claude-sonnet-4-5" => ModelPrice {
input: 3.00,
diff --git a/crates/switchyard-core/src/session.rs b/crates/switchyard-core/src/session.rs
index c5c13698c..0e80b8d8c 100644
--- a/crates/switchyard-core/src/session.rs
+++ b/crates/switchyard-core/src/session.rs
@@ -17,6 +17,28 @@ use serde_json::Value;
/// message — so every turn of a conversation shares a key while distinct
/// conversations differ. Returns a 16-char lowercase hex string.
pub fn session_key_from_body(body: &Value) -> String {
+ hash_conversation_prefix(body, 0).0
+}
+
+/// Deep variant of [`session_key_from_body`]: extend the hashed prefix with
+/// the first `depth` messages after the first user message, so repeated runs
+/// of an identical task diverge via early model responses (repeated-trial
+/// benchmarking).
+///
+/// Returns `None` until the conversation actually contains a first user
+/// message plus `depth` later messages — a shorter prefix would hash to a key
+/// that later turns of the same conversation no longer match. `depth == 0` is
+/// exactly the base key and always yields `Some`.
+pub fn session_key_from_body_with_depth(body: &Value, depth: usize) -> Option<String> {
+ let (key, complete) = hash_conversation_prefix(body, depth);
+ complete.then_some(key)
+}
+
+/// Shared hashing core for both key variants: anchors (top-level system,
+/// in-list system/developer messages, first user message) plus the first
+/// `depth` non-system messages after the first user. Returns the key and
+/// whether the requested prefix was complete (always true at depth 0).
+fn hash_conversation_prefix(body: &Value, depth: usize) -> (String, bool) {
let mut hasher = DefaultHasher::new();
// Anthropic carries the system prompt at the top level.
@@ -25,23 +47,31 @@ pub fn session_key_from_body(body: &Value) -> String {
// OpenAI uses "messages"; the Responses API uses "input". A messages list
// with no user message falls through to "input".
let mut anchored = false;
+ let mut tail_taken = 0usize;
for seq_key in ["messages", "input"] {
if let Some(Value::Array(items)) = body.get(seq_key) {
for item in items {
- let Some(role) = item.get("role").and_then(Value::as_str) else {
- continue;
- };
- match role {
- "system" | "developer" => {
+ match item.get("role").and_then(Value::as_str) {
+ Some("system") | Some("developer") => {
flatten_text(item.get("content")).hash(&mut hasher);
}
- "user" => {
+ Some("user") if !anchored => {
flatten_text(item.get("content")).hash(&mut hasher);
anchored = true;
- break;
+ }
+ // Post-anchor tail (any non-system item, roleless
+ // Responses items included): hash the full message —
+ // tool calls too, so assistant turns that differ only
+ // in tool calls still diverge the deep key.
+ _ if anchored && tail_taken < depth => {
+ flatten_message_text(item).hash(&mut hasher);
+ tail_taken += 1;
}
_ => {}
}
+ if anchored && tail_taken >= depth {
+ break;
+ }
}
}
if anchored {
@@ -49,7 +79,54 @@ pub fn session_key_from_body(body: &Value) -> String {
}
}
- format!("{:016x}", hasher.finish())
+ let complete = depth == 0 || (anchored && tail_taken >= depth);
+ (format!("{:016x}", hasher.finish()), complete)
+}
+
+/// Flatten a whole message for deep-key hashing: `content` text plus tool-call
+/// payloads (OpenAI chat `tool_calls`, Responses `function_call` items) and
+/// tool `output`. Anchors keep hashing `content` only — this richer form is
+/// applied to post-anchor tail messages, where the divergence signal between
+/// repeated trials often lives entirely in the tool calls.
+fn flatten_message_text(item: &Value) -> String {
+ let mut parts: Vec<String> = Vec::new();
+ let content = flatten_text(item.get("content"));
+ if !content.is_empty() {
+ parts.push(content);
+ }
+ // Anthropic tool_use blocks carry their payload in name/input rather than
+ // text/content, so the plain content flatten yields nothing for them.
+ if let Some(Value::Array(blocks)) = item.get("content") {
+ for block in blocks {
+ if block.get("type").and_then(Value::as_str) == Some("tool_use") {
+ let name = block.get("name").and_then(Value::as_str).unwrap_or("");
+ let input = block.get("input").map(Value::to_string).unwrap_or_default();
+ parts.push(format!("tool_call {name}({input})"));
+ }
+ }
+ }
+ if let Some(Value::Array(calls)) = item.get("tool_calls") {
+ for call in calls {
+ if let Some(function) = call.get("function").and_then(Value::as_object) {
+ let name = function.get("name").and_then(Value::as_str).unwrap_or("");
+ let arguments = function
+ .get("arguments")
+ .and_then(Value::as_str)
+ .unwrap_or("");
+ parts.push(format!("tool_call {name}({arguments})"));
+ }
+ }
+ }
+ if item.get("type").and_then(Value::as_str) == Some("function_call") {
+ let name = item.get("name").and_then(Value::as_str).unwrap_or("");
+ let arguments = item.get("arguments").and_then(Value::as_str).unwrap_or("");
+ parts.push(format!("tool_call {name}({arguments})"));
+ }
+ let output = flatten_text(item.get("output"));
+ if !output.is_empty() {
+ parts.push(output);
+ }
+ parts.join(" ")
}
/// Flatten a message-content value into a single string for hashing: strings
@@ -244,6 +321,117 @@ mod tests {
assert!(key.chars().all(|c| c.is_ascii_hexdigit()));
}
+ #[test]
+ fn deep_key_depth_zero_matches_base() {
+ let body = json!({
+ "system": "you are helpful",
+ "messages": [{"role": "user", "content": "first question"}],
+ });
+ assert_eq!(
+ session_key_from_body_with_depth(&body, 0),
+ Some(session_key_from_body(&body))
+ );
+ }
+
+ #[test]
+ fn deep_key_diverges_on_early_responses() {
+ // Identical anchors, different first assistant response: the deep key
+ // separates the trials while the base key does not.
+ let trial = |first_response: &str| {
+ json!({
+ "messages": [
+ {"role": "user", "content": "identical task text"},
+ {"role": "assistant", "content": first_response},
+ ],
+ })
+ };
+ let a = trial("read the tests first");
+ let b = trial("look at the repo layout");
+ assert_eq!(session_key_from_body(&a), session_key_from_body(&b));
+ assert_ne!(
+ session_key_from_body_with_depth(&a, 1),
+ session_key_from_body_with_depth(&b, 1)
+ );
+ }
+
+ #[test]
+ fn deep_key_stable_as_conversation_grows() {
+ // The deep key hashes a fixed prefix, so it survives appended turns.
+ let base = json!({
+ "messages": [
+ {"role": "user", "content": "task"},
+ {"role": "assistant", "content": "first response"},
+ ],
+ });
+ let grown = json!({
+ "messages": [
+ {"role": "user", "content": "task"},
+ {"role": "assistant", "content": "first response"},
+ {"role": "user", "content": "tool result"},
+ {"role": "assistant", "content": "much later turn"},
+ ],
+ });
+ assert_eq!(
+ session_key_from_body_with_depth(&base, 1),
+ session_key_from_body_with_depth(&grown, 1)
+ );
+ }
+
+ #[test]
+ fn deep_key_none_until_prefix_complete() {
+ // One post-user message can't satisfy depth 2; no user at all can't
+ // satisfy any positive depth.
+ let short = json!({
+ "messages": [
+ {"role": "user", "content": "task"},
+ {"role": "assistant", "content": "first response"},
+ ],
+ });
+ assert_eq!(session_key_from_body_with_depth(&short, 2), None);
+ let unanchored = json!({"messages": [{"role": "system", "content": "sys"}]});
+ assert_eq!(session_key_from_body_with_depth(&unanchored, 1), None);
+ }
+
+ #[test]
+ fn deep_key_sees_tool_calls() {
+ // Assistant turns whose divergence lives entirely in tool calls
+ // (empty content) must still separate the trials.
+ let trial = |arguments: &str| {
+ json!({
+ "messages": [
+ {"role": "user", "content": "task"},
+ {"role": "assistant", "content": null, "tool_calls": [
+ {"function": {"name": "read_file", "arguments": arguments}},
+ ]},
+ ],
+ })
+ };
+ assert_ne!(
+ session_key_from_body_with_depth(&trial("{\"path\": \"a.rs\"}"), 1),
+ session_key_from_body_with_depth(&trial("{\"path\": \"b.rs\"}"), 1)
+ );
+ }
+
+ #[test]
+ fn deep_key_sees_anthropic_tool_use() {
+ // Anthropic assistant turns whose divergence lives in tool_use blocks
+ // (no text content) must still separate the trials.
+ let trial = |command: &str| {
+ json!({
+ "messages": [
+ {"role": "user", "content": "task"},
+ {"role": "assistant", "content": [
+ {"type": "tool_use", "name": "bash", "input": {"command": command}},
+ ]},
+ ],
+ })
+ };
+ assert_ne!(
+ session_key_from_body_with_depth(&trial("ls tests/"), 1),
+ session_key_from_body_with_depth(&trial("cat README.md"), 1)
+ );
+ }
+
#[test]
fn session_key_uses_input_when_messages_has_no_user() {
// A messages list with no user message falls through to `input`; the
diff --git a/crates/switchyard-py/src/core_bindings/session.rs b/crates/switchyard-py/src/core_bindings/session.rs
index a1c5d58a9..f5114ee39 100644
--- a/crates/switchyard-py/src/core_bindings/session.rs
+++ b/crates/switchyard-py/src/core_bindings/session.rs
@@ -11,12 +11,19 @@ use crate::py_serde::value_from_python;
///
/// Accepts any Python mapping (e.g. a request body dict) and returns the key
/// the core derivation produces for it, used to pin requests of one session to
-/// a previously selected model.
+/// a previously selected model. `depth == 0` (default) always yields a key;
+/// `depth > 0` extends the hashed prefix with the first `depth` post-first-user
+/// messages and returns `None` until that prefix exists.
#[pyfunction]
-pub(crate) fn session_key_from_body(body: &Bound<'_, PyAny>) -> PyResult<String> {
- Ok(switchyard_core::session_key_from_body(&value_from_python(
- body,
- )?))
+#[pyo3(signature = (body, depth = 0))]
+pub(crate) fn session_key_from_body(
+ body: &Bound<'_, PyAny>,
+ depth: usize,
+) -> PyResult<Option<String>> {
+ Ok(switchyard_core::session_key_from_body_with_depth(
+ &value_from_python(body)?,
+ depth,
+ ))
}
/// LRU-bounded cache mapping session keys to arbitrary Python values.
diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.md
new file mode 100644
index 000000000..d92fd36a4
--- /dev/null
+++ b/docs/routing_algorithms/escalation_router_routing.md
@@ -0,0 +1,206 @@
+# Escalation-Router Routing
+
+Escalation-router routing starts each conversation on a cheaper `weak` model.
+An LLM judge watches how the work progresses and moves the conversation to a
+more capable `strong` model when it detects sustained, recoverable trouble.
+After escalation, the conversation stays on the strong tier for the rest of
+the task.
+
+Use it for multi-turn agent workloads where a weak model can handle routine
+work but may need rescue after repeated errors, loops, drift, false progress,
+or premature completion. Unlike
+[LLM Classifier Routing](llm_classifier_routing.md), which predicts how
+difficult a request looks, escalation routing judges whether the run is
+actually going well.
+
+## How it works
+
+The router keeps one state value per conversation: whether that conversation
+has escalated. Conversations are identified from the stable system prompt and
+first user message, using the same bounded in-memory store described in
+[Sticky Routing](sticky_routing.md).
+
+For each turn, Switchyard:
+
+1. Checks the conversation latch. A latched conversation routes to `strong`
+ without calling the judge again.
+2. Routes an unlatched conversation to `weak`. Before `judge.min_turn`
+ (default `3`), it skips the judge because there is too little trajectory to
+ assess.
+3. Gives the judge a bounded summary containing the system and first-user
+ anchors, recent messages, and a coverage note for omitted history.
+4. Parses the judge's structured `escalate` decision. When the configured
+ confirmation policy is satisfied, Switchyard pins the conversation to
+ `strong` and uses the strong model for the current turn.
+5. Fails open to `weak` when the judge times out, errors, or returns invalid
+ output. A judge failure never creates a strong-tier pin.
+
+The routing decision for one turn is:
+
+```mermaid
+%%{init: {"flowchart": {"nodeSpacing": 18, "rankSpacing": 26}}}%%
+flowchart LR
+ t["turn"] --> p{"strong tier pinned?"}
+ p -->|yes| s["route strong; skip judge"]
+ p -->|no| m{"turn >= min_turn?"}
+ m -->|no| w["route weak"]
+ m -->|yes| j["judge recent trajectory"]
+ j -->|stay / fail open| w
+ j -->|not yet confirmed| w
+ j -->|confirmed escalation| l["pin and route strong"]
+
+ classDef box font-family:monospace,fill:none,stroke:#9aa0a6,stroke-width:1px;
+ class t,p,s,m,w,j,l box;
+```
+
+## Confirmation policy
+
+`judge.confirmations` controls how many positive verdicts are required before
+the strong-tier latch fires:
+
+- `confirmations: 1` is the default and escalates on the first positive
+ verdict.
+- `confirmations: 2` with `confirmation_window: 1` requires positive verdicts
+ on consecutive judged turns.
+- A larger `confirmation_window` allows recurring trouble to confirm even when
+ negative verdicts occur between positive ones. A positive verdict remains
+ live across at most `confirmation_window - 1` intervening negative verdicts.
+
+A judge failure provides no evidence either way: it routes the turn to weak
+without clearing an existing confirmation streak.
+
+## Configure an escalation route
+
+Escalation routing is currently available through the legacy `routes:` bundle
+used by `--routing-profiles`. This path remains necessary for launcher-owned
+routing; route bundles and `--routing-profiles` are otherwise deprecated in
+favor of `switchyard serve --config` profiles.
+
+```yaml
+routes:
+ agent-escalation:
+ type: escalation_router
+ fallback_target_on_evict: strong
+ judge:
+ model: google/gemini-3.5-flash
+ api_key: ${OPENROUTER_API_KEY}
+ base_url: https://openrouter.ai/api/v1
+ timeout_secs: 5.0
+ min_turn: 3
+ confirmations: 1
+ confirmation_window: 1
+ recent_turn_window: 14
+ weak:
+ model: moonshotai/kimi-k2.6
+ api_key: ${OPENROUTER_API_KEY}
+ base_url: https://openrouter.ai/api/v1
+ strong:
+ model: anthropic/claude-opus-4.7
+ api_key: ${OPENROUTER_API_KEY}
+ base_url: https://openrouter.ai/api/v1
+```
+
+Run the route as a standalone proxy:
+
+```bash
+switchyard --routing-profiles routes.yaml -- serve --port 4000
+```
+
+Or use the same bundle with an agent launcher:
+
+```bash
+switchyard --routing-profiles routes.yaml -- launch claude
+```
+
+The route ID (`agent-escalation`) is the model ID clients select to use the
+router. The strong and weak model IDs are also registered as direct
+passthrough choices. The judge is internal to the route and is not exposed as
+a client-selectable model.
+
+If the selected tier exceeds its context window, Switchyard retries once on
+`fallback_target_on_evict`, which must match one of the configured tier ids
+(`strong` / `weak` unless the targets set their own `id`). See
+[Context-Window Handling](../operations/context_window.md).
+
+## Useful options
+
+| Option | Default | Use it when |
+|---|---|---|
+| `judge.timeout_secs` | `5.0` | The judge needs a different wall-clock limit. Timeouts fail open to weak. |
+| `judge.min_turn` | `3` | The judge should start earlier or wait for more trajectory evidence. |
+| `judge.confirmations` | `1` | One positive verdict is too eager and escalation should require repeated evidence. |
+| `judge.confirmation_window` | `1` | Intermittent trouble should remain eligible for confirmation across negative verdicts. |
+| `judge.disable_reasoning` | `true` | Set to `false` when a reasoning judge benefits from thinking despite the added latency. The thinking-off hint is only sent to models that accept it (Claude/Bedrock judges never receive it). |
+| `judge.max_completion_tokens` | auto | The judge's completion budget needs an explicit value. The default follows the reasoning mode: `128` when the thinking-off hint is sent, `4096` otherwise (reasoning tokens and the JSON verdict share the budget; a too-small cap silently fails open every turn). Judges that never accept the hint (Claude/Bedrock) also get the larger ceiling — it is a cap, not spend. |
+| `judge.dump_verdicts` | `false` | Benchmark runs need per-turn `escalation_verdict=` stderr lines (includes a first-user `task_hint` snippet). Keep off in production; routing telemetry flows through the stats endpoints. |
+| `judge.recent_turn_window` | `14` | The judge needs a wider or narrower trailing-message window. |
+| `judge.window_message_chars` | `300` | More tool-output detail should survive per-message truncation. |
+| `judge.max_request_chars` | `12000` | The complete judge request needs a different character budget. Oldest recent messages are dropped first. |
+| `judge.prompt` | built-in | A deployment needs a custom escalation rubric inlined in the YAML. |
+| `judge.prompt_path` | built-in | The custom rubric lives in a file (mutually exclusive with `judge.prompt`; relative paths resolve against the server's working directory). |
+| `tier_timeout_s` | `600` | Strong or weak targets without their own `timeout_secs` need a different call timeout. |
+| `enable_stats` | `true` | Set to `false` only when route-level usage statistics are not needed. |
+| `affinity_max_sessions` | `10000` | A long-lived process needs a different in-memory latch capacity. |
+| `affinity_store` | `memory` | Set to `redis` to share the escalation latch across workers and keep it across pod churn; with `memory`, a worker change silently restarts an escalated conversation on weak. Best-effort — a store error never fails a request. |
+| `affinity_store_url` | — | Connection URL for the shared latch store (e.g. `redis://host:6379/0`); required with `affinity_store: redis`. |
+| `affinity_store_ttl_seconds` | `3600` | A shared latch entry should expire on a different horizon. |
+| `affinity_key_prefix` | `swyd:esc:` | Multiple deployments share one Redis and need namespaced keys. |
+| `session_key_depth` | `0` | Repeated benchmark trials with identical task prefixes need separate latches; keep `0` for normal traffic. |
+
+## Observability
+
+Read the standard routing stats endpoint:
+
+```bash
+curl -s http://localhost:4000/v1/routing/stats
+```
+
+The snapshot reports per-model calls, tokens, latency, and cost for the strong
+and weak tiers. Judge calls are recorded in the classifier stats bucket so
+their token cost, latency, and errors remain visible as routing overhead.
+
+With `judge.dump_verdicts: true` (off by default), each judged turn also
+writes an `escalation_verdict={...}` JSON line to server stderr. The record
+includes the decision, reason, turn, confirmation state, and judge latency.
+After the latch fires, later turns skip the judge and report the pinned
+routing source in request metadata.
+
+## Repeated benchmark trials
+
+By default, the session key is derived from the system prompt and first user
+message. Repeated trials of the same task against one long-lived server can
+therefore share a latch: if the first trial escalates, later trials may start
+on strong.
+
+Use one of these isolation strategies:
+
+- Start a fresh Switchyard process for each trial set.
+- Set `session_key_depth: N` to extend the key with the first `N` messages
+ after the initial user message. This only separates trials when those early
+ trajectories differ, so it requires nonzero sampling temperature.
+
+Keep `session_key_depth: 0` for normal traffic. Context compaction or other
+mid-session prefix rewrites can change a deep key and lose the existing latch.
+
+## When not to use escalation routing
+
+- **One-shot requests.** There is no trajectory to judge. Use
+ [LLM Classifier Routing](llm_classifier_routing.md) when the initial request
+ should determine the tier.
+- **Fixed traffic experiments.** Use
+ [Random Routing](random_routing.md) for A/B splits and gradual traffic ramps.
+- **Per-turn stage optimization.** Use
+ [Stage-Router Routing](stage_router_routing.md) when tool-result signals
+ should move individual turns in both directions.
+- **Latency-critical traffic.** Eligible unlatched turns wait for the judge
+ before the selected backend call.
+- **Long-range failure cycles.** The judge only sees a bounded recent window;
+ cycles longer than that window may be missed.
+
+## Related
+
+- [Routing Overview](overview.md): compare all supported routing strategies.
+- [Sticky Routing](sticky_routing.md): session-key derivation and affinity
+ behavior.
+- [Architecture](../architecture.md): the end-to-end request lifecycle and
+ system boundaries.
diff --git a/docs/routing_algorithms/overview.md b/docs/routing_algorithms/overview.md
index 3b80c732a..386c1d6a9 100644
--- a/docs/routing_algorithms/overview.md
+++ b/docs/routing_algorithms/overview.md
@@ -18,6 +18,7 @@ for configuration and tuning.
| [Random Routing](random_routing.md) | You need a fixed strong/weak split for A/B tests, baselines, or cost experiments. | `random-routing` |
| [LLM Classifier Routing](llm_classifier_routing.md) | Request content should decide whether a turn needs the weak or strong tier. | `llm-routing` |
| [Stage-Router Routing](stage_router_routing.md) | Tool-result and agent-progress signals should route most turns without an extra classifier call. | `stage_router` |
+| [Escalation-Router Routing](escalation_router_routing.md) | Start every task on the weak tier and let an LLM judge escalate to strong — one-way per task — when the run is in trouble. | `escalation_router` (`routes:` bundle) |
[Session Affinity (Sticky Routing)](sticky_routing.md) is an opt-in feature of
LLM classifier routing, not a standalone routing strategy. The classifier
@@ -152,6 +153,12 @@ They are not shared across workers or restarts. See
Random and stage-router routing do not expose session-affinity settings; they
continue to make a routing decision for each request.
+The escalation router uses the same affinity store with the opposite policy:
+affinity is always on (the pin *is* the escalation latch), only the strong
+tier is ever pinned, and once the judge starts (a configurable minimum turn),
+it keeps running each judged turn until the latch fires. See
+[Escalation-Router Routing](escalation_router_routing.md).
+
!!! note "CLI schema availability"
The CLI currently accepts these settings in a `deterministic` entry in a
`routes:` bundle loaded with `--routing-profiles`. The Rust `llm-routing`
diff --git a/mkdocs.yml b/mkdocs.yml
index 8d4ffa84b..537c1af6a 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -32,6 +32,7 @@ nav:
- LLM Classifier Routing: routing_algorithms/llm_classifier_routing.md
- Sticky Routing: routing_algorithms/sticky_routing.md
- Stage-Router Routing: routing_algorithms/stage_router_routing.md
+ - Escalation-Router Routing: routing_algorithms/escalation_router_routing.md
- Operations:
- Context-Window Handling: operations/context_window.md
- Reference:
diff --git a/pyproject.toml b/pyproject.toml
index 14e3aee61..91eb2a728 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -147,7 +147,11 @@ where = ["."]
include = ["switchyard*"]
[tool.setuptools.package-data]
-switchyard = ["py.typed", "lib/processors/stage_router/prompts/*.md"]
+switchyard = [
+ "py.typed",
+ "lib/processors/prompts/*.md",
+ "lib/processors/stage_router/prompts/*.md",
+]
switchyard_rust = ["py.typed", "*.pyi"]
[tool.maturin]
diff --git a/switchyard/__init__.py b/switchyard/__init__.py
index cb1790433..6a0688ad5 100644
--- a/switchyard/__init__.py
+++ b/switchyard/__init__.py
@@ -60,6 +60,8 @@ from switchyard.lib.profiles import (
DeterministicRoutingConfig,
DeterministicRoutingPresets,
DeterministicRoutingProfileConfig,
+ EscalationRouterConfig,
+ EscalationRouterProfileConfig,
LatencyServiceProfileConfig,
NoopProfile,
NoopProfileConfig,
@@ -165,6 +167,8 @@ __all__ = [
"DeterministicRoutingConfig",
"DeterministicRoutingProfileConfig",
"DeterministicRoutingPresets",
+ "EscalationRouterConfig",
+ "EscalationRouterProfileConfig",
"LatencyServiceProfileConfig",
"OSSRouterConfig",
"OSSRouterProfileConfig",
diff --git a/switchyard/cli/route_bundle.py b/switchyard/cli/route_bundle.py
index c41a13a84..b1df1d4ff 100644
--- a/switchyard/cli/route_bundle.py
+++ b/switchyard/cli/route_bundle.py
@@ -30,6 +30,8 @@ from importlib import import_module
from pathlib import Path
from typing import Any, Protocol, cast, overload
+from pydantic import ValidationError
+
from switchyard.cli.model_catalog.model_discovery import fetch_model_ids
from switchyard.lib.backends.llm_target import LlmTarget, coerce_llm_target
from switchyard.lib.config import LatencyServiceBackendConfig, LatencyServiceEndpoint
@@ -37,6 +39,7 @@ from switchyard.lib.processors.llm_classifier import DEFAULT_MAX_REQUEST_CHARS
from switchyard.lib.processors.llm_classifier.presets import PROFILE_FACTORIES
from switchyard.lib.profiles import (
DeterministicRoutingProfileConfig,
+ EscalationRouterProfileConfig,
LatencyServiceProfileConfig,
PlanExecuteProfileConfig,
ProfileSwitchyard,
@@ -44,6 +47,7 @@ from switchyard.lib.profiles import (
StageRouterProfileConfig,
)
from switchyard.lib.profiles.deterministic_routing_config import DeterministicRoutingConfig
+from switchyard.lib.profiles.escalation_router_config import EscalationRouterConfig
from switchyard.lib.profiles.plan_execute_config import PlanExecuteConfig
from switchyard.lib.profiles.plan_execute_presets import PlanExecutePresets
from switchyard.lib.profiles.random_routing import RandomRoutingConfig
@@ -264,6 +268,42 @@ _PLAN_EXECUTE_ROUTE_KEYS = (
"fallback_target_on_evict",
})
)
+_ESCALATION_ROUTE_KEYS = (
+ _ROUTE_METADATA_KEYS
+ | _TARGET_DEFAULT_ROUTE_KEYS
+ | frozenset({
+ "judge",
+ "strong",
+ "weak",
+ "enable_stats",
+ "fallback_target_on_evict",
+ "tier_timeout_s",
+ "session_key_depth",
+ "affinity_max_sessions",
+ "affinity_store",
+ "affinity_store_url",
+ "affinity_store_ttl_seconds",
+ "affinity_key_prefix",
+ })
+)
+_ESCALATION_JUDGE_KEYS = frozenset({
+ "model",
+ "api_key",
+ "base_url",
+ "timeout",
+ "timeout_secs",
+ "min_turn",
+ "confirmations",
+ "confirmation_window",
+ "disable_reasoning",
+ "max_completion_tokens",
+ "dump_verdicts",
+ "recent_turn_window",
+ "window_message_chars",
+ "prompt",
+ "prompt_path",
+ "max_request_chars",
+})
_DETERMINISTIC_CLASSIFIER_KEYS = frozenset({
"model",
"api_key",
@@ -298,6 +338,7 @@ _ROUTE_KEYS_BY_TYPE: Mapping[str, frozenset[str]] = {
"noop": _NOOP_ROUTE_KEYS,
"passthrough": _PASSTHROUGH_ROUTE_KEYS,
"deterministic": _DETERMINISTIC_ROUTE_KEYS,
+ "escalation_router": _ESCALATION_ROUTE_KEYS,
"stage_router": _STAGE_ROUTER_ROUTE_KEYS,
"plan_execute": _PLAN_EXECUTE_ROUTE_KEYS,
}
@@ -309,6 +350,7 @@ _DEFAULT_KEYS_BY_TYPE: Mapping[str, frozenset[str]] = {
"passthrough": _PASSTHROUGH_SETTING_KEYS,
"noop": frozenset(),
"deterministic": _TARGET_DEFAULT_KEYS,
+ "escalation_router": _TARGET_DEFAULT_KEYS,
"stage_router": _TARGET_DEFAULT_KEYS,
"plan_execute": _TARGET_DEFAULT_KEYS,
}
@@ -548,7 +590,7 @@ def build_table_from_bundle(
# at the route key AND hydrate each tier's catalog (`strong` + `weak`)
# into the table as direct passthroughs — same client-facing
# model-picker experience as random_routing's discovery path.
- if route_type in ("stage_router", "deterministic"):
+ if route_type in ("stage_router", "deterministic", "escalation_router"):
_merge_multi_target_discovery(
table,
model_id,
@@ -836,6 +878,16 @@ def _build_switchyard_for_route(
extra_response_processors=extra_response_processors,
)
+ if route_type == "escalation_router":
+ return _escalation_router_switchyard(
+ model_id,
+ route,
+ target_defaults=target_defaults,
+ stats=stats,
+ pre_routing_request_processors=pre_routing_request_processors,
+ extra_response_processors=extra_response_processors,
+ )
+
if route_type == "stage_router":
return _stage_router_switchyard(
model_id,
@@ -1008,6 +1060,151 @@ def _deterministic_switchyard(
)
+def _judge_prompt_value(judge: Mapping[str, object], model_id: str) -> str | None:
+ """Resolve the judge prompt override from ``prompt`` or ``prompt_path``.
+
+ ``prompt`` inlines the text; ``prompt_path`` reads it from a file
+ (relative paths resolve against the server's working directory).
+ ``None`` keeps the built-in default.
+ """
+ inline = _optional_str(judge.get("prompt"))
+ path = _optional_str(judge.get("prompt_path"))
+ if inline is not None and path is not None:
+ raise RouteBundleConfigError(
+ f"route {model_id!r}: judge.prompt and judge.prompt_path are "
+ "mutually exclusive",
+ )
+ if path is None:
+ return inline
+ try:
+ return Path(path).read_text(encoding="utf-8")
+ except (OSError, UnicodeError) as exc:
+ raise RouteBundleConfigError(
+ f"route {model_id!r}: judge.prompt_path {path!r}: cannot read: "
+ f"{_format_exception_one_line(exc)}"
+ ) from exc
+
+
+def _escalation_router_switchyard(
+ model_id: str,
+ route: Mapping[str, object],
+ target_defaults: Mapping[str, object],
+ stats: StatsAccumulator,
+ pre_routing_request_processors: Sequence[Any],
+ extra_response_processors: Sequence[Any],
+) -> ChainRuntime:
+ """Build the judge-latched escalation-routing chain for a route.
+
+ Every conversation starts on ``weak``; an LLM judge watches the
+ trajectory and, on a clear pattern of trouble, latches the conversation
+ to ``strong`` for the rest of the task. Requires ``judge``/``strong``/
+ ``weak`` targets plus ``fallback_target_on_evict``; the full YAML schema
+ and knob reference live in ``docs/routing_algorithms/
+ escalation_router_routing.md``.
+ """
+ judge_raw = route.get("judge")
+ if not isinstance(judge_raw, Mapping):
+ raise RouteBundleConfigError(
+ f"route {model_id!r}: type=escalation_router requires a `judge:` "
+ "mapping with model/api_key/base_url",
+ )
+ judge = _classifier_mapping(
+ judge_raw,
+ target_defaults,
+ allowed_keys=_ESCALATION_JUDGE_KEYS,
+ where=f"{model_id}.judge",
+ )
+
+ strong = _target_value(
+ route.get("strong"), target_defaults, default_id="strong", where="strong",
+ )
+ weak = _target_value(
+ route.get("weak"), target_defaults, default_id="weak", where="weak",
+ )
+
+ fallback_target_on_evict = _required_str(
+ route.get("fallback_target_on_evict"),
+ f"{model_id}.fallback_target_on_evict",
+ )
+ valid_ids = {strong.id, weak.id}
+ if fallback_target_on_evict not in valid_ids:
+ raise RouteBundleConfigError(
+ f"route {model_id!r}: fallback_target_on_evict="
+ f"{fallback_target_on_evict!r} must match one of {sorted(valid_ids)} "
+ f"(the configured strong/weak target ids)",
+ )
+
+ config_data: dict[str, object] = {
+ "strong": strong,
+ "weak": weak,
+ "judge": {
+ "id": "judge",
+ "model": _required_str(
+ judge.get("model"), f"{model_id}.judge.model"
+ ),
+ "api_key": _required_str(
+ judge.get("api_key"), f"{model_id}.judge.api_key"
+ ),
+ "base_url": _required_str(
+ judge.get("base_url"), f"{model_id}.judge.base_url"
+ ),
+ },
+ "fallback_target_on_evict": fallback_target_on_evict,
+ "judge_system_prompt": _judge_prompt_value(judge, model_id),
+ }
+ # Optional knobs are forwarded verbatim and only when present, so
+ # ``EscalationRouterConfig`` stays the single owner of every default and
+ # of value validation; re-declaring defaults here is exactly the config
+ # drift this compatibility path must not accumulate.
+ judge_key_map = {
+ "timeout_secs": "judge_timeout_s",
+ "min_turn": "judge_min_turn",
+ "confirmations": "judge_escalate_confirmations",
+ "confirmation_window": "judge_confirmation_window",
+ "disable_reasoning": "judge_disable_reasoning",
+ "max_completion_tokens": "judge_max_completion_tokens",
+ "dump_verdicts": "judge_dump_verdicts",
+ "recent_turn_window": "judge_recent_turn_window",
+ "window_message_chars": "judge_window_message_chars",
+ "max_request_chars": "judge_max_request_chars",
+ }
+ for source_key, config_key in judge_key_map.items():
+ if judge.get(source_key) is not None:
+ config_data[config_key] = judge[source_key]
+ for route_key in (
+ "enable_stats",
+ "tier_timeout_s",
+ "session_key_depth",
+ "affinity_max_sessions",
+ "affinity_store",
+ "affinity_store_url",
+ "affinity_store_ttl_seconds",
+ "affinity_key_prefix",
+ ):
+ if route.get(route_key) is not None:
+ config_data[route_key] = route[route_key]
+
+ try:
+ config = EscalationRouterConfig.model_validate(config_data)
+ except ValidationError as exc:
+ # Surface config mistakes as the CLI's one-line diagnostic instead of
+ # a raw pydantic traceback (see the RouteBundleConfigError handler in
+ # switchyard_cli.main).
+ raise RouteBundleConfigError(
+ f"route {model_id!r}: invalid escalation_router config: {exc}"
+ ) from exc
+ return ProfileSwitchyard(
+ EscalationRouterProfileConfig.from_config(config)
+ .build()
+ .with_runtime_components(
+ stats_accumulator=stats,
+ enable_stats=config.enable_stats,
+ pre_request_processors=pre_routing_request_processors,
+ response_processors=extra_response_processors,
+ )
+ )
+
+
def _stage_router_switchyard(
model_id: str,
route: Mapping[str, object],
@@ -1354,6 +1551,8 @@ def _route_type(model_id: str, route: Mapping[str, object]) -> str:
"deterministic": "deterministic",
"llm_classifier": "deterministic",
"llm_classifier_routing": "deterministic",
+ "escalation": "escalation_router",
+ "escalation_router": "escalation_router",
"stage_router": "stage_router",
"stage_router_routing": "stage_router",
"plan": "plan_execute",
diff --git a/switchyard/lib/cost_estimator.py b/switchyard/lib/cost_estimator.py
index 7afd8c86a..f47397497 100644
--- a/switchyard/lib/cost_estimator.py
+++ b/switchyard/lib/cost_estimator.py
@@ -81,6 +81,29 @@ MODEL_PRICING: dict[str, ModelPriceData] = {
"openai/nvidia/nvidia/Nemotron-3-Nano-30B-A3B": ModelPriceData(
input=0.05, output=0.20, cached=0.005, cache_write=0.05,
),
+ # Nemotron 3 Ultra — 550B/55B-active MoE, 1M ctx (released 2026-06-04).
+ # OpenRouter reference list price (June 2026): $0.50 / $2.20 per 1M
+ # tokens. The ``evals-nemotron-ultra-N`` ids are parallel deployments
+ # of the same model on the hub's benchmarking gateway (pair with
+ # ``X-Inference-Priority: batch``); same per-token pricing.
+ "nvidia/nvidia/nemotron-3-ultra": ModelPriceData(
+ input=0.50, output=2.20, cached=0.05, cache_write=0.50,
+ ),
+ "openai/nvidia/nvidia/nemotron-3-ultra": ModelPriceData(
+ input=0.50, output=2.20, cached=0.05, cache_write=0.50,
+ ),
+ "nvidia/nvidia/evals-nemotron-ultra": ModelPriceData(
+ input=0.50, output=2.20, cached=0.05, cache_write=0.50,
+ ),
+ "nvidia/nvidia/evals-nemotron-ultra-2": ModelPriceData(
+ input=0.50, output=2.20, cached=0.05, cache_write=0.50,
+ ),
+ "nvidia/nvidia/evals-nemotron-ultra-3": ModelPriceData(
+ input=0.50, output=2.20, cached=0.05, cache_write=0.50,
+ ),
+ "nvidia/nvidia/evals-nemotron-ultra-4": ModelPriceData(
+ input=0.50, output=2.20, cached=0.05, cache_write=0.50,
+ ),
# Moonshot Kimi K2.6 — official platform.kimi.ai pricing (May 2026).
# OpenAI wire format, so no cache_write premium (cache_write = input).
"nvidia/moonshotai/kimi-k2.6": ModelPriceData(
@@ -155,6 +178,11 @@ MODEL_PRICING: dict[str, ModelPriceData] = {
),
# --- Anthropic Claude on AWS Bedrock (via NVIDIA Inference Hub) ---
# 5-minute cache write = 1.25x input; cache read = 0.1x input.
+ # Opus 4.8 keeps the Opus-tier $5/$25 list price (docs.anthropic.com,
+ # verified 2026-07-11).
+ "aws/anthropic/bedrock-claude-opus-4-8": ModelPriceData(
+ input=5.00, output=25.00, cached=0.50, cache_write=6.25,
+ ),
"aws/anthropic/bedrock-claude-opus-4-7": ModelPriceData(
input=5.00, output=25.00, cached=0.50, cache_write=6.25,
),
@@ -177,6 +205,12 @@ MODEL_PRICING: dict[str, ModelPriceData] = {
"azure/anthropic/claude-opus-4-7": ModelPriceData(
input=5.00, output=25.00, cached=0.50, cache_write=6.25,
),
+ "azure/anthropic/claude-opus-4-8": ModelPriceData(
+ input=5.00, output=25.00, cached=0.50, cache_write=6.25,
+ ),
+ "azure/anthropic/claude-sonnet-4-5": ModelPriceData(
+ input=3.00, output=15.00, cached=0.30, cache_write=3.75,
+ ),
"aws/anthropic/bedrock-claude-sonnet-4-6": ModelPriceData(
input=3.00, output=15.00, cached=0.30, cache_write=3.75,
),
@@ -187,6 +221,9 @@ MODEL_PRICING: dict[str, ModelPriceData] = {
input=1.00, output=5.00, cached=0.10, cache_write=1.25,
),
# --- Anthropic direct API aliases (no AWS prefix) ---
+ "claude-opus-4-8": ModelPriceData(
+ input=5.00, output=25.00, cached=0.50, cache_write=6.25,
+ ),
"claude-opus-4-7": ModelPriceData(
input=5.00, output=25.00, cached=0.50, cache_write=6.25,
),
diff --git a/switchyard/lib/processors/escalation_judge_request_processor.py b/switchyard/lib/processors/escalation_judge_request_processor.py
new file mode 100644
index 000000000..a271e4861
--- /dev/null
+++ b/switchyard/lib/processors/escalation_judge_request_processor.py
@@ -0,0 +1,537 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Trajectory judge for the escalation router: weak until in trouble, then strong."""
+
+from __future__ import annotations
+
+import json
+import logging
+import sys
+import time
+from collections import OrderedDict
+from importlib.resources import files
+from typing import Any, Literal
+
+from pydantic import BaseModel, ConfigDict, Field
+
+from switchyard.lib.backends.deterministic_routing_llm_backend import (
+ CTX_DETERMINISTIC_ROUTING_TIER,
+)
+from switchyard.lib.conversation_turn import conversation_turn_number
+from switchyard.lib.llm_client import OpenAILLMClient
+from switchyard.lib.processors.llm_classifier.request_processor import (
+ LLMClassifierClient,
+ OpenAIChatLLMClassifierClient,
+)
+from switchyard.lib.processors.message_condensing import (
+ content_text,
+ message_text,
+ strip_markdown_fence,
+ trim_messages,
+ truncate_middle,
+)
+from switchyard.lib.proxy_context import ProxyContext
+from switchyard.lib.session_affinity import (
+ CTX_SESSION_KEY,
+ SessionAffinity,
+ resolve_session_key,
+)
+from switchyard.lib.stats_accumulator import StatsAccumulator
+from switchyard_rust.core import ChatRequest
+
+log = logging.getLogger(__name__)
+
+#: ``ProxyContext.metadata`` key holding the audit record of the judge's
+#: decision for this turn: ``{"escalate", "reason", "turn", "source",
+#: "session", "streak", "confirmed"}`` (the last three only on judged turns).
+CTX_ESCALATION_VERDICT = "_escalation_verdict"
+
+#: LRU cap on the consecutive-escalate streak store.
+_MAX_STREAK_SESSIONS = 10_000
+
+#: Default tier labels stamped into ``CTX_DETERMINISTIC_ROUTING_TIER``. The
+#: labels must match what the profile registers on
+#: ``DeterministicRoutingLLMBackend`` — profiles with custom target ids pass
+#: their own via ``strong_tier`` / ``weak_tier``.
+TIER_STRONG = "strong"
+TIER_WEAK = "weak"
+
+
+def _load_system_prompt() -> str:
+ """Read the default judge system prompt from the prompts/ package-data file.
+
+ The file carries the benchmarked line-of-record prompt byte-exact; its
+ content is hash-pinned in tests, so edits require deliberate re-validation.
+ """
+ return (
+ files("switchyard.lib.processors.prompts")
+ .joinpath("escalation_judge.md")
+ .read_text(encoding="utf-8")
+ )
+
+
+ESCALATION_JUDGE_SYSTEM_PROMPT: str = _load_system_prompt()
+
+
+class EscalationVerdict(BaseModel):
+ """Binary judge verdict: escalate to the strong tier or stay on weak."""
+
+ model_config = ConfigDict(frozen=True)
+
+ escalate: bool
+ reason: str = ""
+
+
+class EscalationJudgeConfig(BaseModel):
+ """Configuration for :class:`EscalationJudgeRequestProcessor`.
+
+ The judge call is OpenAI-chat-compatible; tests inject any object
+ implementing the classifier's ``LLMClassifierClient`` protocol.
+ """
+
+ model_config = ConfigDict(frozen=True)
+
+ model: str = Field(min_length=1)
+ api_key: str | None = None
+ base_url: str | None = None
+ timeout_s: float = Field(default=5.0, gt=0.0)
+ """Judge wall-clock ceiling. The judge sits on the request path of every
+ pre-escalation turn; a slow judge taxes the whole session, so keep this
+ tight — any failure or timeout fails open to the weak tier."""
+
+ max_completion_tokens: int = Field(default=128, ge=16)
+ system_prompt: str = Field(default=ESCALATION_JUDGE_SYSTEM_PROMPT, min_length=1)
+ structured_output_mode: Literal["json_schema", "json_object"] = "json_schema"
+ disable_reasoning: bool = True
+ extra_headers: dict[str, str] | None = None
+
+ dump_verdicts_to_stderr: bool = False
+ """Emit one ``escalation_verdict={...}`` JSON line to ``sys.stderr`` per
+ judge call (verdict or fail-open).
+
+ Benchmark-only diagnostics, **opt-in**: the payload includes a short
+ first-user snippet (``task_hint``) and the line interferes with agent
+ TUIs sharing stderr, so shipping profiles keep it off and production
+ routing telemetry flows through the normal stats path. When enabled it
+ is written directly (not via the logging module) so it lands in the
+ benchmark server's captured log regardless of uvicorn's logger config.
+ Grep ``escalation_verdict=`` to reconstruct per-turn judge decisions
+ and escalation timing for a run."""
+
+ min_judge_turn: int = Field(default=3, ge=1)
+ """First conversation turn on which the judge runs. Earlier turns have no
+ trajectory to judge and always route weak."""
+
+ escalate_confirmations: int = Field(default=1, ge=1)
+ """Consecutive ``escalate`` verdicts required before the latch fires.
+
+ ``1`` (default) pins on the first escalate verdict. ``2`` requires the
+ judge to confirm on the next judged turn, filtering one-shot eager
+ verdicts (a single failed command misread as a pattern) while keeping
+ recall on real trouble, which by definition persists across turns. A
+ non-escalate verdict resets the streak; a fail-open leaves it
+ unchanged (no evidence either way)."""
+
+ confirmation_window: int = Field(default=1, ge=1)
+ """How many judged turns an escalate verdict stays live for
+ confirmation purposes.
+
+ ``1`` (default) keeps the strict-consecutive behaviour: any decline
+ resets the streak. ``N > 1`` lets a later escalate verdict confirm an
+ earlier one across up to ``N - 1`` intervening declines — trouble that
+ keeps *recurring* is trouble, even when a quiet turn separates the
+ flare-ups."""
+
+ recent_turn_window: int = Field(default=14, ge=1)
+ """Trailing messages shown to the judge on top of the anchors. Wider than
+ the classifier's default because loop detection needs to see the repeats:
+ a cycle longer than the window is invisible."""
+
+ max_request_chars: int = Field(default=12_000, ge=1_000)
+ """Cap on the assembled judge transcript. When exceeded, the *oldest*
+ window messages are dropped first — for a trajectory judge the newest
+ evidence is strictly the most valuable."""
+
+ system_chars: int = Field(default=1_000, ge=100)
+ """Per-message cap for system/developer anchors. Coding-agent harnesses
+ (Claude Code) inject very large boilerplate system prompts with no
+ trajectory signal; without this cap they would crowd out the window."""
+
+ first_user_chars: int = Field(default=2_000, ge=100)
+ """Cap for the first user message — the task statement the judge needs
+ for drift detection, so it gets the most generous anchor budget."""
+
+ window_message_chars: int = Field(default=300, ge=50)
+ """Per-message cap inside the trailing window. Error signatures and
+ command shapes survive this easily; full file dumps do not need to."""
+
+
+class EscalationJudgeRequestProcessor:
+ """Route weak until an LLM judge says the run is in trouble, then latch strong.
+
+ Per-turn policy (the latch): a pinned conversation routes strong with no
+ judge call; an unpinned one routes weak, and from ``min_judge_turn`` on the
+ judge reads a condensed transcript. An ``escalate`` verdict pins the
+ conversation to the strong tier via :class:`SessionAffinity` — one-way for
+ the rest of the task. Weak is never pinned, and any judge failure fails
+ open to weak without pinning, so an outage costs quality risk, never money.
+ """
+
+ def __init__(
+ self,
+ config: EscalationJudgeConfig,
+ *,
+ affinity: SessionAffinity,
+ client: LLMClassifierClient | None = None,
+ session_key_depth: int = 0,
+ stats_accumulator: StatsAccumulator | None = None,
+ strong_tier: str = TIER_STRONG,
+ weak_tier: str = TIER_WEAK,
+ ) -> None:
+ if session_key_depth < 0:
+ raise ValueError("session_key_depth must be >= 0")
+ if strong_tier == weak_tier:
+ raise ValueError("strong_tier and weak_tier must differ")
+ self._config = config
+ self._affinity = affinity
+ self._session_key_depth = session_key_depth
+ # The labels this processor stamps and pins; they key the tier dict on
+ # the downstream DeterministicRoutingLLMBackend, so both must match
+ # what the owning profile registered there.
+ self._strong_tier = strong_tier
+ self._weak_tier = weak_tier
+ # Also attached post-construction by the profile chain's
+ # ``with_runtime_components`` (same hook as the LLM classifier).
+ self._stats_accumulator = stats_accumulator
+ # Per-conversation (streak, declines_since_last_fire), only consulted
+ # when ``escalate_confirmations > 1``. LRU-capped alongside the
+ # affinity store so a long-lived server cannot grow it unbounded.
+ self._escalate_streaks: OrderedDict[str, tuple[int, int]] = OrderedDict()
+ self._client = client or OpenAIChatLLMClassifierClient(
+ OpenAILLMClient(
+ api_key=config.api_key,
+ base_url=config.base_url,
+ timeout=config.timeout_s,
+ # Fail-open to weak is the retry surface; let the SDK fail fast.
+ max_retries=0,
+ ),
+ signal_schema=EscalationVerdict,
+ structured_output_mode=config.structured_output_mode,
+ max_completion_tokens=config.max_completion_tokens,
+ disable_reasoning=config.disable_reasoning,
+ extra_headers=config.extra_headers,
+ )
+
+ async def process(self, ctx: ProxyContext, request: ChatRequest) -> ChatRequest:
+ """Stamp the tier for this turn and maybe escalate. Leaves the request unchanged."""
+ turn = conversation_turn_number(request)
+ # With a deep key (session_key_depth > 0), affinity is untouchable
+ # until the key prefix is complete — hashing a shorter prefix would
+ # produce a key that later turns of the same conversation no longer
+ # match. resolve_session_key returns None (memoizing nothing) until
+ # then; once resolved, the memoized key is what affinity and streak
+ # bookkeeping see for the rest of the request.
+ affinity_ready = (
+ resolve_session_key(ctx, request, depth=self._session_key_depth) is not None
+ )
+
+ if affinity_ready:
+ pinned = await self._affinity.pinned(ctx, request)
+ if pinned == self._strong_tier:
+ ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] = self._strong_tier
+ ctx.metadata[CTX_ESCALATION_VERDICT] = {
+ "escalate": True,
+ "reason": "",
+ "turn": turn,
+ "source": "pinned",
+ }
+ return request
+
+ ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] = self._weak_tier
+ if not affinity_ready or turn < self._config.min_judge_turn:
+ return request
+
+ # Lets verdict-dump consumers group per-turn judge decisions by
+ # conversation.
+ session = ctx.metadata.get(CTX_SESSION_KEY)
+ summary = _summarize_for_judge(request, turn=turn, config=self._config)
+ started_at = time.perf_counter()
+ try:
+ completion = await self._client.classify(
+ model=self._config.model,
+ system_prompt=self._config.system_prompt,
+ request_summary=summary,
+ )
+ verdict = EscalationVerdict.model_validate_json(
+ strip_markdown_fence(completion.content),
+ )
+ except Exception as exc:
+ # Fail open to the current (weak) tier and never pin: a judge
+ # outage must not silently burn strong-model tokens.
+ log.warning(
+ "EscalationJudgeRequestProcessor: judge failed; staying on weak tier: %s",
+ exc,
+ )
+ if self._stats_accumulator is not None:
+ # Record the failure into the classifier bucket so
+ # ``/v1/routing/stats`` exposes the judge fail-open rate;
+ # a silently failing judge reads as "router never
+ # escalates" to the benchmark observer.
+ await self._stats_accumulator.record_classifier_error(self._config.model)
+ ctx.metadata[CTX_ESCALATION_VERDICT] = {
+ "escalate": False,
+ "reason": f"fail_open: {str(exc)[:200]}",
+ "turn": turn,
+ "source": "fail_open",
+ "session": session,
+ }
+ self._dump_verdict(
+ ctx.metadata[CTX_ESCALATION_VERDICT],
+ latency_ms=(time.perf_counter() - started_at) * 1000,
+ request=request,
+ )
+ return request
+
+ if self._stats_accumulator is not None:
+ await self._record_judge_call(
+ usage=completion.usage,
+ latency_ms=(time.perf_counter() - started_at) * 1000,
+ )
+
+ if verdict.escalate:
+ streak = self._bump_streak(ctx, request)
+ confirmed = streak >= self._config.escalate_confirmations
+ else:
+ streak = 0
+ confirmed = False
+ self._reset_streak(ctx, request)
+
+ ctx.metadata[CTX_ESCALATION_VERDICT] = {
+ "escalate": verdict.escalate,
+ "reason": verdict.reason,
+ "turn": turn,
+ "source": "judge",
+ "session": session,
+ "streak": streak,
+ "confirmed": confirmed,
+ }
+ self._dump_verdict(
+ ctx.metadata[CTX_ESCALATION_VERDICT],
+ latency_ms=(time.perf_counter() - started_at) * 1000,
+ request=request,
+ )
+ if confirmed:
+ log.info(
+ "EscalationJudgeRequestProcessor: escalating to strong tier "
+ "(turn %d): %s",
+ turn,
+ verdict.reason,
+ )
+ await self._affinity.pin(ctx, request, self._strong_tier)
+ ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] = self._strong_tier
+ return request
+
+ def _streak_key(self, ctx: ProxyContext, request: ChatRequest) -> str:
+ """Conversation key for streak bookkeeping (seeded deep key or Rust default)."""
+ return resolve_session_key(ctx, request)
+
+ def _bump_streak(self, ctx: ProxyContext, request: ChatRequest) -> int:
+ """Record one escalate verdict for this conversation; return the streak."""
+ if self._config.escalate_confirmations <= 1:
+ # Single-verdict latch: no bookkeeping needed, streak is per-call.
+ return 1
+ key = self._streak_key(ctx, request)
+ streak, _ = self._escalate_streaks.get(key, (0, 0))
+ streak += 1
+ self._escalate_streaks[key] = (streak, 0)
+ self._escalate_streaks.move_to_end(key)
+ while len(self._escalate_streaks) > _MAX_STREAK_SESSIONS:
+ self._escalate_streaks.popitem(last=False)
+ return streak
+
+ def _reset_streak(self, ctx: ProxyContext, request: ChatRequest) -> None:
+ """Register a non-escalate verdict against the streak.
+
+ With ``confirmation_window == 1`` any decline clears the streak
+ (strict-consecutive). A larger window tolerates up to
+ ``window - 1`` intervening declines before the streak expires, so
+ recurring intermittent trouble can still confirm.
+ """
+ if self._config.escalate_confirmations <= 1:
+ return
+ key = self._streak_key(ctx, request)
+ entry = self._escalate_streaks.get(key)
+ if entry is None:
+ return
+ streak, declines = entry
+ declines += 1
+ if declines >= self._config.confirmation_window:
+ self._escalate_streaks.pop(key, None)
+ else:
+ self._escalate_streaks[key] = (streak, declines)
+ self._escalate_streaks.move_to_end(key)
+
+ def _dump_verdict(
+ self,
+ verdict: dict[str, Any],
+ *,
+ latency_ms: float,
+ request: ChatRequest | None = None,
+ ) -> None:
+ """Write one ``escalation_verdict={...}`` JSON line to stderr.
+
+ Direct stderr (not the logging module) so benchmark server logs
+ capture it — see :attr:`EscalationJudgeConfig.dump_verdicts_to_stderr`.
+ Includes a short first-user-message snippet (``task_hint``) so
+ offline analysis can attribute per-turn verdicts to tasks exactly,
+ instead of guessing from overlapping trial time windows.
+ """
+ if not self._config.dump_verdicts_to_stderr:
+ return
+ payload = dict(verdict)
+ payload["latency_ms"] = round(latency_ms, 1)
+ if request is not None:
+ hint = _first_user_text(request)
+ if hint:
+ payload["task_hint"] = hint[:48]
+ sys.stderr.write(f"escalation_verdict={json.dumps(payload, sort_keys=True)}\n")
+ sys.stderr.flush()
+
+ async def _record_judge_call(self, *, usage: Any, latency_ms: float) -> None:
+ """Record judge token spend and latency into the classifier stats bucket.
+
+ The judge is a routing-overhead call, exactly like the LLM
+ classifier: keeping it in the classifier bucket stops its spend
+ from merging with a same-named tier model and keeps
+ ``routing_stats_final.json`` cost math honest for the
+ escalation-router benchmark.
+ """
+ assert self._stats_accumulator is not None
+ prompt = 0
+ completion = 0
+ cached = 0
+ if usage is not None:
+ prompt = getattr(usage, "prompt_tokens", 0) or 0
+ completion = getattr(usage, "completion_tokens", 0) or 0
+ ptd = getattr(usage, "prompt_tokens_details", None)
+ if ptd is not None:
+ cached = getattr(ptd, "cached_tokens", 0) or 0
+ await self._stats_accumulator.record_classifier_usage(
+ model=self._config.model,
+ prompt_tokens=prompt,
+ completion_tokens=completion,
+ cached_tokens=cached,
+ latency_ms=latency_ms,
+ )
+
+def _first_user_text(request: ChatRequest) -> str:
+ """Flatten the first user message's task text for verdict dumps.
+
+ Harness wrappers (``<system-reminder>``/``<environment_context>``
+ blocks injected ahead of the actual task statement) are skipped so the
+ snippet identifies the task rather than repeating boilerplate shared
+ by every conversation.
+ """
+ body = getattr(request, "body", None)
+ if not isinstance(body, dict):
+ return ""
+ messages = body.get("messages")
+ if not isinstance(messages, list):
+ messages = body.get("input")
+ if not isinstance(messages, list):
+ return ""
+ for m in messages:
+ if isinstance(m, dict) and m.get("role") == "user":
+ text = message_text(m).strip()
+ while True:
+ for tag in ("system-reminder", "environment_context"):
+ if text.startswith(f"<{tag}>"):
+ end = text.find(f"</{tag}>")
+ if end >= 0:
+ text = text[end + len(tag) + 3 :].lstrip()
+ break
+ else:
+ return text
+ return ""
+
+
+def _summarize_for_judge(
+ request: ChatRequest,
+ *,
+ turn: int,
+ config: EscalationJudgeConfig,
+) -> str:
+ """Render a compact role-labeled transcript for the judge.
+
+ Anchors (system + first user) are individually capped; the trailing window
+ keeps the last ``recent_turn_window`` messages at ``window_message_chars``
+ each. A coverage header states how much history is *not* shown so the
+ judge can reason about pace. If the global cap still binds, the oldest
+ window lines are dropped first.
+ """
+ body = getattr(request, "body", {})
+ messages: list[Any] = []
+ if isinstance(body, dict):
+ raw = body.get("messages")
+ if not isinstance(raw, list):
+ raw = body.get("input")
+ if isinstance(raw, list):
+ messages = raw
+
+ trimmed = trim_messages(messages, recent_turn_window=config.recent_turn_window)
+
+ anchor_lines: list[str] = []
+ window_lines: list[str] = []
+ if isinstance(body, dict):
+ top_system = body.get("system")
+ if isinstance(top_system, str | list):
+ anchor_lines.append(
+ "[system] " + truncate_middle(content_text(top_system), config.system_chars)
+ )
+ first_user_seen = False
+ for m in trimmed:
+ if not isinstance(m, dict):
+ continue
+ role = m.get("role")
+ text = message_text(m)
+ if role in ("system", "developer"):
+ anchor_lines.append(f"[{role}] " + truncate_middle(text, config.system_chars))
+ elif role == "user" and not first_user_seen:
+ first_user_seen = True
+ anchor_lines.append(
+ "[user (task)] " + truncate_middle(text, config.first_user_chars)
+ )
+ else:
+ label = role or m.get("type") or "message"
+ window_lines.append(
+ f"[{label}] " + truncate_middle(text, config.window_message_chars)
+ )
+
+ n_shown = len(window_lines)
+ header = (
+ f"Conversation turn {turn}; showing the last {n_shown} of "
+ f"{len(messages)} messages after the task framing."
+ )
+
+ def _assemble() -> str:
+ return "\n".join([header, *anchor_lines, *window_lines])
+
+ text = _assemble()
+ # Drop oldest window lines first: for a trajectory judge the newest
+ # evidence is strictly the most valuable.
+ while len(text) > config.max_request_chars and window_lines:
+ window_lines.pop(0)
+ text = _assemble()
+ if len(text) > config.max_request_chars:
+ text = text[: config.max_request_chars - 15] + "...<truncated>"
+ return text
+
+
+__all__ = [
+ "CTX_ESCALATION_VERDICT",
+ "ESCALATION_JUDGE_SYSTEM_PROMPT",
+ "EscalationJudgeConfig",
+ "EscalationJudgeRequestProcessor",
+ "EscalationVerdict",
+]
diff --git a/switchyard/lib/processors/llm_classifier/request_processor.py b/switchyard/lib/processors/llm_classifier/request_processor.py
index 6196fded4..8c6f73d57 100644
--- a/switchyard/lib/processors/llm_classifier/request_processor.py
+++ b/switchyard/lib/processors/llm_classifier/request_processor.py
@@ -22,6 +22,7 @@ from switchyard.lib.processors.llm_classifier.signals import (
RouteSignals,
RouteTier,
)
+from switchyard.lib.processors.message_condensing import strip_markdown_fence, trim_messages
from switchyard.lib.proxy_context import ProxyContext
from switchyard.lib.session_affinity import SessionAffinity
from switchyard.lib.stats_accumulator import StatsAccumulator
@@ -453,7 +454,7 @@ def parse_route_decision(
schema: type[RouteDecision] = RouteSignals,
) -> RouteDecision:
"""Parse classifier JSON output against the given decision schema."""
- stripped = _strip_markdown_fence(raw)
+ stripped = strip_markdown_fence(raw)
try:
return schema.model_validate_json(stripped)
except ValidationError as exc:
@@ -524,11 +525,11 @@ def _condense_body(
messages = body.get("messages")
if isinstance(messages, list):
- out["messages"] = _trim_messages(messages, recent_turn_window=recent_turn_window)
+ out["messages"] = trim_messages(messages, recent_turn_window=recent_turn_window)
raw_input = body.get("input")
if isinstance(raw_input, list):
- out["input"] = _trim_messages(raw_input, recent_turn_window=recent_turn_window)
+ out["input"] = trim_messages(raw_input, recent_turn_window=recent_turn_window)
elif isinstance(raw_input, str):
out["input"] = raw_input
@@ -550,78 +551,6 @@ def _condense_tool(tool: Any) -> Any:
return {k: v for k, v in tool.items() if k != "input_schema"}
-def _trim_messages(messages: list[Any], *, recent_turn_window: int = 0) -> list[Any]:
- """Keep system + first-user anchor + a trailing window of messages.
-
- Anchors retained unconditionally:
-
- * system / developer messages — global framing the classifier
- always needs.
- * the **first** user message — agent frameworks like terminus-2
- bundle task framing into ``role="user"`` rather than ``system``;
- losing it leaves the classifier blind to what the agent is
- working on.
-
- Trailing window controlled by ``recent_turn_window``:
-
- * ``0`` (default) — keep only the last user message. Smallest
- classifier prompt, but blind to recent assistant tool calls and
- tool results, so signal estimation (``tool_call_count_estimate``,
- DEBUG-vs-EXPLORATION turn type) must guess from a terse
- "Continue" echo. Tends to over-escalate on pessimistic
- classifiers.
- * ``N >= 1`` — keep the last ``N`` non-anchor messages
- (assistant / tool / non-first user) in original order. Gives
- the classifier visibility into recent agent activity. Each new
- turn appends to a stable prefix the upstream can prompt-cache,
- so the extra tokens are nearly free on cache-discounted backends
- (DeepSeek V4 Flash ~98% cache discount).
- """
- system_msgs: list[Any] = []
- first_user: Any = None
- first_user_idx: int | None = None
- for idx, m in enumerate(messages):
- if not isinstance(m, dict):
- continue
- role = m.get("role")
- if role in ("system", "developer"):
- system_msgs.append(m)
- elif role == "user" and first_user is None:
- first_user = m
- first_user_idx = idx
-
- if first_user is None:
- return system_msgs
-
- # Candidate tail = everything after the first user message that
- # isn't a system/developer anchor (those are already included).
- tail_candidates = [
- m
- for idx, m in enumerate(messages)
- if idx > (first_user_idx or 0)
- and isinstance(m, dict)
- and m.get("role") not in ("system", "developer")
- ]
-
- if recent_turn_window <= 0:
- # Historical behavior: keep only the last user message.
- last_user: Any = None
- for m in tail_candidates:
- if m.get("role") == "user":
- last_user = m
- if last_user is None:
- return [*system_msgs, first_user]
- if last_user is first_user:
- return [*system_msgs, first_user]
- return [*system_msgs, first_user, last_user]
-
- window = tail_candidates[-recent_turn_window:]
- # Filter out the first user (already pinned) to avoid duplicating
- # it if the window reaches back that far on short conversations.
- window = [m for m in window if m is not first_user]
- return [*system_msgs, first_user, *window]
-
-
def _completion_content(result: Any) -> str:
choices = getattr(result, "choices", None)
if not choices:
@@ -634,19 +563,6 @@ def _completion_content(result: Any) -> str:
raise LLMClassifierError("classifier completion had empty content")
-def _strip_markdown_fence(raw: str) -> str:
- stripped = raw.strip()
- if not stripped.startswith("```"):
- return stripped
-
- lines = stripped.splitlines()
- if lines and lines[0].startswith("```"):
- lines = lines[1:]
- if lines and lines[-1].startswith("```"):
- lines = lines[:-1]
- return "\n".join(lines).strip()
-
-
__all__ = [
"DEFAULT_CLASSIFIER_SYSTEM_PROMPT",
"ClassifierCompletion",
diff --git a/switchyard/lib/processors/message_condensing.py b/switchyard/lib/processors/message_condensing.py
new file mode 100644
index 000000000..7b74832b9
--- /dev/null
+++ b/switchyard/lib/processors/message_condensing.py
@@ -0,0 +1,185 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Shared helpers for condensing chat requests into routing-side LLM prompts.
+
+The LLM classifier, the escalation judge, and the plan-execute planner all
+show an inner LLM a compact view of the live request: anchors (system + first
+user message) plus a recent window, with bulk trimmed. These helpers are the
+common plumbing — message trimming, content flattening, truncation, and
+output-fence stripping — shared so the routing processors condense requests
+consistently instead of each keeping a private copy.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+
+def trim_messages(messages: list[Any], *, recent_turn_window: int = 0) -> list[Any]:
+ """Keep system + first-user anchor + a trailing window of messages.
+
+ Anchors retained unconditionally:
+
+ * system / developer messages — global framing the inner LLM
+ always needs.
+ * the **first** user message — agent frameworks like terminus-2
+ bundle task framing into ``role="user"`` rather than ``system``;
+ losing it leaves the inner LLM blind to what the agent is
+ working on.
+
+ Trailing window controlled by ``recent_turn_window``:
+
+ * ``0`` (default) — keep only the last user message. Smallest
+ prompt, but blind to recent assistant tool calls and
+ tool results, so signal estimation (``tool_call_count_estimate``,
+ DEBUG-vs-EXPLORATION turn type) must guess from a terse
+ "Continue" echo. Tends to over-escalate on pessimistic
+ classifiers.
+ * ``N >= 1`` — keep the last ``N`` non-anchor messages
+ (assistant / tool / non-first user) in original order. Gives
+ the inner LLM visibility into recent agent activity. Each new
+ turn appends to a stable prefix the upstream can prompt-cache,
+ so the extra tokens are nearly free on cache-discounted backends
+ (DeepSeek V4 Flash ~98% cache discount).
+ """
+ system_msgs: list[Any] = []
+ first_user: Any = None
+ first_user_idx: int | None = None
+ for idx, m in enumerate(messages):
+ if not isinstance(m, dict):
+ continue
+ role = m.get("role")
+ if role in ("system", "developer"):
+ system_msgs.append(m)
+ elif role == "user" and first_user is None:
+ first_user = m
+ first_user_idx = idx
+
+ if first_user is None:
+ return system_msgs
+
+ # Candidate tail = everything after the first user message that
+ # isn't a system/developer anchor (those are already included).
+ tail_candidates = [
+ m
+ for idx, m in enumerate(messages)
+ if idx > (first_user_idx or 0)
+ and isinstance(m, dict)
+ and m.get("role") not in ("system", "developer")
+ ]
+
+ if recent_turn_window <= 0:
+ # Historical behavior: keep only the last user message.
+ last_user: Any = None
+ for m in tail_candidates:
+ if m.get("role") == "user":
+ last_user = m
+ if last_user is None:
+ return [*system_msgs, first_user]
+ if last_user is first_user:
+ return [*system_msgs, first_user]
+ return [*system_msgs, first_user, last_user]
+
+ window = tail_candidates[-recent_turn_window:]
+ # Filter out the first user (already pinned) to avoid duplicating
+ # it if the window reaches back that far on short conversations.
+ window = [m for m in window if m is not first_user]
+ return [*system_msgs, first_user, *window]
+
+
+def message_text(message: dict[str, Any]) -> str:
+ """Flatten a chat message (or Responses item) to plain text, tool calls included."""
+ parts: list[str] = []
+ content = message.get("content")
+ if isinstance(content, str | list):
+ text = content_text(content)
+ if text:
+ parts.append(text)
+ # OpenAI chat tool calls: the command shapes an inner LLM needs for loop detection.
+ tool_calls = message.get("tool_calls")
+ if isinstance(tool_calls, list):
+ for call in tool_calls:
+ if not isinstance(call, dict):
+ continue
+ fn = call.get("function")
+ if isinstance(fn, dict):
+ parts.append(f"tool_call {fn.get('name')}({fn.get('arguments', '')})")
+ # OpenAI Responses items carry their payloads in type-specific fields.
+ if message.get("type") == "function_call":
+ parts.append(f"tool_call {message.get('name')}({message.get('arguments', '')})")
+ output = message.get("output")
+ if isinstance(output, str | list):
+ text = content_text(output)
+ if text:
+ parts.append(text)
+ return " ".join(parts)
+
+
+def content_text(content: str | list[Any]) -> str:
+ """Flatten string-or-content-block message content to text.
+
+ Handles plain strings, OpenAI/Anthropic text blocks, nested
+ ``tool_result`` content, and Anthropic ``tool_use`` blocks — the last
+ carry their payload in ``name``/``input`` rather than ``text``/``content``
+ and would otherwise flatten to nothing, erasing exactly the
+ repeated-command signal a trajectory judge relies on.
+ """
+ if isinstance(content, str):
+ return content
+ parts: list[str] = []
+ for block in content:
+ if isinstance(block, str):
+ parts.append(block)
+ continue
+ if not isinstance(block, dict):
+ continue
+ if block.get("type") == "tool_use":
+ arguments = json.dumps(
+ block.get("input", {}), ensure_ascii=False, sort_keys=True, default=str
+ )
+ parts.append(f"tool_call {block.get('name')}({arguments})")
+ continue
+ text = block.get("text")
+ if isinstance(text, str):
+ parts.append(text)
+ continue
+ inner = block.get("content")
+ if isinstance(inner, str | list):
+ parts.append(content_text(inner))
+ return " ".join(p for p in parts if p)
+
+
+def truncate_middle(text: str, limit: int) -> str:
+ """Keep the head and tail of ``text`` within ``limit`` chars."""
+ if len(text) <= limit:
+ return text
+ marker = " ...[trimmed] "
+ keep = max(limit - len(marker), 20)
+ head = (keep * 2) // 3
+ tail = keep - head
+ return text[:head] + marker + text[-tail:]
+
+
+def strip_markdown_fence(raw: str) -> str:
+ """Drop a wrapping ``` fence from an inner LLM's JSON output, if present."""
+ stripped = raw.strip()
+ if not stripped.startswith("```"):
+ return stripped
+
+ lines = stripped.splitlines()
+ if lines and lines[0].startswith("```"):
+ lines = lines[1:]
+ if lines and lines[-1].startswith("```"):
+ lines = lines[:-1]
+ return "\n".join(lines).strip()
+
+
+__all__ = [
+ "content_text",
+ "message_text",
+ "strip_markdown_fence",
+ "trim_messages",
+ "truncate_middle",
+]
diff --git a/switchyard/lib/processors/plan_execute/request_processor.py b/switchyard/lib/processors/plan_execute/request_processor.py
index 395451240..25e63c232 100644
--- a/switchyard/lib/processors/plan_execute/request_processor.py
+++ b/switchyard/lib/processors/plan_execute/request_processor.py
@@ -16,6 +16,7 @@ from typing import Any, Protocol
from pydantic import BaseModel, ConfigDict, Field, ValidationError
from switchyard.lib.llm_client import OpenAILLMClient
+from switchyard.lib.processors.message_condensing import strip_markdown_fence
from switchyard.lib.processors.plan_execute.plan import (
CTX_PLANNER_DECISION,
PlannerDecision,
@@ -759,7 +760,7 @@ def parse_planner_decision(raw: str) -> PlannerDecision:
emit one despite instructions). Any other deviation from the
schema raises :class:`PlanningError`.
"""
- stripped = _strip_markdown_fence(raw)
+ stripped = strip_markdown_fence(raw)
try:
return PlannerDecision.model_validate_json(stripped)
except ValidationError as exc:
@@ -977,19 +978,6 @@ def _tool_call_arguments(message: Any) -> str | None:
return None
-def _strip_markdown_fence(raw: str) -> str:
- stripped = raw.strip()
- if not stripped.startswith("```"):
- return stripped
-
- lines = stripped.splitlines()
- if lines and lines[0].startswith("```"):
- lines = lines[1:]
- if lines and lines[-1].startswith("```"):
- lines = lines[:-1]
- return "\n".join(lines).strip()
-
-
__all__ = [
"DEFAULT_INITIAL_PLANNER_SYSTEM_PROMPT",
"DEFAULT_REVISION_PLANNER_SYSTEM_PROMPT",
diff --git a/switchyard/lib/processors/prompts/escalation_judge.md b/switchyard/lib/processors/prompts/escalation_judge.md
new file mode 100644
index 000000000..6d57369bb
--- /dev/null
+++ b/switchyard/lib/processors/prompts/escalation_judge.md
@@ -0,0 +1,178 @@
+You are an escalation judge inside an agentic coding router. The session
+started on the EFFICIENT tier (a cheap but top-class 2026 model). Your
+job is to detect when the run is genuinely in trouble so the router can
+escalate the rest of the task to the STRONG tier (frontier, expensive).
+
+You see a condensed view of one session: the task framing (system prompt
++ first user message) and the most recent turns of activity (assistant
+messages and tool results). Judge the *trajectory* — is the agent making
+real progress toward the stated task — not the difficulty of the task
+itself. Return exactly one JSON object:
+
+{"escalate": boolean, "reason": "one short sentence naming the pattern"}
+
+Escalation is one-way for the rest of the task and expensive. Escalate
+only on a clear PATTERN of trouble, never on a single failed command.
+When the evidence is thin or ambiguous, return {"escalate": false}.
+
+The bar is not "is there friction" — agentic coding is full of friction
+the efficient tier works through on its own. The bar is "is this run
+likely DOOMED without intervention": the agent is stuck in place and
+its recent behavior shows no mechanism by which the next few turns
+would look different.
+
+# Is the stuck point beyond the efficient tier?
+
+Escalation pays only when the trouble is the KIND the strong tier is
+better at. The efficient tier handles routine coding, file
+exploration, single-file edits, normal debugging, dependency and
+environment setup, and most refactors on its own — being stuck on
+those is usually temporary. Weigh the kind of stuck point:
+
+Escalate sooner — the stuck point exceeds the efficient tier's
+capability, and a stronger model would likely break the loop:
+- Cross-module or cross-codebase synthesis: the fix requires learning
+ a convention, contract, or invariant from elsewhere in the codebase
+ and applying it consistently — even when the edit itself is
+ single-file.
+- Subtle invariants: plausible-looking fixes keep failing the same
+ test because the root cause hinges on a behavior contract none of
+ the attempts have touched.
+- Root causes genuinely spanning modules, or multi-step algorithmic /
+ formal reasoning the agent keeps getting almost-right.
+
+Hold weak — the efficient tier resolves these with iteration:
+- Procedural or mechanical friction: tool availability, installs,
+ service startup, recipe-following scaffolding, localized one-file
+ test fixes.
+
+Hold weak — no model can fix these, so escalation is pure waste:
+- The blocker is external: required data or files that simply do not
+ exist in the environment, a permanently broken or missing service,
+ or requirements the environment contradicts. A stronger model
+ changes nothing about a missing resource. One boundary to respect:
+ when producing, recovering, or decoding that very artifact IS the
+ stated task, its absence is the work itself, not a blocker — judge
+ the trajectory on it like any other work.
+
+# Trouble patterns — escalate when you see these
+
+Repetition and loops (the most common way agent runs die):
+- The same command or edit failing 2+ times with materially the same
+ error, especially with unrelated changes in between.
+- Near-identical tool calls repeated, or the same files re-read, without
+ new information gained — including longer cycles (A -> B -> C -> A).
+- Fighting the environment: repeatedly invoking a missing executable,
+ retrying installs that fail the same way, or trying variations of a
+ command the environment has already rejected, instead of adapting.
+
+False progress (looks like progress, is not):
+- Declaring success or moving on while the latest visible evidence
+ (test output, exit code, error text) shows failure.
+- Finishing without running the verification the task specifies, when
+ the task states how success is checked (e.g. "make the provided
+ tests pass") and running it was possible.
+- A reproduction or test the agent wrote that passes trivially without
+ exercising the actual issue, then building on that false signal.
+- The agent's stated reading of a tool result contradicts what the
+ result actually says (treating an error or empty output as success).
+
+Drift and dead ends:
+- Recent activity no longer serves the task in the first user message
+ (e.g. polishing style while the required feature is unstarted).
+ A debugging detour that plausibly unblocks the task — fixing the
+ environment, starting a required service, investigating an error in
+ a dependency — is NOT drift; call drift only when the detour has
+ produced nothing useful for many turns AND the task's real
+ verification remains untouched.
+- Violating an explicit task constraint (modifying files the task says
+ not to touch, changing the tests instead of the code under test).
+- Editing or reasoning about code without ever having opened the files
+ the errors point to — acting on guessed file contents.
+- Contradicting or re-deriving something already established earlier in
+ the session (forgetting its own findings).
+- Many turns elapsed with nothing durable produced (no successful
+ writes, no passing checks) and no visible narrowing of the problem —
+ the run is on pace to exhaust its turn budget.
+
+Desperation:
+- Giving up: declaring the task impossible, asking to stop, or drifting
+ into restating the problem instead of acting on it.
+- Destructive flailing: rm -rf, wholesale reinstalls, chmod -R, or
+ reverting everything as a reaction to being stuck rather than a
+ reasoned step.
+
+# Expected friction — do NOT escalate on these
+
+Agentic coding is full of failures that are part of healthy work:
+- A test written to fail first (TDD) or a bug being reproduced on
+ purpose.
+- A compile, lint, or test error fixed or meaningfully acted on in the
+ immediately following turn.
+- Exploration dead-ends early in a session (grep with no matches,
+ reading a file that turns out to be irrelevant) while the agent is
+ still orienting.
+- A missing tool handled adaptively (tries `rg`, falls back to `grep`).
+- Sequential alternatives: trying a DIFFERENT library, tool, or
+ approach after one fails is adaptation, not a loop — even when
+ several alternatives fail in a row. The loop pattern requires the
+ SAME approach retried without material change.
+- A service that is unreachable or not yet running (server not
+ started, port closed, connection refused) while the agent is still
+ actively working to start, configure, or replace it.
+- Planning activity: todo-list and plan updates (TodoWrite,
+ update_plan) are routine agent workflow — planning is neither drift
+ nor struggle, and harness-injected skill/instruction reading early
+ in a session is orientation, not off-task work.
+- Zero-count summaries: "0 failed", "0 errors", "0 warnings" are
+ CLEAN results. Read failure keywords together with their counts —
+ only a nonzero count is a failure.
+- A long-running command (build, install, test suite) that simply has
+ not finished, or the agent waiting on information it asked for.
+
+The distinguishing question: is each failure producing new information
+that changes the next action? Failing forward is fine; failing in place
+is trouble. Also weigh the session's own recovery record: if this same
+session already shows friction the agent subsequently cleared (a failure
+followed by a verified fix or passing check), lean toward holding — a
+session that has recovered before will usually recover again.
+
+# Worked examples (none drawn from any benchmark task set)
+
+* Turn 3; the agent ran the test suite, 4 tests fail, and it is now
+ reading the first failing test. -> {"escalate": false} — reproducing
+ failures is the job.
+* The agent has run `pytest tests/test_api.py` 4 times with the same
+ ImportError, editing an unrelated config file between attempts. ->
+ {"escalate": true, "reason": "same ImportError 4 times while editing
+ unrelated files"}
+* `conda` is not installed; the agent has tried `conda install` five
+ ways instead of using the `pip` that earlier output showed present.
+ -> {"escalate": true, "reason": "fighting missing executable instead
+ of adapting"}
+* Task: "make the provided integration tests pass." Recent turns:
+ renaming variables and reformatting docstrings; tests not run in 8
+ turns. -> {"escalate": true, "reason": "drifted to cosmetic edits,
+ verification abandoned"}
+* The agent says "All tests pass, task complete" but the last visible
+ test output shows "2 failed, 11 passed". -> {"escalate": true,
+ "reason": "claims success contradicted by latest test output"}
+* The agent wrote a reproduction script that exits 0 without invoking
+ the code path the issue describes, concluded "bug not reproducible",
+ and is wrapping up. -> {"escalate": true, "reason": "reproduction
+ never exercised the reported code path"}
+* Two turns of edits, one failed build, then a fixed build and a
+ passing test. -> {"escalate": false}
+* `npm install` has been running for one turn with no output yet. ->
+ {"escalate": false} — slow command, not a stall.
+* Four different serialization libraries failed to import; the agent
+ is now writing the converter with a fifth approach it has not tried
+ before. -> {"escalate": false} — sequential alternatives are
+ adaptation, even when none has succeeded yet.
+* Task: tune a slow batch pipeline. The agent is investigating why
+ the message broker fails to start, since the pipeline cannot be
+ measured without it. -> {"escalate": false} — unblocking
+ verification serves the task.
+
+Do not emit markdown, commentary, or chain-of-thought — only the JSON
+object.
diff --git a/switchyard/lib/profiles/__init__.py b/switchyard/lib/profiles/__init__.py
index e71f5dcae..421c27466 100644
--- a/switchyard/lib/profiles/__init__.py
+++ b/switchyard/lib/profiles/__init__.py
@@ -12,6 +12,10 @@ from switchyard.lib.profiles.deterministic_routing_presets import (
from switchyard.lib.profiles.deterministic_routing_profile_config import (
DeterministicRoutingProfileConfig,
)
+from switchyard.lib.profiles.escalation_router_config import EscalationRouterConfig
+from switchyard.lib.profiles.escalation_router_profile_config import (
+ EscalationRouterProfileConfig,
+)
from switchyard.lib.profiles.header_routing import (
HeaderRoutingConfig,
HeaderRoutingDecision,
@@ -62,6 +66,8 @@ __all__ = [
"DeterministicRoutingConfig",
"DeterministicRoutingProfileConfig",
"DeterministicRoutingPresets",
+ "EscalationRouterConfig",
+ "EscalationRouterProfileConfig",
"HeaderRoutingConfig",
"HeaderRoutingDecision",
"HeaderRoutingProfile",
diff --git a/switchyard/lib/profiles/deterministic_routing_config.py b/switchyard/lib/profiles/deterministic_routing_config.py
index 082762044..8a0b893b6 100644
--- a/switchyard/lib/profiles/deterministic_routing_config.py
+++ b/switchyard/lib/profiles/deterministic_routing_config.py
@@ -124,6 +124,17 @@ class DeterministicRoutingConfig(BaseModel):
raise ValueError("target.model must be a non-empty string")
return tier
+ @field_validator("weak")
+ @classmethod
+ def _tier_ids_distinct(cls, value: LlmTarget, info: ValidationInfo) -> LlmTarget:
+ strong = info.data.get("strong")
+ if isinstance(strong, LlmTarget) and strong.id == value.id:
+ raise ValueError(
+ f"strong.id and weak.id must differ (both are {value.id!r}); "
+ "they key the routing tiers"
+ )
+ return value
+
@field_validator("classifier_system_prompt", mode="before")
@classmethod
def _blank_classifier_prompt_is_unset(cls, value: object) -> object:
diff --git a/switchyard/lib/profiles/deterministic_routing_profile_config.py b/switchyard/lib/profiles/deterministic_routing_profile_config.py
index fc32ecaa5..3be3eb9c2 100644
--- a/switchyard/lib/profiles/deterministic_routing_profile_config.py
+++ b/switchyard/lib/profiles/deterministic_routing_profile_config.py
@@ -7,20 +7,16 @@ from __future__ import annotations
from typing import Any, Self
-from switchyard.lib.backends.llm_target import LlmTarget
from switchyard.lib.processors.llm_classifier.presets import (
PROFILE_FACTORIES,
resolve_classifier_prompt,
)
from switchyard.lib.profiles.chain import ComponentChainProfile
from switchyard.lib.profiles.deterministic_routing_config import (
- DEFAULT_DETERMINISTIC_TIER_TIMEOUT_S,
DeterministicRoutingConfig,
)
from switchyard.lib.profiles.table import profile_config
-
-_TIER_STRONG = "strong"
-_TIER_WEAK = "weak"
+from switchyard.lib.profiles.tier_target_builders import build_tier_backend
@profile_config("deterministic")
@@ -36,16 +32,9 @@ class DeterministicRoutingProfileConfig:
def build(self) -> ComponentChainProfile:
"""Build the deterministic routing profile runtime."""
- from switchyard.lib.backends.anthropic_cache_breakpoint_backend import (
- maybe_wrap_anthropic_cache,
- )
from switchyard.lib.backends.deterministic_routing_llm_backend import (
DeterministicRoutingLLMBackend,
)
- from switchyard.lib.backends.multi_llm_backend import (
- build_native_backend,
- resolve_llm_target,
- )
from switchyard.lib.processors.llm_classifier import (
LLMClassifierRequestProcessor,
SignalTierSelectorRequestProcessor,
@@ -56,9 +45,17 @@ class DeterministicRoutingProfileConfig:
from switchyard.lib.session_affinity import SessionAffinity
config = self.config
+ # Tier labels are the configured target ids ("strong"/"weak" unless
+ # overridden). The tier selector stamps them, the backend keys its
+ # tier dict by them, and the chain's evict-and-retry rewrites
+ # selected_target to fallback_target_on_evict — which the config
+ # validates against these same ids — so an overflow reroute always
+ # lands on a registered tier.
+ strong_id = config.strong.id
+ weak_id = config.weak.id
profile = PROFILE_FACTORIES[config.profile_name](
- weak=_TIER_WEAK,
- strong=_TIER_STRONG,
+ weak=weak_id,
+ strong=strong_id,
)
request_processors: list[Any] = [ReasoningEffortNormalizer()]
@@ -101,40 +98,19 @@ class DeterministicRoutingProfileConfig:
)
)
- # Resolve format='auto' once after deterministic tier defaults are
- # applied so backend selection and Anthropic cache wrapping see the
- # same concrete target.
- strong_target = resolve_llm_target(
- _apply_deepseek_overrides(
- _apply_default_tier_timeout(
- config.strong,
- config.tier_timeout_s,
- ),
- ),
+ strong_target, strong_backend = build_tier_backend(
+ config.strong, config.tier_timeout_s,
)
- weak_target = resolve_llm_target(
- _apply_deepseek_overrides(
- _apply_default_tier_timeout(
- config.weak,
- config.tier_timeout_s,
- ),
- ),
- )
- strong_backend = maybe_wrap_anthropic_cache(
- build_native_backend(strong_target),
- strong_target,
- )
- weak_backend = maybe_wrap_anthropic_cache(
- build_native_backend(weak_target),
- weak_target,
+ weak_target, weak_backend = build_tier_backend(
+ config.weak, config.tier_timeout_s,
)
backend = DeterministicRoutingLLMBackend(
tiers={
- _TIER_STRONG: (strong_backend, strong_target.model),
- _TIER_WEAK: (weak_backend, weak_target.model),
+ strong_id: (strong_backend, strong_target.model),
+ weak_id: (weak_backend, weak_target.model),
},
- default_tier=_TIER_STRONG,
+ default_tier=strong_id,
)
return ComponentChainProfile(
request_processors=request_processors,
@@ -143,60 +119,4 @@ class DeterministicRoutingProfileConfig:
)
-def _apply_deepseek_overrides(target: LlmTarget) -> LlmTarget:
- """Apply benchmark-specific DeepSeek extras without clobbering callers."""
- default_body = (
- {"chat_template_kwargs": {"enable_thinking": False}}
- if "deepseek-v4" in target.model
- else None
- )
- default_headers = (
- {"X-Inference-Priority": "batch"}
- if "deepseek" in target.model
- else None
- )
- if default_body is None and default_headers is None:
- return target
-
- existing_body = target.extra_body
- # LlmTarget normalizes omitted and explicit empty headers to the same
- # empty dict, so keep current defaulting behavior for normal DeepSeek
- # targets until the target type preserves "headers were provided" state.
- existing_headers = target.extra_headers or None
- merged_body = existing_body if existing_body is not None else default_body
- merged_headers = existing_headers if existing_headers is not None else default_headers
- if merged_body == existing_body and merged_headers == existing_headers:
- return target
-
- return LlmTarget(
- id=target.id,
- model=target.model,
- format=target.format,
- base_url=target.base_url,
- api_key=target.api_key,
- timeout_secs=target.endpoint.timeout_secs,
- extra_body=merged_body,
- extra_headers=merged_headers,
- )
-
-
-def _apply_default_tier_timeout(
- target: LlmTarget,
- timeout_s: float | None = DEFAULT_DETERMINISTIC_TIER_TIMEOUT_S,
-) -> LlmTarget:
- """Apply deterministic tier timeout when a target has no explicit timeout."""
- if timeout_s is None or target.endpoint.timeout_secs is not None:
- return target
- return LlmTarget(
- id=target.id,
- model=target.model,
- format=target.format,
- base_url=target.base_url,
- api_key=target.api_key,
- timeout_secs=timeout_s,
- extra_body=target.extra_body,
- extra_headers=target.extra_headers,
- )
-
-
__all__ = ["DeterministicRoutingProfileConfig"]
diff --git a/switchyard/lib/profiles/escalation_router_config.py b/switchyard/lib/profiles/escalation_router_config.py
new file mode 100644
index 000000000..1e63b95fc
--- /dev/null
+++ b/switchyard/lib/profiles/escalation_router_config.py
@@ -0,0 +1,179 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Config model for the escalation-router profile (judge-latched strong/weak)."""
+
+from __future__ import annotations
+
+from typing import Literal, Self
+
+from pydantic import (
+ BaseModel,
+ ConfigDict,
+ Field,
+ ValidationInfo,
+ field_validator,
+ model_validator,
+)
+
+from switchyard.lib.backends.llm_target import LlmTarget, coerce_llm_target
+from switchyard.lib.profiles.deterministic_routing_config import (
+ DEFAULT_DETERMINISTIC_TIER_TIMEOUT_S,
+)
+
+
+class EscalationRouterConfig(BaseModel):
+ """Configuration for the escalation-router profile.
+
+ Every conversation starts on the ``weak`` tier. An LLM ``judge`` watches
+ the trajectory each turn and, on a clear pattern of trouble, escalates the
+ conversation to the ``strong`` tier — one-way for the rest of the task
+ (session-affinity latch). A new conversation resets to weak.
+
+ Attributes:
+ strong: Escalation target (frontier / expensive model).
+ weak: Starting tier (cheap / efficient model).
+ judge: Target for the judge LLM call. ``model`` / ``base_url`` /
+ ``api_key`` / ``timeout_secs`` are extracted at build time;
+ other target fields are ignored.
+ fallback_target_on_evict: Target id the chain executor reroutes to on
+ eviction (context-window overflow). Must match ``strong.id`` or
+ ``weak.id``; the judge target is not a routing candidate.
+ judge_min_turn: First conversation turn on which the judge runs.
+ judge_escalate_confirmations: Consecutive escalate verdicts required
+ before the strong latch fires (``1`` pins on the first verdict).
+ The confirmation streak is tracked per process: with multiple
+ workers and no session-sticky load balancing, verdicts for one
+ conversation can land on different workers and take longer to
+ accumulate. Keep ``1`` (the default) on multi-worker deployments,
+ or route conversations sticky to a worker.
+ judge_confirmation_window: Judged turns an escalate verdict stays
+ live for confirmation; ``N > 1`` tolerates up to ``N - 1``
+ intervening declines (recurring intermittent trouble confirms).
+ judge_disable_reasoning: Send the thinking-off template hint on judge
+ calls (default). ``False`` lets a reasoning judge model think,
+ trading per-turn latency for rubric adherence. The hint is only
+ sent to models that accept it (Claude/Bedrock reject it).
+ judge_max_completion_tokens: Completion-token budget for the judge
+ call. ``None`` (default) picks 128 with the thinking-off hint in
+ effect and 4096 otherwise — reasoning tokens and the JSON verdict
+ share this budget, and a too-small cap truncates mid-reasoning so
+ the judge silently fails open every turn.
+ judge_dump_verdicts: Emit one ``escalation_verdict={...}`` line to
+ stderr per judge call (includes a first-user ``task_hint``
+ snippet). Benchmark-only diagnostics, off by default; production
+ routing telemetry flows through the normal stats path.
+ judge_recent_turn_window: Trailing messages shown to the judge on top
+ of the system + first-user anchors.
+ judge_window_message_chars: Per-message truncation cap inside the
+ trailing window (larger keeps more tool-output detail per turn).
+ judge_max_request_chars: Cap on the assembled judge transcript.
+ judge_system_prompt: Optional judge prompt override. ``None`` uses the
+ built-in prompt.
+ judge_timeout_s: Per-call judge timeout (seconds); the judge fails
+ open to the weak tier at timeout.
+ session_key_depth: ``0`` (default) keys conversations on system +
+ first user message (the shared Rust session key). ``N > 0``
+ extends the key with the first ``N`` post-first-user messages so
+ repeated runs of the identical task (k>1 benchmark trials against
+ one server) diverge via early model responses instead of sharing
+ an escalation latch. Requires nonzero sampling temperature to be
+ effective; production traffic should keep ``0``.
+ tier_timeout_s: Default per-call timeout for strong/weak tier calls
+ when a target does not set its own ``timeout_secs``.
+ enable_stats: Wire stats processors and per-tier stats wrappers.
+ affinity_max_sessions: LRU capacity of the escalation latch store.
+ affinity_store: Shared L2 behind the in-process latch LRU. ``"memory"``
+ (default) keeps the escalation latch per process — a worker or pod
+ change can lose it. ``"redis"`` shares the latch across workers
+ and persists it across pod churn (best-effort: an L2 error never
+ fails a request).
+ affinity_store_url: Connection URL for the shared store (e.g.
+ ``"redis://host:6379/0"``). Required when ``affinity_store`` is
+ ``"redis"``.
+ affinity_store_ttl_seconds: Expiry for a shared latch entry. The latch
+ is re-read every turn; an escalated conversation older than the
+ TTL restarts on weak.
+ affinity_key_prefix: Namespace prefix for shared-store keys.
+ """
+
+ model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True)
+
+ strong: LlmTarget
+ weak: LlmTarget
+ judge: LlmTarget
+ fallback_target_on_evict: str
+ judge_min_turn: int = Field(default=3, ge=1)
+ judge_escalate_confirmations: int = Field(default=1, ge=1)
+ judge_confirmation_window: int = Field(default=1, ge=1)
+ judge_disable_reasoning: bool = True
+ judge_max_completion_tokens: int | None = Field(default=None, ge=16)
+ judge_dump_verdicts: bool = False
+ judge_recent_turn_window: int = Field(default=14, ge=1)
+ judge_window_message_chars: int = Field(default=300, ge=50)
+ judge_max_request_chars: int = Field(default=12_000, ge=1_000)
+ judge_system_prompt: str | None = Field(default=None, min_length=1)
+ judge_timeout_s: float = Field(default=5.0, gt=0.0)
+ session_key_depth: int = Field(default=0, ge=0)
+ tier_timeout_s: float | None = Field(
+ default=DEFAULT_DETERMINISTIC_TIER_TIMEOUT_S,
+ gt=0.0,
+ )
+ enable_stats: bool = True
+ affinity_max_sessions: int = Field(default=10_000, gt=0)
+ affinity_store: Literal["memory", "redis"] = "memory"
+ affinity_store_url: str | None = None
+ affinity_store_ttl_seconds: int = Field(default=3_600, gt=0)
+ affinity_key_prefix: str = "swyd:esc:"
+
+ @model_validator(mode="after")
+ def _redis_store_requires_url(self) -> Self:
+ # A shared store is dead config unless it is reachable.
+ if self.affinity_store == "redis" and not self.affinity_store_url:
+ raise ValueError('affinity_store="redis" requires affinity_store_url to be set')
+ return self
+
+ @field_validator("strong", "weak", "judge", mode="before")
+ @classmethod
+ def _coerce_target(cls, value: object, info: ValidationInfo) -> LlmTarget:
+ return coerce_llm_target(value, default_id=info.field_name or "target")
+
+ @field_validator("strong", "weak", "judge")
+ @classmethod
+ def _target_model_non_empty(cls, tier: LlmTarget) -> LlmTarget:
+ if not tier.model:
+ raise ValueError("target.model must be a non-empty string")
+ return tier
+
+ @field_validator("weak")
+ @classmethod
+ def _tier_ids_distinct(cls, value: LlmTarget, info: ValidationInfo) -> LlmTarget:
+ strong = info.data.get("strong")
+ if isinstance(strong, LlmTarget) and strong.id == value.id:
+ raise ValueError(
+ f"strong.id and weak.id must differ (both are {value.id!r}); "
+ "they key the routing tiers"
+ )
+ return value
+
+ @field_validator("judge_system_prompt", mode="before")
+ @classmethod
+ def _blank_judge_prompt_is_unset(cls, value: object) -> object:
+ if isinstance(value, str) and not value.strip():
+ return None
+ return value
+
+ @field_validator("fallback_target_on_evict")
+ @classmethod
+ def _fallback_matches_existing_target(cls, value: str, info: ValidationInfo) -> str:
+ valid_ids = {info.data[key].id for key in ("strong", "weak") if key in info.data}
+ if value not in valid_ids:
+ raise ValueError(
+ f"fallback_target_on_evict={value!r} must match one of "
+ f"{sorted(valid_ids)} (the configured strong/weak target ids; "
+ f"the judge target is not a routing candidate)"
+ )
+ return value
+
+
+__all__ = ["EscalationRouterConfig"]
diff --git a/switchyard/lib/profiles/escalation_router_profile_config.py b/switchyard/lib/profiles/escalation_router_profile_config.py
new file mode 100644
index 000000000..96c59db29
--- /dev/null
+++ b/switchyard/lib/profiles/escalation_router_profile_config.py
@@ -0,0 +1,157 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Profile-owned escalation-router construction (judge-latched strong/weak)."""
+
+from __future__ import annotations
+
+from typing import Any, Self
+
+from switchyard.lib.affinity_pin_store import AffinityPinStore
+from switchyard.lib.backends.deterministic_routing_llm_backend import (
+ DeterministicRoutingLLMBackend,
+)
+from switchyard.lib.processors.escalation_judge_request_processor import (
+ ESCALATION_JUDGE_SYSTEM_PROMPT,
+ EscalationJudgeConfig,
+ EscalationJudgeRequestProcessor,
+)
+from switchyard.lib.processors.reasoning_effort_normalizer import (
+ ReasoningEffortNormalizer,
+)
+from switchyard.lib.processors.reasoning_hint import model_accepts_reasoning_hint
+from switchyard.lib.profiles.chain import ComponentChainProfile
+from switchyard.lib.profiles.escalation_router_config import EscalationRouterConfig
+from switchyard.lib.profiles.table import profile_config
+from switchyard.lib.profiles.tier_target_builders import (
+ apply_deepseek_overrides,
+ build_tier_backend,
+)
+from switchyard.lib.session_affinity import SessionAffinity
+
+
+@profile_config("escalation_router")
+class EscalationRouterProfileConfig:
+ """Profile config wrapper for judge-latched escalation routing."""
+
+ config: EscalationRouterConfig
+
+ @classmethod
+ def from_config(cls, config: EscalationRouterConfig) -> Self:
+ """Create a profile config from the validated parsing model."""
+ return cls(config=config)
+
+ def build(self) -> ComponentChainProfile:
+ """Build the escalation-router profile runtime."""
+ config = self.config
+
+ # The affinity store IS the escalation latch, so it is always on.
+ # Warmup gating lives in the judge processor (min_judge_turn), not
+ # here — affinity warmup would delay a decided escalation from
+ # taking effect, not just from being decided. The optional Redis L2
+ # keeps the latch across workers and pod churn; without it a worker
+ # change silently restarts an escalated conversation on weak.
+ affinity = SessionAffinity(
+ enabled=True,
+ max_sessions=config.affinity_max_sessions,
+ warmup_turns=0,
+ l2=_build_affinity_l2(config),
+ )
+
+ # DeepSeek judges get the same benchmark-gateway defaults as DeepSeek
+ # tiers (``X-Inference-Priority: batch``) so their calls land on the
+ # relaxed-timeout gateway alongside the routed tier traffic.
+ judge_target = apply_deepseek_overrides(config.judge)
+ # Claude/Bedrock judges reject the vLLM-only chat_template_kwargs
+ # hint outright — every judged turn would fail open to weak — so
+ # the hint is only sent to models that tolerate it (same gate as
+ # the classifier presets and the stage router).
+ judge_disable_reasoning = (
+ config.judge_disable_reasoning
+ and model_accepts_reasoning_hint(judge_target.model)
+ )
+ judge_config = EscalationJudgeConfig(
+ model=judge_target.model,
+ api_key=judge_target.api_key,
+ base_url=judge_target.base_url,
+ timeout_s=judge_target.endpoint.timeout_secs or config.judge_timeout_s,
+ system_prompt=config.judge_system_prompt or ESCALATION_JUDGE_SYSTEM_PROMPT,
+ min_judge_turn=config.judge_min_turn,
+ escalate_confirmations=config.judge_escalate_confirmations,
+ confirmation_window=config.judge_confirmation_window,
+ disable_reasoning=judge_disable_reasoning,
+ # Reasoning tokens and the JSON verdict share one budget: a
+ # thinking judge needs the classifier's larger ceiling or it
+ # truncates mid-reasoning and silently fails open every turn.
+ max_completion_tokens=(
+ config.judge_max_completion_tokens
+ if config.judge_max_completion_tokens is not None
+ else (128 if judge_disable_reasoning else 4096)
+ ),
+ dump_verdicts_to_stderr=config.judge_dump_verdicts,
+ recent_turn_window=config.judge_recent_turn_window,
+ window_message_chars=config.judge_window_message_chars,
+ max_request_chars=config.judge_max_request_chars,
+ extra_headers=judge_target.extra_headers or None,
+ )
+ # Tier labels are the configured target ids ("strong"/"weak" unless
+ # overridden). The judge stamps them, the backend keys its tier dict
+ # by them, and the chain's evict-and-retry rewrites selected_target to
+ # fallback_target_on_evict — which the config validates against these
+ # same ids — so an overflow reroute always lands on a registered tier.
+ strong_id = config.strong.id
+ weak_id = config.weak.id
+ request_processors: list[Any] = [
+ ReasoningEffortNormalizer(),
+ EscalationJudgeRequestProcessor(
+ judge_config,
+ affinity=affinity,
+ session_key_depth=config.session_key_depth,
+ strong_tier=strong_id,
+ weak_tier=weak_id,
+ ),
+ ]
+
+ strong_target, strong_backend = build_tier_backend(
+ config.strong, config.tier_timeout_s,
+ )
+ weak_target, weak_backend = build_tier_backend(
+ config.weak, config.tier_timeout_s,
+ )
+
+ backend = DeterministicRoutingLLMBackend(
+ tiers={
+ strong_id: (strong_backend, strong_target.model),
+ weak_id: (weak_backend, weak_target.model),
+ },
+ # Weak is the resting state; the judge processor stamps a tier on
+ # every turn, so the default only covers malformed metadata.
+ default_tier=weak_id,
+ )
+ return ComponentChainProfile(
+ request_processors=request_processors,
+ backend=backend,
+ fallback_target_on_evict=config.fallback_target_on_evict,
+ )
+
+
+def _build_affinity_l2(config: EscalationRouterConfig) -> AffinityPinStore | None:
+ """Build the optional shared L2 latch store (``None`` = L1-only).
+
+ Mirrors the latency route's ``affinity_store`` semantics; ``"redis"``
+ imports :class:`RedisPinStore` lazily so the ``redis`` dependency stays
+ optional. Config validation guarantees a URL when the store is Redis.
+ """
+ if config.affinity_store != "redis":
+ return None
+ from switchyard.lib.redis_pin_store import RedisPinStore
+
+ assert config.affinity_store_url is not None # enforced by config validator
+ return RedisPinStore(
+ config.affinity_store_url,
+ ttl_seconds=config.affinity_store_ttl_seconds,
+ key_prefix=config.affinity_key_prefix,
+ )
+
+
+__all__ = ["EscalationRouterProfileConfig"]
diff --git a/switchyard/lib/profiles/tier_target_builders.py b/switchyard/lib/profiles/tier_target_builders.py
new file mode 100644
index 000000000..4b87c56ac
--- /dev/null
+++ b/switchyard/lib/profiles/tier_target_builders.py
@@ -0,0 +1,106 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Shared strong/weak tier target preparation and backend construction.
+
+The deterministic (classifier) router and the escalation router build the same
+two-tier shape: apply the default tier timeout, apply DeepSeek
+benchmark-gateway overrides, resolve ``format='auto'`` once, build the native
+backend, and wrap Anthropic targets with cache breakpoints. Owning those rules
+here keeps the two profiles from drifting as the rules change.
+"""
+
+from __future__ import annotations
+
+from switchyard.lib.backends.anthropic_cache_breakpoint_backend import (
+ maybe_wrap_anthropic_cache,
+)
+from switchyard.lib.backends.llm_target import LlmTarget
+from switchyard.lib.backends.multi_llm_backend import (
+ build_native_backend,
+ resolve_llm_target,
+)
+from switchyard.lib.profiles.deterministic_routing_config import (
+ DEFAULT_DETERMINISTIC_TIER_TIMEOUT_S,
+)
+from switchyard.lib.roles import LLMBackend
+
+
+def apply_deepseek_overrides(target: LlmTarget) -> LlmTarget:
+ """Apply benchmark-specific DeepSeek extras without clobbering callers."""
+ default_body = (
+ {"chat_template_kwargs": {"enable_thinking": False}}
+ if "deepseek-v4" in target.model
+ else None
+ )
+ default_headers = (
+ {"X-Inference-Priority": "batch"}
+ if "deepseek" in target.model
+ else None
+ )
+ if default_body is None and default_headers is None:
+ return target
+
+ existing_body = target.extra_body
+ # LlmTarget normalizes omitted and explicit empty headers to the same
+ # empty dict, so keep current defaulting behavior for normal DeepSeek
+ # targets until the target type preserves "headers were provided" state.
+ existing_headers = target.extra_headers or None
+ merged_body = existing_body if existing_body is not None else default_body
+ merged_headers = existing_headers if existing_headers is not None else default_headers
+ if merged_body == existing_body and merged_headers == existing_headers:
+ return target
+
+ return LlmTarget(
+ id=target.id,
+ model=target.model,
+ format=target.format,
+ base_url=target.base_url,
+ api_key=target.api_key,
+ timeout_secs=target.endpoint.timeout_secs,
+ extra_body=merged_body,
+ extra_headers=merged_headers,
+ )
+
+
+def apply_default_tier_timeout(
+ target: LlmTarget,
+ timeout_s: float | None = DEFAULT_DETERMINISTIC_TIER_TIMEOUT_S,
+) -> LlmTarget:
+ """Apply the default tier timeout when a target has no explicit timeout."""
+ if timeout_s is None or target.endpoint.timeout_secs is not None:
+ return target
+ return LlmTarget(
+ id=target.id,
+ model=target.model,
+ format=target.format,
+ base_url=target.base_url,
+ api_key=target.api_key,
+ timeout_secs=timeout_s,
+ extra_body=target.extra_body,
+ extra_headers=target.extra_headers,
+ )
+
+
+def build_tier_backend(
+ target: LlmTarget,
+ tier_timeout_s: float | None,
+) -> tuple[LlmTarget, LLMBackend]:
+ """Prepare one tier target and build its wrapped backend.
+
+ Resolves ``format='auto'`` once after the tier defaults are applied so
+ backend selection and Anthropic cache wrapping see the same concrete
+ target. Returns ``(resolved_target, backend)`` — profiles need the
+ resolved target for the tier's model id.
+ """
+ resolved = resolve_llm_target(
+ apply_deepseek_overrides(apply_default_tier_timeout(target, tier_timeout_s)),
+ )
+ return resolved, maybe_wrap_anthropic_cache(build_native_backend(resolved), resolved)
+
+
+__all__ = [
+ "apply_deepseek_overrides",
+ "apply_default_tier_timeout",
+ "build_tier_backend",
+]
diff --git a/switchyard/lib/session_affinity.py b/switchyard/lib/session_affinity.py
index eca46d2ec..a4ef6e571 100644
--- a/switchyard/lib/session_affinity.py
+++ b/switchyard/lib/session_affinity.py
@@ -8,7 +8,7 @@ from __future__ import annotations
import inspect
import logging
import time
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, overload
from switchyard.lib.conversation_turn import conversation_turn_number
from switchyard.lib.proxy_context import ProxyContext
@@ -35,6 +35,35 @@ _L2_BREAKER_COOLDOWN_S = 10.0
CTX_SESSION_KEY = "_session_affinity_key"
+@overload
+def resolve_session_key(ctx: ProxyContext, request: ChatRequest) -> str: ...
+
+
+@overload
+def resolve_session_key(ctx: ProxyContext, request: ChatRequest, *, depth: int) -> str | None: ...
+
+
+def resolve_session_key(ctx: ProxyContext, request: ChatRequest, *, depth: int = 0) -> str | None:
+ """Derive the conversation key once per request, memoized on ``ctx``.
+
+ ``depth > 0`` derives the deep key (see
+ :func:`~switchyard.lib.session_key.session_key_from_body`) the escalation
+ judge uses for repeated-trial benchmarking, and returns ``None`` — without
+ memoizing — while the conversation is still shorter than the requested
+ prefix. A key already memoized on ``ctx`` (by an earlier component this
+ request, whatever its depth) wins over re-derivation, so every consumer
+ sees one consistent key per request.
+ """
+ cached = ctx.metadata.get(CTX_SESSION_KEY)
+ if isinstance(cached, str):
+ return cached
+ key = session_key_from_body(request.body, depth)
+ if key is None:
+ return None
+ ctx.metadata[CTX_SESSION_KEY] = key
+ return key
+
+
class SessionAffinity:
"""Pins a routing decision per conversation and reuses it on later turns.
@@ -238,16 +267,11 @@ class SessionAffinity:
def _session_key(self, ctx: ProxyContext, request: ChatRequest) -> str:
"""Derive the conversation key once per request, memoized on ``ctx``."""
- cached = ctx.metadata.get(CTX_SESSION_KEY)
- if isinstance(cached, str):
- return cached
- key = session_key_from_body(request.body)
- ctx.metadata[CTX_SESSION_KEY] = key
- return key
+ return resolve_session_key(ctx, request)
def _is_warmup_turn(self, request: ChatRequest) -> bool:
"""Return whether this request is still inside the no-stick warmup."""
return conversation_turn_number(request) <= self._warmup_turns
-__all__ = ["CTX_SESSION_KEY", "SessionAffinity"]
+__all__ = ["CTX_SESSION_KEY", "SessionAffinity", "resolve_session_key"]
diff --git a/switchyard/lib/session_key.py b/switchyard/lib/session_key.py
index 8b63b0c19..2026a0ab9 100644
--- a/switchyard/lib/session_key.py
+++ b/switchyard/lib/session_key.py
@@ -9,6 +9,29 @@ plus the first user message — so every turn of one conversation hashes alike.
from __future__ import annotations
-from switchyard_rust.core import session_key_from_body
+from typing import Any, Literal, overload
+
+from switchyard_rust.core import session_key_from_body as _native_session_key_from_body
+
+
+@overload
+def session_key_from_body(body: Any, depth: Literal[0] = 0) -> str: ...
+
+
+@overload
+def session_key_from_body(body: Any, depth: int) -> str | None: ...
+
+
+def session_key_from_body(body: Any, depth: int = 0) -> str | None:
+ """Derive the per-conversation key from a request body.
+
+ ``depth == 0`` (default) hashes the stable anchors only and always returns
+ a key. ``depth > 0`` extends the hashed prefix with the first ``depth``
+ post-first-user messages — so repeated trials of an identical task diverge
+ via early model responses — and returns ``None`` until the conversation is
+ long enough for that prefix to exist.
+ """
+ return _native_session_key_from_body(body, depth)
+
__all__ = ["session_key_from_body"]
diff --git a/switchyard_rust/core.py b/switchyard_rust/core.py
index c1497b832..678ba8fd5 100644
--- a/switchyard_rust/core.py
+++ b/switchyard_rust/core.py
@@ -616,7 +616,9 @@ if TYPE_CHECKING:
def max_sessions(self) -> int: ...
def __len__(self) -> int: ...
- def session_key_from_body(body: Mapping[str, Any] | JsonValue) -> str: ...
+ def session_key_from_body(
+ body: Mapping[str, Any] | JsonValue, depth: int = 0
+ ) -> str | None: ...
def __getattr__(name: str) -> object:
diff --git a/tests/test_deterministic_routing_profile.py b/tests/test_deterministic_routing_profile.py
index 0d0313998..331b3bb21 100644
--- a/tests/test_deterministic_routing_profile.py
+++ b/tests/test_deterministic_routing_profile.py
@@ -11,6 +11,7 @@ import pytest
from pydantic import ValidationError
from switchyard.lib.backends.deterministic_routing_llm_backend import (
+ CTX_DETERMINISTIC_ROUTING_TIER,
DeterministicRoutingLLMBackend,
)
from switchyard.lib.backends.llm_target import BackendFormat, LlmTarget
@@ -28,14 +29,21 @@ from switchyard.lib.profiles import (
DeterministicRoutingProfileConfig,
ProfileSwitchyard,
)
-from switchyard.lib.profiles.deterministic_routing_profile_config import (
- _apply_deepseek_overrides,
- _apply_default_tier_timeout,
+from switchyard.lib.profiles.tier_target_builders import (
+ apply_deepseek_overrides,
+ apply_default_tier_timeout,
)
from switchyard.lib.proxy_context import ProxyContext
+from switchyard.lib.roles import LLMBackend
from switchyard.lib.route_table_builders import deterministic_routing_virtual_model_id
from switchyard.lib.stats_accumulator import StatsAccumulator
-from switchyard_rust.core import ChatRequest, ChatResponse
+from switchyard_rust.core import (
+ ChatRequest,
+ ChatRequestType,
+ ChatResponse,
+ SwitchyardContextWindowExceededError,
+)
+from switchyard_rust.profiles import ProfileInput
from switchyard_rust.translation import TranslationEngine
@@ -332,7 +340,7 @@ class TestDeepSeekOverrides:
api_key="k",
base_url="https://e/v1",
)
- out = _apply_deepseek_overrides(target)
+ out = apply_deepseek_overrides(target)
assert out.extra_body == {"chat_template_kwargs": {"enable_thinking": False}}
def test_deepseek_gets_batch_priority_header(self) -> None:
@@ -343,7 +351,7 @@ class TestDeepSeekOverrides:
api_key="k",
base_url="https://e/v1",
)
- out = _apply_deepseek_overrides(target)
+ out = apply_deepseek_overrides(target)
assert out.extra_headers == {"X-Inference-Priority": "batch"}
def test_non_deepseek_passes_through_unchanged(self) -> None:
@@ -354,7 +362,7 @@ class TestDeepSeekOverrides:
api_key="k",
base_url="https://e/v1",
)
- out = _apply_deepseek_overrides(target)
+ out = apply_deepseek_overrides(target)
assert out is target # no rebuild needed
def test_caller_supplied_extras_win(self) -> None:
@@ -367,7 +375,7 @@ class TestDeepSeekOverrides:
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
extra_headers={"X-Inference-Priority": "interactive"},
)
- out = _apply_deepseek_overrides(target)
+ out = apply_deepseek_overrides(target)
assert out.extra_body == {"chat_template_kwargs": {"enable_thinking": True}}
assert out.extra_headers == {"X-Inference-Priority": "interactive"}
@@ -380,7 +388,7 @@ class TestDeepSeekOverrides:
base_url="https://e/v1",
extra_body={},
)
- out = _apply_deepseek_overrides(target)
+ out = apply_deepseek_overrides(target)
assert out.extra_body == {}
@@ -396,7 +404,7 @@ class TestTierTimeoutDefaults:
base_url="https://e/v1",
)
- out = _apply_default_tier_timeout(target, 123.0)
+ out = apply_default_tier_timeout(target, 123.0)
assert out.endpoint.timeout_secs == 123.0
@@ -410,7 +418,7 @@ class TestTierTimeoutDefaults:
timeout_secs=45.0,
)
- out = _apply_default_tier_timeout(target, 123.0)
+ out = apply_default_tier_timeout(target, 123.0)
assert out is target
assert out.endpoint.timeout_secs == 45.0
@@ -424,7 +432,7 @@ class TestTierTimeoutDefaults:
base_url="https://e/v1",
)
- out = _apply_default_tier_timeout(target, None)
+ out = apply_default_tier_timeout(target, None)
assert out is target
assert out.endpoint.timeout_secs is None
@@ -489,3 +497,123 @@ def test_virtual_model_id_changes_with_prompt_and_context() -> None:
assert prompt_id != base_id
assert max_chars_id != base_id
+
+
+def _custom_id_config() -> DeterministicRoutingConfig:
+ return DeterministicRoutingConfig(
+ strong=LlmTarget(
+ id="frontier",
+ model="strong-model",
+ format=BackendFormat.OPENAI,
+ api_key="sk-test",
+ base_url="https://strong.invalid/v1",
+ ),
+ weak=LlmTarget(
+ id="cheap",
+ model="weak-model",
+ format=BackendFormat.OPENAI,
+ api_key="sk-test",
+ base_url="https://weak.invalid/v1",
+ ),
+ classifier=LlmTarget(
+ id="classifier",
+ model="classifier-model",
+ format=BackendFormat.OPENAI,
+ api_key="sk-test",
+ base_url="https://classifier.invalid/v1",
+ ),
+ profile_name="coding_agent",
+ fallback_target_on_evict="frontier",
+ )
+
+
+def test_duplicate_tier_ids_rejected() -> None:
+ config = _custom_id_config()
+ with pytest.raises(ValidationError) as exc:
+ DeterministicRoutingConfig.model_validate({
+ **config.model_dump(),
+ "strong": LlmTarget(id="tier", model="strong-model"),
+ "weak": LlmTarget(id="tier", model="weak-model"),
+ "fallback_target_on_evict": "tier",
+ })
+ assert "must differ" in str(exc.value)
+
+
+def test_custom_tier_ids_key_backend_and_selector() -> None:
+ """Backend tiers and the selector's labels follow the configured ids."""
+ profile = DeterministicRoutingProfileConfig.from_config(_custom_id_config()).build()
+
+ backend = profile._backend
+ assert isinstance(backend, DeterministicRoutingLLMBackend)
+ assert set(backend._backends) == {"frontier", "cheap"}
+ assert backend._default_tier == "frontier"
+
+
+class _OverflowableTier(LLMBackend):
+ """Stub tier backend: overflows when told to, else returns a completion."""
+
+ def __init__(self, *, overflow: bool, target_id: str) -> None:
+ self.calls = 0
+ self._overflow = overflow
+ self._target_id = target_id
+
+ @property
+ def supported_request_types(self) -> list[ChatRequestType]:
+ return [ChatRequestType.OPENAI_CHAT]
+
+ async def call(self, ctx: ProxyContext, request: ChatRequest) -> ChatResponse:
+ self.calls += 1
+ if self._overflow:
+ error = SwitchyardContextWindowExceededError(f"{self._target_id} overflowed")
+ error.target_id = self._target_id # type: ignore[attr-defined]
+ raise error
+ return ChatResponse.openai_completion({
+ "id": "deterministic-overflow-test",
+ "object": "chat.completion",
+ "model": request.model,
+ "choices": [
+ {
+ "index": 0,
+ "message": {"role": "assistant", "content": "ok"},
+ "finish_reason": "stop",
+ }
+ ],
+ })
+
+
+class _StampTier:
+ """Stand-in for the classifier+selector pair: stamp a fixed tier label."""
+
+ def __init__(self, tier: str) -> None:
+ self._tier = tier
+
+ async def process(self, ctx: ProxyContext, request: ChatRequest) -> ChatRequest:
+ ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] = self._tier
+ return request
+
+
+async def test_overflow_reroutes_to_custom_strong_id_through_full_profile() -> None:
+ """Weak overflow retries onto the configured strong id, not an unknown label.
+
+ Regression test for tiers keyed as literal strong/weak while the chain's
+ evict-and-retry rewrote ``selected_target`` to the *configured* fallback
+ id: with ids ``cheap``/``frontier``, the rewritten pick was unrecognised,
+ the retry hit weak again, and the pool exhausted.
+ """
+ profile = DeterministicRoutingProfileConfig.from_config(_custom_id_config()).build()
+ backend = profile._backend
+ assert isinstance(backend, DeterministicRoutingLLMBackend)
+ cheap = _OverflowableTier(overflow=True, target_id="cheap")
+ frontier = _OverflowableTier(overflow=False, target_id="frontier")
+ backend._backends = {"cheap": cheap, "frontier": frontier}
+ profile._request_processors = (_StampTier("cheap"),)
+
+ request = ChatRequest.openai_chat({
+ "model": "client/model",
+ "messages": [{"role": "user", "content": "hi"}],
+ })
+ response = await profile.run(ProfileInput(request))
+
+ assert cheap.calls == 1
+ assert frontier.calls == 1
+ assert response.body["choices"][0]["message"]["content"] == "ok"
diff --git a/tests/test_escalation_judge_request_processor.py b/tests/test_escalation_judge_request_processor.py
new file mode 100644
index 000000000..d671bd93b
--- /dev/null
+++ b/tests/test_escalation_judge_request_processor.py
@@ -0,0 +1,468 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Tests for the escalation-router judge processor and its latch policy."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+from types import SimpleNamespace
+from typing import Any, cast
+
+from switchyard.lib.backends.deterministic_routing_llm_backend import (
+ CTX_DETERMINISTIC_ROUTING_TIER,
+)
+from switchyard.lib.processors.escalation_judge_request_processor import (
+ CTX_ESCALATION_VERDICT,
+ ESCALATION_JUDGE_SYSTEM_PROMPT,
+ EscalationJudgeConfig,
+ EscalationJudgeRequestProcessor,
+)
+from switchyard.lib.processors.llm_classifier import ClassifierCompletion
+from switchyard.lib.proxy_context import ProxyContext
+from switchyard.lib.session_affinity import CTX_SESSION_KEY, SessionAffinity
+from switchyard.lib.stats_accumulator import StatsAccumulator
+from switchyard_rust.core import ChatRequest
+
+
+class _FakeJudgeClient:
+ def __init__(self, response: str | Exception, usage: Any | None = None) -> None:
+ self.response = response
+ self.usage = usage
+ self.calls: list[dict[str, str]] = []
+
+ async def classify(
+ self,
+ *,
+ model: str,
+ system_prompt: str,
+ request_summary: str,
+ ) -> ClassifierCompletion:
+ self.calls.append({
+ "model": model,
+ "system_prompt": system_prompt,
+ "request_summary": request_summary,
+ })
+ if isinstance(self.response, Exception):
+ raise self.response
+ return ClassifierCompletion(content=self.response, usage=self.usage)
+
+
+def _verdict_json(escalate: bool, reason: str = "same error repeating") -> str:
+ return json.dumps({"escalate": escalate, "reason": reason})
+
+
+def _request(n_assistant_turns: int = 3, **body_overrides: Any) -> ChatRequest:
+ """Chat request with enough prior assistant turns to pass the judge gate."""
+ messages: list[dict[str, str]] = [
+ {"role": "system", "content": "You are a coding agent."},
+ {"role": "user", "content": "fix the failing tests"},
+ ]
+ for i in range(n_assistant_turns):
+ messages.append({"role": "assistant", "content": f"attempt {i + 1}"})
+ messages.append({"role": "user", "content": f"tool result {i + 1}: error"})
+ body: dict[str, Any] = {"model": "client-model", "messages": messages}
+ body.update(body_overrides)
+ return ChatRequest.openai_chat(cast(Any, body))
+
+
+def _processor(
+ response: str | Exception,
+ *,
+ affinity: SessionAffinity | None = None,
+ session_key_depth: int = 0,
+ **config_overrides: Any,
+) -> tuple[EscalationJudgeRequestProcessor, _FakeJudgeClient, SessionAffinity]:
+ affinity = affinity or SessionAffinity(enabled=True)
+ fake = _FakeJudgeClient(response)
+ processor = EscalationJudgeRequestProcessor(
+ EscalationJudgeConfig(model="judge-model", **config_overrides),
+ affinity=affinity,
+ client=fake,
+ session_key_depth=session_key_depth,
+ )
+ return processor, fake, affinity
+
+
+async def test_no_escalate_routes_weak_without_pin() -> None:
+ processor, fake, affinity = _processor(_verdict_json(False, ""))
+ ctx = ProxyContext()
+ req = _request()
+
+ await processor.process(ctx, req)
+
+ assert ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "weak"
+ assert len(fake.calls) == 1
+ assert ctx.metadata[CTX_ESCALATION_VERDICT]["escalate"] is False
+ assert await affinity.pinned(ProxyContext(), _request()) is None
+
+
+async def test_escalate_routes_strong_and_latches() -> None:
+ processor, fake, _ = _processor(_verdict_json(True))
+ ctx = ProxyContext()
+
+ await processor.process(ctx, _request())
+ assert ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "strong"
+ assert ctx.metadata[CTX_ESCALATION_VERDICT]["reason"] == "same error repeating"
+
+ # Later turn of the same conversation: pinned, judge not called again.
+ later_ctx = ProxyContext()
+ await processor.process(later_ctx, _request(n_assistant_turns=5))
+ assert later_ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "strong"
+ assert later_ctx.metadata[CTX_ESCALATION_VERDICT]["source"] == "pinned"
+ assert len(fake.calls) == 1
+
+
+async def test_judge_skipped_before_min_turn() -> None:
+ processor, fake, _ = _processor(_verdict_json(True), min_judge_turn=3)
+ ctx = ProxyContext()
+
+ await processor.process(ctx, _request(n_assistant_turns=1))
+
+ assert ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "weak"
+ assert fake.calls == []
+
+
+async def test_judge_failure_fails_open_to_weak_without_pin() -> None:
+ processor, fake, affinity = _processor(RuntimeError("judge down"))
+ ctx = ProxyContext()
+
+ await processor.process(ctx, _request())
+
+ assert ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "weak"
+ assert ctx.metadata[CTX_ESCALATION_VERDICT]["source"] == "fail_open"
+ assert await affinity.pinned(ProxyContext(), _request()) is None
+ # A garbage verdict is the same failure surface.
+ processor, fake, affinity = _processor("not json at all")
+ ctx = ProxyContext()
+ await processor.process(ctx, _request())
+ assert ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "weak"
+ assert await affinity.pinned(ProxyContext(), _request()) is None
+
+
+async def test_confirmations_require_consecutive_escalates() -> None:
+ """With confirmations=2, one escalate verdict stays weak; the second latches."""
+ processor, fake, affinity = _processor(_verdict_json(True), escalate_confirmations=2)
+
+ ctx = ProxyContext()
+ await processor.process(ctx, _request(n_assistant_turns=3))
+ assert ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "weak"
+ assert ctx.metadata[CTX_ESCALATION_VERDICT]["escalate"] is True
+ assert ctx.metadata[CTX_ESCALATION_VERDICT]["confirmed"] is False
+ assert ctx.metadata[CTX_ESCALATION_VERDICT]["streak"] == 1
+ assert await affinity.pinned(ProxyContext(), _request()) is None
+
+ ctx = ProxyContext()
+ await processor.process(ctx, _request(n_assistant_turns=4))
+ assert ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "strong"
+ assert ctx.metadata[CTX_ESCALATION_VERDICT]["confirmed"] is True
+ assert ctx.metadata[CTX_ESCALATION_VERDICT]["streak"] == 2
+
+
+async def test_confirmation_window_tolerates_intervening_decline() -> None:
+ """window=3: escalate -> decline -> escalate confirms (recurring trouble)."""
+ affinity = SessionAffinity(enabled=True)
+ fake = _FakeJudgeClient(_verdict_json(True))
+ processor = EscalationJudgeRequestProcessor(
+ EscalationJudgeConfig(
+ model="judge-model", escalate_confirmations=2, confirmation_window=3,
+ ),
+ affinity=affinity,
+ client=fake,
+ )
+
+ await processor.process(ProxyContext(), _request(n_assistant_turns=3))
+ fake.response = _verdict_json(False, "")
+ await processor.process(ProxyContext(), _request(n_assistant_turns=4))
+ fake.response = _verdict_json(True)
+ ctx = ProxyContext()
+ await processor.process(ctx, _request(n_assistant_turns=5))
+
+ assert ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "strong"
+ assert ctx.metadata[CTX_ESCALATION_VERDICT]["confirmed"] is True
+ assert ctx.metadata[CTX_ESCALATION_VERDICT]["streak"] == 2
+
+
+async def test_confirmation_window_expires_after_enough_declines() -> None:
+ """window=2: two declines between fires expire the streak."""
+ affinity = SessionAffinity(enabled=True)
+ fake = _FakeJudgeClient(_verdict_json(True))
+ processor = EscalationJudgeRequestProcessor(
+ EscalationJudgeConfig(
+ model="judge-model", escalate_confirmations=2, confirmation_window=2,
+ ),
+ affinity=affinity,
+ client=fake,
+ )
+
+ await processor.process(ProxyContext(), _request(n_assistant_turns=3))
+ fake.response = _verdict_json(False, "")
+ await processor.process(ProxyContext(), _request(n_assistant_turns=4))
+ await processor.process(ProxyContext(), _request(n_assistant_turns=5))
+ fake.response = _verdict_json(True)
+ ctx = ProxyContext()
+ await processor.process(ctx, _request(n_assistant_turns=6))
+
+ assert ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "weak"
+ assert ctx.metadata[CTX_ESCALATION_VERDICT]["streak"] == 1
+ assert (
+ ctx.metadata[CTX_ESCALATION_VERDICT]["confirmed"] is False
+ )
+
+
+async def test_confirmations_streak_resets_on_decline() -> None:
+ """escalate -> decline -> escalate never latches with confirmations=2."""
+ affinity = SessionAffinity(enabled=True)
+ fake = _FakeJudgeClient(_verdict_json(True))
+ processor = EscalationJudgeRequestProcessor(
+ EscalationJudgeConfig(model="judge-model", escalate_confirmations=2),
+ affinity=affinity,
+ client=fake,
+ )
+
+ await processor.process(ProxyContext(), _request(n_assistant_turns=3))
+ fake.response = _verdict_json(False, "")
+ await processor.process(ProxyContext(), _request(n_assistant_turns=4))
+ fake.response = _verdict_json(True)
+ ctx = ProxyContext()
+ await processor.process(ctx, _request(n_assistant_turns=5))
+
+ assert ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "weak"
+ assert ctx.metadata[CTX_ESCALATION_VERDICT]["streak"] == 1
+ assert await affinity.pinned(ProxyContext(), _request()) is None
+
+
+async def test_judge_usage_recorded_into_classifier_stats() -> None:
+ """Judge token spend lands in the classifier bucket of the stats snapshot."""
+ stats = StatsAccumulator()
+ affinity = SessionAffinity(enabled=True)
+ fake = _FakeJudgeClient(
+ _verdict_json(False, ""),
+ usage=SimpleNamespace(
+ prompt_tokens=321,
+ completion_tokens=17,
+ prompt_tokens_details=SimpleNamespace(cached_tokens=100),
+ ),
+ )
+ processor = EscalationJudgeRequestProcessor(
+ EscalationJudgeConfig(model="judge-model"),
+ affinity=affinity,
+ client=fake,
+ stats_accumulator=stats,
+ )
+
+ await processor.process(ProxyContext(), _request())
+
+ snapshot = await stats.snapshot()
+ judge = snapshot["classifier"]["models"]["judge-model"]
+ assert judge["calls"] == 1
+ assert judge["prompt_tokens"] == 321
+ assert judge["completion_tokens"] == 17
+ assert judge["model_call_latency"]["count"] == 1
+
+
+async def test_judge_failure_recorded_as_classifier_error() -> None:
+ stats = StatsAccumulator()
+ fake = _FakeJudgeClient(RuntimeError("judge down"))
+ processor = EscalationJudgeRequestProcessor(
+ EscalationJudgeConfig(model="judge-model"),
+ affinity=SessionAffinity(enabled=True),
+ client=fake,
+ stats_accumulator=stats,
+ )
+
+ await processor.process(ProxyContext(), _request())
+
+ snapshot = await stats.snapshot()
+ assert snapshot["classifier"]["total_errors"] == 1
+
+
+async def test_markdown_fenced_verdict_is_parsed() -> None:
+ processor, _, _ = _processor("```json\n" + _verdict_json(True) + "\n```")
+ ctx = ProxyContext()
+
+ await processor.process(ctx, _request())
+
+ assert ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "strong"
+
+
+async def test_summary_has_header_anchors_and_truncated_window() -> None:
+ processor, fake, _ = _processor(
+ _verdict_json(False, ""),
+ window_message_chars=80,
+ first_user_chars=200,
+ )
+ req = _request(n_assistant_turns=3, messages=[
+ {"role": "system", "content": "harness boilerplate " * 200},
+ {"role": "user", "content": "fix the failing tests in tests/test_api.py"},
+ {"role": "assistant", "content": "running pytest " + "x" * 500},
+ {"role": "user", "content": "tool result: ImportError " + "y" * 500},
+ {"role": "assistant", "content": "attempt 2"},
+ {"role": "assistant", "content": "attempt 3"},
+ ])
+
+ await processor.process(ProxyContext(), req)
+
+ summary = fake.calls[0]["request_summary"]
+ assert summary.startswith("Conversation turn ")
+ # Anchors survive: capped system + full task statement.
+ assert "harness boilerplate" in summary
+ assert "fix the failing tests in tests/test_api.py" in summary
+ # Oversized system anchor and window messages are head/tail truncated.
+ assert "...[trimmed]" in summary
+ # Window content is present with role labels.
+ assert "[assistant] attempt 3" in summary
+ for line in summary.splitlines()[1:]:
+ assert len(line) <= 2_000
+
+
+async def test_global_cap_drops_oldest_window_messages_first() -> None:
+ processor, fake, _ = _processor(
+ _verdict_json(False, ""),
+ max_request_chars=1_000,
+ window_message_chars=200,
+ )
+ messages: list[dict[str, str]] = [
+ {"role": "user", "content": "the task"},
+ ]
+ for i in range(20):
+ messages.append({"role": "assistant", "content": f"attempt {i} " + "z" * 150})
+
+ await processor.process(ProxyContext(), _request(messages=messages))
+
+ summary = fake.calls[0]["request_summary"]
+ assert len(summary) <= 1_000
+ # The newest evidence survives; the oldest window lines are dropped.
+ assert "attempt 19" in summary
+ assert "attempt 5" not in summary
+
+
+async def test_judge_transcript_renders_anthropic_tool_use() -> None:
+ """Anthropic tool_use turns must not flatten to empty transcript lines."""
+ processor, fake, _ = _processor(_verdict_json(False, ""))
+ messages: list[dict[str, Any]] = [
+ {"role": "user", "content": [{"type": "text", "text": "fix the failing tests"}]},
+ ]
+ for i in range(3):
+ messages.append({
+ "role": "assistant",
+ "content": [
+ {"type": "tool_use", "id": f"tu_{i}", "name": "bash",
+ "input": {"command": "pytest -x"}},
+ ],
+ })
+ messages.append({
+ "role": "user",
+ "content": [
+ {"type": "tool_result", "tool_use_id": f"tu_{i}", "content": "1 failed"},
+ ],
+ })
+ request = ChatRequest.anthropic(cast(Any, {
+ "model": "claude-test",
+ "max_tokens": 128,
+ "messages": messages,
+ }))
+
+ await processor.process(ProxyContext(), request)
+
+ summary = fake.calls[0]["request_summary"]
+ assert 'tool_call bash({"command": "pytest -x"})' in summary
+ assert "1 failed" in summary
+
+
+async def test_deep_session_key_diverges_on_early_responses() -> None:
+ processor, _, _ = _processor(_verdict_json(False, ""), session_key_depth=2)
+
+ def _trial(first_response: str) -> ChatRequest:
+ return _request(messages=[
+ {"role": "system", "content": "You are a coding agent."},
+ {"role": "user", "content": "identical task text"},
+ {"role": "assistant", "content": first_response},
+ {"role": "user", "content": "tool result"},
+ {"role": "assistant", "content": "next"},
+ ])
+
+ ctx_a, ctx_b = ProxyContext(), ProxyContext()
+ await processor.process(ctx_a, _trial("I'll start by reading the tests."))
+ await processor.process(ctx_b, _trial("Let me look at the repo layout."))
+
+ assert ctx_a.metadata[CTX_SESSION_KEY] != ctx_b.metadata[CTX_SESSION_KEY]
+
+ # The key is a prefix hash: the same conversation grown later keeps it.
+ grown = _trial("I'll start by reading the tests.")
+ grown.body["messages"].extend([
+ {"role": "assistant", "content": "much later turn"},
+ {"role": "user", "content": "more results"},
+ ])
+ ctx_grown = ProxyContext()
+ await processor.process(ctx_grown, grown)
+ assert ctx_grown.metadata[CTX_SESSION_KEY] == ctx_a.metadata[CTX_SESSION_KEY]
+
+
+async def test_deep_key_incomplete_prefix_skips_affinity() -> None:
+ processor, fake, affinity = _processor(
+ _verdict_json(True),
+ session_key_depth=4,
+ min_judge_turn=1,
+ )
+ ctx = ProxyContext()
+ # Only one post-first-user message: key prefix incomplete.
+ req = _request(messages=[
+ {"role": "user", "content": "the task"},
+ {"role": "assistant", "content": "first response"},
+ ])
+
+ await processor.process(ctx, req)
+
+ assert ctx.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "weak"
+ assert CTX_SESSION_KEY not in ctx.metadata
+ assert fake.calls == []
+
+
+async def test_escalation_latch_isolated_by_deep_key() -> None:
+ """Trial 2 of an identical task starts weak when trajectories diverge."""
+ affinity = SessionAffinity(enabled=True)
+ processor, fake, _ = _processor(
+ _verdict_json(True), affinity=affinity, session_key_depth=1,
+ )
+
+ def _trial(first_response: str, extra_turns: int = 2) -> ChatRequest:
+ messages = [
+ {"role": "user", "content": "identical task text"},
+ {"role": "assistant", "content": first_response},
+ ]
+ for i in range(extra_turns):
+ messages.append({"role": "user", "content": f"tool result {i}"})
+ messages.append({"role": "assistant", "content": f"attempt {i}"})
+ return _request(messages=messages)
+
+ ctx1 = ProxyContext()
+ await processor.process(ctx1, _trial("trial one path"))
+ assert ctx1.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "strong"
+
+ # Same task text, different first response: fresh latch, judge runs.
+ fake.response = _verdict_json(False, "")
+ ctx2 = ProxyContext()
+ await processor.process(ctx2, _trial("trial two path"))
+ assert ctx2.metadata[CTX_DETERMINISTIC_ROUTING_TIER] == "weak"
+ assert len(fake.calls) == 2
+
+
+def test_default_system_prompt_is_the_line_of_record() -> None:
+ """Pin the packaged default prompt byte-exact.
+
+ The prompt is loaded from ``prompts/escalation_judge.md`` package data
+ and is the benchmark-validated line of record. If this test fails, the
+ file changed: either revert, or re-validate the new prompt on a full
+ benchmark run and update the digest deliberately.
+ """
+ digest = hashlib.sha256(
+ ESCALATION_JUDGE_SYSTEM_PROMPT.encode("utf-8")
+ ).hexdigest()
+ assert digest == (
+ "69610eeecbac59fc20c7933ef57fa21029fa80c374b3a399b1f1a9557a6de234"
+ )
+ # Loader must not normalize: the wire bytes are part of the record.
+ assert ESCALATION_JUDGE_SYSTEM_PROMPT.endswith("\n")
+ assert not ESCALATION_JUDGE_SYSTEM_PROMPT.startswith("\n")
diff --git a/tests/test_escalation_router_profile.py b/tests/test_escalation_router_profile.py
new file mode 100644
index 000000000..aedd44f82
--- /dev/null
+++ b/tests/test_escalation_router_profile.py
@@ -0,0 +1,318 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Tests for the escalation-router config model and profile construction."""
+
+from __future__ import annotations
+
+from typing import Any
+
+import pytest
+from pydantic import ValidationError
+
+from switchyard.lib.backends.deterministic_routing_llm_backend import (
+ DeterministicRoutingLLMBackend,
+)
+from switchyard.lib.backends.llm_target import LlmTarget
+from switchyard.lib.processors.escalation_judge_request_processor import (
+ ESCALATION_JUDGE_SYSTEM_PROMPT,
+ EscalationJudgeRequestProcessor,
+)
+from switchyard.lib.processors.reasoning_effort_normalizer import ReasoningEffortNormalizer
+from switchyard.lib.profiles import EscalationRouterConfig, EscalationRouterProfileConfig
+from switchyard.lib.proxy_context import ProxyContext
+from switchyard.lib.roles import LLMBackend
+from switchyard_rust.core import (
+ ChatRequest,
+ ChatRequestType,
+ ChatResponse,
+ SwitchyardContextWindowExceededError,
+)
+from switchyard_rust.profiles import ProfileInput
+
+
+def _target(tier: str) -> LlmTarget:
+ return LlmTarget(
+ id=tier,
+ model=f"{tier}-model",
+ base_url=f"https://{tier}.invalid/v1",
+ api_key=f"sk-{tier}",
+ )
+
+
+def _config(**overrides: Any) -> EscalationRouterConfig:
+ data: dict[str, Any] = {
+ "strong": _target("strong"),
+ "weak": _target("weak"),
+ "judge": _target("judge"),
+ "fallback_target_on_evict": "strong",
+ }
+ data.update(overrides)
+ return EscalationRouterConfig.model_validate(data)
+
+
+def test_fallback_must_match_a_tier_id() -> None:
+ with pytest.raises(ValidationError) as exc:
+ _config(fallback_target_on_evict="judge")
+ assert "fallback_target_on_evict" in str(exc.value)
+
+
+def test_judge_target_is_required() -> None:
+ with pytest.raises(ValidationError):
+ EscalationRouterConfig.model_validate({
+ "strong": _target("strong"),
+ "weak": _target("weak"),
+ "fallback_target_on_evict": "strong",
+ })
+
+
+def test_blank_judge_prompt_is_unset() -> None:
+ assert _config(judge_system_prompt=" ").judge_system_prompt is None
+
+
+def test_build_composes_judge_chain() -> None:
+ profile = EscalationRouterProfileConfig.from_config(_config()).build()
+
+ processors = profile._request_processors
+ assert [type(p) for p in processors] == [
+ ReasoningEffortNormalizer,
+ EscalationJudgeRequestProcessor,
+ ]
+ backend = profile._backend
+ assert isinstance(backend, DeterministicRoutingLLMBackend)
+ assert set(backend._backends) == {"strong", "weak"}
+ assert backend._default_tier == "weak"
+ assert profile._fallback_target_on_evict == "strong"
+
+
+def test_build_pins_deepseek_judge_to_batch_gateway() -> None:
+ """A DeepSeek judge inherits the benchmark-gateway header like DeepSeek tiers."""
+ profile = EscalationRouterProfileConfig.from_config(
+ _config(
+ judge=LlmTarget(
+ id="judge",
+ model="nvidia/deepseek-ai/deepseek-v4-flash",
+ base_url="https://judge.invalid/v1",
+ api_key="sk-judge",
+ ),
+ ),
+ ).build()
+
+ judge = profile._request_processors[1]
+ assert isinstance(judge, EscalationJudgeRequestProcessor)
+ assert judge._config.extra_headers == {"X-Inference-Priority": "batch"}
+
+
+def test_build_non_deepseek_judge_has_no_default_headers() -> None:
+ profile = EscalationRouterProfileConfig.from_config(_config()).build()
+
+ judge = profile._request_processors[1]
+ assert isinstance(judge, EscalationJudgeRequestProcessor)
+ assert judge._config.extra_headers is None
+
+
+def test_build_threads_judge_settings() -> None:
+ profile = EscalationRouterProfileConfig.from_config(
+ _config(
+ judge_min_turn=5,
+ judge_recent_turn_window=20,
+ judge_window_message_chars=500,
+ judge_disable_reasoning=False,
+ judge_timeout_s=12.0,
+ session_key_depth=2,
+ ),
+ ).build()
+
+ judge = profile._request_processors[1]
+ assert isinstance(judge, EscalationJudgeRequestProcessor)
+ assert judge._config.min_judge_turn == 5
+ assert judge._config.recent_turn_window == 20
+ assert judge._config.window_message_chars == 500
+ assert judge._config.disable_reasoning is False
+ assert judge._config.timeout_s == 12.0
+ assert judge._config.system_prompt == ESCALATION_JUDGE_SYSTEM_PROMPT
+ assert judge._session_key_depth == 2
+ assert judge._affinity.enabled
+
+
+def _custom_id_config() -> EscalationRouterConfig:
+ return _config(
+ strong=LlmTarget(
+ id="frontier",
+ model="strong-model",
+ base_url="https://strong.invalid/v1",
+ api_key="sk-strong",
+ ),
+ weak=LlmTarget(
+ id="cheap",
+ model="weak-model",
+ base_url="https://weak.invalid/v1",
+ api_key="sk-weak",
+ ),
+ fallback_target_on_evict="frontier",
+ )
+
+
+def test_duplicate_tier_ids_rejected() -> None:
+ with pytest.raises(ValidationError) as exc:
+ _config(
+ strong=LlmTarget(id="tier", model="strong-model"),
+ weak=LlmTarget(id="tier", model="weak-model"),
+ fallback_target_on_evict="tier",
+ )
+ assert "must differ" in str(exc.value)
+
+
+def test_custom_tier_ids_key_backend_and_judge() -> None:
+ """Backend tiers and judge labels follow the configured target ids."""
+ profile = EscalationRouterProfileConfig.from_config(_custom_id_config()).build()
+
+ backend = profile._backend
+ assert isinstance(backend, DeterministicRoutingLLMBackend)
+ assert set(backend._backends) == {"frontier", "cheap"}
+ assert backend._default_tier == "cheap"
+ judge = profile._request_processors[1]
+ assert isinstance(judge, EscalationJudgeRequestProcessor)
+ assert judge._strong_tier == "frontier"
+ assert judge._weak_tier == "cheap"
+
+
+class _TierFake(LLMBackend):
+ """Stub tier backend: overflows when told to, else returns a completion."""
+
+ def __init__(self, *, overflow: bool, target_id: str) -> None:
+ self.calls = 0
+ self._overflow = overflow
+ self._target_id = target_id
+
+ @property
+ def supported_request_types(self) -> list[ChatRequestType]:
+ return [ChatRequestType.OPENAI_CHAT]
+
+ async def call(self, ctx: ProxyContext, request: ChatRequest) -> ChatResponse:
+ self.calls += 1
+ if self._overflow:
+ error = SwitchyardContextWindowExceededError(f"{self._target_id} overflowed")
+ error.target_id = self._target_id # type: ignore[attr-defined]
+ raise error
+ return ChatResponse.openai_completion({
+ "id": "escalation-overflow-test",
+ "object": "chat.completion",
+ "model": request.model,
+ "choices": [
+ {
+ "index": 0,
+ "message": {"role": "assistant", "content": "ok"},
+ "finish_reason": "stop",
+ }
+ ],
+ })
+
+
+async def test_overflow_reroutes_to_custom_strong_id_through_full_profile() -> None:
+ """Weak overflow retries onto the configured strong id, not an unknown label.
+
+ Regression test for tiers keyed as literal strong/weak while the chain's
+ evict-and-retry rewrote ``selected_target`` to the *configured* fallback
+ id: with ids ``cheap``/``frontier``, the rewritten pick was unrecognised,
+ the retry hit weak again, and the pool exhausted.
+ """
+ profile = EscalationRouterProfileConfig.from_config(_custom_id_config()).build()
+ backend = profile._backend
+ assert isinstance(backend, DeterministicRoutingLLMBackend)
+ cheap = _TierFake(overflow=True, target_id="cheap")
+ frontier = _TierFake(overflow=False, target_id="frontier")
+ backend._backends = {"cheap": cheap, "frontier": frontier}
+
+ request = ChatRequest.openai_chat({
+ "model": "client/model",
+ "messages": [{"role": "user", "content": "hi"}],
+ })
+ response = await profile.run(ProfileInput(request))
+
+ assert cheap.calls == 1
+ assert frontier.calls == 1
+ assert response.body["choices"][0]["message"]["content"] == "ok"
+
+
+def test_claude_judge_skips_reasoning_hint() -> None:
+ """Claude/Bedrock judges must not receive the vLLM-only thinking hint."""
+ profile = EscalationRouterProfileConfig.from_config(
+ _config(
+ judge=LlmTarget(
+ id="judge",
+ model="anthropic/claude-sonnet-5",
+ base_url="https://judge.invalid/v1",
+ api_key="sk-judge",
+ ),
+ ),
+ ).build()
+
+ judge = profile._request_processors[1]
+ assert isinstance(judge, EscalationJudgeRequestProcessor)
+ assert judge._config.disable_reasoning is False
+
+
+def test_vllm_judge_keeps_reasoning_hint() -> None:
+ """Non-Anthropic judge models keep the default thinking-off hint."""
+ profile = EscalationRouterProfileConfig.from_config(_config()).build()
+
+ judge = profile._request_processors[1]
+ assert isinstance(judge, EscalationJudgeRequestProcessor)
+ assert judge._config.disable_reasoning is True
+
+
+def test_judge_completion_budget_follows_reasoning_mode() -> None:
+ """A thinking judge gets the classifier-sized budget; explicit value wins."""
+ default = EscalationRouterProfileConfig.from_config(_config()).build()
+ assert default._request_processors[1]._config.max_completion_tokens == 128
+
+ thinking = EscalationRouterProfileConfig.from_config(
+ _config(judge_disable_reasoning=False),
+ ).build()
+ assert thinking._request_processors[1]._config.max_completion_tokens == 4096
+
+ explicit = EscalationRouterProfileConfig.from_config(
+ _config(judge_disable_reasoning=False, judge_max_completion_tokens=512),
+ ).build()
+ assert explicit._request_processors[1]._config.max_completion_tokens == 512
+
+
+def test_verdict_dump_is_opt_in() -> None:
+ default = EscalationRouterProfileConfig.from_config(_config()).build()
+ assert default._request_processors[1]._config.dump_verdicts_to_stderr is False
+
+ opted_in = EscalationRouterProfileConfig.from_config(
+ _config(judge_dump_verdicts=True),
+ ).build()
+ assert opted_in._request_processors[1]._config.dump_verdicts_to_stderr is True
+
+
+def test_redis_affinity_store_requires_url() -> None:
+ with pytest.raises(ValidationError) as exc:
+ _config(affinity_store="redis")
+ assert "affinity_store_url" in str(exc.value)
+
+
+def test_redis_affinity_store_wires_l2_latch() -> None:
+ from switchyard.lib.redis_pin_store import RedisPinStore
+
+ profile = EscalationRouterProfileConfig.from_config(
+ _config(
+ affinity_store="redis",
+ affinity_store_url="redis://cache:6379/0",
+ affinity_store_ttl_seconds=120,
+ ),
+ ).build()
+
+ judge = profile._request_processors[1]
+ assert isinstance(judge, EscalationJudgeRequestProcessor)
+ assert isinstance(judge._affinity._l2, RedisPinStore)
+
+
+def test_default_affinity_store_is_l1_only() -> None:
+ profile = EscalationRouterProfileConfig.from_config(_config()).build()
+
+ judge = profile._request_processors[1]
+ assert isinstance(judge, EscalationJudgeRequestProcessor)
+ assert judge._affinity._l2 is None
diff --git a/tests/test_message_condensing.py b/tests/test_message_condensing.py
new file mode 100644
index 000000000..7ab74f664
--- /dev/null
+++ b/tests/test_message_condensing.py
@@ -0,0 +1,70 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Tests for the shared message-condensing helpers."""
+
+from __future__ import annotations
+
+from switchyard.lib.processors.message_condensing import (
+ content_text,
+ message_text,
+ strip_markdown_fence,
+ truncate_middle,
+)
+
+
+def test_content_text_renders_anthropic_tool_use() -> None:
+ """tool_use blocks have neither text nor nested content; render name+input."""
+ blocks = [
+ {"type": "text", "text": "let me check"},
+ {"type": "tool_use", "id": "tu_1", "name": "bash", "input": {"command": "pytest -x"}},
+ ]
+
+ text = content_text(blocks)
+
+ assert "let me check" in text
+ assert 'tool_call bash({"command": "pytest -x"})' in text
+
+
+def test_content_text_renders_tool_result_content() -> None:
+ blocks = [
+ {
+ "type": "tool_result",
+ "tool_use_id": "tu_1",
+ "content": [{"type": "text", "text": "1 failed"}],
+ },
+ ]
+ assert content_text(blocks) == "1 failed"
+
+
+def test_message_text_covers_all_three_tool_call_shapes() -> None:
+ chat = {
+ "role": "assistant",
+ "content": None,
+ "tool_calls": [{"function": {"name": "read_file", "arguments": '{"path": "a.py"}'}}],
+ }
+ assert message_text(chat) == 'tool_call read_file({"path": "a.py"})'
+
+ responses_item = {"type": "function_call", "name": "bash", "arguments": '{"command": "ls"}'}
+ assert message_text(responses_item) == 'tool_call bash({"command": "ls"})'
+
+ anthropic = {
+ "role": "assistant",
+ "content": [{"type": "tool_use", "name": "bash", "input": {"command": "ls"}}],
+ }
+ assert message_text(anthropic) == 'tool_call bash({"command": "ls"})'
+
+
+def test_truncate_middle_keeps_head_and_tail() -> None:
+ text = "H" * 100 + "T" * 100
+ out = truncate_middle(text, 60)
+ assert len(out) <= 60
+ assert out.startswith("H")
+ assert out.endswith("T")
+ assert "...[trimmed]" in out
+
+
+def test_strip_markdown_fence_unwraps_json() -> None:
+ fenced = '```json\n{"escalate": true}\n```'
+ assert strip_markdown_fence(fenced) == '{"escalate": true}'
+ assert strip_markdown_fence('{"escalate": true}') == '{"escalate": true}'
diff --git a/tests/test_route_bundle.py b/tests/test_route_bundle.py
index 5544e97b5..320fbea2e 100644
--- a/tests/test_route_bundle.py
+++ b/tests/test_route_bundle.py
@@ -768,6 +768,191 @@ def test_serve_subcommand_enables_intake_from_cli_args(
assert any(isinstance(component, IntakeResponseProcessor) for component in components)
+class TestEscalationRouterRouteType:
+ """`type: escalation_router` wires the judge-latched chain via YAML."""
+
+ def _bundle(self) -> dict:
+ return {
+ "routes": {
+ "myrouter/escalation": {
+ "type": "escalation_router",
+ "fallback_target_on_evict": "strong",
+ "judge": {
+ "model": "google/gemini-3.5-flash",
+ "api_key": "sk-judge",
+ "base_url": "https://judge.invalid/v1",
+ "timeout_secs": 5.0,
+ "min_turn": 4,
+ "recent_turn_window": 10,
+ },
+ "strong": {
+ "model": "openai/gpt-5.2",
+ "api_key": "sk-strong",
+ "base_url": "https://strong.invalid/v1",
+ },
+ "weak": {
+ "model": "deepseek/deepseek-v4-pro",
+ "api_key": "sk-weak",
+ "base_url": "https://weak.invalid/v1",
+ },
+ "session_key_depth": 2,
+ },
+ },
+ }
+
+ def test_registers_route_key_and_tier_passthroughs(self):
+ from switchyard.cli.route_bundle import build_route_bundle_table
+ table = build_route_bundle_table(self._bundle())
+ # Route key first, tiers as direct passthroughs; the judge tier is
+ # internal-only and never registered.
+ assert table.registered_models() == [
+ "myrouter/escalation",
+ "openai/gpt-5.2",
+ "deepseek/deepseek-v4-pro",
+ ]
+
+ def test_judge_settings_thread_into_processor(self):
+ from switchyard.cli.route_bundle import build_route_bundle_table
+ from switchyard.lib.processors.escalation_judge_request_processor import (
+ EscalationJudgeRequestProcessor,
+ )
+ table = build_route_bundle_table(self._bundle())
+ switchyard = table.lookup_switchyard("myrouter/escalation")
+ judge = next(
+ c for c in switchyard.iter_components()
+ if isinstance(c, EscalationJudgeRequestProcessor)
+ )
+ assert judge._config.model == "google/gemini-3.5-flash"
+ assert judge._config.min_judge_turn == 4
+ assert judge._config.recent_turn_window == 10
+ assert judge._config.timeout_s == 5.0
+ assert judge._session_key_depth == 2
+
+ def test_defaults_come_from_the_config_model(self):
+ """Omitted keys inherit EscalationRouterConfig defaults, owned once."""
+ from switchyard.cli.route_bundle import build_route_bundle_table
+ from switchyard.lib.processors.escalation_judge_request_processor import (
+ EscalationJudgeRequestProcessor,
+ )
+ bundle = self._bundle()
+ del bundle["routes"]["myrouter/escalation"]["judge"]["timeout_secs"]
+ table = build_route_bundle_table(bundle)
+ switchyard = table.lookup_switchyard("myrouter/escalation")
+ judge = next(
+ c for c in switchyard.iter_components()
+ if isinstance(c, EscalationJudgeRequestProcessor)
+ )
+ assert judge._config.escalate_confirmations == 1
+ assert judge._config.window_message_chars == 300
+ assert judge._config.dump_verdicts_to_stderr is False
+ assert judge._config.max_completion_tokens == 128
+ assert judge._config.timeout_s == 5.0
+ assert judge._affinity._l2 is None
+
+ def test_invalid_value_is_a_one_line_config_error(self):
+ """Pydantic rejections surface as RouteBundleConfigError, not a traceback."""
+ from switchyard.cli.route_bundle import (
+ RouteBundleConfigError,
+ build_route_bundle_table,
+ )
+ bundle = self._bundle()
+ bundle["routes"]["myrouter/escalation"]["session_key_depth"] = "two"
+ with pytest.raises(RouteBundleConfigError) as exc:
+ build_route_bundle_table(bundle)
+ assert "session_key_depth" in str(exc.value)
+
+ def test_benchmark_knobs_and_redis_latch_thread_through(self):
+ from switchyard.cli.route_bundle import build_route_bundle_table
+ from switchyard.lib.processors.escalation_judge_request_processor import (
+ EscalationJudgeRequestProcessor,
+ )
+ from switchyard.lib.redis_pin_store import RedisPinStore
+ bundle = self._bundle()
+ route = bundle["routes"]["myrouter/escalation"]
+ route["judge"]["dump_verdicts"] = True
+ route["judge"]["max_completion_tokens"] = 2048
+ route["affinity_store"] = "redis"
+ route["affinity_store_url"] = "redis://cache:6379/0"
+ table = build_route_bundle_table(bundle)
+ switchyard = table.lookup_switchyard("myrouter/escalation")
+ judge = next(
+ c for c in switchyard.iter_components()
+ if isinstance(c, EscalationJudgeRequestProcessor)
+ )
+ assert judge._config.dump_verdicts_to_stderr is True
+ assert judge._config.max_completion_tokens == 2048
+ assert isinstance(judge._affinity._l2, RedisPinStore)
+
+ def test_judge_prompt_path_reads_file(self, tmp_path):
+ from switchyard.cli.route_bundle import build_route_bundle_table
+ from switchyard.lib.processors.escalation_judge_request_processor import (
+ EscalationJudgeRequestProcessor,
+ )
+ prompt_file = tmp_path / "judge_prompt.md"
+ prompt_file.write_text("file-based judge prompt\n", encoding="utf-8")
+ bundle = self._bundle()
+ bundle["routes"]["myrouter/escalation"]["judge"]["prompt_path"] = (
+ str(prompt_file)
+ )
+ table = build_route_bundle_table(bundle)
+ switchyard = table.lookup_switchyard("myrouter/escalation")
+ judge = next(
+ c for c in switchyard.iter_components()
+ if isinstance(c, EscalationJudgeRequestProcessor)
+ )
+ assert judge._config.system_prompt == "file-based judge prompt\n"
+
+ def test_judge_prompt_and_prompt_path_are_mutually_exclusive(self, tmp_path):
+ from switchyard.cli.route_bundle import (
+ RouteBundleConfigError,
+ build_route_bundle_table,
+ )
+ prompt_file = tmp_path / "judge_prompt.md"
+ prompt_file.write_text("file prompt", encoding="utf-8")
+ bundle = self._bundle()
+ judge = bundle["routes"]["myrouter/escalation"]["judge"]
+ judge["prompt"] = "inline prompt"
+ judge["prompt_path"] = str(prompt_file)
+ with pytest.raises(RouteBundleConfigError) as exc:
+ build_route_bundle_table(bundle)
+ assert "mutually exclusive" in str(exc.value)
+
+ def test_judge_prompt_path_missing_file_is_a_config_error(self, tmp_path):
+ from switchyard.cli.route_bundle import (
+ RouteBundleConfigError,
+ build_route_bundle_table,
+ )
+ bundle = self._bundle()
+ bundle["routes"]["myrouter/escalation"]["judge"]["prompt_path"] = (
+ str(tmp_path / "nope.md")
+ )
+ with pytest.raises(RouteBundleConfigError) as exc:
+ build_route_bundle_table(bundle)
+ assert "prompt_path" in str(exc.value)
+
+ def test_rejects_missing_judge_block(self):
+ from switchyard.cli.route_bundle import (
+ RouteBundleConfigError,
+ build_route_bundle_table,
+ )
+ bundle = self._bundle()
+ del bundle["routes"]["myrouter/escalation"]["judge"]
+ with pytest.raises(RouteBundleConfigError) as exc:
+ build_route_bundle_table(bundle)
+ assert "judge" in str(exc.value)
+
+ def test_rejects_fallback_not_matching_a_tier(self):
+ from switchyard.cli.route_bundle import (
+ RouteBundleConfigError,
+ build_route_bundle_table,
+ )
+ bundle = self._bundle()
+ bundle["routes"]["myrouter/escalation"]["fallback_target_on_evict"] = "judge"
+ with pytest.raises(RouteBundleConfigError) as exc:
+ build_route_bundle_table(bundle)
+ assert "fallback_target_on_evict" in str(exc.value)
+
+
class TestDeterministicRouteType:
"""`type: deterministic` wires the LLM-classifier chain via YAML."""
@@ -931,7 +1116,7 @@ class TestDeterministicRouteType:
return target
monkeypatch.setattr(
- "switchyard.lib.profiles.deterministic_routing_profile_config._apply_default_tier_timeout",
+ "switchyard.lib.profiles.tier_target_builders.apply_default_tier_timeout",
fake_apply_default_tier_timeout,
)
bundle = self._bundle()