Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

feat: task affinity based llm classifier with message hash fallback (#168)

Ayush Agarwalf3e29af0650e
diff --git a/crates/libsy/README.md b/crates/libsy/README.md
index 2d1cc32bf..4f622649f 100644
--- a/crates/libsy/README.md
+++ b/crates/libsy/README.md
@@ -8,11 +8,11 @@ so it drops into a proxy, gateway, or agent runtime.
## Example
-Build a target set, pick an algorithm, run a request:
+Build the targets, pick an algorithm, run a request:
```rust
-use switchyard_libsy::{Algorithm, Context, RoutedLlmClient, LlmTarget, LlmTargetSet, Request, State};
-use switchyard_libsy::algorithms::{FallThrough, LlmTaskClassifier};
+use switchyard_libsy::{Algorithm, Context, RoutedLlmClient, LlmTarget, Request};
+use switchyard_libsy::algorithms::{LlmTaskClassifier, TaskClassifierConfig};
use switchyard_protocol::{completion_text, text_request};
use std::sync::Arc;
@@ -20,15 +20,13 @@ use std::sync::Arc;
let client = Arc::new(MyClient { /* .. */ }) as Arc<dyn RoutedLlmClient>;
let target = |name: &str| LlmTarget { semantic_name: name.into(), llm_client: Some(client.clone()) };
-let classifier = target("classifier");
-let strong = target("strong");
-let weak = target("weak");
-let targets = LlmTargetSet::new(vec![strong.clone(), weak.clone()]);
-let algo: Arc<dyn Algorithm> = Arc::new(
- FallThrough::<State>::new_with_state(targets).with_classifier(Arc::new(LlmTaskClassifier::new(
- classifier, weak, strong, 0.5,
- )?)),
-);
+let algo: Arc<dyn Algorithm> = Arc::new(LlmTaskClassifier::new(
+ target("classifier"), // the judge
+ target("weak"), // efficient tier
+ target("strong"), // capable tier
+ // Every field but `base_threshold` has a default; see "Tuning the classifier" below.
+ TaskClassifierConfig { base_threshold: 0.5, ..Default::default() },
+)?);
let req = Request {
// `text_request` is the single-turn shortcut; build an `LlmRequest` directly for
@@ -222,15 +220,39 @@ pub trait Decision: Send + Sync {
}
```
-Compose a classifier into `FallThrough`; the fall-through algorithm calls the judge,
-applies its policy, and then invokes the selected target:
+`LlmTaskClassifier` owns its fall-through cascade: it calls the judge, applies its policy, and
+then invokes the selected target:
```rust
-let classifier = LlmTaskClassifier::new(judge_target, weak_target, strong_target, 0.5)?;
-let router =
- FallThrough::<State>::new_with_state(targets).with_classifier(Arc::new(classifier));
+let router = LlmTaskClassifier::new(
+ judge_target,
+ weak_target,
+ strong_target,
+ TaskClassifierConfig { base_threshold: 0.5, ..Default::default() },
+)?;
```
+### Tuning the classifier
+
+The judge returns a `p_solve` — its estimate that the efficient tier completes the task
+correctly — plus a confidence and a capability boundary. `TaskClassifierConfig` decides what
+to do with that verdict. Anything invalid, abstained, or below a threshold routes to the
+capable target, so every knob below trades cost against quality in the same direction.
+
+| Field | Default | Meaning |
+|---|---|---|
+| `base_threshold` | *required* | Lowest `p_solve` that routes a supported task to the efficient target. Raise it to send less traffic to the cheap model. |
+| `min_confidence` | `0.0` | Lowest judge confidence that permits efficient routing. `0.0` disables the gate. |
+| `capability_elevated_floor` | `None` | A higher `p_solve` floor applied only when the judge marks the task `uncertain`, `unsupported`, or `unmatched`. `None` reuses `base_threshold`. |
+| `session_affinity` | `false` | Retains the first decision per session and reuses it on later turns, so the judge is called once per session instead of once per turn. |
+| `message_hash_fallback` | `false` | Extends affinity to clients that send no session header, keying on a hash of the first user message. Requires `session_affinity`. |
+
+Two things to understand before enabling affinity. The retained decision is **sticky for the
+process lifetime** — including a fail-closed `capable` verdict produced when the judge was
+unreachable — so a transient judge failure pins that identity until restart. And
+`message_hash_fallback` keys on request *content*, not a correlation id, so unrelated callers
+sending identical text share one assignment.
+
## Errors
All libsy-owned APIs return [`switchyard_libsy::Result<T>`], whose error is
@@ -301,8 +323,8 @@ agents live in [`examples`](examples/) folder.
- [`Random`](src/algorithms/rand.rs) — uniform or weighted random over the set
(one call).
-- [`LlmTaskClassifier`](src/algorithms/llm_class.rs) — capability judge for a
- [`FallThrough`](src/algorithms/fall_through.rs) classifier cascade.
+- [`LlmTaskClassifier`](src/algorithms/llm_class.rs) — task-level capability routing with an
+ internal [`FallThrough`](src/algorithms/fall_through.rs) cascade.
- [`EnsembleOrchAlgo`](examples/ensemble.rs) — stateful: fan out to
candidates, judge the best, commit to the winner after N exploration turns.
diff --git a/crates/libsy/examples/research_agent.rs b/crates/libsy/examples/research_agent.rs
index 0056ac02b..885a3882f 100644
--- a/crates/libsy/examples/research_agent.rs
+++ b/crates/libsy/examples/research_agent.rs
@@ -6,7 +6,7 @@
//! Every target owns an `RoutedLlmClient`, so the agent runs each request to completion with
//! [`Algorithm::run`]: it serves each offloaded call with the routed
//! target's `default_client` and returns the final response — no stream to drive. The
-//! classifier cascade runs inside `FallThrough`; the agent never sees it. To drive the step
+//! classifier cascade runs inside `LlmTaskClassifier`; the agent never sees it. To drive the step
//! stream yourself instead, use
//! `Algorithm::run_stream`. Run with:
//! cargo run -p libsy --example research_agent
@@ -14,10 +14,10 @@
use std::sync::Arc;
use async_trait::async_trait;
-use switchyard_libsy::algorithms::{FallThrough, LlmTaskClassifier};
+use switchyard_libsy::algorithms::{LlmTaskClassifier, TaskClassifierConfig};
use switchyard_libsy::{
Algorithm, Context, Decision, LibsyError, LlmResponse, LlmTarget, LlmTargetSet, Request,
- Response, Result, RoutedLlmClient, State,
+ Response, Result, RoutedLlmClient,
};
use switchyard_protocol::{completion_text, text_request, text_response};
@@ -25,7 +25,7 @@ const CLASSIFIER: &str = "classifier/model";
const STRONG: &str = "strong/model";
const WEAK: &str = "weak/model";
/// Lowest judge-estimated solve probability that still routes to the weak model.
-const THRESHOLD: f64 = 0.5;
+const BASE_THRESHOLD: f64 = 0.5;
/// Stub transport. Real integrators implement `RoutedLlmClient` over their own HTTP.
struct StubClient;
@@ -102,11 +102,18 @@ async fn main() -> Result<()> {
let classifier = target_set.get_target(CLASSIFIER)?;
let weak = target_set.get_target(WEAK)?;
let strong = target_set.get_target(STRONG)?;
- let algo: Arc<dyn Algorithm> = Arc::new(
- FallThrough::<State>::new_with_state(target_set).with_classifier(Arc::new(
- LlmTaskClassifier::new(classifier, weak, strong, THRESHOLD)?,
- )),
- );
+ let algo: Arc<dyn Algorithm> = Arc::new(LlmTaskClassifier::new(
+ classifier,
+ weak,
+ strong,
+ TaskClassifierConfig {
+ base_threshold: BASE_THRESHOLD,
+ min_confidence: 0.0,
+ capability_elevated_floor: None,
+ session_affinity: false,
+ message_hash_fallback: false,
+ },
+ )?);
let agent = ResearchAgent { algo };
println!("{}", agent.run("what is switchyard?").await?);
diff --git a/crates/libsy/examples/research_agent_core.rs b/crates/libsy/examples/research_agent_core.rs
index 475cf468f..3c0571b13 100644
--- a/crates/libsy/examples/research_agent_core.rs
+++ b/crates/libsy/examples/research_agent_core.rs
@@ -11,10 +11,10 @@
use std::sync::Arc;
-use switchyard_libsy::algorithms::{FallThrough, LlmTaskClassifier};
+use switchyard_libsy::algorithms::{LlmTaskClassifier, TaskClassifierConfig};
use switchyard_libsy::{
Algorithm, Context, Decision, LibsyError, LlmResponse, LlmTarget, LlmTargetSet, Request,
- Response, Result, State, Step,
+ Response, Result, Step,
};
use switchyard_protocol::{completion_text, text_request, text_response};
use tokio_stream::StreamExt;
@@ -23,7 +23,7 @@ const CLASSIFIER: &str = "classifier/model";
const STRONG: &str = "strong/model";
const WEAK: &str = "weak/model";
/// Lowest judge-estimated solve probability that still routes to the weak model.
-const THRESHOLD: f64 = 0.5;
+const BASE_THRESHOLD: f64 = 0.5;
/// The "real" model call the agent makes to fulfill a promise. The core never
/// makes the call itself — it hands back a request and waits for the response.
@@ -110,11 +110,18 @@ async fn main() -> Result<()> {
let classifier = target_set.get_target(CLASSIFIER)?;
let weak = target_set.get_target(WEAK)?;
let strong = target_set.get_target(STRONG)?;
- let algo: Arc<dyn Algorithm> = Arc::new(
- FallThrough::<State>::new_with_state(target_set).with_classifier(Arc::new(
- LlmTaskClassifier::new(classifier, weak, strong, THRESHOLD)?,
- )),
- );
+ let algo: Arc<dyn Algorithm> = Arc::new(LlmTaskClassifier::new(
+ classifier,
+ weak,
+ strong,
+ TaskClassifierConfig {
+ base_threshold: BASE_THRESHOLD,
+ min_confidence: 0.0,
+ capability_elevated_floor: None,
+ session_affinity: false,
+ message_hash_fallback: false,
+ },
+ )?);
let mut agent = ResearchAgent { algo };
println!("{}", agent.run("what is switchyard?").await?);
diff --git a/crates/libsy/src/algorithms.rs b/crates/libsy/src/algorithms.rs
index 1d64e936f..b36b542f8 100644
--- a/crates/libsy/src/algorithms.rs
+++ b/crates/libsy/src/algorithms.rs
@@ -13,7 +13,7 @@ pub mod passthrough;
pub mod rand;
pub use fall_through::{FallThrough, FallThroughDecision};
-pub use llm_class::LlmTaskClassifier;
+pub use llm_class::{LlmTaskClassifier, TaskClassifierConfig};
pub use noop::{Noop, NoopDecision};
pub use passthrough::{Passthrough, PassthroughDecision};
pub use rand::{Random, RandomClassifier, RandomDecision};
diff --git a/crates/libsy/src/algorithms/fall_through.rs b/crates/libsy/src/algorithms/fall_through.rs
index 0152af477..8a6e70fdb 100644
--- a/crates/libsy/src/algorithms/fall_through.rs
+++ b/crates/libsy/src/algorithms/fall_through.rs
@@ -263,7 +263,6 @@ where
self.execute(ctx, driver, request).await
}
}
-
#[cfg(test)]
mod tests {
use super::*;
diff --git a/crates/libsy/src/algorithms/llm_class.rs b/crates/libsy/src/algorithms/llm_class.rs
index a3dc63c57..255a0c43b 100644
--- a/crates/libsy/src/algorithms/llm_class.rs
+++ b/crates/libsy/src/algorithms/llm_class.rs
@@ -1,39 +1,44 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
-//! Capability routing with a judge-backed classifier.
+//! Task-level capability routing with a judge-backed classifier.
//!
-//! The classifier judges the full inbound request and emits one decisive target for a
-//! [`FallThrough`](super::FallThrough) cascade. Invalid, abstained, or unavailable judge output
-//! always selects the capable target.
+//! The algorithm owns a [`FallThrough`](super::FallThrough) cascade. Its classifier judges the
+//! full inbound request and selects one decisive target. Invalid, abstained, or unavailable judge
+//! output always selects the capable target.
+
+use std::sync::Arc;
use async_trait::async_trait;
use serde::Deserialize;
use serde_json::Value;
use switchyard_protocol::{LlmRequest, Message, OutputParams, Role};
-use super::util::{Judge, JudgeClassifier, JudgeConfig, JudgePolicy};
+use super::util::{AffinityRouter, Judge, JudgeClassifier, JudgeConfig, JudgePolicy};
+use super::FallThrough;
use crate::{
- Classification, Classifier, Driver, LibsyError, LlmTarget, Request, Result, Score, State,
+ Algorithm, Classification, Classifier, Context, Driver, LibsyError, LlmTarget, LlmTargetSet,
+ Request, Response, Result, RoutedLlmClient, Score, State,
};
// TODO: As a first implementation, keeping the prompt and schema paths hardcoded. Add a way to dynamically load and parse user passed prompt and schema.
const PROMPT_TEMPLATE: &str = include_str!("../prompts/capability-classifier/prompt.md");
const SCHEMA_TEMPLATE: &str = include_str!("../prompts/capability-classifier/schema.json");
-// TODO: There can be more knobs to tune the classifier after its verdict is parsed. Add more later.
-const RECENT_MESSAGE_WINDOW: usize = 5;
+/// Telemetry label for this algorithm's spans, metrics, and logs.
+const ALGORITHM_NAME: &str = "llm_task_classifier";
#[derive(Deserialize)]
#[serde(deny_unknown_fields)]
/// Parsed judge output. Only confidence, abstention, and solve probability affect v0 routing.
+/// These fields are parsed from the judge output and used to route the request.
+/// For supporting a new schema, we need to add a new Verdict struct and parse the new
struct TaskClassifierVerdict {
#[serde(rename = "recommended_route")]
_recommended_route: String,
p_solve: f64,
confidence: f64,
abstain: bool,
- #[serde(rename = "capability_boundary")]
- _capability_boundary: String,
+ capability_boundary: String,
#[serde(rename = "primary_rule")]
_primary_rule: String,
#[serde(rename = "crux")]
@@ -44,52 +49,28 @@ impl TaskClassifierVerdict {
/// Reject out-of-range probabilities before the policy can route efficiently. Range
/// containment also rejects NaN and the infinities, which compare false against both bounds.
fn is_valid(&self) -> bool {
- (0.0..=1.0).contains(&self.p_solve) && (0.0..=1.0).contains(&self.confidence)
+ (0.0..=1.0).contains(&self.p_solve)
+ && (0.0..=1.0).contains(&self.confidence)
+ && matches!(
+ self.capability_boundary.as_str(),
+ "supported" | "uncertain" | "unsupported" | "unmatched"
+ )
}
-}
-/// Keeps client instructions, the initial task, and bounded recent dialogue for the judge.
-fn trim_messages(messages: &[Message], recent_turn_window: usize) -> Vec<Message> {
- let mut system = Vec::new();
- let mut first_user = None;
- let mut first_user_idx = None;
- for (idx, message) in messages.iter().enumerate() {
- match message.role {
- Role::System | Role::Developer => system.push(message.clone()),
- Role::User if first_user.is_none() => {
- first_user = Some(message.clone());
- first_user_idx = Some(idx);
- }
- _ => {}
- }
- }
- let Some(first_user) = first_user else {
- return system;
- };
- let tail = messages
- .iter()
- .enumerate()
- .filter(|(idx, message)| {
- *idx > first_user_idx.unwrap_or(0)
- && !matches!(message.role, Role::System | Role::Developer)
- })
- .map(|(_, message)| message.clone())
- .collect::<Vec<_>>();
- if recent_turn_window == 0 {
- let mut out = system;
- out.push(first_user);
- if let Some(last_user) = tail.iter().rev().find(|message| message.role == Role::User) {
- out.push(last_user.clone());
- }
- return out;
+ /// Whether this verdict needs the elevated capability threshold.
+ /// When capability boundary is "uncertain", "unsupported", or "unmatched", we need to use the elevated capability threshold to route the request to weak model
+ /// This is to ensure that we are not routing too many requests to the weak model.
+ fn is_capability_elevated(&self) -> bool {
+ matches!(
+ self.capability_boundary.as_str(),
+ "uncertain" | "unsupported" | "unmatched"
+ )
}
- let mut out = system;
- out.push(first_user);
- let start = tail.len().saturating_sub(recent_turn_window);
- out.extend_from_slice(&tail[start..]);
- out
}
+/// The judge is responsible for any kind of llm judge based calls
+/// Example: A judge can be a capability classifier, a escalation classifier etc
+/// Builds capability-specific judge requests from shared classifier configuration.
struct CapabilityJudge {
config: JudgeConfig,
}
@@ -97,9 +78,20 @@ struct CapabilityJudge {
impl Judge for CapabilityJudge {
type Verdict = TaskClassifierVerdict;
+ /// For different judges, the request building logic can be different
+ /// To have a single interface for all judges, we may make a common request building logic here.
fn build_request(&self, _state: &State, request: &Request) -> Request {
- // The judge owns the leading system prompt; client instructions and task context follow.
- let mut messages = trim_messages(&request.llm_request.messages, RECENT_MESSAGE_WINDOW);
+ // For task-based routing, classify only the newest user message with the judge prompt.
+ // For any turn window size setting, use TaskClassifierConfig to define the window size.
+ let mut messages = request
+ .llm_request
+ .messages
+ .iter()
+ .rev()
+ .find(|message| message.role == Role::User)
+ .cloned()
+ .into_iter()
+ .collect::<Vec<_>>();
messages.insert(
0,
Message::text(Role::System, self.config.system_prompt.clone()),
@@ -123,20 +115,30 @@ impl Judge for CapabilityJudge {
struct TaskClassifierPolicy {
efficient_target: String,
capable_target: String,
- /// Lowest `p_solve` that still routes to the efficient target.
- threshold: f64,
+ config: TaskClassifierConfig,
}
impl TaskClassifierPolicy {
fn new(
efficient_target: impl Into<String>,
capable_target: impl Into<String>,
- threshold: f64,
+ config: TaskClassifierConfig,
) -> Self {
Self {
efficient_target: efficient_target.into(),
capable_target: capable_target.into(),
- threshold,
+ config,
+ }
+ }
+
+ /// Returns the required solve probability for one validated verdict.
+ fn threshold(&self, verdict: &TaskClassifierVerdict) -> f64 {
+ if verdict.is_capability_elevated() {
+ self.config
+ .capability_elevated_floor
+ .unwrap_or(self.config.base_threshold)
+ } else {
+ self.config.base_threshold
}
}
}
@@ -145,11 +147,14 @@ impl JudgePolicy for TaskClassifierPolicy {
type Verdict = TaskClassifierVerdict;
fn to_classification(&self, verdict: Option<&Self::Verdict>) -> Classification {
- // Judge output is untrusted. Anything incomplete, invalid, abstained, or below the
- // threshold routes to the capable target rather than risking an underpowered route.
+ // Judge output is untrusted. Anything incomplete, invalid, abstained, or below its
+ // configured confidence or capability threshold routes to the capable target.
let target = match verdict {
Some(verdict)
- if verdict.is_valid() && !verdict.abstain && verdict.p_solve >= self.threshold =>
+ if verdict.is_valid()
+ && !verdict.abstain
+ && verdict.confidence >= self.config.min_confidence
+ && verdict.p_solve >= self.threshold(verdict) =>
{
&self.efficient_target
}
@@ -162,44 +167,128 @@ impl JudgePolicy for TaskClassifierPolicy {
}
}
-/// A full-request capability classifier configured with the packaged prompt and schema.
-pub struct LlmTaskClassifier {
+#[derive(Clone, Debug, Default, Deserialize)]
+/// Thresholds that control capability classifier routing.
+pub struct TaskClassifierConfig {
+ /// Lowest solve probability that routes a supported task to the efficient target.
+ pub base_threshold: f64,
+ /// Lowest judge confidence that permits efficient routing.
+ #[serde(default)]
+ pub min_confidence: f64,
+ /// Higher solve-probability floor for uncertain, unmatched, and unsupported tasks.
+ #[serde(default)]
+ pub capability_elevated_floor: Option<f64>,
+ /// Enables session affinity before the judge-backed classifier.
+ #[serde(default)]
+ pub session_affinity: bool,
+ /// Uses the first user message as the SessionKey for sticky routing when session metadata is unavailable.
+ #[serde(default)]
+ pub message_hash_fallback: bool,
+}
+
+impl TaskClassifierConfig {
+ /// Validates routing thresholds before the classifier is constructed.
+ fn validate(&self) -> Result<()> {
+ if !(0.0..=1.0).contains(&self.base_threshold) {
+ return Err(LibsyError::AlgorithmError {
+ message: format!(
+ "base_threshold must be between 0 and 1, got {}",
+ self.base_threshold
+ ),
+ });
+ }
+ if !(0.0..=1.0).contains(&self.min_confidence) {
+ return Err(LibsyError::AlgorithmError {
+ message: format!(
+ "min_confidence must be between 0 and 1, got {}",
+ self.min_confidence
+ ),
+ });
+ }
+ if let Some(floor) = self.capability_elevated_floor {
+ if !(0.0..=1.0).contains(&floor) {
+ return Err(LibsyError::AlgorithmError {
+ message: format!(
+ "capability_elevated_floor must be between 0 and 1, got {floor}"
+ ),
+ });
+ }
+ if floor <= self.base_threshold {
+ return Err(LibsyError::AlgorithmError {
+ message: format!(
+ "capability_elevated_floor must be greater than base_threshold, got {floor}"
+ ),
+ });
+ }
+ }
+ if self.message_hash_fallback && !self.session_affinity {
+ return Err(LibsyError::AlgorithmError {
+ message: "message_hash_fallback requires session_affinity".to_string(),
+ });
+ }
+ Ok(())
+ }
+}
+
+struct TaskClassifier {
classifier: JudgeClassifier<CapabilityJudge, TaskClassifierPolicy>,
efficient_target: String,
capable_target: String,
}
+/// A task-level capability routing algorithm with an internal fall-through cascade.
+pub struct LlmTaskClassifier {
+ route: FallThrough<State>,
+ classifier: Arc<TaskClassifier>,
+}
+
impl LlmTaskClassifier {
- /// Selects `efficient_target` when the judge's `p_solve` reaches `threshold`, and
- /// `capable_target` otherwise. Errors if `threshold` is outside `[0.0, 1.0]`.
+ /// Routes according to `config` and returns errors for invalid thresholds.
pub fn new(
judge_target: LlmTarget,
efficient_target: LlmTarget,
capable_target: LlmTarget,
- threshold: f64,
+ config: TaskClassifierConfig,
) -> Result<Self> {
- if !(0.0..=1.0).contains(&threshold) {
- return Err(LibsyError::AlgorithmError {
- message: format!("threshold must be between 0 and 1, got {threshold}"),
- });
- }
- let judge = CapabilityJudge {
- config: Self::load_judge_config()?,
- };
- let efficient_target = efficient_target.semantic_name;
- let capable_target = capable_target.semantic_name;
- Ok(Self {
+ config.validate()?;
+ let judge_config = Self::load_judge_config()?;
+ let targets = LlmTargetSet::new(vec![efficient_target.clone(), capable_target.clone()]);
+ let session_affinity = config.session_affinity;
+ let message_hash_fallback = config.message_hash_fallback;
+ let classifier = Arc::new(TaskClassifier {
classifier: JudgeClassifier::new(
- judge,
+ CapabilityJudge {
+ config: judge_config,
+ },
judge_target,
TaskClassifierPolicy::new(
- efficient_target.clone(),
- capable_target.clone(),
- threshold,
+ efficient_target.semantic_name.clone(),
+ capable_target.semantic_name.clone(),
+ config,
),
),
- efficient_target,
- capable_target,
+ efficient_target: efficient_target.semantic_name,
+ capable_target: capable_target.semantic_name,
+ });
+
+ // The cascade is an internal detail: callers drive the algorithm, not its parts.
+ // Affinity comes first so a retained assignment short-circuits the judge call.
+ let mut route = FallThrough::<State>::new_with_state(targets).with_name(ALGORITHM_NAME);
+ if session_affinity {
+ let affinity = if message_hash_fallback {
+ AffinityRouter::new().with_message_hash_fallback()
+ } else {
+ AffinityRouter::new()
+ };
+ // Both roles must share one `Arc` so the classifier reads what the processor wrote.
+ let affinity = Arc::new(affinity);
+ route = route
+ .with_processor(affinity.clone())
+ .with_classifier(affinity);
+ }
+ Ok(Self {
+ route: route.with_classifier(classifier.clone()),
+ classifier,
})
}
@@ -230,7 +319,7 @@ impl LlmTaskClassifier {
}
#[async_trait]
-impl Classifier<State> for LlmTaskClassifier {
+impl Classifier<State> for TaskClassifier {
fn routing_tier(&self, selected_model: &str) -> Option<&'static str> {
if self.efficient_target == self.capable_target {
None
@@ -253,6 +342,42 @@ impl Classifier<State> for LlmTaskClassifier {
}
}
+#[async_trait]
+impl Classifier<State> for LlmTaskClassifier {
+ fn routing_tier(&self, selected_model: &str) -> Option<&'static str> {
+ self.classifier.routing_tier(selected_model)
+ }
+
+ async fn score(
+ &self,
+ state: &mut State,
+ request: &mut Request,
+ driver: Option<&Driver>,
+ ) -> Result<Classification> {
+ self.classifier.score(state, request, driver).await
+ }
+}
+
+#[async_trait]
+impl Algorithm for LlmTaskClassifier {
+ fn name(&self) -> &str {
+ "llm_task_classifier"
+ }
+
+ fn count_tokens_client(&self) -> Option<Arc<dyn RoutedLlmClient>> {
+ self.route.count_tokens_client()
+ }
+
+ async fn create_run_task(
+ self: Arc<Self>,
+ ctx: Context,
+ driver: Driver,
+ request: Request,
+ ) -> Result<Response> {
+ self.route.execute(ctx, driver, request).await
+ }
+}
+
#[cfg(test)]
mod tests {
use std::sync::Arc;
@@ -260,14 +385,24 @@ mod tests {
use parking_lot::Mutex;
use super::*;
- use switchyard_protocol::{completion_text, text_response, LlmClientError};
+ use switchyard_protocol::{completion_text, text_response, LlmClientError, Metadata};
- use crate::{Algorithm, Context, LlmResponse, LlmTargetSet, Response, RoutedLlmClient};
+ use crate::{Algorithm, Context, LlmResponse, Response, RoutedLlmClient};
const TEST_THRESHOLD: f64 = 0.5;
+ fn test_config(base_threshold: f64) -> TaskClassifierConfig {
+ TaskClassifierConfig {
+ base_threshold,
+ min_confidence: 0.0,
+ capability_elevated_floor: None,
+ session_affinity: false,
+ message_hash_fallback: false,
+ }
+ }
+
fn policy() -> TaskClassifierPolicy {
- TaskClassifierPolicy::new("efficient", "capable", TEST_THRESHOLD)
+ TaskClassifierPolicy::new("efficient", "capable", test_config(TEST_THRESHOLD))
}
/// A verdict whose non-routing fields are fixed — only the three the policy reads vary.
@@ -277,7 +412,7 @@ mod tests {
p_solve,
confidence,
abstain,
- _capability_boundary: "supported".to_string(),
+ capability_boundary: "supported".to_string(),
_primary_rule: "SUP-1".to_string(),
_crux: "test crux".to_string(),
}
@@ -352,19 +487,17 @@ mod tests {
}
}
- fn router(client: Arc<dyn RoutedLlmClient>) -> Result<Arc<super::super::FallThrough<State>>> {
+ fn router(client: Arc<dyn RoutedLlmClient>) -> Result<Arc<LlmTaskClassifier>> {
let target = |name: &str| LlmTarget {
semantic_name: name.to_string(),
llm_client: Some(client.clone()),
};
- let targets = LlmTargetSet::new(vec![target("efficient"), target("capable")]);
- let efficient = targets.get_target("efficient")?;
- let capable = targets.get_target("capable")?;
- Ok(Arc::new(
- super::super::FallThrough::<State>::new_with_state(targets).with_classifier(Arc::new(
- LlmTaskClassifier::new(target("judge"), efficient, capable, TEST_THRESHOLD)?,
- )),
- ))
+ Ok(Arc::new(LlmTaskClassifier::new(
+ target("judge"),
+ target("efficient"),
+ target("capable"),
+ test_config(TEST_THRESHOLD),
+ )?))
}
fn classify_request() -> Request {
@@ -378,6 +511,29 @@ mod tests {
}
}
+ fn classify_session_request() -> Request {
+ Request {
+ metadata: Some(Metadata {
+ session_id: Some("session-1".to_string()),
+ ..Metadata::default()
+ }),
+ ..classify_request()
+ }
+ }
+
+ fn classify_follow_up_request() -> Request {
+ let mut request = classify_request();
+ request
+ .llm_request
+ .messages
+ .push(Message::text(Role::Assistant, "I will add the test."));
+ request.llm_request.messages.push(Message::text(
+ Role::User,
+ "Now run the test suite and report the result.",
+ ));
+ request
+ }
+
#[tokio::test]
async fn an_unreachable_judge_routes_capable_instead_of_failing_the_request() -> Result<()> {
let router = router(Arc::new(UnreachableJudgeClient))?;
@@ -408,6 +564,67 @@ mod tests {
Ok(())
}
+ #[tokio::test]
+ async fn classifier_config_enables_session_affinity() -> Result<()> {
+ let client = Arc::new(PerRequestClient::default());
+ let target = |name: &str| LlmTarget {
+ semantic_name: name.to_string(),
+ llm_client: Some(client.clone()),
+ };
+ let router = Arc::new(LlmTaskClassifier::new(
+ target("judge"),
+ target("efficient"),
+ target("capable"),
+ TaskClassifierConfig {
+ session_affinity: true,
+ ..test_config(TEST_THRESHOLD)
+ },
+ )?);
+
+ router
+ .clone()
+ .run(Context::default(), classify_session_request())
+ .await?;
+ router
+ .clone()
+ .run(Context::default(), classify_session_request())
+ .await?;
+
+ assert_eq!(client.calls(), vec!["judge", "efficient", "efficient"]);
+ Ok(())
+ }
+
+ #[tokio::test]
+ async fn classifier_config_reuses_message_hash_affinity_for_a_follow_up() -> Result<()> {
+ let client = Arc::new(PerRequestClient::default());
+ let target = |name: &str| LlmTarget {
+ semantic_name: name.to_string(),
+ llm_client: Some(client.clone()),
+ };
+ let router = Arc::new(LlmTaskClassifier::new(
+ target("judge"),
+ target("efficient"),
+ target("capable"),
+ TaskClassifierConfig {
+ session_affinity: true,
+ message_hash_fallback: true,
+ ..test_config(TEST_THRESHOLD)
+ },
+ )?);
+
+ router
+ .clone()
+ .run(Context::default(), classify_request())
+ .await?;
+ router
+ .clone()
+ .run(Context::default(), classify_follow_up_request())
+ .await?;
+
+ assert_eq!(client.calls(), vec!["judge", "efficient", "efficient"]);
+ Ok(())
+ }
+
#[test]
fn the_threshold_boundary_is_inclusive() -> Result<()> {
let policy = policy();
@@ -421,27 +638,60 @@ mod tests {
#[test]
fn the_threshold_moves_the_routing_boundary() -> Result<()> {
let borderline = verdict(0.5, 1.0, false);
- let strict = TaskClassifierPolicy::new("efficient", "capable", 0.9);
- let lenient = TaskClassifierPolicy::new("efficient", "capable", 0.1);
+ let strict = TaskClassifierPolicy::new("efficient", "capable", test_config(0.9));
+ let lenient = TaskClassifierPolicy::new("efficient", "capable", test_config(0.1));
assert_eq!(selected(&strict, Some(&borderline))?, "capable");
assert_eq!(selected(&lenient, Some(&borderline))?, "efficient");
Ok(())
}
#[test]
- fn an_out_of_range_threshold_is_rejected() -> Result<()> {
+ fn invalid_classifier_config_is_rejected() -> Result<()> {
let target = |name: &str| LlmTarget {
semantic_name: name.to_string(),
llm_client: None,
};
for bad in [1.5, -0.1, f64::NAN, f64::INFINITY] {
assert!(
- LlmTaskClassifier::new(target("judge"), target("e"), target("c"), bad).is_err(),
- "threshold {bad} should be rejected"
+ LlmTaskClassifier::new(
+ target("judge"),
+ target("e"),
+ target("c"),
+ test_config(bad),
+ )
+ .is_err(),
+ "base threshold {bad} should be rejected"
);
}
- LlmTaskClassifier::new(target("judge"), target("e"), target("c"), 0.0)?;
- LlmTaskClassifier::new(target("judge"), target("e"), target("c"), 1.0)?;
+ for config in [
+ TaskClassifierConfig {
+ base_threshold: 0.5,
+ min_confidence: 1.1,
+ capability_elevated_floor: None,
+ session_affinity: false,
+ message_hash_fallback: false,
+ },
+ TaskClassifierConfig {
+ base_threshold: 0.5,
+ min_confidence: 0.0,
+ capability_elevated_floor: Some(0.5),
+ session_affinity: false,
+ message_hash_fallback: false,
+ },
+ TaskClassifierConfig {
+ base_threshold: 0.5,
+ min_confidence: 0.0,
+ capability_elevated_floor: None,
+ session_affinity: false,
+ message_hash_fallback: true,
+ },
+ ] {
+ assert!(
+ LlmTaskClassifier::new(target("judge"), target("e"), target("c"), config).is_err()
+ );
+ }
+ LlmTaskClassifier::new(target("judge"), target("e"), target("c"), test_config(0.0))?;
+ LlmTaskClassifier::new(target("judge"), target("e"), target("c"), test_config(1.0))?;
Ok(())
}
@@ -450,12 +700,43 @@ mod tests {
let policy = policy();
let invalid_probability = verdict(1.1, 1.0, false);
let abstained = verdict(1.0, 1.0, true);
+ let invalid_boundary = TaskClassifierVerdict {
+ capability_boundary: "unknown".to_string(),
+ ..verdict(1.0, 1.0, false)
+ };
assert_eq!(selected(&policy, Some(&invalid_probability))?, "capable");
assert_eq!(selected(&policy, Some(&abstained))?, "capable");
+ assert_eq!(selected(&policy, Some(&invalid_boundary))?, "capable");
assert_eq!(selected(&policy, None)?, "capable");
Ok(())
}
+ #[test]
+ fn elevated_capability_floor_is_a_targeted_safety_brake() -> Result<()> {
+ let policy = TaskClassifierPolicy::new(
+ "efficient",
+ "capable",
+ TaskClassifierConfig {
+ capability_elevated_floor: Some(0.45),
+ ..test_config(0.25)
+ },
+ );
+ let supported = verdict(0.30, 1.0, false);
+ let elevated = TaskClassifierVerdict {
+ capability_boundary: "uncertain".to_string(),
+ ..verdict(0.30, 1.0, false)
+ };
+ let strong_elevated = TaskClassifierVerdict {
+ capability_boundary: "unsupported".to_string(),
+ ..verdict(0.50, 1.0, false)
+ };
+
+ assert_eq!(selected(&policy, Some(&supported))?, "efficient");
+ assert_eq!(selected(&policy, Some(&elevated))?, "capable");
+ assert_eq!(selected(&policy, Some(&strong_elevated))?, "efficient");
+ Ok(())
+ }
+
#[test]
fn capability_judge_builds_a_structured_request() -> Result<()> {
let judge = CapabilityJudge {
@@ -484,23 +765,17 @@ mod tests {
let judge_request = judge.build_request(&State::default(), &request);
assert_eq!(judge_request.llm_request.model, request.llm_request.model);
- assert_eq!(
- judge_request.llm_request.messages.len(),
- RECENT_MESSAGE_WINDOW + 4
- );
+ assert_eq!(judge_request.llm_request.messages.len(), 2);
let contents = judge_request
.llm_request
.messages
.iter()
.filter_map(|message| message.text_content("\n"))
.collect::<Vec<_>>();
- assert!(contents.contains(&"initial task".to_string()));
- assert!(contents.contains(&"recent 1".to_string()));
- assert!(contents.contains(&"recent 5".to_string()));
- assert!(contents.contains(&"client instructions".to_string()));
- assert!(contents.contains(&"client developer instructions".to_string()));
- assert!(!contents.contains(&"old response".to_string()));
- assert!(!contents.contains(&"old follow-up".to_string()));
+ assert!(contents.contains(&"recent 4".to_string()));
+ assert!(!contents.contains(&"initial task".to_string()));
+ assert!(!contents.contains(&"recent 5".to_string()));
+ assert!(!contents.contains(&"client instructions".to_string()));
assert_eq!(
judge_request.llm_request.output.response_format,
judge.config.response_schema
diff --git a/crates/libsy/src/algorithms/util/affinity.rs b/crates/libsy/src/algorithms/util/affinity.rs
index 8f33db35a..c71b9fb0a 100644
--- a/crates/libsy/src/algorithms/util/affinity.rs
+++ b/crates/libsy/src/algorithms/util/affinity.rs
@@ -17,11 +17,12 @@
//! session. [`AffinityRouter::for_subagents`] narrows affinity to explicitly identified
//! child agents, leaving root traffic to later classifiers on every turn.
-use std::collections::{HashMap, HashSet};
+use std::collections::{hash_map::DefaultHasher, HashMap, HashSet};
+use std::hash::{Hash, Hasher};
use async_trait::async_trait;
use parking_lot::Mutex;
-use switchyard_protocol::Request;
+use switchyard_protocol::{Request, Role};
use crate::{Classification, Classifier, Event, Processor, Score};
@@ -57,6 +58,8 @@ pub struct AffinityRouter {
latch_only: Option<HashSet<String>>,
/// Whether root-session requests should abstain instead of being retained.
subagents_only: bool,
+ /// In absence of headers, use the message hash based fallback key to do task based routing
+ message_hash_fallback: bool,
/// Retained assignments, shared across this router's processor and classifier roles.
///
/// Held on the instance so the two roles share one process-local map through a
@@ -80,6 +83,12 @@ impl AffinityRouter {
}
}
+ /// Uses the first user-message text as a fallback key when metadata has no session.
+ pub fn with_message_hash_fallback(mut self) -> Self {
+ self.message_hash_fallback = true;
+ self
+ }
+
/// Restricts latching to `models`; a decision for any other model routes but is not
/// retained.
pub fn with_latch_only(mut self, models: impl IntoIterator<Item = impl Into<String>>) -> Self {
@@ -96,18 +105,34 @@ impl AffinityRouter {
/// Derives the stable identity this router should retain for `request`.
fn affinity_key(&self, request: &Request) -> Option<AffinityKey> {
- let metadata = request.metadata.as_ref()?;
- let session = metadata.session_id.clone()?;
- if metadata.is_subagent {
- Some(AffinityKey::Subagent {
- session,
- agent: metadata.agent_id.clone()?,
- })
- } else if self.subagents_only {
- None
- } else {
- Some(AffinityKey::Session(session))
+ if let Some(metadata) = request.metadata.as_ref() {
+ if let Some(session) = metadata.session_id.clone() {
+ return if metadata.is_subagent {
+ metadata
+ .agent_id
+ .clone()
+ .map(|agent| AffinityKey::Subagent { session, agent })
+ } else if self.subagents_only {
+ None
+ } else {
+ Some(AffinityKey::Session(session))
+ };
+ }
}
+
+ // If headers are not present and we are not a subagent, use the message hash based fallback key to do task based routing
+ let is_subagent = request
+ .metadata
+ .as_ref()
+ .is_some_and(|metadata| metadata.is_subagent);
+ (!self.subagents_only && !is_subagent && self.message_hash_fallback)
+ .then(|| {
+ first_user_message_hash(request).map(|hash| {
+ tracing::debug!(affinity_key = %hash, "affinity using message hash fallback");
+ AffinityKey::Session(hash)
+ })
+ })
+ .flatten()
}
}
@@ -131,6 +156,20 @@ where
}
}
+/// Hashes the first user message so later turns retain the initial task's affinity.
+/// For benchmarking purpose with harnesses, task instructions are added as a user prompt to the request so we hash the initial user message.
+/// TODO: Have not considered multi-modal payloads yet. That needs to be handled separately.
+fn first_user_message_hash(request: &Request) -> Option<String> {
+ let message = request
+ .llm_request
+ .messages
+ .iter()
+ .find(|message| message.role == Role::User)?;
+ let mut hasher = DefaultHasher::new();
+ message.text_content("")?.hash(&mut hasher);
+ Some(format!("{:016x}", hasher.finish()))
+}
+
#[async_trait]
impl<S> Classifier<S> for AffinityRouter
where
@@ -171,7 +210,9 @@ mod tests {
use std::sync::Arc;
- use switchyard_protocol::{text_request, Decision, Metadata};
+ use switchyard_protocol::{
+ text_request, ContentBlock, Decision, LlmRequest, Message, Metadata,
+ };
/// Boxed, thread-safe error type keeping the test helpers ergonomic.
type BoxErr = Box<dyn std::error::Error + Send + Sync>;
@@ -201,6 +242,30 @@ mod tests {
}
}
+ fn task_request(
+ metadata: Option<Metadata>,
+ first_user: &str,
+ follow_up: Option<&str>,
+ ) -> Request {
+ let mut messages = vec![
+ Message::text(Role::System, "follow repository instructions"),
+ Message::text(Role::User, first_user),
+ ];
+ if let Some(follow_up) = follow_up {
+ messages.push(Message::text(Role::Assistant, "I will inspect the code."));
+ messages.push(Message::text(Role::User, follow_up));
+ }
+ Request {
+ llm_request: LlmRequest {
+ model: Some("auto".to_string()),
+ messages,
+ ..LlmRequest::default()
+ },
+ raw_request: None,
+ metadata,
+ }
+ }
+
fn session(session_id: &str, agent_id: &str) -> Metadata {
Metadata {
session_id: Some(session_id.to_string()),
@@ -374,6 +439,112 @@ mod tests {
Ok(())
}
+ #[tokio::test]
+ async fn message_hash_fallback_uses_the_first_user_message() -> Result<(), BoxErr> {
+ let router = AffinityRouter::new().with_message_hash_fallback();
+ let mut state = ();
+
+ let mut first = task_request(
+ None,
+ "Add a unit test for this function.",
+ Some("Now run the test suite."),
+ );
+ retain(&router, &mut state, &mut first, "weak").await?;
+
+ let mut follow_up = task_request(
+ None,
+ "Add a unit test for this function.",
+ Some("Now file a pull request."),
+ );
+ assert_eq!(
+ scores(&router, &mut state, &mut follow_up)
+ .await?
+ .first()
+ .map(|score| score.target.as_str()),
+ Some("weak")
+ );
+
+ let mut other_task = task_request(
+ None,
+ "Reimplement this binary from two input/output pairs.",
+ Some("Now run the test suite."),
+ );
+ assert!(scores(&router, &mut state, &mut other_task)
+ .await?
+ .is_empty());
+ Ok(())
+ }
+
+ #[test]
+ fn user_message_hash_ignores_non_text_provider_payloads() {
+ let request = |user_message| Request {
+ llm_request: LlmRequest {
+ messages: vec![user_message],
+ ..LlmRequest::default()
+ },
+ raw_request: None,
+ metadata: None,
+ };
+ let text_only = request(Message::text(Role::User, "Implement the parser."));
+ let text_with_reasoning = request(Message {
+ role: Role::User,
+ content: vec![
+ ContentBlock::Text {
+ text: "Implement the parser.".to_string(),
+ },
+ ContentBlock::Reasoning {
+ text: "Internal provider reasoning.".to_string(),
+ signature: Some("provider-signature".to_string()),
+ },
+ ],
+ });
+
+ assert_eq!(
+ first_user_message_hash(&text_only),
+ first_user_message_hash(&text_with_reasoning)
+ );
+ }
+
+ #[tokio::test]
+ async fn metadata_session_takes_precedence_over_message_hash() -> Result<(), BoxErr> {
+ let router = AffinityRouter::new().with_message_hash_fallback();
+ let mut state = ();
+
+ let mut first = task_request(
+ Some(session("session-1", "agent-a")),
+ "Implement the parser.",
+ None,
+ );
+ retain(&router, &mut state, &mut first, "strong").await?;
+
+ let mut other_session = task_request(
+ Some(session("session-2", "agent-a")),
+ "Implement the parser.",
+ None,
+ );
+ assert!(scores(&router, &mut state, &mut other_session)
+ .await?
+ .is_empty());
+ Ok(())
+ }
+
+ #[tokio::test]
+ async fn message_hash_fallback_abstains_for_subagents() -> Result<(), BoxErr> {
+ let router = AffinityRouter::new().with_message_hash_fallback();
+ let mut state = ();
+ let mut subagent = task_request(
+ Some(Metadata {
+ is_subagent: true,
+ ..Metadata::default()
+ }),
+ "Implement the parser.",
+ None,
+ );
+
+ assert!(scores(&router, &mut state, &mut subagent).await?.is_empty());
+ Ok(())
+ }
+
#[tokio::test]
async fn one_router_serves_both_roles() -> Result<(), BoxErr> {
// The same instance is registered under both SDK roles; a decision folded in via
diff --git a/crates/libsy/src/lib.rs b/crates/libsy/src/lib.rs
index e4a6bea86..0c54c3015 100644
--- a/crates/libsy/src/lib.rs
+++ b/crates/libsy/src/lib.rs
@@ -74,8 +74,7 @@
//!
//! [`algorithms::Random`] provides uniform or weighted random routing.
//!
-//! [`algorithms::LlmTaskClassifier`] classifies with one model; compose it with
-//! [`algorithms::FallThrough`] to route to the selected target.
+//! [`algorithms::LlmTaskClassifier`] uses one model to classify and route to its selected target.
mod core;
pub use core::*;
diff --git a/crates/libsy/tests/observability.rs b/crates/libsy/tests/observability.rs
index 70dee0411..e22cef8eb 100644
--- a/crates/libsy/tests/observability.rs
+++ b/crates/libsy/tests/observability.rs
@@ -26,10 +26,10 @@ use tracing_subscriber::layer::{Context as LayerContext, SubscriberExt};
use tracing_subscriber::registry::LookupSpan;
use tracing_subscriber::Layer;
-use switchyard_libsy::algorithms::{FallThrough, LlmTaskClassifier};
+use switchyard_libsy::algorithms::{LlmTaskClassifier, TaskClassifierConfig};
use switchyard_libsy::{
AggLlmResponse, Algorithm, Context, Decision, Driver, LibsyError, LlmResponse, LlmTarget,
- LlmTargetSet, Metadata, Request, Response, RoutedLlmClient, State, Step, Usage,
+ LlmTargetSet, Metadata, Request, Response, RoutedLlmClient, Step, Usage,
};
use switchyard_protocol::{text_request, text_response, LlmClientError};
@@ -725,11 +725,18 @@ async fn classifier_metrics_count_only_the_final_routed_call() -> switchyard_lib
let targets = LlmTargetSet::new(vec![target("weak"), target("strong")]);
let weak = targets.get_target("weak")?;
let strong = targets.get_target("strong")?;
- let router = Arc::new(
- FallThrough::<State>::new_with_state(targets).with_classifier(Arc::new(
- LlmTaskClassifier::new(target("classifier"), weak, strong, 0.5)?,
- )),
- );
+ let router = Arc::new(LlmTaskClassifier::new(
+ target("classifier"),
+ weak,
+ strong,
+ TaskClassifierConfig {
+ base_threshold: 0.5,
+ min_confidence: 0.0,
+ capability_elevated_floor: None,
+ session_affinity: false,
+ message_hash_fallback: false,
+ },
+ )?);
let (trace, _response) = router
.run(
diff --git a/crates/switchyard-server/README.md b/crates/switchyard-server/README.md
index 8cfec5844..f83760b0b 100644
--- a/crates/switchyard-server/README.md
+++ b/crates/switchyard-server/README.md
@@ -34,7 +34,7 @@ type = "llm_classifier"
classifier_target = "model_a"
strong_target = "model_a"
weak_target = "model_b"
-threshold = 0.5
+base_threshold = 0.5
[routes.passthrough]
id = "switchyard/passthrough"
@@ -64,6 +64,23 @@ never contains the secret itself. If omitted, the client sends no authentication
Random-route `weights` are relative, follow target order, and do not need to sum to one. Omit them
for equal weighting. The optional `seed` reproduces the selection sequence for the same call order.
+An `llm_classifier` route sends each task to `classifier_target` for a capability verdict, then
+routes to `weak_target` or `strong_target`. Beyond the three targets it accepts these keys; only
+`base_threshold` is required, and anything the judge cannot decide routes to `strong_target`:
+
+| Key | Default | Meaning |
+|---|---|---|
+| `base_threshold` | *required* | Lowest solve probability that routes a task to `weak_target`. Raise it to send less traffic to the weak model. |
+| `min_confidence` | `0.0` | Lowest judge confidence that permits weak routing. `0.0` disables the gate. |
+| `capability_elevated_floor` | unset | Higher solve-probability floor applied only to tasks the judge marks uncertain, unsupported, or unmatched. Unset reuses `base_threshold`. |
+| `session_affinity` | `false` | Reuses a session's first routing decision on later turns, so the judge is called once per session rather than once per turn. |
+| `message_hash_fallback` | `false` | Extends affinity to clients that send no session header, keying on the first user message. Requires `session_affinity = true`. |
+
+Session affinity retains a decision for the process lifetime, including a `strong_target`
+fallback produced while the judge was unreachable. `message_hash_fallback` keys on request
+content rather than a session id, so unrelated callers sending identical text share one
+assignment.
+
## Metrics
`GET /metrics` exposes Prometheus text from the server's process-wide OpenTelemetry provider.
diff --git a/crates/switchyard-server/src/config.rs b/crates/switchyard-server/src/config.rs
index 3315a8914..5074b4ff5 100644
--- a/crates/switchyard-server/src/config.rs
+++ b/crates/switchyard-server/src/config.rs
@@ -8,8 +8,8 @@ use std::fs;
use std::path::Path;
use std::sync::Arc;
-use libsy::algorithms::{FallThrough, LlmTaskClassifier, Noop, Passthrough, Random};
-use libsy::{Algorithm, LlmTarget, LlmTargetSet, RoutedLlmClient, State};
+use libsy::algorithms::{LlmTaskClassifier, Noop, Passthrough, Random, TaskClassifierConfig};
+use libsy::{Algorithm, LlmTarget, LlmTargetSet, RoutedLlmClient};
use serde::Deserialize;
use switchyard_llm_client::{Backend, HttpBackendConfig, ModelConfig, TranslatingLlmClient};
@@ -180,7 +180,8 @@ enum RouteConfig {
classifier_target: String,
strong_target: String,
weak_target: String,
- threshold: f64,
+ #[serde(flatten)]
+ classifier_config: TaskClassifierConfig,
},
}
@@ -263,22 +264,19 @@ fn build_algorithm(
classifier_target,
strong_target,
weak_target,
- threshold,
+ classifier_config,
..
} => {
let classifier = resolve_target(route_name, classifier_target, targets)?;
let strong = resolve_target(route_name, strong_target, targets)?;
let weak = resolve_target(route_name, weak_target, targets)?;
- // The judge is called through its own target, so it is not a routing destination.
- let target_set = LlmTargetSet::new(vec![strong.clone(), weak.clone()]);
// The weak model is the efficient tier; the strong model is the capable one.
- let judge =
- LlmTaskClassifier::new(classifier, weak, strong, *threshold).map_err(|error| {
- ServerError::new(format!("llm_classifier route {route_name}: {error}"))
- })?;
- let router =
- FallThrough::<State>::new_with_state(target_set).with_classifier(Arc::new(judge));
- Ok(Arc::new(router))
+ let algorithm =
+ LlmTaskClassifier::new(classifier, weak, strong, classifier_config.clone())
+ .map_err(|error| {
+ ServerError::new(format!("llm_classifier route {route_name}: {error}"))
+ })?;
+ Ok(Arc::new(algorithm))
}
}
}
@@ -362,7 +360,7 @@ type = "llm_classifier"
classifier_target = "classifier"
strong_target = "strong"
weak_target = "weak"
-threshold = 0.5
+base_threshold = 0.5
[routes.passthrough]
id = "switchyard/passthrough"
@@ -439,8 +437,15 @@ target = "weak"
"at least one weight must be positive",
),
(
- VALID_CONFIG.replace("threshold = 0.5", "threshold = 1.5"),
- "threshold must be between 0 and 1",
+ VALID_CONFIG.replace("base_threshold = 0.5", "base_threshold = 1.5"),
+ "base_threshold must be between 0 and 1",
+ ),
+ (
+ VALID_CONFIG.replace(
+ "base_threshold = 0.5",
+ "base_threshold = 0.5\nmessage_hash_fallback = true",
+ ),
+ "message_hash_fallback requires session_affinity",
),
(
VALID_CONFIG.replace("schema_version = 1", "schema_version = 2"),
@@ -470,6 +475,16 @@ target = "weak"
Ok(())
}
+ #[test]
+ fn accepts_session_affinity_with_message_hash_fallback() -> ServerResult<()> {
+ let configured = VALID_CONFIG.replace(
+ "base_threshold = 0.5",
+ "base_threshold = 0.25\nmin_confidence = 0.0\ncapability_elevated_floor = 0.45\nsession_affinity = true\nmessage_hash_fallback = true",
+ );
+ server_state_from_toml(&configured)?;
+ Ok(())
+ }
+
#[test]
fn api_key_environment_reference_is_validated() {
let missing = VALID_CONFIG.replacen(
diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rs
index c963b393b..95173c7dd 100644
--- a/crates/switchyard-server/tests/server.rs
+++ b/crates/switchyard-server/tests/server.rs
@@ -321,7 +321,7 @@ type = "llm_classifier"
classifier_target = "classifier"
strong_target = "strong"
weak_target = "weak"
-threshold = 0.5
+base_threshold = 0.5
[routes.passthrough]
id = "switchyard/passthrough"