Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

fix(libsy): allow message_hash_fallback with classify_trigger = user_turn (#815)

Deepak Jain (DJ)9d0ae83a14ae
diff --git a/crates/libsy/src/algorithms/llm_class.rs b/crates/libsy/src/algorithms/llm_class.rs
index 12d9241d0..488bc6585 100644
--- a/crates/libsy/src/algorithms/llm_class.rs
+++ b/crates/libsy/src/algorithms/llm_class.rs
@@ -418,10 +418,12 @@ impl TaskClassifierConfig {
message: "max_output_tokens must be at least 1".to_string(),
});
}
- if self.message_hash_fallback && self.classify_trigger != ClassifyTrigger::NewSession {
+ // Only `every_request` is rejected: it retains no target, so a fallback identity has nothing to key on. Both retaining triggers can key the retained target on a message hash when the caller sends no session id.
+ if self.message_hash_fallback && self.classify_trigger == ClassifyTrigger::EveryRequest {
return Err(LibsyError::AlgorithmError {
- message: "message_hash_fallback requires classify_trigger = new_session"
- .to_string(),
+ message:
+ "message_hash_fallback requires classify_trigger = new_session or user_turn"
+ .to_string(),
});
}
Ok(())
@@ -490,10 +492,12 @@ impl CustomClassifierConfig {
message: "max_output_tokens must be at least 1".to_string(),
});
}
- if self.message_hash_fallback && self.classify_trigger != ClassifyTrigger::NewSession {
+ // Only `every_request` is rejected: it retains no target, so a fallback identity has nothing to key on. Both retaining triggers can key the retained target on a message hash when the caller sends no session id.
+ if self.message_hash_fallback && self.classify_trigger == ClassifyTrigger::EveryRequest {
return Err(LibsyError::AlgorithmError {
- message: "message_hash_fallback requires classify_trigger = new_session"
- .to_string(),
+ message:
+ "message_hash_fallback requires classify_trigger = new_session or user_turn"
+ .to_string(),
});
}
Ok(())
@@ -713,10 +717,13 @@ impl LlmTaskClassifier {
inner: Arc<dyn Classifier<State>>,
config: ClassifierRouteConfig,
) -> Result<Self> {
- if config.message_hash_fallback && config.classify_trigger != ClassifyTrigger::NewSession {
+ // Only `every_request` is rejected: it retains no target, so a fallback identity has nothing to key on. Both retaining triggers can key the retained target on a message hash when the caller sends no session id.
+ if config.message_hash_fallback && config.classify_trigger == ClassifyTrigger::EveryRequest
+ {
return Err(LibsyError::AlgorithmError {
- message: "message_hash_fallback requires classify_trigger = new_session"
- .to_string(),
+ message:
+ "message_hash_fallback requires classify_trigger = new_session or user_turn"
+ .to_string(),
});
}
// Affinity comes first so a retained assignment short-circuits the judge call.
@@ -1277,6 +1284,40 @@ mod tests {
Ok(())
}
+ #[test]
+ fn message_hash_fallback_accepts_retaining_triggers() -> Result<()> {
+ // Only `every_request` is rejected: it retains no target, so a fallback
+ // identity has nothing to key on. Both retaining triggers may key the
+ // retained target on a message hash when the caller sends no session id.
+ for trigger in [ClassifyTrigger::NewSession, ClassifyTrigger::UserTurn] {
+ let config = TaskClassifierConfig {
+ base_threshold: 0.5,
+ classify_trigger: trigger,
+ message_hash_fallback: true,
+ ..TaskClassifierConfig::default()
+ };
+ LlmTaskClassifier::new(LlmClassifierConfig::Capability { config }).map_err(
+ |error| LibsyError::AlgorithmError {
+ message: format!("{trigger:?} with message_hash_fallback rejected: {error}"),
+ },
+ )?;
+ }
+ let every_request = TaskClassifierConfig {
+ base_threshold: 0.5,
+ classify_trigger: ClassifyTrigger::EveryRequest,
+ message_hash_fallback: true,
+ ..TaskClassifierConfig::default()
+ };
+ assert!(
+ LlmTaskClassifier::new(LlmClassifierConfig::Capability {
+ config: every_request
+ })
+ .is_err(),
+ "every_request with message_hash_fallback should stay rejected"
+ );
+ Ok(())
+ }
+
#[test]
fn an_unusable_verdict_is_ambiguous() -> Result<()> {
let policy = policy();
diff --git a/crates/switchyard-server/README.md b/crates/switchyard-server/README.md
index 45176afb6..7cf6930e8 100644
--- a/crates/switchyard-server/README.md
+++ b/crates/switchyard-server/README.md
@@ -131,7 +131,7 @@ routes to `weak_target` or `strong_target`. Beyond the three targets it accepts
| `base_threshold` | *required* | Lowest solve probability that routes a task to `weak_target`. Raise it to send less traffic to the weak model. |
| `threshold_step` | `0.0` | Finite, non-negative amount added once for uncertain or unmatched verdicts and twice for unsupported verdicts. `base_threshold + 2 * threshold_step` must be at most `1`. |
| `classify_trigger` | `every_request` | When the judge runs. `every_request` judges every request including tool continuations, `user_turn` judges each new user message and holds that target across the tool calls between, `new_session` judges once and reuses that target for the session. |
-| `message_hash_fallback` | `false` | Extends affinity to clients that send no session header, keying on the first user message. Requires `classify_trigger = "new_session"`. |
+| `message_hash_fallback` | `false` | Extends affinity to clients that send no session header, keying on the first user message. Requires `classify_trigger = "new_session"` or `"user_turn"`. |
Session affinity retains a decision for the process lifetime, including a `strong_target`
fallback produced while the judge was unreachable. `message_hash_fallback` keys on request
diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.md
index f644cae30..f0c7d5cc5 100644
--- a/docs/reference/toml_schema.md
+++ b/docs/reference/toml_schema.md
@@ -241,7 +241,7 @@ Capability mode classifies before serving. See
| `base_threshold` | Yes | — | Lowest solve probability that routes to the weak target. In `[0, 1]`. |
| `threshold_step` | No | `0.0` | Finite, non-negative amount added once for uncertain or unmatched verdicts and twice for unsupported verdicts. `base_threshold + 2 * threshold_step` must be at most `1`. |
| `classify_trigger` | No | `every_request` | When the judge runs. `every_request` judges every request, tool continuations included. `user_turn` judges each new user message and retains that target across intervening tool calls only when requests carry a session ID; without a session ID, it behaves like `every_request`. `new_session` judges once and reuses that target for the session. |
-| `message_hash_fallback` | No | `false` | Keys affinity on the first user message. Requires `classify_trigger = "new_session"`. |
+| `message_hash_fallback` | No | `false` | Retains the target against a hash of the first user message when a request carries no session ID. Requires `classify_trigger = "new_session"` or `"user_turn"`. |
| `recent_turn_window` | No | unset | When unset, the judge sees the opening task and latest user follow-up, when present. When set, it also sees trailing turns. |
| `prompt` | No | packaged prompt | Replaces the capability prompt. The packaged schema is sent separately as structured-output configuration. |
@@ -282,7 +282,7 @@ how one route chooses between more than two models.
| `response_schema` | Yes | — | Inner JSON Schema encoded as a TOML string. Switchyard adds the provider wrapper. |
| `policy` | Yes | — | Policy table. `target_selector` accepts a JSON Pointer such as `/decision/target`. |
| `classify_trigger` | No | `every_request` | When the judge runs. `every_request` judges every request, tool continuations included. `user_turn` judges each new user message and retains that target across intervening tool calls only when requests carry a session ID; without a session ID, it behaves like `every_request`. `new_session` judges once and reuses that target for the session. |
-| `message_hash_fallback` | No | `false` | Keys affinity on the first user message. Requires `classify_trigger = "new_session"`. |
+| `message_hash_fallback` | No | `false` | Retains the target against a hash of the first user message when a request carries no session ID. Requires `classify_trigger = "new_session"` or `"user_turn"`. |
| `recent_turn_window` | No | unset | When unset, the judge sees the opening task and latest user follow-up, when present. When set, it also sees trailing turns. |
The selected JSON label must name a configured group. A label naming a target
diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.md
index d57d70dde..c1d63bfb4 100644
--- a/docs/routing_algorithms/llm_classifier_routing.md
+++ b/docs/routing_algorithms/llm_classifier_routing.md
@@ -123,7 +123,7 @@ for the server merge behavior.
| `threshold_step` | `0.0` | Amount added for each boundary step. Must be finite and non-negative, and `base_threshold + 2 * threshold_step` must not exceed `1`. |
| `recent_turn_window` | unset | When unset, the judge sees the opening user task and the latest user message when they differ. When set to `N`, it sees the opening user task and the last `N` conversation messages after that task. `0` keeps only the opening task. Client system and developer instructions are not shown to the judge. |
| `classify_trigger` | `every_request` | When the judge runs. `every_request` judges every request, tool continuations included. `user_turn` judges each new user message and holds that target across the tool calls between. `new_session` judges once and reuses that target for the session. |
-| `message_hash_fallback` | `false` | When session metadata is absent, keys affinity from the first user-message text. Requires `classify_trigger = "new_session"`. |
+| `message_hash_fallback` | `false` | When session metadata is absent, keys affinity from the first user-message text. Requires `classify_trigger = "new_session"` or `"user_turn"`. |
| `prompt` | packaged capability prompt | Replaces the classifier's system prompt. The packaged verdict schema and routing policy remain active. |
| `response_format_type` | `json_schema` | Structured-output mode for capability and escalation judges. Use `json_object` for providers without JSON Schema support. |
| `max_output_tokens` | `4096` | Maximum completion tokens available to the classifier verdict. Must be at least `1`. |