Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

feat(libsy): let routing judges fail open on client errors and deadlines (#888)

Graham King8a2195279d93
diff --git a/crates/libsy-llm-client/README.md b/crates/libsy-llm-client/README.md
index 8b0ef430d..9f028be2b 100644
--- a/crates/libsy-llm-client/README.md
+++ b/crates/libsy-llm-client/README.md
@@ -266,9 +266,11 @@ fn build_multi_format_client(
`run` and `decide` collect streams used during routing, including answers that an
algorithm must inspect, before returning them to the algorithm. They retain the
provider events for replay. After the configured retries, a client failure stops
-`run` or `decide` before the algorithm can choose another routing candidate. This
-also applies when `timeout` is `None` or an advisor has `fail_open = true`.
-`libsy` and custom hosts that drive it directly are unchanged.
+`run` or `decide` by default. Calls with `CallModel::recover_errors` enabled return
+the error to the algorithm so it can apply its fallback policy. Capability
+classifiers and advisor consults default to `fail_open = true`. Set `fail_open = false`
+to stop the request on their client failures. Recovery includes client errors and
+deadlines during stream collection. Custom hosts can honor the same flag when serving calls.
Retries replay the same upstream request to the same model. After routing completes,
non-timeout failures may try another completion candidate. A timeout stops the call.
diff --git a/crates/libsy-llm-client/src/run.rs b/crates/libsy-llm-client/src/run.rs
index 691e169cf..1ff16bc3a 100644
--- a/crates/libsy-llm-client/src/run.rs
+++ b/crates/libsy-llm-client/src/run.rs
@@ -11,7 +11,8 @@
//! libsy owns the stream mechanics; what this module adds is per-target request preparation,
//! ordered candidate fallback, and the `libsy.client_call` span around each candidate. Each
//! completion candidate exhausts its backend retry budget before fallback advances. Routing
-//! calls stop on the first candidate's failure. A timeout stops either kind of call.
+//! calls stop on the first candidate's failure unless the algorithm enables error recovery.
+//! A timeout stops completion calls.
//!
//! A Responses continuation can refer to state through `previous_response_id` or `conversation`.
//! [`ClientRouter`] sends native Responses state back to its provider; for Chat or Anthropic
@@ -49,7 +50,7 @@ use crate::{metrics, observability};
/// a per-call lookup, not one client for the whole run. Use
/// [`ClientRouter::single`](ClientRouter::single) when one client serves every target.
///
-/// Routing calls are buffered; a client failure stops the request before the algorithm continues.
+/// Routing calls are buffered; client failures stop the request unless the call enables recovery.
/// Once routing completes, non-timeout failures may try the outcome's ordered fallback candidates.
pub async fn run(
algorithm: Arc<dyn Algorithm>,
@@ -172,7 +173,7 @@ fn emit_routing_observations(
/// Serve one offloaded call and fulfill its promise.
///
-/// A client failure stops the driver before the algorithm can issue another call.
+/// A client failure stops the driver unless this call opts into algorithm-level recovery.
async fn serve(
clients: ClientRouter,
call: CallModel,
@@ -198,6 +199,7 @@ async fn serve(
.await
{
Ok(response) => call.respond(Ok(response)),
+ Err(error) if call.recover_errors => call.respond(Err(error)),
Err(error) => call.fail(error),
}
}
diff --git a/crates/libsy-llm-client/tests/observability.rs b/crates/libsy-llm-client/tests/observability.rs
index 93c12adb5..30adc407c 100644
--- a/crates/libsy-llm-client/tests/observability.rs
+++ b/crates/libsy-llm-client/tests/observability.rs
@@ -607,6 +607,7 @@ fn classifier_router() -> switchyard_libsy::Result<Arc<dyn Algorithm>> {
LlmClassifierConfig::Capability {
config: TaskClassifierConfig {
base_threshold: 0.5,
+ fail_open: false,
..TaskClassifierConfig::default()
},
},
@@ -1855,6 +1856,101 @@ async fn classifier_stops_on_client_errors_and_records_verdict_fallback()
Ok(())
}
+#[tokio::test]
+async fn classifier_can_fail_open_on_judge_client_errors() -> switchyard_libsy::Result<()> {
+ let _guard = serialize_test().lock().await;
+ assert!(TaskClassifierConfig::default().fail_open);
+ for (judge, reason) in [
+ (
+ Arc::new(TimeoutClient) as Arc<dyn RoutedLlmClient>,
+ "timeout",
+ ),
+ (
+ Arc::new(JudgeClient {
+ judge_model: "judge".into(),
+ outcome: JudgeOutcome::CallFailure,
+ }),
+ "upstream_5xx",
+ ),
+ (
+ Arc::new(JudgeClient {
+ judge_model: "judge".into(),
+ outcome: JudgeOutcome::StreamDecodeFailure,
+ }),
+ "invalid_response",
+ ),
+ ] {
+ let algorithm = |fail_open: Option<bool>| -> switchyard_libsy::Result<Arc<dyn Algorithm>> {
+ let mut config = serde_json::json!({"base_threshold": 0.5});
+ if let Some(fail_open) = fail_open {
+ config["fail_open"] = fail_open.into();
+ }
+ Ok(Arc::new(LlmTaskClassifier::new(
+ LlmClassifierConfig::Capability {
+ config: serde_json::from_value(config).unwrap(),
+ },
+ )?))
+ };
+ let clients = ClientRouter::new(std::collections::HashMap::from([
+ (ModelId::from("judge"), judge),
+ (
+ ModelId::from("capable"),
+ Arc::new(JudgeClient {
+ judge_model: "judge".into(),
+ outcome: JudgeOutcome::CallFailure,
+ }) as Arc<dyn RoutedLlmClient>,
+ ),
+ ]));
+ let models = classifier_models("judge", "efficient", "capable");
+ let stopped = switchyard_llm_client::decide(
+ algorithm(Some(false))?,
+ clients.clone(),
+ classifier_request(),
+ models.clone(),
+ )
+ .await;
+ assert!(
+ matches!(stopped, Err(LibsyError::ClientCall { .. })),
+ "{reason}"
+ );
+
+ let outcome = switchyard_llm_client::decide(
+ algorithm(None)?,
+ clients.clone(),
+ classifier_request(),
+ models.clone(),
+ )
+ .await?;
+ assert_eq!(outcome.selected_model_id()?.as_str(), "capable");
+ assert_eq!(
+ outcome.metadata.and_then(|metadata| metadata.evidence),
+ Some(serde_json::json!({"source": "fail_open", "reason_code": reason})),
+ );
+ let (selected, _) = switchyard_llm_client::run(
+ algorithm(Some(true))?,
+ clients,
+ classifier_request(),
+ models.clone(),
+ None,
+ )
+ .await?;
+ assert_eq!(selected.as_str(), "capable");
+
+ let answer_failure = switchyard_llm_client::run(
+ algorithm(None)?,
+ ClientRouter::single(Arc::new(TimeoutClient)),
+ classifier_request(),
+ models,
+ None,
+ )
+ .await;
+ assert!(matches!(answer_failure, Err(LibsyError::ClientCall {
+ target, source: LlmClientError::Timeout { .. },
+ }) if target.as_str() == "capable"));
+ }
+ Ok(())
+}
+
#[tokio::test]
async fn in_flight_gauge_reads_a_run_parked_on_an_unanswered_routing_call()
-> switchyard_libsy::Result<()> {
diff --git a/crates/libsy/src/algorithms/advisor_gate.rs b/crates/libsy/src/algorithms/advisor_gate.rs
index 4551ac703..9f0c35ef2 100644
--- a/crates/libsy/src/algorithms/advisor_gate.rs
+++ b/crates/libsy/src/algorithms/advisor_gate.rs
@@ -408,7 +408,14 @@ impl AdvisorGate {
Ok(advisor) => {
let advisor = advisor.clone();
let advisor_models = driver.models_for(&Category::Judge).to_vec();
- match driver.call_model(consult_request, advisor_models).await {
+ match driver
+ .call_model_with_error_recovery(
+ consult_request,
+ advisor_models,
+ self.config.fail_open,
+ )
+ .await
+ {
Ok(response) => {
let served_advisor = response
.served_model()
diff --git a/crates/libsy/src/algorithms/llm_class.rs b/crates/libsy/src/algorithms/llm_class.rs
index 488bc6585..40d9ad3d9 100644
--- a/crates/libsy/src/algorithms/llm_class.rs
+++ b/crates/libsy/src/algorithms/llm_class.rs
@@ -300,6 +300,8 @@ fn capability_evidence(
#[derive(Clone, Debug)]
/// Settings that control capability classifier prompting and routing.
pub struct TaskClassifierConfig {
+ /// Routes to the capable tier on judge client failures and deadlines. Defaults to true.
+ pub fail_open: bool,
/// Lowest solve probability that routes a supported task to the efficient target.
pub base_threshold: f64,
/// Amount added per capability-boundary step.
@@ -329,6 +331,8 @@ pub struct TaskClassifierConfig {
#[serde(deny_unknown_fields)]
struct TaskClassifierConfigWire {
base_threshold: f64,
+ #[serde(default = "default_fail_open")]
+ fail_open: bool,
#[serde(default)]
threshold_step: f64,
#[serde(default)]
@@ -358,6 +362,7 @@ impl<'de> Deserialize<'de> for TaskClassifierConfig {
contract = contract.with_response_format_type(wire.response_format_type);
Ok(Self {
base_threshold: wire.base_threshold,
+ fail_open: wire.fail_open,
threshold_step: wire.threshold_step,
classify_trigger: wire.classify_trigger,
message_hash_fallback: wire.message_hash_fallback,
@@ -368,6 +373,10 @@ impl<'de> Deserialize<'de> for TaskClassifierConfig {
}
}
+const fn default_fail_open() -> bool {
+ true
+}
+
const fn default_judge_max_output_tokens() -> u64 {
DEFAULT_JUDGE_MAX_OUTPUT_TOKENS
}
@@ -376,6 +385,7 @@ impl Default for TaskClassifierConfig {
fn default() -> Self {
Self {
base_threshold: 0.0,
+ fail_open: default_fail_open(),
threshold_step: 0.0,
classify_trigger: ClassifyTrigger::default(),
message_hash_fallback: false,
@@ -644,6 +654,7 @@ impl LlmTaskClassifier {
),
TaskClassifierPolicy::new(&config),
)
+ .with_error_recovery(config.fail_open)
.with_evidence(capability_evidence),
);
Self::from_classifier(
diff --git a/crates/libsy/src/algorithms/util/llm_judge.rs b/crates/libsy/src/algorithms/util/llm_judge.rs
index cb8277dae..d3fbe34e0 100644
--- a/crates/libsy/src/algorithms/util/llm_judge.rs
+++ b/crates/libsy/src/algorithms/util/llm_judge.rs
@@ -216,6 +216,7 @@ where
{
judge: J,
policy: P,
+ recover_errors: bool,
evidence: Option<EvidenceFn<J::Verdict, P>>,
}
@@ -229,10 +230,17 @@ where
Self {
judge,
policy,
+ recover_errors: false,
evidence: None,
}
}
+ /// Lets the judge policy recover from client failures, including deadlines.
+ pub(crate) fn with_error_recovery(mut self, enabled: bool) -> Self {
+ self.recover_errors = enabled;
+ self
+ }
+
/// Enables bounded evidence for built-in judges without widening the public policy trait.
pub(crate) fn with_evidence(mut self, evidence: EvidenceFn<J::Verdict, P>) -> Self {
self.evidence = Some(evidence);
@@ -256,11 +264,9 @@ where
/// Consults the judge, yielding `None` when it is unavailable or unintelligible.
///
- /// A judge is an optimization, not a dependency: failing the caller's request because the
- /// judge is down would be worse than routing without it, so every failure — transport,
- /// mid-stream, or unparseable reply — is logged and folded into `None` for the policy's
- /// fallback branch. A closed driver stream is folded too; the algorithm's next driver
- /// call surfaces it, so nothing is masked.
+ /// Errors delivered by the host and invalid verdicts are logged and folded into `None`
+ /// for the policy's fallback branch. The HTTP driver delivers client errors here when
+ /// error recovery is enabled; otherwise it stops the run.
pub(crate) async fn verdict(
&self,
state: &mut State,
@@ -272,9 +278,10 @@ where
tracing::info!(target = judge_model, "consulting llm judge");
let response = driver
- .call_model(
+ .call_model_with_error_recovery(
self.judge.build_request(state, request),
judge_models.to_vec(),
+ self.recover_errors,
)
.await
.inspect_err(|error| {
diff --git a/crates/libsy/src/core/algorithm.rs b/crates/libsy/src/core/algorithm.rs
index 959621db3..5c3ebda92 100644
--- a/crates/libsy/src/core/algorithm.rs
+++ b/crates/libsy/src/core/algorithm.rs
@@ -109,6 +109,8 @@ pub struct CallModel {
pub request: Request,
/// Candidate models, tried in order until one answers. Never empty.
pub models: Vec<ModelId>,
+ /// Return client errors to the algorithm so it can apply its fallback policy.
+ pub recover_errors: bool,
/// How to send the response back to the algorithm. `None` once the call is recorded.
reply: Option<oneshot::Sender<Result<Response>>>,
started: Instant,
@@ -275,6 +277,12 @@ impl Driver {
/// through [`CallModel::respond`] or [`CallModel::fail`]; outcome and token usage
/// are recorded on the span when the promise resolves. The provider call itself is the
/// host's, and is instrumented by whoever makes it.
+ pub async fn call_model(&self, request: Request, models: Vec<ModelId>) -> Result<Response> {
+ self.call_model_with_error_recovery(request, models, false)
+ .await
+ }
+
+ /// Allows a routing policy to handle a failed call when recovery is enabled.
#[tracing::instrument(
target = "libsy",
name = "libsy.llm_call",
@@ -290,7 +298,12 @@ impl Driver {
reasoning_tokens = tracing::field::Empty,
)
)]
- pub async fn call_model(&self, mut request: Request, models: Vec<ModelId>) -> Result<Response> {
+ pub(crate) async fn call_model_with_error_recovery(
+ &self,
+ mut request: Request,
+ models: Vec<ModelId>,
+ recover_errors: bool,
+ ) -> Result<Response> {
let Some(selected_model_id) = models.first() else {
return Err(LibsyError::NoTargets);
};
@@ -301,6 +314,7 @@ impl Driver {
algorithm: self.algorithm.clone(),
request,
models,
+ recover_errors,
reply: Some(reply),
started,
};
diff --git a/crates/switchyard-py/src/libsy_bindings.rs b/crates/switchyard-py/src/libsy_bindings.rs
index 23c90d7ad..e0d86d365 100644
--- a/crates/switchyard-py/src/libsy_bindings.rs
+++ b/crates/switchyard-py/src/libsy_bindings.rs
@@ -328,6 +328,7 @@ impl PyTaskClassifierConfig {
) -> PyResult<Self> {
Ok(Self {
inner: TaskClassifierConfig {
+ fail_open: true,
base_threshold,
threshold_step,
classify_trigger: classify_trigger(session_affinity),
diff --git a/crates/switchyard-runner/src/algorithm.rs b/crates/switchyard-runner/src/algorithm.rs
index e742d48ba..acb2644c8 100644
--- a/crates/switchyard-runner/src/algorithm.rs
+++ b/crates/switchyard-runner/src/algorithm.rs
@@ -99,6 +99,7 @@ enum LlmClassifierModeConfig {
#[derive(Clone, Debug)]
struct CapabilityClassifierRouteConfig {
classifier_target: String,
+ fail_open: bool,
strong_target: String,
weak_target: String,
base_threshold: f64,
@@ -237,6 +238,9 @@ pub struct LlmClassifierRouteConfig {
/// Capability mode: how much to raise the threshold when the judge is
/// uncertain. Added once for an uncertain verdict and twice for unsupported.
pub threshold_step: Option<f64>,
+ /// Capability mode: routes to the capable tier on judge client failures and deadlines.
+ /// Defaults to true when capability mode is selected.
+ pub fail_open: Option<bool>,
/// How often the judge runs: every request, once per user turn, or once per session.
pub classify_trigger: ClassifyTrigger,
/// Reuses the session's target by hashing the first user message when no
@@ -522,6 +526,7 @@ pub struct StageTierConfig {
impl StageClassifierConfig {
fn task_classifier_config(&self) -> TaskClassifierConfig {
TaskClassifierConfig {
+ fail_open: true,
base_threshold: self.base_threshold,
threshold_step: self.threshold_step,
classify_trigger: self.classify_trigger,
@@ -878,6 +883,7 @@ impl LlmClassifierRouteConfig {
weak_target,
base_threshold,
threshold_step,
+ fail_open,
classify_trigger,
message_hash_fallback,
recent_turn_window,
@@ -897,6 +903,12 @@ impl LlmClassifierRouteConfig {
(None, false) => ClassifierMode::Capability,
};
+ if !matches!(selected_mode, ClassifierMode::Capability) && *fail_open == Some(true) {
+ return Err(AlgorithmConfigError::new(format!(
+ "llm_classifier route {route_name}: fail_open = true requires capability mode"
+ )));
+ }
+
match selected_mode {
ClassifierMode::Capability => {
if escalation.is_some() {
@@ -917,6 +929,7 @@ impl LlmClassifierRouteConfig {
Ok(LlmClassifierModeConfig::Capability(
CapabilityClassifierRouteConfig {
classifier_target: classifier_target.clone(),
+ fail_open: fail_open.unwrap_or(true),
strong_target: required_classifier_field(
route_name,
"strong_target",
@@ -1229,6 +1242,7 @@ fn build_algorithm(
let algorithm = match mode {
LlmClassifierModeConfig::Capability(config) => {
let classifier_config = TaskClassifierConfig {
+ fail_open: config.fail_open,
base_threshold: config.base_threshold,
threshold_step: config.threshold_step,
classify_trigger: config.classify_trigger,
diff --git a/crates/switchyard-server/tests/client_deadline.rs b/crates/switchyard-server/tests/client_deadline.rs
index 58bd06fc3..395f07d8a 100644
--- a/crates/switchyard-server/tests/client_deadline.rs
+++ b/crates/switchyard-server/tests/client_deadline.rs
@@ -139,6 +139,11 @@ fn app(
"type = \"llm_classifier\"\nclassifier_target = \"judge\"\nstrong_target = \"strong\"\nweak_target = \"weak\"\nbase_threshold = 0.5"
}
};
+ let route = if mode == "classifier" {
+ format!("{route}\nfail_open = false")
+ } else {
+ route.to_owned()
+ };
let mut config = tempfile::Builder::new().suffix(".toml").tempfile()?;
write!(
config,
@@ -233,6 +238,7 @@ async fn check(
async fn client_deadline_stops_routing_and_counts_attempts() -> TestResult {
let upstream = Upstream::start().await?;
let stalled = app(&upstream, "classifier", "stalled", "weak", 100)?;
+ let fail_open = app(&upstream, "classifier-default", "stalled", "weak", 100)?;
let streaming = app(&upstream, "terminal", "judge", "streaming", 500)?;
let slow_stream = app(&upstream, "terminal", "judge", "slow-body", 100)?;
for endpoint in [
@@ -245,6 +251,13 @@ async fn client_deadline_stops_routing_and_counts_attempts() -> TestResult {
check(&upstream, &stalled, endpoint, &["stalled"], [0., 0., 1.]).await?;
assert_eq!(status, StatusCode::GATEWAY_TIMEOUT, "{body}");
assert!(body.contains("100 ms"), "{body}");
+ let (calls, counts) = if endpoint == "/v1/decision" {
+ (&["stalled"][..], [0., 0., 1.])
+ } else {
+ (&["stalled", "strong"][..], [1., 0., 1.])
+ };
+ let (status, body) = check(&upstream, &fail_open, endpoint, calls, counts).await?;
+ assert_eq!(status, StatusCode::OK, "{body}");
if endpoint == "/v1/decision" {
continue;
}
diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rs
index 3c2656257..5236ff559 100644
--- a/crates/switchyard-server/tests/server.rs
+++ b/crates/switchyard-server/tests/server.rs
@@ -4977,7 +4977,7 @@ async fn advisor_route_redo_client_error_and_stats_projection() -> TestResult {
assert!(feedback.starts_with("A senior reviewer examined your work"));
assert!(feedback.ends_with("run the tests"));
- // A failed HTTP advisor call stops the request before the algorithm can approve it.
+ // Fail-open returns the buffered executor turn when the HTTP advisor call fails.
let response = send_with_headers(
&app,
"POST",
@@ -4986,8 +4986,8 @@ async fn advisor_route_redo_client_error_and_stats_projection() -> TestResult {
&[("proxy_x_session_id", "fail-flow")],
)
.await?;
- assert_eq!(response.status, StatusCode::SERVICE_UNAVAILABLE);
- assert_eq!(response.json()?["error"]["type"], "upstream_error");
+ assert_eq!(response.status, StatusCode::OK);
+ assert_eq!(response.json()?["choices"][0]["message"]["content"], "ok");
assert_eq!(
upstream.models().await,
[
@@ -5000,8 +5000,8 @@ async fn advisor_route_redo_client_error_and_stats_projection() -> TestResult {
);
let stats = send(&app, "GET", "/v1/stats", None).await?.json()?;
- // Only the successful redo request returns an executor answer.
- assert_eq!(stats["models"]["model/executor"]["calls"], 1);
+ // Both requests return an executor answer.
+ assert_eq!(stats["models"]["model/executor"]["calls"], 2);
assert_eq!(stats["classifier"]["total_errors"], 1);
// Projection deltas for the metrics only this test emits.
let redo = gate_count(&stats, &["reviews", "redo", "total"])
@@ -5028,10 +5028,9 @@ async fn advisor_route_redo_client_error_and_stats_projection() -> TestResult {
gate_count(&stats, &["discarded", "tokens", "output"]),
gate_count(&before, &["discarded", "tokens", "output"]) + 2
);
- // The host stops before the algorithm records a fail-open advisor decision.
assert_eq!(
gate_count(&stats, &["consult_failures", "upstream_5xx"]),
- gate_count(&before, &["consult_failures", "upstream_5xx"])
+ gate_count(&before, &["consult_failures", "upstream_5xx"]) + 1
);
// Reset re-baselines the projection: the redo/discard counts this test
diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.md
index 4b51a7918..bbd03af38 100644
--- a/docs/reference/toml_schema.md
+++ b/docs/reference/toml_schema.md
@@ -62,14 +62,18 @@ must exist and be non-empty when the server loads.
`timeout_ms` applies separately to every call through the client, including judge
verdicts and answers. To give a judge a short deadline without limiting the
answering models, put the judge on its own `[llm_clients]` entry; two entries may
-share a `base_url`. When the deadline expires, the server returns `504` without
-trying another target. If the final answer has already started streaming, the
+share a `base_url`. When an answer deadline expires, the server returns `504`.
+If the final answer has already started streaming, the
server sends a framed error and ends the stream without a success marker.
The Rust runner collects streams used during routing before the algorithm
continues, preserving provider events for replay. After the configured retries,
-an HTTP client failure stops routing. This also applies when `timeout_ms` is
-unset or an advisor has `fail_open = true`.
+an HTTP client failure stops routing unless the call enables recovery.
+Capability classifiers and advisor gates default to `fail_open = true`.
+A capability classifier routes to the capable tier on judge failures, including
+deadlines. An advisor returns the buffered executor turn when its consult fails.
+Set `fail_open = false` to stop the request on these failures.
+Answer calls keep their usual error behavior.
Set `forward_auth = true` to use each caller's credential instead of a
server-owned key:
@@ -239,6 +243,7 @@ Capability mode classifies before serving. See
| `strong_target` | Yes | — | Capable tier. |
| `weak_target` | Yes | — | Efficient tier. |
| `base_threshold` | Yes | — | Lowest solve probability that routes to the weak target. In `[0, 1]`. |
+| `fail_open` | No | `true` | Capability mode only. When `true`, judge client failures and deadlines route to the capable tier and record fail-open evidence. When `false`, they stop the request. Invalid verdicts use the capable tier with either setting. |
| `threshold_step` | No | `0.0` | Finite, non-negative amount added once for uncertain or unmatched verdicts and twice for unsupported verdicts. `base_threshold + 2 * threshold_step` must be at most `1`. |
| `classify_trigger` | No | `every_request` | When the judge runs. `every_request` judges every request, tool continuations included. `user_turn` judges each new user message and retains that target across intervening tool calls only when requests carry a session ID; without a session ID, it behaves like `every_request`. `new_session` judges once and reuses that target for the session. |
| `message_hash_fallback` | No | `false` | Retains the target against a hash of the first user message when a request carries no session ID. Requires `classify_trigger = "new_session"` or `"user_turn"`. |