Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

chore(docs): llm classifier reasoning note (#286)

Ayush Agarwal6e63ad9d569d
diff --git a/crates/switchyard-server/CONFIGURATION.md b/crates/switchyard-server/CONFIGURATION.md
index 44ed47d95..efd438109 100644
--- a/crates/switchyard-server/CONFIGURATION.md
+++ b/crates/switchyard-server/CONFIGURATION.md
@@ -20,6 +20,11 @@ extra_body = { chat_template_kwargs = { enable_thinking = false } }
`extra_body` is target-specific. It shallow-merges top-level provider options into
the outbound request, while explicit request fields win on conflicts.
+The `chat_template_kwargs.enable_thinking` example is a provider/model-specific
+vLLM option. It is not a portable Switchyard reasoning switch. Use it on a judge
+target only when the upstream supports it and would otherwise return the verdict
+outside normal assistant `content`.
+
To support another wire format, add its `ClientFormat` variant and explicit construction match in
`src/config.rs`. Add a client type only when a second implementation exists.
diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.md
index 089d313d8..ff94db027 100644
--- a/docs/routing_algorithms/escalation_router_routing.md
+++ b/docs/routing_algorithms/escalation_router_routing.md
@@ -88,6 +88,12 @@ A judge that times out, errors, or returns an unparseable verdict fails open: th
turn serves the buffered weak reply and the existing streak is held rather than
cleared. A judge failure never creates a strong-tier latch.
+## Judge model compatibility
+
+The trajectory judge uses the same response contract and provider/model
+compatibility guidance as the LLM classifier judge. See
+[Judge model compatibility](llm_classifier_routing.md#judge-model-compatibility).
+
## Tuning options
The judge exposes three settings. Their defaults are the benchmarked
diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.md
index e44f9a372..ff9c8e863 100644
--- a/docs/routing_algorithms/llm_classifier_routing.md
+++ b/docs/routing_algorithms/llm_classifier_routing.md
@@ -69,6 +69,30 @@ An abstention, an invalid or unparseable verdict, a judge failure, or confidence
below `min_confidence` routes to `strong_target`. Raising either threshold sends
more traffic to the strong model.
+## Judge model compatibility
+
+The judge must return complete, schema-valid JSON in normal assistant `content`.
+Switchyard does not parse provider-specific reasoning fields such as
+`reasoning_content`. If `content` is empty or unparseable, the route falls back
+to `strong_target` even when the judge request returned HTTP 200. With session
+affinity, that fallback can be reused without another judge call.
+
+When a vLLM-compatible provider supports `enable_thinking`, configure it on the
+judge target through `extra_body`:
+
+```toml
+[targets.classifier]
+extra_body = { chat_template_kwargs = { enable_thinking = false } }
+```
+
+`enable_thinking` is a provider-specific vLLM option, not a general requirement
+for reasoning models. Other model/provider pairs may work with reasoning enabled
+or use a different control. Verify the judge response shape before deployment.
+If reasoning remains enabled, set `max_output_tokens` high enough for both the
+reasoning and final JSON. A truncated verdict has the same fail-open result. See
+the [target-level `extra_body` reference](../../crates/switchyard-server/CONFIGURATION.md#add-an-llm-client-and-target)
+for the server merge behavior.
+
## Tuning options
| Key | Default | Meaning |