Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
chore(docs): llm classifier reasoning note (#286)
Ayush Agarwal6e63ad9d569d
diff --git a/crates/switchyard-server/CONFIGURATION.md b/crates/switchyard-server/CONFIGURATION.mdindex 44ed47d95..efd438109 100644--- a/crates/switchyard-server/CONFIGURATION.md+++ b/crates/switchyard-server/CONFIGURATION.md@@ -20,6 +20,11 @@ extra_body = { chat_template_kwargs = { enable_thinking = false } }`extra_body` is target-specific. It shallow-merges top-level provider options intothe outbound request, while explicit request fields win on conflicts.+The `chat_template_kwargs.enable_thinking` example is a provider/model-specific+vLLM option. It is not a portable Switchyard reasoning switch. Use it on a judge+target only when the upstream supports it and would otherwise return the verdict+outside normal assistant `content`.+To support another wire format, add its `ClientFormat` variant and explicit construction match in`src/config.rs`. Add a client type only when a second implementation exists.diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.mdindex 089d313d8..ff94db027 100644--- a/docs/routing_algorithms/escalation_router_routing.md+++ b/docs/routing_algorithms/escalation_router_routing.md@@ -88,6 +88,12 @@ A judge that times out, errors, or returns an unparseable verdict fails open: thturn serves the buffered weak reply and the existing streak is held rather thancleared. A judge failure never creates a strong-tier latch.+## Judge model compatibility++The trajectory judge uses the same response contract and provider/model+compatibility guidance as the LLM classifier judge. See+[Judge model compatibility](llm_classifier_routing.md#judge-model-compatibility).+## Tuning optionsThe judge exposes three settings. Their defaults are the benchmarkeddiff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.mdindex e44f9a372..ff9c8e863 100644--- a/docs/routing_algorithms/llm_classifier_routing.md+++ b/docs/routing_algorithms/llm_classifier_routing.md@@ -69,6 +69,30 @@ An abstention, an invalid or unparseable verdict, a judge failure, or confidencebelow `min_confidence` routes to `strong_target`. Raising either threshold sendsmore traffic to the strong model.+## Judge model compatibility++The judge must return complete, schema-valid JSON in normal assistant `content`.+Switchyard does not parse provider-specific reasoning fields such as+`reasoning_content`. If `content` is empty or unparseable, the route falls back+to `strong_target` even when the judge request returned HTTP 200. With session+affinity, that fallback can be reused without another judge call.++When a vLLM-compatible provider supports `enable_thinking`, configure it on the+judge target through `extra_body`:++```toml+[targets.classifier]+extra_body = { chat_template_kwargs = { enable_thinking = false } }+```++`enable_thinking` is a provider-specific vLLM option, not a general requirement+for reasoning models. Other model/provider pairs may work with reasoning enabled+or use a different control. Verify the judge response shape before deployment.+If reasoning remains enabled, set `max_output_tokens` high enough for both the+reasoning and final JSON. A truncated verdict has the same fail-open result. See+the [target-level `extra_body` reference](../../crates/switchyard-server/CONFIGURATION.md#add-an-llm-client-and-target)+for the server merge behavior.+## Tuning options| Key | Default | Meaning |