Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs: correct stage routing guidance and qualify prefill router (#742)

ryan-lempka8d5af02994a6
diff --git a/crates/switchyard-server/README.md b/crates/switchyard-server/README.md
index c700c32ce..7e3dbb404 100644
--- a/crates/switchyard-server/README.md
+++ b/crates/switchyard-server/README.md
@@ -80,7 +80,9 @@ Each target references an entry under `llm_clients`. All configured clients use
`TranslatingLlmClient`; supported formats are `openai_chat`, `openai_responses`, and
`anthropic_messages`. Supported algorithms are `noop`, `random`, `passthrough`,
`llm_classifier`, and `stage_router`. The optional `prefill-router` feature also enables
-`prefill_router`. An `api_key_env` value names an environment variable; the TOML never contains the
+the experimental `prefill_router`. See its
+[artifact requirements](../../docs/reference/toml_schema.md#prefill_router).
+An `api_key_env` value names an environment variable. The TOML never contains the
secret itself. If omitted, the client sends no authentication.
A client can set `forward_auth = true` instead of `api_key_env` to send the
caller's credential to the configured upstream. OpenAI clients forward
diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.md
index d5c78c17e..cecb8b7af 100644
--- a/docs/reference/toml_schema.md
+++ b/docs/reference/toml_schema.md
@@ -172,6 +172,14 @@ Splits traffic across targets. See
### `prefill_router`
+!!! warning "Experimental in v0.3.0"
+
+ Switchyard v0.3.0 does not provide or support a router checkpoint, an
+ exporter, or compatible encoder assets. You must obtain or train a compatible
+ checkpoint and obtain its encoder and tokenizer yourself. There is no
+ supported end-to-end checkpoint export and compatibility contract.
+ The example below shows configuration syntax, not a ready-to-run deployment.
+
Routes the latest non-empty user message with a checkpoint-backed prefill classifier. Build
`switchyard-server` with `--features prefill-router` and make the prefill router's Python
dependencies available in the active virtual environment.
diff --git a/docs/routing_algorithms/prefill_routing.md b/docs/routing_algorithms/prefill_routing.md
index 9cd7f1a63..4a67374cd 100644
--- a/docs/routing_algorithms/prefill_routing.md
+++ b/docs/routing_algorithms/prefill_routing.md
@@ -1,3 +1,9 @@
+> **Experimental in v0.3.0.** Switchyard does not provide or support a router
+> checkpoint, an exporter, or compatible encoder assets. You must obtain or train
+> a compatible checkpoint and obtain its encoder and tokenizer yourself. There is
+> no supported end-to-end checkpoint export and compatibility contract. The steps
+> below describe a research workflow, not a ready-to-run deployment.
+
Prefill router runs the request through `pytorch` and `transformers` to get hidden states. It uses those states as input to a classifier which you have to train. For each model the classifier predicts if it will succeed at the task. This is a variation on using embeddings as input to the classifier.
This makes it different from the other routers in that it both requires running inference (with `transformers`) and training a classifier for the specific models you intend to use and the specific workload you will send it. This makes it much more complex to deploy than the other routers, but potentially better if you know your workload.
@@ -12,4 +18,3 @@ This makes it different from the other routers in that it both requires running
There is some documentation for an older version here:
- [Collect training data](https://github.com/NVIDIA-AI-Blueprints/llm-router/tree/v3#collect-training-data)
- [Train a router](https://github.com/NVIDIA-AI-Blueprints/llm-router/tree/v3#train-a-router)
-
diff --git a/docs/routing_algorithms/stage_router_routing.md b/docs/routing_algorithms/stage_router_routing.md
index f9f600af1..362b7c540 100644
--- a/docs/routing_algorithms/stage_router_routing.md
+++ b/docs/routing_algorithms/stage_router_routing.md
@@ -144,32 +144,34 @@ Stratify the pure-capable results across four quadrant candidates before running
| Category | Criterion | Count | Value |
|---|---|---|---|
| Easy + clean | Capable passes, small diff, clear spec | ~5 | Establishes SAFE floor |
-| Easy + tricky | Capable passes, subtle logic | ~5 | Catches LOSS false-positives |
+| Easy + tricky | Capable passes, subtle logic | ~5 | Finds RESCUE candidates |
| Hard + structural | Capable fails, large multi-file diff | ~5 | HARD noise baseline |
-| Hard + localized | Capable fails, small targeted fix | ~5 | Best RESCUE signal |
+| Hard + localized | Capable fails, small targeted fix | ~5 | Finds LOSS candidates |
Sample across repos and diff sizes. Don't over-represent one project.
**Building RESCUE / LOSS quadrants**
-From the overlap tasks (those with both capable and efficient results):
+For efficient-to-capable escalation, group the overlap tasks (those with both
+capable and efficient results):
-- `RESCUE` = capable-fail ∩ efficient-pass → escalation is beneficial here
-- `LOSS` = capable-pass ∩ efficient-fail → do NOT escalate here
+- `RESCUE` = efficient-fail ∩ capable-pass, a candidate for beneficial escalation
+- `LOSS` = efficient-pass ∩ capable-fail, a candidate for harmful escalation
- `SAFE` = both pass
- `HARD` = both fail
-Sweep a few candidate thresholds in representative benchmark runs. Choose the
-lowest threshold that rescues the RESCUE quadrant without over-escalating the
-LOSS quadrant. Because the scorer is corroborative, a `0.5` threshold takes
+Sweep a few candidate thresholds in routed benchmark runs. For
+`efficient_first`, choose the lowest threshold that improves outcomes on RESCUE
+tasks without turning passing LOSS tasks into failures. Because the scorer is
+corroborative, a `0.5` threshold takes
roughly 1.5 signals of agreement.
-**Caveat on efficient outcomes in stage-router vs. pure-efficient**
+**Caveat on fixed-model outcomes vs. routed outcomes**
-In stage-router, the efficient model may inherit partial context from the capable arm
-(conversation history up to the escalation point). Pure-efficient runs start
-fresh, so RESCUE is a conservative lower bound. Efficient performs at least as
-well in stage-router as it does alone.
+After a switch, the selected model inherits conversation history from the other
+model. Fixed-model runs do not measure this effect. Inherited context may help
+or hurt, so these quadrants identify candidates, not guaranteed outcomes.
+Validate the threshold with routed runs on the same tasks.
## Route configuration