Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(relay): document known upstream error propagation issue (#761)
Bryan Bednarski082e68ea1456
diff --git a/crates/switchyard-nemo-relay-plugin/README.md b/crates/switchyard-nemo-relay-plugin/README.mdindex d889afada..0bf2103ad 100644--- a/crates/switchyard-nemo-relay-plugin/README.md+++ b/crates/switchyard-nemo-relay-plugin/README.md@@ -12,6 +12,12 @@ algorithm construction, retry policy, and route validation.The plugin requires NeMo Relay `>=0.8.0, <1.0.0`.+Relay 0.8.x and 0.9.0 can lose upstream error status and details when this native+plugin is enabled, including for models outside its configured routes. This is+a known issue tracked by [NeMo Relay PR #1109](https://github.com/NVIDIA/NeMo-Relay/pull/1109).+Until a fix is available, isolate unmanaged traffic in a plugin-disabled gateway.+See the [upstream error compatibility note](../../docs/integrations/nemo_relay.md#upstream-error-compatibility).+### Install a released bundleOfficial plugin bundles are distributed through thediff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.mdindex e451466b7..31ce7a640 100644--- a/docs/integrations/nemo_relay.md+++ b/docs/integrations/nemo_relay.md@@ -6,6 +6,26 @@ loads Switchyard into an existingdeployment through Relay's[native dynamic plugin system](https://docs.nvidia.com/nemo/relay/build-plugins/native/about).+## Upstream Error Compatibility++Relay 0.8.x and 0.9.0 have a native-plugin error propagation issue. Enabling the+Switchyard plugin can change an upstream 401 or 403 into a generic 400 for a+non-streaming request. A streaming request can receive HTTP 200 followed by an+aborted body. This is not a successful response or an authentication bypass.++This also affects **unmanaged models**: requested model names that do not match+a configured Switchyard route. For example, if the route is `switchyard/core`+and its target is `azure/openai/gpt-5.5`, requesting the target name directly+still delegates the request to Relay.++This is a known issue. A proposed correction is tracked in+[NeMo Relay PR #1109](https://github.com/NVIDIA/NeMo-Relay/pull/1109).+The correction has not been released. Until a fix is available, send unmanaged+traffic through a separate Relay instance with the plugin disabled.+Do not assume that upgrading Switchyard alone fixes this.+The plugin's `>=0.8.0, <1.0.0` compatibility range describes which hosts can load+it; it does not mean those older hosts preserve upstream errors correctly.+## Why Use Switchyard with NeMo Relay?Switchyard's routing algorithms select a model for each LLM request or step in@@ -26,8 +46,9 @@ latency, and token use through Relay telemetry. Use it to answer:The plugin runs inside Relay, so the agent does not need to change andSwitchyard does not need to run as a separate service.-Requests for models that Switchyard does not manage continue through Relay as-usual.+Requests for models that Switchyard does not manage are passed to Relay.+See the [upstream error compatibility note](#upstream-error-compatibility) before+using older Relay versions.Routing does not require Relay. You can instead run the[standalone server](../getting_started.md#server-path) or embed