Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs(relay): document known upstream error propagation issue (#761)

Bryan Bednarski082e68ea1456
diff --git a/crates/switchyard-nemo-relay-plugin/README.md b/crates/switchyard-nemo-relay-plugin/README.md
index d889afada..0bf2103ad 100644
--- a/crates/switchyard-nemo-relay-plugin/README.md
+++ b/crates/switchyard-nemo-relay-plugin/README.md
@@ -12,6 +12,12 @@ algorithm construction, retry policy, and route validation.
The plugin requires NeMo Relay `>=0.8.0, <1.0.0`.
+Relay 0.8.x and 0.9.0 can lose upstream error status and details when this native
+plugin is enabled, including for models outside its configured routes. This is
+a known issue tracked by [NeMo Relay PR #1109](https://github.com/NVIDIA/NeMo-Relay/pull/1109).
+Until a fix is available, isolate unmanaged traffic in a plugin-disabled gateway.
+See the [upstream error compatibility note](../../docs/integrations/nemo_relay.md#upstream-error-compatibility).
+
### Install a released bundle
Official plugin bundles are distributed through the
diff --git a/docs/integrations/nemo_relay.md b/docs/integrations/nemo_relay.md
index e451466b7..31ce7a640 100644
--- a/docs/integrations/nemo_relay.md
+++ b/docs/integrations/nemo_relay.md
@@ -6,6 +6,26 @@ loads Switchyard into an existing
deployment through Relay's
[native dynamic plugin system](https://docs.nvidia.com/nemo/relay/build-plugins/native/about).
+## Upstream Error Compatibility
+
+Relay 0.8.x and 0.9.0 have a native-plugin error propagation issue. Enabling the
+Switchyard plugin can change an upstream 401 or 403 into a generic 400 for a
+non-streaming request. A streaming request can receive HTTP 200 followed by an
+aborted body. This is not a successful response or an authentication bypass.
+
+This also affects **unmanaged models**: requested model names that do not match
+a configured Switchyard route. For example, if the route is `switchyard/core`
+and its target is `azure/openai/gpt-5.5`, requesting the target name directly
+still delegates the request to Relay.
+
+This is a known issue. A proposed correction is tracked in
+[NeMo Relay PR #1109](https://github.com/NVIDIA/NeMo-Relay/pull/1109).
+The correction has not been released. Until a fix is available, send unmanaged
+traffic through a separate Relay instance with the plugin disabled.
+Do not assume that upgrading Switchyard alone fixes this.
+The plugin's `>=0.8.0, <1.0.0` compatibility range describes which hosts can load
+it; it does not mean those older hosts preserve upstream errors correctly.
+
## Why Use Switchyard with NeMo Relay?
Switchyard's routing algorithms select a model for each LLM request or step in
@@ -26,8 +46,9 @@ latency, and token use through Relay telemetry. Use it to answer:
The plugin runs inside Relay, so the agent does not need to change and
Switchyard does not need to run as a separate service.
-Requests for models that Switchyard does not manage continue through Relay as
-usual.
+Requests for models that Switchyard does not manage are passed to Relay.
+See the [upstream error compatibility note](#upstream-error-compatibility) before
+using older Relay versions.
Routing does not require Relay. You can instead run the
[standalone server](../getting_started.md#server-path) or embed