Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Switchyard NeMo Relay Plugin
switchyard-nemo-relay-plugin is a native NeMo Relay dynamic plugin. It loads
a standard Switchyard TOML deployment from a file or Relay's nested plugin
configuration and executes its configured routes through switchyard-runner.
The plugin does not define a second routing or target configuration language.
switchyard-server and Relay therefore use the same targets, client pooling,
algorithm construction, retry policy, and route validation.
Install
The plugin uses nemo-relay-plugin 0.10.0 and requires NeMo Relay
>=0.10.0, <1.0.0. Relay 0.10 changes the native callback layout, so older
hosts cannot load this build. The manifest's native plugin API label stays 1.
Relay 0.8.x and 0.9.0 can lose upstream error status and details when this native plugin is enabled, including for models outside its configured routes. This is a known issue on those older hosts. This build requires Relay 0.10 or later. See the upstream error compatibility note.
Install a released bundle
Official plugin bundles are distributed through the
NeMo Relay Plugins repository.
Each bundle includes the native library, a completed relay-plugin.toml, the
configuration schema, and license notices. Installing a bundle requires no
Rust build or separate switchyard-nemo-relay-plugin crate installation.
The repository currently requires NVIDIA GitHub repository access. Sign in
with an account that has access and complete organization SSO/SAML authorization
where required, including for credentials used by gh. An HTTP 404 can mean
that your account or credentials lack access; it does not prove that a bundle
is missing.
- Select a published
switchyard-plugin-<version>release for Switchyard0.3.0. Plugin versions are managed separately and can match the Switchyard version. Confirm the source commit in the release notes and.jsonmetadata rather than relying on the version number alone. A Switchyard release does not itself publish a plugin bundle. If no matching published bundle is available, use Build from source. - Download the archive for your platform and its matching
.sha256and.jsonsidecars. For Linux x86_64, the archive is namedswitchyard-plugin-<version>-linux-x86_64.tar.gz. Check the metadata'ssource_commit,platform, andrelayfields for the source and tested host. - Verify the downloaded archive against its
.sha256file before extracting it. On Linux, runsha256sum -c <archive>.sha256from the download directory, replacing<archive>with the archive's filename. - Extract the archive into a directory you plan to keep, then follow
Register and enable the plugin. Use the
actual path to the extracted
relay-plugin.toml; the examples below use./plugins/switchyard/relay-plugin.toml.
Build from source
Build from source when you need a Switchyard commit that does not have a released bundle or when you need to customize the plugin. This path requires a Rust toolchain and Python 3 for the packaging script. Run every command from the repository root.
1. Build the shared library.
cargo build --release -p switchyard-nemo-relay-plugin
The artifact is target/release/libswitchyard_nemo_relay_plugin.so on Linux
and target/release/libswitchyard_nemo_relay_plugin.dylib on macOS.
2. Package the bundle. The script copies the library, the config schema,
and license files into an empty directory and writes a relay-plugin.toml
manifest with the library name and its SHA-256 digest filled in. Relay verifies
that digest before it loads the library, so rebuild the bundle after every
rebuild of the library.
Linux:
python crates/switchyard-nemo-relay-plugin/scripts/package_bundle.py \
--library target/release/libswitchyard_nemo_relay_plugin.so \
--output ./plugins/switchyard
macOS:
python crates/switchyard-nemo-relay-plugin/scripts/package_bundle.py \
--library target/release/libswitchyard_nemo_relay_plugin.dylib \
--output ./plugins/switchyard
Pass --archive switchyard-plugin.tar.gz (or .zip) to also produce an
archive for distribution.
Register and enable the plugin
These steps apply to both a downloaded bundle and a bundle built from source.
1. Register the plugin.
nemo-relay plugins validate ./plugins/switchyard/relay-plugin.toml
nemo-relay plugins add --user ./plugins/switchyard/relay-plugin.toml
add writes a [[plugins.dynamic]] entry to your user plugins.toml
(~/.config/nemo-relay/plugins.toml or $XDG_CONFIG_HOME/nemo-relay/plugins.toml).
The plugin is not enabled yet; enabling before the deployment is configured
fails validation because the plugin requires a Switchyard configuration.
2. Configure the deployment and trust policy in that plugins.toml, as
described in Configure Relay.
3. Enable and validate the plugin, then restart Relay. The manifest ships
with enabled = false; Relay validates a disabled plugin but never loads it.
nemo-relay plugins enable nvidia.switchyard
nemo-relay plugins validate nvidia.switchyard
validate evaluates the manifest, the plugin configuration, the artifact
digest, and the host trust policy.
Configure Relay
Add a config table to the [[plugins.dynamic]] entry that plugins add
wrote, and a policy override for the plugin. Use exactly one Switchyard
deployment source. To share an existing deployment file with
switchyard-server, configure its path:
[[plugins.dynamic]]
manifest = "./plugins/switchyard/relay-plugin.toml"
[plugins.dynamic.config]
priority = 0
switchyard_config_path = "/etc/switchyard/routes.toml"
[plugins.policy.overrides."nvidia.switchyard"]
attestation = "integrity_only"
The policy override is required. The generated manifest carries a SHA-256
digest but no signature, and Relay refuses to activate an unsigned dynamic
plugin at gateway start unless its host policy says otherwise; plugins validate still passes without the override, so the failure only shows up at
startup as requires integrity.signature under host policy. Native plugins
run inside the Relay process without a sandbox, so only install a bundle you
built or obtained from a source you trust. To require a signature instead, sign
the artifact with an Ed25519 key and list it in trusted_public_keys; see
Relay's
discoverable plugins guide
for the policy keys.
switchyard_config_path is a Switchyard version-1 TOML deployment, accepted by both
switchyard-server and switchyard-runner. See the
TOML schema reference for every key and
the routing overview for the
algorithms.
To keep the deployment in the Relay configuration, nest the same version-1
Switchyard configuration under switchyard_config:
[[plugins.dynamic]]
manifest = "./plugins/switchyard/relay-plugin.toml"
[plugins.dynamic.config]
priority = 0
[plugins.dynamic.config.switchyard_config]
schema_version = 1
[plugins.dynamic.config.switchyard_config.llm_clients.primary]
format = "openai_chat"
base_url = "https://example.test/v1"
[plugins.dynamic.config.switchyard_config.targets.default]
id = "example/model"
llm_client = "primary"
[plugins.dynamic.config.switchyard_config.routes.default]
id = "switchyard/default"
type = "passthrough"
target = "default"
Request handling
For OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages calls, the plugin decodes the Relay request and checks the requested model against the deployment's route IDs.
- A configured route is executed by
switchyard-runner. - An unknown model calls Relay's continuation unchanged.
- The returned provider response is encoded back into the caller's wire format.
- Streaming responses are returned as unpolled translated streams; Relay owns cancellation and the outer serving-call lifecycle.
The caller and selected target may use different supported API formats:
openai_chat, openai_responses, or anthropic_messages. Switchyard translates
the request into the selected target's configured format and returns buffered
or streaming responses in the caller's original format. With server-owned
credentials, one route targeting an openai_chat client can serve all three
caller formats. Separate targets and routes are not required solely for format
translation.
The native Relay plugin rejects routes that use forward_auth = true during
configuration validation and activation. This includes routing-model calls and
alternate targets. Relay does not provide caller credentials to the plugin's
provider calls. Secure forwarding support is tracked in
NeMo Relay #1108.
For deployment-owned credentials, remove forward_auth or set it to false
and configure api_key_env on each authenticated client. The two options cannot
be enabled together. If each caller must use its own provider credential, use
standalone switchyard-server. Standalone forwarding requires the caller and
target to use the same credential family: OpenAI-compatible (Chat Completions
and Responses) or Anthropic (Messages). A route may mix both families when all
of its forwarding clients use the same scheme, host, and port, such as one LLM
gateway; such a route serves Chat Completions and Responses callers.
Support for provider-specific fields depends on the source and target formats. Test any fields that your application relies on before deploying a translated route. See the integration guide for request handling, header forwarding, and streaming details.
The plugin emits routing request, model-call, measured-overhead, and decision marks. Call marks distinguish routing from answer calls; decisions distinguish selected from served models. Token metrics cover both call roles, while Relay retains ownership of the outer LLM lifecycle.
The plugin also captures the existing libsy.client_call,
libsy.upstream_attempt, and libsy.decision_client_call spans for each routed
request. It replays completed
spans beneath Relay's outer LLM scope with their original timestamps. The
projection contains allowlisted routing metadata by default. To inspect the
final classifier/judge or typed decision request and buffered response, enable
routing payload capture
in the plugin configuration. Capture uses Relay's existing exporters and writes
no payload files inside the plugin. It is disabled by default and omits bodies
above the configured size limit.
Observability
When Relay is configured with OTLP logs and metrics exporters, the plugin emits typed telemetry through Relay's native plugin runtime:
- Routing request, decision, and overhead marks are Info logs.
- Per-model call marks are Debug logs with
call_role, outcome, and latency, but no token usage. Streaming marks cover stream creation; later failures are reported separately. - Terminal routing and response-finalization failures are Error logs. Their payload contains only the safe Switchyard failure summary; it excludes provider response bodies and free-form provider messages.
- Metrics use bounded attributes only: algorithm for
switchyard.routing.requests; outcome forswitchyard.routing.llm_callsandswitchyard.routing.llm_call.duration; and safe failure kind, category, phase, and optional upstream HTTP status forswitchyard.routing.failures.switchyard.routing.overheadrecords total routing latency, including routing-model calls; durations use milliseconds. switchyard.routing.llm_tokensrecords normalized token usage withcall_role(routingoranswer), configuredtarget_model, andtoken_typeattributes. A provider may omit usage for streaming responses; the plugin does not synthesize zero-value measurements.
The plugin does not attach sessions, requests, or provider messages as metric
attributes. target_model comes from the configured Switchyard target set,
rather than arbitrary caller input, keeping the metric cardinality bounded by
the deployment.
Every non-metric mark sets data_schema.name to the mark name and
data_schema.version to 1. Consumers should tolerate unknown fields and
values. Removing or renaming fields, changing their type, or changing their
meaning requires a new schema version.
| Mark | Data fields |
|---|---|
switchyard.routing.requested |
algorithm |
switchyard.routing.llm_call |
call_index, selected_model, call_role, outcome, latency_ms |
switchyard.routing.overhead |
latency_ms |
switchyard.routing.decision |
algorithm, optional outcome_id, selected_model, nullable served_model, nullable fallback_used, and optional evidence |
switchyard.routing.error |
failure_kind; route-execution failures also include category, phase, nullable upstream_status and target, and may include outcome_id and evidence |
served_model and fallback_used are null when serving metadata is unavailable.
outcome_id is present when the algorithm runner supplies outcome metadata.
evidence is an object containing the supported string fields source, verdict,
trigger, and reason_code, and numeric fields score, confidence, threshold,
capable_score, and capable_threshold.
String values longer than 64 bytes are omitted and should be stable, non-sensitive labels.
Failure policy
Provider credential redaction
The plugin replaces configured api_key_env credentials with [REDACTED] in
buffered responses and each translated stream event before returning them to
Relay. It also redacts returned error strings, routing mark data and metadata,
metric attributes and metadata, and plugin telemetry-emission diagnostics.
The intended upstream still receives the original credential. Upstream response
headers are not returned through the plugin's JSON execution intercepts.
The plugin and standalone server share the credential replacement helpers. Redaction runs after translation, including on preserved provider fields and JSON member names. It does not buffer the response stream or change cancellation. Each event is handled independently: a credential split across events or separate JSON strings is not reconstructed or redacted as a whole. String values and error text are checked for both raw credentials and their JSON-escaped forms, including one embedded JSON serialization layer such as serialized tool arguments. Further repeated escaping and other encodings or transformations are outside this policy. Preventing reconstruction across stream events requires a separate stateful policy for each logical text or tool argument field; per-event redaction does not provide that guarantee.
Execution failures
switchyard-llm-client owns provider retry and route-candidate fallback
behavior. The plugin does not maintain a separate trusted-default target or
rerun routing after an execution failure. Failures outside the shared runner,
including response translation failures, are returned to Relay.