Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

feat(node): expose configured algorithm execution (#1235)

Alex Fournier6b9acf33ca57
diff --git a/Cargo.lock b/Cargo.lock
index 32419e042..f9411ca10 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -2606,6 +2606,7 @@ name = "switchyard-node"
version = "0.3.0"
dependencies = [
"futures",
+ "http",
"napi",
"napi-build",
"napi-derive",
@@ -2614,7 +2615,9 @@ dependencies = [
"serde_json",
"switchyard-libsy",
"switchyard-protocol",
+ "switchyard-runner",
"switchyard-translation",
+ "tokio",
]
[[package]]
diff --git a/bindings/typescript/README.md b/bindings/typescript/README.md
index 793327e39..9886ece79 100644
--- a/bindings/typescript/README.md
+++ b/bindings/typescript/README.md
@@ -1,13 +1,9 @@
# Switchyard libsy for Node.js
-TypeScript bindings for composite routing and buffered JSON translation.
-Composite routing uses the caller's decision response and local tool signals to
-select a model. The caller handles classifier calls, completion requests, and
-branch state.
-
-Capable-first routing defaults to pinning the efficient tier after switching to
-it. Set `stage.efficient_hold_turns: 0` to disable this hold. Efficient-first
-defaults to no efficient hold. Persist the returned state to retain holds.
+TypeScript bindings for routing algorithms and buffered JSON translation.
+The host handles model and decision calls. `AlgorithmFactory` builds configured
+algorithms from deployment TOML; `CompositeRouter` provides the smaller composite
+routing API.
Requires Node.js 22.19 or newer and the repository's Rust toolchain. The package
is private. Its native addon is built for the local host. Linux is covered in CI.
@@ -24,8 +20,8 @@ npm --prefix bindings/typescript test
```
The build uses Cargo's development profile and copies the addon and license files
-into the package. Rebuild after changing Rust code. Tests use local decision
-callbacks.
+into the package. Rebuild after changing Rust code. Tests use local callbacks and
+HTTP fixtures.
Install the built package in a local Node project:
@@ -33,7 +29,71 @@ Install the built package in a local Node project:
npm install /absolute/path/to/Switchyard/bindings/typescript
```
-## API
+## Configured algorithms
+
+```ts
+import { AlgorithmFactory, LlmResponse, Step } from "@switchyard/libsy";
+
+const routes = AlgorithmFactory.fromToml(source).build();
+const steps = routes["public/main"].runStream(request, { headers });
+```
+
+`source` is deployment TOML and `request` is a normalized Switchyard request.
+The factory validates configuration without resolving credentials or creating
+provider clients. Reuse configured algorithms to retain algorithm-owned state.
+Configurations that require target eligibility checks must use Runner admission;
+the factory rejects them.
+
+The iterator yields Python-compatible routing steps:
+
+- `Step.CallModel`: serve `call.request` using `call.models[0]`, then pass a Promise
+ of `LlmResponse.Agg` or `LlmResponse.Stream` to `call.respond`.
+- `Step.CallDecision`: serve `call.request` using `call.model`, then pass a
+ Promise of `DecisionResponse` to `call.respond`.
+- `Step.Done`: read the selected models, rewritten requests, optional response,
+ and metadata from `step.outcome`.
+
+For routing-time model calls, use only the first candidate and pass failures to
+`call.respond` or `call.fail`. The reply has no field for the actual served logical
+target. When `Step.Done` has no response, the host may try `outcome.selected_model_ids`
+in order to serve the final answer, applying the outcome's requests.
+
+Model calls and selected models use logical target names from the TOML.
+Decision calls use configured provider IDs. The host owns provider settings,
+target prompts, transport, and final completion calls when the outcome has no
+response. Responses and streams use normalized protocol JSON, not provider wire
+formats or SSE bytes. `ContextWindowExceededError` reports a typed context-window
+failure to the algorithm.
+
+Calls can overlap. Keep reading steps while serving concurrent calls, and handle
+rejections from every `respond` or `fail` Promise. A non-recovering call rejects
+its responder; ignoring that error loses the original failure.
+
+Returning the routing iterator stops Rust from waiting on outstanding calls.
+JavaScript cannot cancel a supplied Promise. For buffered calls, use a host-owned
+controller; here `callProvider` returns the response expected by `call.respond`:
+
+```ts
+const controller = new AbortController();
+const work = callProvider(call.request, { signal: controller.signal });
+try {
+ await call.respond(work);
+} finally {
+ controller.abort();
+ await work.catch(() => {});
+}
+```
+
+The provider client must honor the signal and have a request timeout. This also
+cleans up work when the algorithm stops waiting before the response arrives.
+For `LlmResponse.Stream`, `respond` finishes when the stream is delivered. Keep
+its HTTP request alive until stream completion or cancellation.
+
+## CompositeRouter
+
+Capable-first routing defaults to pinning the efficient tier after switching to
+it. Set `stage.efficient_hold_turns: 0` to disable this hold. Efficient-first
+defaults to no efficient hold. Persist the returned state to retain holds.
```ts
import { CompositeRouter, type DecisionRequest, type DecisionResponse } from "@switchyard/libsy";
@@ -92,7 +152,7 @@ Treat `CompositeState` as opaque. It carries the judge's tier and stage holds.
| Code | Meaning |
| --- | --- |
-| `ERR_CONFIG` | Invalid composite options |
+| `ERR_CONFIG` | Invalid options or deployment configuration |
| `ERR_INVALID_REQUEST` | Invalid messages, state, or decision response JSON |
| `ERR_ROUTING` | The algorithm or classifier call failed |
| `ABORT_ERR` | Routing was cancelled; the error name is `AbortError` |
diff --git a/bindings/typescript/src/algorithm.ts b/bindings/typescript/src/algorithm.ts
new file mode 100644
index 000000000..2c2ba45ec
--- /dev/null
+++ b/bindings/typescript/src/algorithm.ts
@@ -0,0 +1,288 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+import { createRequire } from "node:module";
+import type { DecisionRequest, DecisionResponse } from "./index.js";
+
+type JsonObject = Record<string, unknown>;
+
+export class LibsyError extends Error {
+ readonly code = "ERR_ROUTING";
+ override name = "LibsyError";
+}
+
+export class ContextWindowExceededError extends Error {
+ override name = "ContextWindowExceededError";
+}
+
+export namespace LlmResponse {
+ export class Agg {
+ constructor(readonly response: JsonObject) {}
+ }
+
+ export class Stream {
+ constructor(readonly stream: AsyncIterable<JsonObject>) {}
+ }
+}
+
+export interface ModelCall {
+ readonly algorithm: string;
+ readonly request: JsonObject;
+ readonly models: readonly string[];
+ respond(work: PromiseLike<LlmResponse.Agg | LlmResponse.Stream>): Promise<void>;
+ fail(error: Error): Promise<void>;
+}
+
+export interface DecisionCall {
+ readonly algorithm: string;
+ readonly request: DecisionRequest;
+ readonly model: string;
+ respond(work: PromiseLike<DecisionResponse>): Promise<void>;
+ fail(error: Error): Promise<void>;
+}
+
+export interface OutcomeMetadata {
+ readonly outcome_id: string;
+ readonly algorithm: string;
+ readonly evidence: unknown | null;
+}
+
+export interface RoutingOutcome {
+ readonly selected_model_ids: readonly string[];
+ readonly request: JsonObject;
+ readonly fallback_request: JsonObject | null;
+ readonly candidate_requests: Readonly<Record<string, JsonObject>>;
+ readonly response: LlmResponse.Agg | LlmResponse.Stream | null;
+ readonly metadata: OutcomeMetadata | null;
+}
+
+export namespace Step {
+ export class CallModel {
+ constructor(readonly call: ModelCall) {}
+ }
+
+ export class CallDecision {
+ constructor(readonly call: DecisionCall) {}
+ }
+
+ export class Done {
+ constructor(readonly outcome: RoutingOutcome) {}
+ }
+}
+
+type AlgorithmStep = Step.CallModel | Step.CallDecision | Step.Done;
+
+interface NativeReader<T> {
+ next(): Promise<T | null>;
+ close(): void;
+}
+
+interface NativeInput {
+ push(json: string): Promise<void>;
+ finish(message: string | null, isContextWindow: boolean): Promise<void>;
+ closed(): Promise<void>;
+}
+
+interface NativeCall {
+ respond(work: Promise<string>): Promise<void>;
+ input(): NativeInput;
+}
+
+interface NativeStep {
+ kind: "decision" | "model" | "done";
+ data: string;
+ call: NativeCall | null;
+ stream: NativeReader<string> | null;
+}
+
+interface NativeConfigured {
+ start(request: string, headers: string | null): NativeReader<NativeStep>;
+}
+
+interface NativeFactory {
+ build(): Record<string, NativeConfigured>;
+}
+
+const native: { NativeAlgorithmFactory: new (source: string) => NativeFactory } =
+ createRequire(import.meta.url)("./switchyard.node");
+
+function message(error: unknown): string {
+ return error instanceof Error ? error.message : String(error);
+}
+
+function publicError(error: unknown): Error {
+ const match = /^(ERR_CONFIG|ERR_INVALID_REQUEST|ERR_ROUTING|ABORT_ERR): (.*)$/s.exec(message(error));
+ if (match?.[1] === "ABORT_ERR") {
+ return Object.assign(new Error(match[2]), { name: "AbortError", code: "ABORT_ERR" });
+ }
+ if (!match || match[1] === "ERR_ROUTING") return new LibsyError(match?.[2] ?? message(error));
+ return Object.assign(new Error(match[2]), { code: match[1] });
+}
+
+function json(value: unknown): string {
+ const result = JSON.stringify(value);
+ if (result === undefined) throw new TypeError("Expected a JSON value");
+ return result;
+}
+
+function failure(error: unknown): string {
+ return json({
+ kind: "error", error: message(error),
+ context_window_exceeded: error instanceof ContextWindowExceededError,
+ });
+}
+
+async function pump(source: AsyncIterable<JsonObject>, input: NativeInput): Promise<void> {
+ let iterator: AsyncIterator<JsonObject> | undefined;
+ let isFinished = false;
+ try {
+ iterator = source[Symbol.asyncIterator]();
+ const closed = input.closed().then(() => null);
+ for (;;) {
+ const item = await Promise.race([iterator.next(), closed]);
+ if (item === null) return;
+ if (item.done) break;
+ await input.push(json(item.value));
+ }
+ isFinished = true;
+ await input.finish(null, false);
+ } catch (error) {
+ await input.finish(message(error), error instanceof ContextWindowExceededError).catch(() => {});
+ } finally {
+ if (!isFinished) void Promise.resolve(iterator?.return?.()).catch(() => {});
+ }
+}
+
+function reader<T, U>(native: NativeReader<T>, decode: (value: T) => U): AsyncIterableIterator<U> {
+ let isClosed = false;
+ let tail: Promise<void> = Promise.resolve();
+ return {
+ [Symbol.asyncIterator]() { return this; },
+ next() {
+ const result = tail.then(async (): Promise<IteratorResult<U>> => {
+ if (isClosed) return { done: true, value: undefined };
+ try {
+ const value = await native.next();
+ if (isClosed) return { done: true, value: undefined };
+ if (value === null) {
+ isClosed = true;
+ native.close();
+ return { done: true, value: undefined };
+ }
+ return { done: false, value: decode(value) };
+ } catch (error) {
+ if (isClosed) return { done: true, value: undefined };
+ isClosed = true;
+ native.close();
+ throw publicError(error);
+ }
+ });
+ tail = result.then(() => {}, () => {});
+ return result;
+ },
+ async return() {
+ isClosed = true;
+ native.close();
+ return { done: true, value: undefined };
+ },
+ async throw(error: unknown) {
+ isClosed = true;
+ native.close();
+ throw error;
+ },
+ };
+}
+
+function callMethods(call: NativeCall, isModel: boolean) {
+ let isComplete = false;
+ async function respond(work: PromiseLike<unknown>): Promise<void> {
+ if (isComplete) {
+ void Promise.resolve(work).catch(() => {});
+ throw new LibsyError("Call has already been completed");
+ }
+ isComplete = true;
+ const reply = Promise.resolve(work).then((value) => {
+ if (!isModel) return json({ kind: "decision", response: value });
+ if (value instanceof LlmResponse.Agg) return json({ kind: "agg", response: value.response });
+ if (value instanceof LlmResponse.Stream) {
+ const input = call.input();
+ void pump(value.stream, input).catch(() => {});
+ return json({ kind: "stream" });
+ }
+ throw new TypeError("Expected LlmResponse.Agg or LlmResponse.Stream");
+ }).catch(failure);
+ try {
+ await call.respond(reply);
+ } catch (error) {
+ throw publicError(error);
+ }
+ }
+ return {
+ respond,
+ fail(error: Error): Promise<void> {
+ if (!(error instanceof Error)) throw new TypeError("error must be an Error");
+ return respond(Promise.reject(error));
+ },
+ };
+}
+
+function decodeStep(step: NativeStep): AlgorithmStep {
+ const data = JSON.parse(step.data);
+ if (step.kind === "model") {
+ return new Step.CallModel(Object.freeze({ ...data, ...callMethods(step.call!, true) }));
+ }
+ if (step.kind === "decision") {
+ return new Step.CallDecision(Object.freeze({ ...data, ...callMethods(step.call!, false) }));
+ }
+ if (data.response?.kind === "agg") data.response = new LlmResponse.Agg(data.response.response);
+ else if (data.response?.kind === "stream") {
+ data.response = new LlmResponse.Stream(reader(step.stream!, (json) => JSON.parse(json)));
+ }
+ return new Step.Done(Object.freeze(data));
+}
+
+let configured: (native: NativeConfigured) => ConfiguredAlgorithm;
+
+export class AlgorithmFactory {
+ readonly #native: NativeFactory;
+
+ private constructor(source: string) {
+ this.#native = new native.NativeAlgorithmFactory(source);
+ }
+
+ static fromToml(source: string): AlgorithmFactory {
+ try {
+ return new AlgorithmFactory(source);
+ } catch (error) {
+ throw publicError(error);
+ }
+ }
+
+ build(): Record<string, ConfiguredAlgorithm> {
+ try {
+ return Object.fromEntries(Object.entries(this.#native.build()).map(([id, route]) => [id, configured(route)]));
+ } catch (error) {
+ throw publicError(error);
+ }
+ }
+}
+
+export class ConfiguredAlgorithm {
+ readonly #native: NativeConfigured;
+
+ private constructor(native: NativeConfigured) {
+ this.#native = native;
+ }
+
+ static {
+ configured = (native) => new ConfiguredAlgorithm(native);
+ }
+
+ runStream(request: JsonObject, options: { headers?: Record<string, string> } = {}): AsyncIterableIterator<AlgorithmStep> {
+ try {
+ return reader(this.#native.start(json(request), options.headers === undefined ? null : json(options.headers)), decodeStep);
+ } catch (error) {
+ throw publicError(error);
+ }
+ }
+}
diff --git a/bindings/typescript/src/index.ts b/bindings/typescript/src/index.ts
index 699ba12ba..a75f16db1 100644
--- a/bindings/typescript/src/index.ts
+++ b/bindings/typescript/src/index.ts
@@ -3,6 +3,7 @@
import { createRequire } from "node:module";
+export * from "./algorithm.js";
export { TranslationEngine, TranslationError, WireFormat } from "./translation.js";
/** Opaque routing state. Store it unchanged and pass it back on the same branch. */
diff --git a/bindings/typescript/test/algorithm.test.mjs b/bindings/typescript/test/algorithm.test.mjs
new file mode 100644
index 000000000..b8d4dbf37
--- /dev/null
+++ b/bindings/typescript/test/algorithm.test.mjs
@@ -0,0 +1,260 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+import assert from "node:assert/strict";
+import { createServer } from "node:http";
+import { test } from "node:test";
+import { AlgorithmFactory, ConfiguredAlgorithm, LibsyError, LlmResponse, Step } from "../dist/index.js";
+
+const deployment = `
+schema_version = 1
+[llm_clients.answers]
+format = "openai_chat"
+base_url = "https://example.test/v1"
+api_key_env = "SWITCHYARD_FACTORY_TEST_KEY_THAT_IS_NOT_SET"
+[targets]
+fast = { id = "upstream/fast", llm_client = "answers" }
+capable = { id = "upstream/capable", llm_client = "answers" }
+[decision_clients.judge]
+format = "openai_decisions"
+endpoint = "https://example.test/v1/decisions"
+api_key_env = "SWITCHYARD_FACTORY_TEST_KEY_THAT_IS_NOT_SET"
+timeout_ms = 2000
+[decision_targets.judge]
+id = "upstream/judge"
+decision_client = "judge"
+[routes.parent]
+id = "public/main"
+type = "passthrough"
+target = "fast"
+[routes.parent.subagents]
+type = "passthrough"
+target = "capable"
+[routes.classifier]
+id = "public/classify"
+type = "llm_classifier"
+classifier_target = "judge"
+strong_target = "capable"
+weak_target = "fast"
+[routes.advisor]
+id = "public/advisor"
+type = "advisor"
+executor_target = "fast"
+advisor_target = "capable"
+gate_min_tool_results = 1
+`;
+const routes = AlgorithmFactory.fromToml(deployment).build();
+const request = { model: "public/main", messages: [{ role: "user", content: [{ type: "text", text: "hello" }] }] };
+const aggregate = (text = "hello") => ({
+ model: "upstream/fast", outputs: [{ role: "assistant", content: [{ type: "text", text }], stop_reason: "end_turn" }],
+});
+const answer = { answers: { route: { value: {
+ type: "choice", data: { selected: "capable", probabilities: { capable: 0.9, efficient: 0.1 } },
+} } } };
+
+async function collect(source) {
+ const result = [];
+ for await (const item of source) result.push(item);
+ return result;
+}
+
+async function start(t) {
+ const run = routes["public/advisor"].runStream(request);
+ t.after(() => run.return());
+ const step = (await run.next()).value;
+ assert.ok(step instanceof Step.CallModel);
+ return { run, call: step.call };
+}
+
+test("factory returns logical targets, full outcomes, and header-driven subagent selection", async () => {
+ assert.deepEqual(Object.keys(routes), ["public/advisor", "public/classify", "public/main"]);
+ assert.ok(routes["public/main"] instanceof ConfiguredAlgorithm);
+ for (const [headers, expected] of [[undefined, "fast"], [{ "x-switchyard-is-subagent": "true" }, "capable"]]) {
+ const steps = await collect(routes["public/main"].runStream(request, { headers }));
+ assert.equal(steps.length, 1);
+ assert.ok(steps[0] instanceof Step.Done);
+ const outcome = steps[0].outcome;
+ assert.deepEqual(outcome.selected_model_ids, [expected]);
+ assert.equal(outcome.request.model, expected);
+ assert.deepEqual(outcome.request.messages, request.messages);
+ assert.equal(outcome.fallback_request, null);
+ assert.deepEqual(outcome.candidate_requests, {});
+ assert.equal(outcome.response, null);
+ assert.equal(typeof outcome.metadata.outcome_id, "string");
+ }
+ assert.throws(() => AlgorithmFactory.fromToml("invalid"), { code: "ERR_CONFIG" });
+ assert.throws(() => routes["public/main"].runStream({ messages: [{ role: "invalid", content: [] }] }), { code: "ERR_INVALID_REQUEST" });
+});
+
+test("factory rejects configuration requiring Runner admission", () => {
+ assert.throws(() => AlgorithmFactory.fromToml(`${deployment}\n[routes.parent.eligibility]\n`).build(), {
+ code: "ERR_CONFIG", message: /requires Runner admission/,
+ });
+});
+
+test("decision calls accept async replies and failures", async (t) => {
+ for (const fail of [false, true]) {
+ const run = routes["public/classify"].runStream(request);
+ t.after(() => run.return());
+ const step = (await run.next()).value;
+ assert.ok(step instanceof Step.CallDecision);
+ assert.equal(step.call.model, "upstream/judge");
+ assert.equal(step.call.request.model, "upstream/judge");
+ if (fail) await step.call.fail(new Error("judge unavailable"));
+ else await step.call.respond(Promise.resolve(answer));
+ await assert.rejects(step.call.respond(Promise.resolve(answer)), LibsyError);
+ const done = (await run.next()).value;
+ assert.ok(done instanceof Step.Done);
+ assert.deepEqual(done.outcome.selected_model_ids, ["capable", "fast"]);
+ assert.equal(done.outcome.metadata.evidence.source, fail ? "fail_open" : "decision_classifier");
+ }
+});
+
+test("model calls accept normalized aggregates and response streams", async (t) => {
+ const events = [
+ { preservation: null, normalized: [{ MessageStart: { id: "answer-1", model: "upstream/fast" } }] },
+ { preservation: null, normalized: [{ TextDelta: { index: 0, text: "hello" } }] },
+ { preservation: null, normalized: [{ Usage: {
+ input_tokens: 2, cached_input_tokens: null, cache_creation_input_tokens: null,
+ output_tokens: 1, total_tokens: 3, reasoning_tokens: null,
+ } }] },
+ { preservation: null, normalized: [{ MessageStop: { reason: "end_turn" } }] },
+ ];
+ for (const isStream of [false, true]) {
+ const { run, call } = await start(t);
+ assert.deepEqual(call.models, ["fast"]);
+ assert.equal(call.request.model, "fast");
+ await call.respond(Promise.resolve(isStream
+ ? new LlmResponse.Stream((async function* () { yield* events; })())
+ : new LlmResponse.Agg(aggregate())));
+ let isPolled = false;
+ const duplicate = new LlmResponse.Stream({ async *[Symbol.asyncIterator]() { isPolled = true; } });
+ await assert.rejects(call.respond(Promise.resolve(duplicate)), LibsyError);
+ assert.equal(isPolled, false);
+ const step = (await run.next()).value;
+ assert.ok(step instanceof Step.Done);
+ const outcome = step.outcome;
+ if (isStream) {
+ assert.ok(outcome.response instanceof LlmResponse.Stream);
+ const replayed = await collect(outcome.response.stream);
+ assert.deepEqual(replayed.map((event) => event.normalized), events.map((event) => event.normalized));
+ } else {
+ assert.ok(outcome.response instanceof LlmResponse.Agg);
+ assert.equal(outcome.response.response.outputs[0].content[0].text, "hello");
+ }
+ }
+});
+
+test("response stream setup and read failures reach the routing iterator", { timeout: 5000 }, async (t) => {
+ for (const isSetupFailure of [true, false]) {
+ const { run, call } = await start(t);
+ const source = isSetupFailure
+ ? { [Symbol.asyncIterator]() { throw new Error("stream unavailable"); } }
+ : { async *[Symbol.asyncIterator]() { throw new Error("stream unavailable"); } };
+ const rejected = assert.rejects(run.next(), {
+ name: "LibsyError", code: "ERR_ROUTING", message: /stream unavailable/,
+ });
+ await call.respond(Promise.resolve(new LlmResponse.Stream(source)));
+ await rejected;
+ }
+});
+
+test("non-recovering model responders reject with the original host failure", { timeout: 5000 }, async (t) => {
+ for (const fail of [false, true]) {
+ const { call } = await start(t);
+ const cause = new Error("executor unavailable");
+ await assert.rejects(fail ? call.fail(cause) : call.respond(Promise.reject(cause)), {
+ name: "LibsyError", code: "ERR_ROUTING", message: /executor unavailable/,
+ });
+ }
+});
+
+test("concurrent model calls keep replies on their originating runs", { timeout: 5000 }, async (t) => {
+ const pending = await Promise.all([start(t), start(t)]);
+ const work = [Promise.withResolvers(), Promise.withResolvers()];
+ const responding = pending.map(({ call }, index) => call.respond(work[index].promise));
+ const outcomes = pending.map(({ run }) => run.next());
+ for (const [index, text] of [[1, "second"], [0, "first"]]) {
+ work[index].resolve(new LlmResponse.Agg(aggregate(text)));
+ await responding[index];
+ const step = (await outcomes[index]).value;
+ assert.ok(step instanceof Step.Done);
+ assert.equal(step.outcome.response.response.outputs[0].content[0].text, text);
+ }
+ for (const { run } of pending) assert.deepEqual(await run.next(), { done: true, value: undefined });
+});
+
+test("iterator return releases native waits and closes input readers without cancelling host promises", { timeout: 5000 }, async (t) => {
+ const { run: concurrent, call: first } = await start(t);
+ const p1 = concurrent.next();
+ const p2 = concurrent.next();
+ await first.respond(Promise.resolve(new LlmResponse.Agg(aggregate())));
+ assert.ok((await p1).value instanceof Step.Done);
+ assert.deepEqual(await p2, { done: true, value: undefined });
+
+ const { run, call } = await start(t);
+ const work = Promise.withResolvers();
+ let isHostFinished = false;
+ void work.promise.then(() => { isHostFinished = true; }, () => { isHostFinished = true; });
+ const responding = call.respond(work.promise);
+ await run.return();
+ await responding;
+ assert.equal(isHostFinished, false);
+ work.reject(new Error("late host failure"));
+ await Promise.resolve();
+ assert.equal(isHostFinished, true);
+
+ const { run: streamRun, call: streamCall } = await start(t);
+ const entered = Promise.withResolvers();
+ const returned = Promise.withResolvers();
+ const pending = Promise.withResolvers();
+ const source = {
+ [Symbol.asyncIterator]() { return this; },
+ next() { entered.resolve(); return pending.promise; },
+ return() { returned.resolve(); return Promise.resolve({ done: true }); },
+ };
+ await streamCall.respond(Promise.resolve(new LlmResponse.Stream(source)));
+ await entered.promise;
+ const next = streamRun.next();
+ await streamRun.return();
+ assert.equal((await next).done, true);
+ await returned.promise;
+ pending.resolve({ done: true });
+});
+
+test("host-owned AbortController closes HTTP work while a concurrent route survives", { timeout: 5000 }, async (t) => {
+ const entered = Promise.withResolvers();
+ const closed = Promise.withResolvers();
+ const controller = new AbortController();
+ const server = createServer((_req, res) => {
+ res.once("close", closed.resolve);
+ entered.resolve();
+ });
+ t.after(async () => {
+ controller.abort();
+ const closing = new Promise((resolve) => server.close(resolve));
+ server.closeAllConnections();
+ await closing;
+ });
+ await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
+ const base = `http://127.0.0.1:${server.address().port}`;
+ const cancelled = await start(t);
+ const work = fetch(base, { signal: controller.signal })
+ .then((response) => response.json()).then((response) => new LlmResponse.Agg(response));
+ const responding = cancelled.call.respond(work).finally(async () => {
+ controller.abort();
+ await work.catch(() => {});
+ });
+ await entered.promise;
+ const healthy = await start(t);
+ const healthyNext = healthy.run.next();
+ await cancelled.run.return();
+ await responding;
+ await closed.promise;
+ await healthy.call.respond(Promise.resolve(new LlmResponse.Agg(aggregate("healthy"))));
+ const done = (await healthyNext).value;
+ assert.ok(done instanceof Step.Done);
+ assert.equal(done.outcome.response.response.outputs[0].content[0].text, "healthy");
+ assert.deepEqual(await healthy.run.next(), { done: true, value: undefined });
+ assert.deepEqual(await cancelled.run.next(), { done: true, value: undefined });
+});
diff --git a/crates/switchyard-node/Cargo.toml b/crates/switchyard-node/Cargo.toml
index b35608ebf..f5b6ba67a 100644
--- a/crates/switchyard-node/Cargo.toml
+++ b/crates/switchyard-node/Cargo.toml
@@ -19,12 +19,15 @@ crate-type = ["cdylib"]
napi = { version = "3", default-features = false, features = ["napi8", "tokio_rt", "dyn-symbols"] }
napi-derive = "3"
futures.workspace = true
+http.workspace = true
parking_lot.workspace = true
serde.workspace = true
serde_json.workspace = true
switchyard-libsy.workspace = true
switchyard-protocol.workspace = true
+switchyard-runner.workspace = true
switchyard-translation.workspace = true
+tokio.workspace = true
[build-dependencies]
napi-build = "2"
diff --git a/crates/switchyard-node/src/algorithm.rs b/crates/switchyard-node/src/algorithm.rs
new file mode 100644
index 000000000..432ea6b3d
--- /dev/null
+++ b/crates/switchyard-node/src/algorithm.rs
@@ -0,0 +1,556 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+//! Configured routing steps and normalized responses for host-owned execution.
+
+use std::collections::BTreeMap;
+use std::pin::Pin;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicBool, Ordering};
+
+use futures::future::{AbortHandle, Abortable};
+use futures::{Stream, StreamExt};
+use http::{HeaderMap, HeaderName, HeaderValue};
+use napi::bindgen_prelude::Promise;
+use napi_derive::napi;
+use parking_lot::Mutex;
+use serde::{Deserialize, Serialize};
+use serde_json::Value;
+use switchyard_libsy::{
+ Algorithm, CallDecision, CallModel, LibsyError, RuntimeModels, State, Step,
+};
+use switchyard_protocol::{
+ AggLlmResponse, DecisionRequest, DecisionResponse, LlmClientError, LlmRequest, LlmResponse,
+ LlmResponseStream, LlmResponseStreamEvent, Metadata, ModelId, Request, Response,
+};
+use switchyard_runner::AlgorithmFactory;
+use tokio::sync::mpsc;
+
+use crate::{error, parse, routing_error};
+
+#[napi]
+pub struct NativeAlgorithmFactory {
+ inner: AlgorithmFactory,
+}
+
+#[napi]
+impl NativeAlgorithmFactory {
+ #[napi(constructor)]
+ pub fn new(source: String) -> napi::Result<Self> {
+ Ok(Self {
+ inner: AlgorithmFactory::from_toml(&source).map_err(|err| error("ERR_CONFIG", err))?,
+ })
+ }
+
+ #[napi]
+ pub fn build(&self) -> napi::Result<BTreeMap<String, NativeConfiguredAlgorithm>> {
+ Ok(self
+ .inner
+ .build()
+ .map_err(|err| error("ERR_CONFIG", err))?
+ .into_iter()
+ .map(|(id, (algorithm, models))| {
+ (
+ id,
+ NativeConfiguredAlgorithm {
+ algorithm,
+ models: Arc::new(models),
+ },
+ )
+ })
+ .collect())
+ }
+}
+
+#[napi]
+pub struct NativeConfiguredAlgorithm {
+ algorithm: Arc<dyn Algorithm>,
+ models: Arc<RuntimeModels>,
+}
+
+#[napi]
+impl NativeConfiguredAlgorithm {
+ #[napi]
+ pub fn start(
+ &self,
+ request_json: String,
+ headers_json: Option<String>,
+ ) -> napi::Result<NativeAlgorithmRun> {
+ let headers: Option<BTreeMap<String, String>> =
+ headers_json.as_deref().map(parse).transpose()?;
+ let headers = headers
+ .map(|headers| {
+ let mut result = HeaderMap::new();
+ for (name, value) in headers {
+ let name = HeaderName::from_bytes(name.as_bytes())
+ .map_err(|err| error("ERR_INVALID_REQUEST", err))?;
+ let value = HeaderValue::from_str(&value)
+ .map_err(|err| error("ERR_INVALID_REQUEST", err))?;
+ result
+ .try_append(name, value)
+ .map_err(|err| error("ERR_INVALID_REQUEST", err))?;
+ }
+ Ok::<_, napi::Error>(result)
+ })
+ .transpose()?;
+ let mut llm_request = parse(&request_json)?;
+ switchyard_translation::normalize_compaction(&mut llm_request, headers.as_ref());
+ let request = Request {
+ llm_request,
+ raw_request: None,
+ metadata: headers.map(|headers| Metadata::from_headers(&headers)),
+ };
+ let algorithm = Arc::clone(&self.algorithm);
+ let models = Arc::clone(&self.models);
+ // Algorithm startup must run on napi's Tokio runtime.
+ let stream = futures::stream::once(async move {
+ algorithm.run_stream(request, models, State::default())
+ })
+ .flatten()
+ .map(|step| step.map_err(routing_error));
+ Ok(NativeAlgorithmRun {
+ reader: Reader::new(stream),
+ })
+ }
+}
+
+type Items<T> = Pin<Box<dyn Stream<Item = napi::Result<T>> + Send>>;
+
+struct ReaderSlot<T> {
+ stream: Option<Items<T>>,
+ abort: Option<AbortHandle>,
+}
+
+struct Reader<T> {
+ slot: Mutex<ReaderSlot<T>>,
+}
+
+impl<T: Send + 'static> Reader<T> {
+ fn new(stream: impl Stream<Item = napi::Result<T>> + Send + 'static) -> Self {
+ Self {
+ slot: Mutex::new(ReaderSlot {
+ stream: Some(Box::pin(stream)),
+ abort: None,
+ }),
+ }
+ }
+
+ async fn next(&self) -> napi::Result<Option<T>> {
+ let (abort, registration) = AbortHandle::new_pair();
+ let mut stream = {
+ let mut slot = self.slot.lock();
+ let stream = match slot.stream.take() {
+ Some(stream) => stream,
+ None if slot.abort.is_none() => return Ok(None),
+ None => return Err(error("ERR_INVALID_REQUEST", "next is already pending")),
+ };
+ slot.abort = Some(abort.clone());
+ stream
+ };
+ // close() aborts the future that owns the reader, including a pending next().
+ let (stream, item) = Abortable::new(
+ async move {
+ let item = stream.next().await;
+ (stream, item)
+ },
+ registration,
+ )
+ .await
+ .map_err(|_| error("ABORT_ERR", "Routing cancelled"))?;
+ let mut slot = self.slot.lock();
+ if abort.is_aborted() {
+ return Err(error("ABORT_ERR", "Routing cancelled"));
+ }
+ slot.abort = None;
+ if item.as_ref().is_some_and(Result::is_ok) {
+ slot.stream = Some(stream);
+ }
+ item.transpose()
+ }
+
+ fn close(&self) {
+ let mut slot = self.slot.lock();
+ if let Some(abort) = slot.abort.take() {
+ abort.abort();
+ }
+ slot.stream = None;
+ }
+}
+
+#[napi]
+pub struct NativeAlgorithmRun {
+ reader: Reader<Step>,
+}
+
+#[napi(object, object_from_js = false, use_nullable = true)]
+pub struct NativeAlgorithmStep {
+ pub kind: String,
+ pub data: String,
+ pub call: Option<NativeAlgorithmCall>,
+ pub stream: Option<NativeResponseStream>,
+}
+
+#[napi]
+impl NativeAlgorithmRun {
+ #[napi]
+ pub async fn next(&self) -> napi::Result<Option<NativeAlgorithmStep>> {
+ self.reader.next().await?.map(step_to_native).transpose()
+ }
+
+ #[napi]
+ pub fn close(&self) {
+ self.reader.close();
+ }
+}
+
+enum HostCall {
+ Model(Box<CallModel>),
+ Decision(Box<CallDecision>),
+}
+
+type InputItem = Option<Result<LlmResponseStreamEvent, LlmClientError>>;
+
+#[napi]
+pub struct NativeAlgorithmCall {
+ inner: Mutex<Option<HostCall>>,
+ input: Mutex<Option<mpsc::Receiver<InputItem>>>,
+ model: ModelId,
+ is_complete: AtomicBool,
+}
+
+#[derive(Deserialize)]
+#[serde(tag = "kind", rename_all = "snake_case")]
+enum Reply {
+ Agg {
+ response: AggLlmResponse,
+ },
+ Stream,
+ Decision {
+ response: DecisionResponse,
+ },
+ Error {
+ error: String,
+ #[serde(default)]
+ context_window_exceeded: bool,
+ },
+}
+
+#[derive(Debug)]
+struct ForeignError(String);
+
+impl std::fmt::Display for ForeignError {
+ fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ formatter.write_str(&self.0)
+ }
+}
+
+impl std::error::Error for ForeignError {}
+
+fn client_error(model: &ModelId, message: String, is_context_window: bool) -> LlmClientError {
+ if is_context_window {
+ LlmClientError::ContextWindowExceeded {
+ model: model.clone(),
+ message,
+ }
+ } else {
+ LlmClientError::Ffi {
+ source: Box::new(ForeignError(message)),
+ }
+ }
+}
+
+#[napi]
+impl NativeAlgorithmCall {
+ #[napi]
+ pub fn input(&self) -> napi::Result<NativeInputStream> {
+ let mut input = self.input.lock();
+ if self.is_complete.load(Ordering::Acquire) {
+ return Err(error("ABORT_ERR", "Call is no longer waiting"));
+ }
+ if input.is_some() {
+ return Err(error(
+ "ERR_INVALID_REQUEST",
+ "Response stream already supplied",
+ ));
+ }
+ let (sender, receiver) = mpsc::channel(1);
+ *input = Some(receiver);
+ Ok(NativeInputStream {
+ sender,
+ model: self.model.clone(),
+ })
+ }
+
+ #[napi]
+ pub async fn respond(&self, work: Promise<String>) -> napi::Result<()> {
+ let call = self
+ .inner
+ .lock()
+ .take()
+ .ok_or_else(|| routing_error("Call has already been completed"))?;
+ let model = &self.model;
+ let work = async {
+ let json = work.await.map_err(|err| {
+ LibsyError::client_call(model.clone(), client_error(model, err.to_string(), false))
+ })?;
+ let reply: Reply = serde_json::from_str(&json).map_err(|err| {
+ LibsyError::client_call(model.clone(), client_error(model, err.to_string(), false))
+ })?;
+ if let Reply::Error {
+ error,
+ context_window_exceeded,
+ } = reply
+ {
+ return Err(LibsyError::client_call(
+ model.clone(),
+ client_error(model, error, context_window_exceeded),
+ ));
+ }
+ Ok(reply)
+ };
+ let result = match call {
+ HostCall::Decision(call) => {
+ call.respond(async {
+ match work.await? {
+ Reply::Decision { response } => Ok(response),
+ _ => Err(LibsyError::client_call(
+ model.clone(),
+ client_error(model, "Expected a decision response".into(), false),
+ )),
+ }
+ })
+ .await
+ }
+ HostCall::Model(call) => {
+ let metadata = call.request.metadata.clone();
+ call.respond(async {
+ let llm_response = match work.await? {
+ Reply::Agg { response } => LlmResponse::Agg(response),
+ Reply::Stream => {
+ let receiver = self.input.lock().take().ok_or_else(|| {
+ LibsyError::client_call(
+ model.clone(),
+ client_error(model, "Response stream is missing".into(), false),
+ )
+ })?;
+ let stream = futures::stream::unfold(Some(receiver), |state| async {
+ let mut receiver = state?;
+ let item = receiver.recv().await??;
+ let state = item.is_ok().then_some(receiver);
+ Some((item, state))
+ });
+ LlmResponse::Stream(Box::pin(stream))
+ }
+ _ => {
+ return Err(LibsyError::client_call(
+ model.clone(),
+ client_error(
+ model,
+ "Expected LlmResponse.Agg or LlmResponse.Stream".into(),
+ false,
+ ),
+ ));
+ }
+ };
+ Ok(Response {
+ llm_response,
+ metadata,
+ upstream_headers: HeaderMap::new(),
+ })
+ })
+ .await
+ }
+ };
+ self.is_complete.store(true, Ordering::Release);
+ self.input.lock().take();
+ result.map_err(routing_error)
+ }
+}
+
+#[napi]
+pub struct NativeInputStream {
+ sender: mpsc::Sender<InputItem>,
+ model: ModelId,
+}
+
+#[napi]
+impl NativeInputStream {
+ #[napi]
+ pub async fn push(&self, json: String) -> napi::Result<()> {
+ let event = serde_json::from_str(&json).map_err(routing_error)?;
+ self.sender
+ .send(Some(Ok(event)))
+ .await
+ .map_err(|_| error("ABORT_ERR", "Response stream closed"))
+ }
+
+ #[napi]
+ pub async fn finish(
+ &self,
+ message: Option<String>,
+ is_context_window: bool,
+ ) -> napi::Result<()> {
+ let item =
+ message.map(|message| Err(client_error(&self.model, message, is_context_window)));
+ self.sender
+ .send(item)
+ .await
+ .map_err(|_| error("ABORT_ERR", "Response stream closed"))
+ }
+
+ #[napi]
+ pub async fn closed(&self) {
+ self.sender.closed().await;
+ }
+}
+
+#[napi]
+pub struct NativeResponseStream {
+ reader: Reader<LlmResponseStreamEvent>,
+}
+
+impl NativeResponseStream {
+ fn new(stream: LlmResponseStream) -> Self {
+ Self {
+ reader: Reader::new(stream.map(|item| item.map_err(routing_error))),
+ }
+ }
+}
+
+#[napi]
+impl NativeResponseStream {
+ #[napi]
+ pub async fn next(&self) -> napi::Result<Option<String>> {
+ self.reader
+ .next()
+ .await?
+ .map(|event| serde_json::to_string(&event).map_err(routing_error))
+ .transpose()
+ }
+
+ #[napi]
+ pub fn close(&self) {
+ self.reader.close();
+ }
+}
+
+#[derive(Serialize)]
+#[serde(untagged)]
+enum StepPayload<'a> {
+ Decision {
+ algorithm: &'a str,
+ request: &'a DecisionRequest,
+ model: &'a ModelId,
+ },
+ Model {
+ algorithm: &'a str,
+ request: &'a LlmRequest,
+ models: &'a [ModelId],
+ },
+ Done {
+ selected_model_ids: &'a [ModelId],
+ request: &'a LlmRequest,
+ fallback_request: Option<&'a LlmRequest>,
+ candidate_requests: BTreeMap<&'a str, &'a LlmRequest>,
+ response: Option<ResponsePayload<'a>>,
+ metadata: Option<MetadataPayload<'a>>,
+ },
+}
+
+#[derive(Serialize)]
+#[serde(tag = "kind", rename_all = "snake_case")]
+enum ResponsePayload<'a> {
+ Agg { response: &'a AggLlmResponse },
+ Stream,
+}
+
+#[derive(Serialize)]
+struct MetadataPayload<'a> {
+ outcome_id: &'a str,
+ algorithm: &'a str,
+ evidence: Option<&'a Value>,
+}
+
+fn step_to_native(step: Step) -> napi::Result<NativeAlgorithmStep> {
+ let mut result = NativeAlgorithmStep {
+ kind: String::new(),
+ data: String::new(),
+ call: None,
+ stream: None,
+ };
+ match step {
+ Step::CallDecision(call) => {
+ result.kind = "decision".into();
+ result.data = serde_json::to_string(&StepPayload::Decision {
+ algorithm: &call.algorithm,
+ request: &call.request,
+ model: &call.model,
+ })
+ .map_err(routing_error)?;
+ result.call = Some(NativeAlgorithmCall {
+ model: call.model.clone(),
+ inner: Mutex::new(Some(HostCall::Decision(call))),
+ input: Mutex::new(None),
+ is_complete: AtomicBool::new(false),
+ });
+ }
+ Step::CallModel(call) => {
+ result.kind = "model".into();
+ result.data = serde_json::to_string(&StepPayload::Model {
+ algorithm: &call.algorithm,
+ request: &call.request.llm_request,
+ models: &call.models,
+ })
+ .map_err(routing_error)?;
+ result.call = Some(NativeAlgorithmCall {
+ model: call
+ .request
+ .llm_request
+ .model
+ .as_ref()
+ .map(ModelId::new)
+ .ok_or_else(|| routing_error("Model call has no selected model"))?,
+ inner: Mutex::new(Some(HostCall::Model(call))),
+ input: Mutex::new(None),
+ is_complete: AtomicBool::new(false),
+ });
+ }
+ Step::Done(outcome) => {
+ result.kind = "done".into();
+ result.data = serde_json::to_string(&StepPayload::Done {
+ selected_model_ids: &outcome.selected_model_ids,
+ request: &outcome.request.llm_request,
+ fallback_request: outcome
+ .fallback_request
+ .as_ref()
+ .map(|request| &request.llm_request),
+ candidate_requests: outcome
+ .candidate_requests
+ .iter()
+ .map(|(id, request)| (id.as_str(), &request.llm_request))
+ .collect(),
+ response: outcome
+ .response
+ .as_ref()
+ .map(|response| match &response.llm_response {
+ LlmResponse::Agg(response) => ResponsePayload::Agg { response },
+ LlmResponse::Stream(_) => ResponsePayload::Stream,
+ }),
+ metadata: outcome.metadata.as_ref().map(|metadata| MetadataPayload {
+ outcome_id: metadata.outcome_id(),
+ algorithm: &metadata.algorithm,
+ evidence: metadata.evidence.as_ref(),
+ }),
+ })
+ .map_err(routing_error)?;
+ if let Some(Response {
+ llm_response: LlmResponse::Stream(stream),
+ ..
+ }) = outcome.response
+ {
+ result.stream = Some(NativeResponseStream::new(stream));
+ }
+ }
+ }
+ Ok(result)
+}
diff --git a/crates/switchyard-node/src/lib.rs b/crates/switchyard-node/src/lib.rs
index 7291e1fd0..32bf423c3 100644
--- a/crates/switchyard-node/src/lib.rs
+++ b/crates/switchyard-node/src/lib.rs
@@ -1,8 +1,9 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
-//! Node bindings for composite routing and buffered JSON translation.
+//! Node bindings for routing algorithms and buffered JSON translation.
+mod algorithm;
mod translation;
use std::future::ready;