Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
feat(node): expose configured algorithm execution (#1235)
Alex Fournier6b9acf33ca57
diff --git a/Cargo.lock b/Cargo.lockindex 32419e042..f9411ca10 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -2606,6 +2606,7 @@ name = "switchyard-node"version = "0.3.0"dependencies = ["futures",+ "http","napi","napi-build","napi-derive",@@ -2614,7 +2615,9 @@ dependencies = ["serde_json","switchyard-libsy","switchyard-protocol",+ "switchyard-runner","switchyard-translation",+ "tokio",][[package]]diff --git a/bindings/typescript/README.md b/bindings/typescript/README.mdindex 793327e39..9886ece79 100644--- a/bindings/typescript/README.md+++ b/bindings/typescript/README.md@@ -1,13 +1,9 @@# Switchyard libsy for Node.js-TypeScript bindings for composite routing and buffered JSON translation.-Composite routing uses the caller's decision response and local tool signals to-select a model. The caller handles classifier calls, completion requests, and-branch state.--Capable-first routing defaults to pinning the efficient tier after switching to-it. Set `stage.efficient_hold_turns: 0` to disable this hold. Efficient-first-defaults to no efficient hold. Persist the returned state to retain holds.+TypeScript bindings for routing algorithms and buffered JSON translation.+The host handles model and decision calls. `AlgorithmFactory` builds configured+algorithms from deployment TOML; `CompositeRouter` provides the smaller composite+routing API.Requires Node.js 22.19 or newer and the repository's Rust toolchain. The packageis private. Its native addon is built for the local host. Linux is covered in CI.@@ -24,8 +20,8 @@ npm --prefix bindings/typescript test```The build uses Cargo's development profile and copies the addon and license files-into the package. Rebuild after changing Rust code. Tests use local decision-callbacks.+into the package. Rebuild after changing Rust code. Tests use local callbacks and+HTTP fixtures.Install the built package in a local Node project:@@ -33,7 +29,71 @@ Install the built package in a local Node project:npm install /absolute/path/to/Switchyard/bindings/typescript```-## API+## Configured algorithms++```ts+import { AlgorithmFactory, LlmResponse, Step } from "@switchyard/libsy";++const routes = AlgorithmFactory.fromToml(source).build();+const steps = routes["public/main"].runStream(request, { headers });+```++`source` is deployment TOML and `request` is a normalized Switchyard request.+The factory validates configuration without resolving credentials or creating+provider clients. Reuse configured algorithms to retain algorithm-owned state.+Configurations that require target eligibility checks must use Runner admission;+the factory rejects them.++The iterator yields Python-compatible routing steps:++- `Step.CallModel`: serve `call.request` using `call.models[0]`, then pass a Promise+ of `LlmResponse.Agg` or `LlmResponse.Stream` to `call.respond`.+- `Step.CallDecision`: serve `call.request` using `call.model`, then pass a+ Promise of `DecisionResponse` to `call.respond`.+- `Step.Done`: read the selected models, rewritten requests, optional response,+ and metadata from `step.outcome`.++For routing-time model calls, use only the first candidate and pass failures to+`call.respond` or `call.fail`. The reply has no field for the actual served logical+target. When `Step.Done` has no response, the host may try `outcome.selected_model_ids`+in order to serve the final answer, applying the outcome's requests.++Model calls and selected models use logical target names from the TOML.+Decision calls use configured provider IDs. The host owns provider settings,+target prompts, transport, and final completion calls when the outcome has no+response. Responses and streams use normalized protocol JSON, not provider wire+formats or SSE bytes. `ContextWindowExceededError` reports a typed context-window+failure to the algorithm.++Calls can overlap. Keep reading steps while serving concurrent calls, and handle+rejections from every `respond` or `fail` Promise. A non-recovering call rejects+its responder; ignoring that error loses the original failure.++Returning the routing iterator stops Rust from waiting on outstanding calls.+JavaScript cannot cancel a supplied Promise. For buffered calls, use a host-owned+controller; here `callProvider` returns the response expected by `call.respond`:++```ts+const controller = new AbortController();+const work = callProvider(call.request, { signal: controller.signal });+try {+ await call.respond(work);+} finally {+ controller.abort();+ await work.catch(() => {});+}+```++The provider client must honor the signal and have a request timeout. This also+cleans up work when the algorithm stops waiting before the response arrives.+For `LlmResponse.Stream`, `respond` finishes when the stream is delivered. Keep+its HTTP request alive until stream completion or cancellation.++## CompositeRouter++Capable-first routing defaults to pinning the efficient tier after switching to+it. Set `stage.efficient_hold_turns: 0` to disable this hold. Efficient-first+defaults to no efficient hold. Persist the returned state to retain holds.```tsimport { CompositeRouter, type DecisionRequest, type DecisionResponse } from "@switchyard/libsy";@@ -92,7 +152,7 @@ Treat `CompositeState` as opaque. It carries the judge's tier and stage holds.| Code | Meaning || --- | --- |-| `ERR_CONFIG` | Invalid composite options |+| `ERR_CONFIG` | Invalid options or deployment configuration || `ERR_INVALID_REQUEST` | Invalid messages, state, or decision response JSON || `ERR_ROUTING` | The algorithm or classifier call failed || `ABORT_ERR` | Routing was cancelled; the error name is `AbortError` |diff --git a/bindings/typescript/src/algorithm.ts b/bindings/typescript/src/algorithm.tsnew file mode 100644index 000000000..2c2ba45ec--- /dev/null+++ b/bindings/typescript/src/algorithm.ts@@ -0,0 +1,288 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++import { createRequire } from "node:module";+import type { DecisionRequest, DecisionResponse } from "./index.js";++type JsonObject = Record<string, unknown>;++export class LibsyError extends Error {+ readonly code = "ERR_ROUTING";+ override name = "LibsyError";+}++export class ContextWindowExceededError extends Error {+ override name = "ContextWindowExceededError";+}++export namespace LlmResponse {+ export class Agg {+ constructor(readonly response: JsonObject) {}+ }++ export class Stream {+ constructor(readonly stream: AsyncIterable<JsonObject>) {}+ }+}++export interface ModelCall {+ readonly algorithm: string;+ readonly request: JsonObject;+ readonly models: readonly string[];+ respond(work: PromiseLike<LlmResponse.Agg | LlmResponse.Stream>): Promise<void>;+ fail(error: Error): Promise<void>;+}++export interface DecisionCall {+ readonly algorithm: string;+ readonly request: DecisionRequest;+ readonly model: string;+ respond(work: PromiseLike<DecisionResponse>): Promise<void>;+ fail(error: Error): Promise<void>;+}++export interface OutcomeMetadata {+ readonly outcome_id: string;+ readonly algorithm: string;+ readonly evidence: unknown | null;+}++export interface RoutingOutcome {+ readonly selected_model_ids: readonly string[];+ readonly request: JsonObject;+ readonly fallback_request: JsonObject | null;+ readonly candidate_requests: Readonly<Record<string, JsonObject>>;+ readonly response: LlmResponse.Agg | LlmResponse.Stream | null;+ readonly metadata: OutcomeMetadata | null;+}++export namespace Step {+ export class CallModel {+ constructor(readonly call: ModelCall) {}+ }++ export class CallDecision {+ constructor(readonly call: DecisionCall) {}+ }++ export class Done {+ constructor(readonly outcome: RoutingOutcome) {}+ }+}++type AlgorithmStep = Step.CallModel | Step.CallDecision | Step.Done;++interface NativeReader<T> {+ next(): Promise<T | null>;+ close(): void;+}++interface NativeInput {+ push(json: string): Promise<void>;+ finish(message: string | null, isContextWindow: boolean): Promise<void>;+ closed(): Promise<void>;+}++interface NativeCall {+ respond(work: Promise<string>): Promise<void>;+ input(): NativeInput;+}++interface NativeStep {+ kind: "decision" | "model" | "done";+ data: string;+ call: NativeCall | null;+ stream: NativeReader<string> | null;+}++interface NativeConfigured {+ start(request: string, headers: string | null): NativeReader<NativeStep>;+}++interface NativeFactory {+ build(): Record<string, NativeConfigured>;+}++const native: { NativeAlgorithmFactory: new (source: string) => NativeFactory } =+ createRequire(import.meta.url)("./switchyard.node");++function message(error: unknown): string {+ return error instanceof Error ? error.message : String(error);+}++function publicError(error: unknown): Error {+ const match = /^(ERR_CONFIG|ERR_INVALID_REQUEST|ERR_ROUTING|ABORT_ERR): (.*)$/s.exec(message(error));+ if (match?.[1] === "ABORT_ERR") {+ return Object.assign(new Error(match[2]), { name: "AbortError", code: "ABORT_ERR" });+ }+ if (!match || match[1] === "ERR_ROUTING") return new LibsyError(match?.[2] ?? message(error));+ return Object.assign(new Error(match[2]), { code: match[1] });+}++function json(value: unknown): string {+ const result = JSON.stringify(value);+ if (result === undefined) throw new TypeError("Expected a JSON value");+ return result;+}++function failure(error: unknown): string {+ return json({+ kind: "error", error: message(error),+ context_window_exceeded: error instanceof ContextWindowExceededError,+ });+}++async function pump(source: AsyncIterable<JsonObject>, input: NativeInput): Promise<void> {+ let iterator: AsyncIterator<JsonObject> | undefined;+ let isFinished = false;+ try {+ iterator = source[Symbol.asyncIterator]();+ const closed = input.closed().then(() => null);+ for (;;) {+ const item = await Promise.race([iterator.next(), closed]);+ if (item === null) return;+ if (item.done) break;+ await input.push(json(item.value));+ }+ isFinished = true;+ await input.finish(null, false);+ } catch (error) {+ await input.finish(message(error), error instanceof ContextWindowExceededError).catch(() => {});+ } finally {+ if (!isFinished) void Promise.resolve(iterator?.return?.()).catch(() => {});+ }+}++function reader<T, U>(native: NativeReader<T>, decode: (value: T) => U): AsyncIterableIterator<U> {+ let isClosed = false;+ let tail: Promise<void> = Promise.resolve();+ return {+ [Symbol.asyncIterator]() { return this; },+ next() {+ const result = tail.then(async (): Promise<IteratorResult<U>> => {+ if (isClosed) return { done: true, value: undefined };+ try {+ const value = await native.next();+ if (isClosed) return { done: true, value: undefined };+ if (value === null) {+ isClosed = true;+ native.close();+ return { done: true, value: undefined };+ }+ return { done: false, value: decode(value) };+ } catch (error) {+ if (isClosed) return { done: true, value: undefined };+ isClosed = true;+ native.close();+ throw publicError(error);+ }+ });+ tail = result.then(() => {}, () => {});+ return result;+ },+ async return() {+ isClosed = true;+ native.close();+ return { done: true, value: undefined };+ },+ async throw(error: unknown) {+ isClosed = true;+ native.close();+ throw error;+ },+ };+}++function callMethods(call: NativeCall, isModel: boolean) {+ let isComplete = false;+ async function respond(work: PromiseLike<unknown>): Promise<void> {+ if (isComplete) {+ void Promise.resolve(work).catch(() => {});+ throw new LibsyError("Call has already been completed");+ }+ isComplete = true;+ const reply = Promise.resolve(work).then((value) => {+ if (!isModel) return json({ kind: "decision", response: value });+ if (value instanceof LlmResponse.Agg) return json({ kind: "agg", response: value.response });+ if (value instanceof LlmResponse.Stream) {+ const input = call.input();+ void pump(value.stream, input).catch(() => {});+ return json({ kind: "stream" });+ }+ throw new TypeError("Expected LlmResponse.Agg or LlmResponse.Stream");+ }).catch(failure);+ try {+ await call.respond(reply);+ } catch (error) {+ throw publicError(error);+ }+ }+ return {+ respond,+ fail(error: Error): Promise<void> {+ if (!(error instanceof Error)) throw new TypeError("error must be an Error");+ return respond(Promise.reject(error));+ },+ };+}++function decodeStep(step: NativeStep): AlgorithmStep {+ const data = JSON.parse(step.data);+ if (step.kind === "model") {+ return new Step.CallModel(Object.freeze({ ...data, ...callMethods(step.call!, true) }));+ }+ if (step.kind === "decision") {+ return new Step.CallDecision(Object.freeze({ ...data, ...callMethods(step.call!, false) }));+ }+ if (data.response?.kind === "agg") data.response = new LlmResponse.Agg(data.response.response);+ else if (data.response?.kind === "stream") {+ data.response = new LlmResponse.Stream(reader(step.stream!, (json) => JSON.parse(json)));+ }+ return new Step.Done(Object.freeze(data));+}++let configured: (native: NativeConfigured) => ConfiguredAlgorithm;++export class AlgorithmFactory {+ readonly #native: NativeFactory;++ private constructor(source: string) {+ this.#native = new native.NativeAlgorithmFactory(source);+ }++ static fromToml(source: string): AlgorithmFactory {+ try {+ return new AlgorithmFactory(source);+ } catch (error) {+ throw publicError(error);+ }+ }++ build(): Record<string, ConfiguredAlgorithm> {+ try {+ return Object.fromEntries(Object.entries(this.#native.build()).map(([id, route]) => [id, configured(route)]));+ } catch (error) {+ throw publicError(error);+ }+ }+}++export class ConfiguredAlgorithm {+ readonly #native: NativeConfigured;++ private constructor(native: NativeConfigured) {+ this.#native = native;+ }++ static {+ configured = (native) => new ConfiguredAlgorithm(native);+ }++ runStream(request: JsonObject, options: { headers?: Record<string, string> } = {}): AsyncIterableIterator<AlgorithmStep> {+ try {+ return reader(this.#native.start(json(request), options.headers === undefined ? null : json(options.headers)), decodeStep);+ } catch (error) {+ throw publicError(error);+ }+ }+}diff --git a/bindings/typescript/src/index.ts b/bindings/typescript/src/index.tsindex 699ba12ba..a75f16db1 100644--- a/bindings/typescript/src/index.ts+++ b/bindings/typescript/src/index.ts@@ -3,6 +3,7 @@import { createRequire } from "node:module";+export * from "./algorithm.js";export { TranslationEngine, TranslationError, WireFormat } from "./translation.js";/** Opaque routing state. Store it unchanged and pass it back on the same branch. */diff --git a/bindings/typescript/test/algorithm.test.mjs b/bindings/typescript/test/algorithm.test.mjsnew file mode 100644index 000000000..b8d4dbf37--- /dev/null+++ b/bindings/typescript/test/algorithm.test.mjs@@ -0,0 +1,260 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++import assert from "node:assert/strict";+import { createServer } from "node:http";+import { test } from "node:test";+import { AlgorithmFactory, ConfiguredAlgorithm, LibsyError, LlmResponse, Step } from "../dist/index.js";++const deployment = `+schema_version = 1+[llm_clients.answers]+format = "openai_chat"+base_url = "https://example.test/v1"+api_key_env = "SWITCHYARD_FACTORY_TEST_KEY_THAT_IS_NOT_SET"+[targets]+fast = { id = "upstream/fast", llm_client = "answers" }+capable = { id = "upstream/capable", llm_client = "answers" }+[decision_clients.judge]+format = "openai_decisions"+endpoint = "https://example.test/v1/decisions"+api_key_env = "SWITCHYARD_FACTORY_TEST_KEY_THAT_IS_NOT_SET"+timeout_ms = 2000+[decision_targets.judge]+id = "upstream/judge"+decision_client = "judge"+[routes.parent]+id = "public/main"+type = "passthrough"+target = "fast"+[routes.parent.subagents]+type = "passthrough"+target = "capable"+[routes.classifier]+id = "public/classify"+type = "llm_classifier"+classifier_target = "judge"+strong_target = "capable"+weak_target = "fast"+[routes.advisor]+id = "public/advisor"+type = "advisor"+executor_target = "fast"+advisor_target = "capable"+gate_min_tool_results = 1+`;+const routes = AlgorithmFactory.fromToml(deployment).build();+const request = { model: "public/main", messages: [{ role: "user", content: [{ type: "text", text: "hello" }] }] };+const aggregate = (text = "hello") => ({+ model: "upstream/fast", outputs: [{ role: "assistant", content: [{ type: "text", text }], stop_reason: "end_turn" }],+});+const answer = { answers: { route: { value: {+ type: "choice", data: { selected: "capable", probabilities: { capable: 0.9, efficient: 0.1 } },+} } } };++async function collect(source) {+ const result = [];+ for await (const item of source) result.push(item);+ return result;+}++async function start(t) {+ const run = routes["public/advisor"].runStream(request);+ t.after(() => run.return());+ const step = (await run.next()).value;+ assert.ok(step instanceof Step.CallModel);+ return { run, call: step.call };+}++test("factory returns logical targets, full outcomes, and header-driven subagent selection", async () => {+ assert.deepEqual(Object.keys(routes), ["public/advisor", "public/classify", "public/main"]);+ assert.ok(routes["public/main"] instanceof ConfiguredAlgorithm);+ for (const [headers, expected] of [[undefined, "fast"], [{ "x-switchyard-is-subagent": "true" }, "capable"]]) {+ const steps = await collect(routes["public/main"].runStream(request, { headers }));+ assert.equal(steps.length, 1);+ assert.ok(steps[0] instanceof Step.Done);+ const outcome = steps[0].outcome;+ assert.deepEqual(outcome.selected_model_ids, [expected]);+ assert.equal(outcome.request.model, expected);+ assert.deepEqual(outcome.request.messages, request.messages);+ assert.equal(outcome.fallback_request, null);+ assert.deepEqual(outcome.candidate_requests, {});+ assert.equal(outcome.response, null);+ assert.equal(typeof outcome.metadata.outcome_id, "string");+ }+ assert.throws(() => AlgorithmFactory.fromToml("invalid"), { code: "ERR_CONFIG" });+ assert.throws(() => routes["public/main"].runStream({ messages: [{ role: "invalid", content: [] }] }), { code: "ERR_INVALID_REQUEST" });+});++test("factory rejects configuration requiring Runner admission", () => {+ assert.throws(() => AlgorithmFactory.fromToml(`${deployment}\n[routes.parent.eligibility]\n`).build(), {+ code: "ERR_CONFIG", message: /requires Runner admission/,+ });+});++test("decision calls accept async replies and failures", async (t) => {+ for (const fail of [false, true]) {+ const run = routes["public/classify"].runStream(request);+ t.after(() => run.return());+ const step = (await run.next()).value;+ assert.ok(step instanceof Step.CallDecision);+ assert.equal(step.call.model, "upstream/judge");+ assert.equal(step.call.request.model, "upstream/judge");+ if (fail) await step.call.fail(new Error("judge unavailable"));+ else await step.call.respond(Promise.resolve(answer));+ await assert.rejects(step.call.respond(Promise.resolve(answer)), LibsyError);+ const done = (await run.next()).value;+ assert.ok(done instanceof Step.Done);+ assert.deepEqual(done.outcome.selected_model_ids, ["capable", "fast"]);+ assert.equal(done.outcome.metadata.evidence.source, fail ? "fail_open" : "decision_classifier");+ }+});++test("model calls accept normalized aggregates and response streams", async (t) => {+ const events = [+ { preservation: null, normalized: [{ MessageStart: { id: "answer-1", model: "upstream/fast" } }] },+ { preservation: null, normalized: [{ TextDelta: { index: 0, text: "hello" } }] },+ { preservation: null, normalized: [{ Usage: {+ input_tokens: 2, cached_input_tokens: null, cache_creation_input_tokens: null,+ output_tokens: 1, total_tokens: 3, reasoning_tokens: null,+ } }] },+ { preservation: null, normalized: [{ MessageStop: { reason: "end_turn" } }] },+ ];+ for (const isStream of [false, true]) {+ const { run, call } = await start(t);+ assert.deepEqual(call.models, ["fast"]);+ assert.equal(call.request.model, "fast");+ await call.respond(Promise.resolve(isStream+ ? new LlmResponse.Stream((async function* () { yield* events; })())+ : new LlmResponse.Agg(aggregate())));+ let isPolled = false;+ const duplicate = new LlmResponse.Stream({ async *[Symbol.asyncIterator]() { isPolled = true; } });+ await assert.rejects(call.respond(Promise.resolve(duplicate)), LibsyError);+ assert.equal(isPolled, false);+ const step = (await run.next()).value;+ assert.ok(step instanceof Step.Done);+ const outcome = step.outcome;+ if (isStream) {+ assert.ok(outcome.response instanceof LlmResponse.Stream);+ const replayed = await collect(outcome.response.stream);+ assert.deepEqual(replayed.map((event) => event.normalized), events.map((event) => event.normalized));+ } else {+ assert.ok(outcome.response instanceof LlmResponse.Agg);+ assert.equal(outcome.response.response.outputs[0].content[0].text, "hello");+ }+ }+});++test("response stream setup and read failures reach the routing iterator", { timeout: 5000 }, async (t) => {+ for (const isSetupFailure of [true, false]) {+ const { run, call } = await start(t);+ const source = isSetupFailure+ ? { [Symbol.asyncIterator]() { throw new Error("stream unavailable"); } }+ : { async *[Symbol.asyncIterator]() { throw new Error("stream unavailable"); } };+ const rejected = assert.rejects(run.next(), {+ name: "LibsyError", code: "ERR_ROUTING", message: /stream unavailable/,+ });+ await call.respond(Promise.resolve(new LlmResponse.Stream(source)));+ await rejected;+ }+});++test("non-recovering model responders reject with the original host failure", { timeout: 5000 }, async (t) => {+ for (const fail of [false, true]) {+ const { call } = await start(t);+ const cause = new Error("executor unavailable");+ await assert.rejects(fail ? call.fail(cause) : call.respond(Promise.reject(cause)), {+ name: "LibsyError", code: "ERR_ROUTING", message: /executor unavailable/,+ });+ }+});++test("concurrent model calls keep replies on their originating runs", { timeout: 5000 }, async (t) => {+ const pending = await Promise.all([start(t), start(t)]);+ const work = [Promise.withResolvers(), Promise.withResolvers()];+ const responding = pending.map(({ call }, index) => call.respond(work[index].promise));+ const outcomes = pending.map(({ run }) => run.next());+ for (const [index, text] of [[1, "second"], [0, "first"]]) {+ work[index].resolve(new LlmResponse.Agg(aggregate(text)));+ await responding[index];+ const step = (await outcomes[index]).value;+ assert.ok(step instanceof Step.Done);+ assert.equal(step.outcome.response.response.outputs[0].content[0].text, text);+ }+ for (const { run } of pending) assert.deepEqual(await run.next(), { done: true, value: undefined });+});++test("iterator return releases native waits and closes input readers without cancelling host promises", { timeout: 5000 }, async (t) => {+ const { run: concurrent, call: first } = await start(t);+ const p1 = concurrent.next();+ const p2 = concurrent.next();+ await first.respond(Promise.resolve(new LlmResponse.Agg(aggregate())));+ assert.ok((await p1).value instanceof Step.Done);+ assert.deepEqual(await p2, { done: true, value: undefined });++ const { run, call } = await start(t);+ const work = Promise.withResolvers();+ let isHostFinished = false;+ void work.promise.then(() => { isHostFinished = true; }, () => { isHostFinished = true; });+ const responding = call.respond(work.promise);+ await run.return();+ await responding;+ assert.equal(isHostFinished, false);+ work.reject(new Error("late host failure"));+ await Promise.resolve();+ assert.equal(isHostFinished, true);++ const { run: streamRun, call: streamCall } = await start(t);+ const entered = Promise.withResolvers();+ const returned = Promise.withResolvers();+ const pending = Promise.withResolvers();+ const source = {+ [Symbol.asyncIterator]() { return this; },+ next() { entered.resolve(); return pending.promise; },+ return() { returned.resolve(); return Promise.resolve({ done: true }); },+ };+ await streamCall.respond(Promise.resolve(new LlmResponse.Stream(source)));+ await entered.promise;+ const next = streamRun.next();+ await streamRun.return();+ assert.equal((await next).done, true);+ await returned.promise;+ pending.resolve({ done: true });+});++test("host-owned AbortController closes HTTP work while a concurrent route survives", { timeout: 5000 }, async (t) => {+ const entered = Promise.withResolvers();+ const closed = Promise.withResolvers();+ const controller = new AbortController();+ const server = createServer((_req, res) => {+ res.once("close", closed.resolve);+ entered.resolve();+ });+ t.after(async () => {+ controller.abort();+ const closing = new Promise((resolve) => server.close(resolve));+ server.closeAllConnections();+ await closing;+ });+ await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));+ const base = `http://127.0.0.1:${server.address().port}`;+ const cancelled = await start(t);+ const work = fetch(base, { signal: controller.signal })+ .then((response) => response.json()).then((response) => new LlmResponse.Agg(response));+ const responding = cancelled.call.respond(work).finally(async () => {+ controller.abort();+ await work.catch(() => {});+ });+ await entered.promise;+ const healthy = await start(t);+ const healthyNext = healthy.run.next();+ await cancelled.run.return();+ await responding;+ await closed.promise;+ await healthy.call.respond(Promise.resolve(new LlmResponse.Agg(aggregate("healthy"))));+ const done = (await healthyNext).value;+ assert.ok(done instanceof Step.Done);+ assert.equal(done.outcome.response.response.outputs[0].content[0].text, "healthy");+ assert.deepEqual(await healthy.run.next(), { done: true, value: undefined });+ assert.deepEqual(await cancelled.run.next(), { done: true, value: undefined });+});diff --git a/crates/switchyard-node/Cargo.toml b/crates/switchyard-node/Cargo.tomlindex b35608ebf..f5b6ba67a 100644--- a/crates/switchyard-node/Cargo.toml+++ b/crates/switchyard-node/Cargo.toml@@ -19,12 +19,15 @@ crate-type = ["cdylib"]napi = { version = "3", default-features = false, features = ["napi8", "tokio_rt", "dyn-symbols"] }napi-derive = "3"futures.workspace = true+http.workspace = trueparking_lot.workspace = trueserde.workspace = trueserde_json.workspace = trueswitchyard-libsy.workspace = trueswitchyard-protocol.workspace = true+switchyard-runner.workspace = trueswitchyard-translation.workspace = true+tokio.workspace = true[build-dependencies]napi-build = "2"diff --git a/crates/switchyard-node/src/algorithm.rs b/crates/switchyard-node/src/algorithm.rsnew file mode 100644index 000000000..432ea6b3d--- /dev/null+++ b/crates/switchyard-node/src/algorithm.rs@@ -0,0 +1,556 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++//! Configured routing steps and normalized responses for host-owned execution.++use std::collections::BTreeMap;+use std::pin::Pin;+use std::sync::Arc;+use std::sync::atomic::{AtomicBool, Ordering};++use futures::future::{AbortHandle, Abortable};+use futures::{Stream, StreamExt};+use http::{HeaderMap, HeaderName, HeaderValue};+use napi::bindgen_prelude::Promise;+use napi_derive::napi;+use parking_lot::Mutex;+use serde::{Deserialize, Serialize};+use serde_json::Value;+use switchyard_libsy::{+ Algorithm, CallDecision, CallModel, LibsyError, RuntimeModels, State, Step,+};+use switchyard_protocol::{+ AggLlmResponse, DecisionRequest, DecisionResponse, LlmClientError, LlmRequest, LlmResponse,+ LlmResponseStream, LlmResponseStreamEvent, Metadata, ModelId, Request, Response,+};+use switchyard_runner::AlgorithmFactory;+use tokio::sync::mpsc;++use crate::{error, parse, routing_error};++#[napi]+pub struct NativeAlgorithmFactory {+ inner: AlgorithmFactory,+}++#[napi]+impl NativeAlgorithmFactory {+ #[napi(constructor)]+ pub fn new(source: String) -> napi::Result<Self> {+ Ok(Self {+ inner: AlgorithmFactory::from_toml(&source).map_err(|err| error("ERR_CONFIG", err))?,+ })+ }++ #[napi]+ pub fn build(&self) -> napi::Result<BTreeMap<String, NativeConfiguredAlgorithm>> {+ Ok(self+ .inner+ .build()+ .map_err(|err| error("ERR_CONFIG", err))?+ .into_iter()+ .map(|(id, (algorithm, models))| {+ (+ id,+ NativeConfiguredAlgorithm {+ algorithm,+ models: Arc::new(models),+ },+ )+ })+ .collect())+ }+}++#[napi]+pub struct NativeConfiguredAlgorithm {+ algorithm: Arc<dyn Algorithm>,+ models: Arc<RuntimeModels>,+}++#[napi]+impl NativeConfiguredAlgorithm {+ #[napi]+ pub fn start(+ &self,+ request_json: String,+ headers_json: Option<String>,+ ) -> napi::Result<NativeAlgorithmRun> {+ let headers: Option<BTreeMap<String, String>> =+ headers_json.as_deref().map(parse).transpose()?;+ let headers = headers+ .map(|headers| {+ let mut result = HeaderMap::new();+ for (name, value) in headers {+ let name = HeaderName::from_bytes(name.as_bytes())+ .map_err(|err| error("ERR_INVALID_REQUEST", err))?;+ let value = HeaderValue::from_str(&value)+ .map_err(|err| error("ERR_INVALID_REQUEST", err))?;+ result+ .try_append(name, value)+ .map_err(|err| error("ERR_INVALID_REQUEST", err))?;+ }+ Ok::<_, napi::Error>(result)+ })+ .transpose()?;+ let mut llm_request = parse(&request_json)?;+ switchyard_translation::normalize_compaction(&mut llm_request, headers.as_ref());+ let request = Request {+ llm_request,+ raw_request: None,+ metadata: headers.map(|headers| Metadata::from_headers(&headers)),+ };+ let algorithm = Arc::clone(&self.algorithm);+ let models = Arc::clone(&self.models);+ // Algorithm startup must run on napi's Tokio runtime.+ let stream = futures::stream::once(async move {+ algorithm.run_stream(request, models, State::default())+ })+ .flatten()+ .map(|step| step.map_err(routing_error));+ Ok(NativeAlgorithmRun {+ reader: Reader::new(stream),+ })+ }+}++type Items<T> = Pin<Box<dyn Stream<Item = napi::Result<T>> + Send>>;++struct ReaderSlot<T> {+ stream: Option<Items<T>>,+ abort: Option<AbortHandle>,+}++struct Reader<T> {+ slot: Mutex<ReaderSlot<T>>,+}++impl<T: Send + 'static> Reader<T> {+ fn new(stream: impl Stream<Item = napi::Result<T>> + Send + 'static) -> Self {+ Self {+ slot: Mutex::new(ReaderSlot {+ stream: Some(Box::pin(stream)),+ abort: None,+ }),+ }+ }++ async fn next(&self) -> napi::Result<Option<T>> {+ let (abort, registration) = AbortHandle::new_pair();+ let mut stream = {+ let mut slot = self.slot.lock();+ let stream = match slot.stream.take() {+ Some(stream) => stream,+ None if slot.abort.is_none() => return Ok(None),+ None => return Err(error("ERR_INVALID_REQUEST", "next is already pending")),+ };+ slot.abort = Some(abort.clone());+ stream+ };+ // close() aborts the future that owns the reader, including a pending next().+ let (stream, item) = Abortable::new(+ async move {+ let item = stream.next().await;+ (stream, item)+ },+ registration,+ )+ .await+ .map_err(|_| error("ABORT_ERR", "Routing cancelled"))?;+ let mut slot = self.slot.lock();+ if abort.is_aborted() {+ return Err(error("ABORT_ERR", "Routing cancelled"));+ }+ slot.abort = None;+ if item.as_ref().is_some_and(Result::is_ok) {+ slot.stream = Some(stream);+ }+ item.transpose()+ }++ fn close(&self) {+ let mut slot = self.slot.lock();+ if let Some(abort) = slot.abort.take() {+ abort.abort();+ }+ slot.stream = None;+ }+}++#[napi]+pub struct NativeAlgorithmRun {+ reader: Reader<Step>,+}++#[napi(object, object_from_js = false, use_nullable = true)]+pub struct NativeAlgorithmStep {+ pub kind: String,+ pub data: String,+ pub call: Option<NativeAlgorithmCall>,+ pub stream: Option<NativeResponseStream>,+}++#[napi]+impl NativeAlgorithmRun {+ #[napi]+ pub async fn next(&self) -> napi::Result<Option<NativeAlgorithmStep>> {+ self.reader.next().await?.map(step_to_native).transpose()+ }++ #[napi]+ pub fn close(&self) {+ self.reader.close();+ }+}++enum HostCall {+ Model(Box<CallModel>),+ Decision(Box<CallDecision>),+}++type InputItem = Option<Result<LlmResponseStreamEvent, LlmClientError>>;++#[napi]+pub struct NativeAlgorithmCall {+ inner: Mutex<Option<HostCall>>,+ input: Mutex<Option<mpsc::Receiver<InputItem>>>,+ model: ModelId,+ is_complete: AtomicBool,+}++#[derive(Deserialize)]+#[serde(tag = "kind", rename_all = "snake_case")]+enum Reply {+ Agg {+ response: AggLlmResponse,+ },+ Stream,+ Decision {+ response: DecisionResponse,+ },+ Error {+ error: String,+ #[serde(default)]+ context_window_exceeded: bool,+ },+}++#[derive(Debug)]+struct ForeignError(String);++impl std::fmt::Display for ForeignError {+ fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ formatter.write_str(&self.0)+ }+}++impl std::error::Error for ForeignError {}++fn client_error(model: &ModelId, message: String, is_context_window: bool) -> LlmClientError {+ if is_context_window {+ LlmClientError::ContextWindowExceeded {+ model: model.clone(),+ message,+ }+ } else {+ LlmClientError::Ffi {+ source: Box::new(ForeignError(message)),+ }+ }+}++#[napi]+impl NativeAlgorithmCall {+ #[napi]+ pub fn input(&self) -> napi::Result<NativeInputStream> {+ let mut input = self.input.lock();+ if self.is_complete.load(Ordering::Acquire) {+ return Err(error("ABORT_ERR", "Call is no longer waiting"));+ }+ if input.is_some() {+ return Err(error(+ "ERR_INVALID_REQUEST",+ "Response stream already supplied",+ ));+ }+ let (sender, receiver) = mpsc::channel(1);+ *input = Some(receiver);+ Ok(NativeInputStream {+ sender,+ model: self.model.clone(),+ })+ }++ #[napi]+ pub async fn respond(&self, work: Promise<String>) -> napi::Result<()> {+ let call = self+ .inner+ .lock()+ .take()+ .ok_or_else(|| routing_error("Call has already been completed"))?;+ let model = &self.model;+ let work = async {+ let json = work.await.map_err(|err| {+ LibsyError::client_call(model.clone(), client_error(model, err.to_string(), false))+ })?;+ let reply: Reply = serde_json::from_str(&json).map_err(|err| {+ LibsyError::client_call(model.clone(), client_error(model, err.to_string(), false))+ })?;+ if let Reply::Error {+ error,+ context_window_exceeded,+ } = reply+ {+ return Err(LibsyError::client_call(+ model.clone(),+ client_error(model, error, context_window_exceeded),+ ));+ }+ Ok(reply)+ };+ let result = match call {+ HostCall::Decision(call) => {+ call.respond(async {+ match work.await? {+ Reply::Decision { response } => Ok(response),+ _ => Err(LibsyError::client_call(+ model.clone(),+ client_error(model, "Expected a decision response".into(), false),+ )),+ }+ })+ .await+ }+ HostCall::Model(call) => {+ let metadata = call.request.metadata.clone();+ call.respond(async {+ let llm_response = match work.await? {+ Reply::Agg { response } => LlmResponse::Agg(response),+ Reply::Stream => {+ let receiver = self.input.lock().take().ok_or_else(|| {+ LibsyError::client_call(+ model.clone(),+ client_error(model, "Response stream is missing".into(), false),+ )+ })?;+ let stream = futures::stream::unfold(Some(receiver), |state| async {+ let mut receiver = state?;+ let item = receiver.recv().await??;+ let state = item.is_ok().then_some(receiver);+ Some((item, state))+ });+ LlmResponse::Stream(Box::pin(stream))+ }+ _ => {+ return Err(LibsyError::client_call(+ model.clone(),+ client_error(+ model,+ "Expected LlmResponse.Agg or LlmResponse.Stream".into(),+ false,+ ),+ ));+ }+ };+ Ok(Response {+ llm_response,+ metadata,+ upstream_headers: HeaderMap::new(),+ })+ })+ .await+ }+ };+ self.is_complete.store(true, Ordering::Release);+ self.input.lock().take();+ result.map_err(routing_error)+ }+}++#[napi]+pub struct NativeInputStream {+ sender: mpsc::Sender<InputItem>,+ model: ModelId,+}++#[napi]+impl NativeInputStream {+ #[napi]+ pub async fn push(&self, json: String) -> napi::Result<()> {+ let event = serde_json::from_str(&json).map_err(routing_error)?;+ self.sender+ .send(Some(Ok(event)))+ .await+ .map_err(|_| error("ABORT_ERR", "Response stream closed"))+ }++ #[napi]+ pub async fn finish(+ &self,+ message: Option<String>,+ is_context_window: bool,+ ) -> napi::Result<()> {+ let item =+ message.map(|message| Err(client_error(&self.model, message, is_context_window)));+ self.sender+ .send(item)+ .await+ .map_err(|_| error("ABORT_ERR", "Response stream closed"))+ }++ #[napi]+ pub async fn closed(&self) {+ self.sender.closed().await;+ }+}++#[napi]+pub struct NativeResponseStream {+ reader: Reader<LlmResponseStreamEvent>,+}++impl NativeResponseStream {+ fn new(stream: LlmResponseStream) -> Self {+ Self {+ reader: Reader::new(stream.map(|item| item.map_err(routing_error))),+ }+ }+}++#[napi]+impl NativeResponseStream {+ #[napi]+ pub async fn next(&self) -> napi::Result<Option<String>> {+ self.reader+ .next()+ .await?+ .map(|event| serde_json::to_string(&event).map_err(routing_error))+ .transpose()+ }++ #[napi]+ pub fn close(&self) {+ self.reader.close();+ }+}++#[derive(Serialize)]+#[serde(untagged)]+enum StepPayload<'a> {+ Decision {+ algorithm: &'a str,+ request: &'a DecisionRequest,+ model: &'a ModelId,+ },+ Model {+ algorithm: &'a str,+ request: &'a LlmRequest,+ models: &'a [ModelId],+ },+ Done {+ selected_model_ids: &'a [ModelId],+ request: &'a LlmRequest,+ fallback_request: Option<&'a LlmRequest>,+ candidate_requests: BTreeMap<&'a str, &'a LlmRequest>,+ response: Option<ResponsePayload<'a>>,+ metadata: Option<MetadataPayload<'a>>,+ },+}++#[derive(Serialize)]+#[serde(tag = "kind", rename_all = "snake_case")]+enum ResponsePayload<'a> {+ Agg { response: &'a AggLlmResponse },+ Stream,+}++#[derive(Serialize)]+struct MetadataPayload<'a> {+ outcome_id: &'a str,+ algorithm: &'a str,+ evidence: Option<&'a Value>,+}++fn step_to_native(step: Step) -> napi::Result<NativeAlgorithmStep> {+ let mut result = NativeAlgorithmStep {+ kind: String::new(),+ data: String::new(),+ call: None,+ stream: None,+ };+ match step {+ Step::CallDecision(call) => {+ result.kind = "decision".into();+ result.data = serde_json::to_string(&StepPayload::Decision {+ algorithm: &call.algorithm,+ request: &call.request,+ model: &call.model,+ })+ .map_err(routing_error)?;+ result.call = Some(NativeAlgorithmCall {+ model: call.model.clone(),+ inner: Mutex::new(Some(HostCall::Decision(call))),+ input: Mutex::new(None),+ is_complete: AtomicBool::new(false),+ });+ }+ Step::CallModel(call) => {+ result.kind = "model".into();+ result.data = serde_json::to_string(&StepPayload::Model {+ algorithm: &call.algorithm,+ request: &call.request.llm_request,+ models: &call.models,+ })+ .map_err(routing_error)?;+ result.call = Some(NativeAlgorithmCall {+ model: call+ .request+ .llm_request+ .model+ .as_ref()+ .map(ModelId::new)+ .ok_or_else(|| routing_error("Model call has no selected model"))?,+ inner: Mutex::new(Some(HostCall::Model(call))),+ input: Mutex::new(None),+ is_complete: AtomicBool::new(false),+ });+ }+ Step::Done(outcome) => {+ result.kind = "done".into();+ result.data = serde_json::to_string(&StepPayload::Done {+ selected_model_ids: &outcome.selected_model_ids,+ request: &outcome.request.llm_request,+ fallback_request: outcome+ .fallback_request+ .as_ref()+ .map(|request| &request.llm_request),+ candidate_requests: outcome+ .candidate_requests+ .iter()+ .map(|(id, request)| (id.as_str(), &request.llm_request))+ .collect(),+ response: outcome+ .response+ .as_ref()+ .map(|response| match &response.llm_response {+ LlmResponse::Agg(response) => ResponsePayload::Agg { response },+ LlmResponse::Stream(_) => ResponsePayload::Stream,+ }),+ metadata: outcome.metadata.as_ref().map(|metadata| MetadataPayload {+ outcome_id: metadata.outcome_id(),+ algorithm: &metadata.algorithm,+ evidence: metadata.evidence.as_ref(),+ }),+ })+ .map_err(routing_error)?;+ if let Some(Response {+ llm_response: LlmResponse::Stream(stream),+ ..+ }) = outcome.response+ {+ result.stream = Some(NativeResponseStream::new(stream));+ }+ }+ }+ Ok(result)+}diff --git a/crates/switchyard-node/src/lib.rs b/crates/switchyard-node/src/lib.rsindex 7291e1fd0..32bf423c3 100644--- a/crates/switchyard-node/src/lib.rs+++ b/crates/switchyard-node/src/lib.rs@@ -1,8 +1,9 @@// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.// SPDX-License-Identifier: Apache-2.0-//! Node bindings for composite routing and buffered JSON translation.+//! Node bindings for routing algorithms and buffered JSON translation.+mod algorithm;mod translation;use std::future::ready;