Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

feat: Pi extension now uses Composite router with Decision classifier (#1059)

Graham Kingc9476199cff9
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 63717b1a5..f5dd62e40 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -147,11 +147,9 @@ jobs:
- run: npm --prefix bindings/typescript run build
- run: npm --prefix bindings/typescript run typecheck
- run: npm --prefix bindings/typescript test
- # Re-enable when the Pi extension and its tests use @switchyard/libsy's
- # CompositeRouter, JSON config, and Pi-owned classifier calls.
- # - run: npm --prefix examples/pi ci
- # - run: npm --prefix examples/pi run typecheck
- # - run: npm --prefix examples/pi test
+ - run: npm --prefix examples/pi ci
+ - run: npm --prefix examples/pi run typecheck
+ - run: npm --prefix examples/pi test
typecheck:
name: Typecheck (mypy)
diff --git a/examples/pi/README.md b/examples/pi/README.md
index 59f0d9f8b..2d6eca869 100644
--- a/examples/pi/README.md
+++ b/examples/pi/README.md
@@ -1,42 +1,45 @@
-# Pi virtual model using Switchyard auto
+# Pi virtual model using Switchyard composite routing
-This example registers `switchyard/auto` and asks Switchyard to select a model on
-every request:
+This extension registers `switchyard/auto`. Pi calls a Decision classifier on
+user turns. Switchyard's composite algorithm uses that answer and recent tool
+activity to select a completion model. Thinking stays `medium`.
-- **Efficient:** Luna (`openai-codex/gpt-6-luna`).
-- **Capable:** Sol (`openai-codex/gpt-6.1-sol`).
-- **Thinking:** always `medium`.
-
-`auto` currently uses Switchyard's stage router with an efficient-first default
-and a confidence threshold of 0.5. It reads tool activity and results to choose a
-model. It can escalate to Sol during recovery and return to Luna as work proceeds.
-The same routing applies to user requests, continuations, retries, and direct
-requests such as compaction summaries.
+The fixed policy selects the capable tier when the classifier's `advantage`
+probability exceeds 0.4. Exactly 0.4 selects efficient. Stage routing then picks
+the model on every request, using the classifier's tier as its fallback. Its
+efficient-first and capable-first thresholds are 0.5 and 0.2. A tool failure can
+escalate to the capable model, with a two-request hold during recovery.
## Configuration
-`switchyard.toml` contains just the algorithm settings:
+Create `<agent-dir>/switchyard-router.json`. The agent directory defaults to
+`~/.pi/agent`; `PI_CODING_AGENT_DIR` overrides it.
-```toml
-type = "auto"
-efficient_target = "openai-codex/gpt-6-luna"
-capable_target = "openai-codex/gpt-6.1-sol"
+```json
+{
+ "decision_model": "openrouter/inception/mercury-decide:free",
+ "capable_model": "openai-codex/gpt-6.1-sol",
+ "efficient_model": "openai-codex/gpt-6-luna"
+}
```
-Set either target to a `provider/model` reference available in Pi's catalog, such
-as `openai-codex/gpt-6-astra`. The first slash separates the provider from the model
-ID; `openrouter/vendor/model` keeps `vendor/model` as the ID. Both providers and
-models come from this file. Thinking remains `medium`.
+Use `provider/model` references from Pi's catalog. The first slash separates the
+provider from the model ID, so `openrouter/vendor/model` keeps `vendor/model` as
+the ID. The Decision model must be registered as a classifier in Pi.
+
+The policy's evidence describes candidate a (capable) as a larger,
+reasoning-oriented model for complex work. Candidate b (efficient) is a smaller,
+fast model for simple questions, local reads and edits, routine tests, and
+straightforward fixes. Pick completion models that fit these roles.
-The extension generates the runner's target entries, route ID, and credential-free
-placeholder client. Pi uses the selected model's catalog limits. Set
-`SWITCHYARD_CONFIG` to use another algorithm-settings file, and reload the
-extension after changing its configuration.
+The extension reads the config on its first routing request and caches it.
+Reload the extension after editing the config. Pi owns credentials and all model
+calls, including classifier calls. Classifier and completion calls may cost money.
## Local setup
Requires Node.js 22.19 or newer, the repository's Rust toolchain, and Pi with the
-virtual-model API. Development checks use Pi 1.0.4.
+virtual-model and classifier APIs. Development checks use Pi 1.0.4.
From the repository root:
@@ -48,8 +51,8 @@ npm --prefix examples/pi run typecheck
npm --prefix examples/pi test
```
-The tests exercise the actual auto algorithm locally, check message conversion,
-and load the extension through Pi's public resource loader.
+Tests run the real composite algorithm with a local classifier mock, check
+message conversion, and load the extension through Pi's public resource loader.
Sign in to the configured providers through Pi's `/login` and confirm the models
are in your catalog. Then:
@@ -58,23 +61,27 @@ are in your catalog. Then:
pi -e ./examples/pi/switchyard-router.ts --model switchyard/auto
```
-Completion calls can incur provider charges. The auto routing decision runs
-locally; Pi owns completion credentials and connections.
-
## Conversation handling
The extension passes visible text, tool-call arguments, paired tool results, and
failure flags to Switchyard. Pi keeps the original images and reasoning for the
-chosen completion model. Each decision uses the current branch's transcript,
-including the history available after compaction or resume.
+chosen completion model. The classifier sees the opening user task and latest
+user message. Its instructions come from libsy and are sent to Pi as a JSON
+string.
+
+Pi stores routing state on the session branch. It follows forks and `/tree`, and
+survives compaction, resume, and extension reload. Tool continuations reuse the
+stored tier and capable hold while stage signals select the model. State is
+returned only when it changes.
-The binding uses fresh routing state for each request. Auto's cross-request
-capable hold requires session identity; this extension relies on the recent tool
-history for recovery signals instead. It keeps its routing decisions independent
-across branches.
+Retries reuse stored state. Retrying a user-turn request calls the classifier
+again. Direct requests, such as compaction summaries, start with fresh state;
+Pi discards their returned state. They call the classifier when their last
+message contains user text. Image-only user messages retain the previous tier
+because routing uses text and tool activity.
-Routing errors and cancellation propagate to Pi. The extension checks target
-identities and resolves the selected model through Pi's catalog.
+Classifier failures fail the request, and cancellation propagates to Pi. A valid
+answer without a usable `advantage` probability retains the previous tier.
-For the binding's API and configuration restrictions, see
+For the binding API, see
[`bindings/typescript`](../../bindings/typescript/README.md).
diff --git a/examples/pi/package-lock.json b/examples/pi/package-lock.json
index 51517d8a5..6002c0c4b 100644
--- a/examples/pi/package-lock.json
+++ b/examples/pi/package-lock.json
@@ -9,8 +9,7 @@
"version": "0.3.0",
"license": "Apache-2.0",
"dependencies": {
- "@switchyard/runner": "file:../../bindings/typescript",
- "smol-toml": "1.9.0"
+ "@switchyard/libsy": "file:../../bindings/typescript"
},
"devDependencies": {
"@earendil-works/pi-ai": "1.0.4",
@@ -27,7 +26,7 @@
}
},
"../../bindings/typescript": {
- "name": "@switchyard/runner",
+ "name": "@switchyard/libsy",
"version": "0.3.0",
"license": "Apache-2.0",
"devDependencies": {
@@ -1205,7 +1204,7 @@
"dev": true,
"license": "MIT"
},
- "node_modules/@switchyard/runner": {
+ "node_modules/@switchyard/libsy": {
"resolved": "../../bindings/typescript",
"link": true
},
@@ -2019,18 +2018,6 @@
"dev": true,
"license": "ISC"
},
- "node_modules/smol-toml": {
- "version": "1.9.0",
- "resolved": "https://registry.npmjs.org/smol-toml/-/smol-toml-1.9.0.tgz",
- "integrity": "sha512-hpd+HLON7HdZXqYchMM/+LaTTbdK0AU3NngIJ4KVyWbY9bfQqdL9cD+4yf6dUoU2Ap4VsU0JkQi6FxAI1B2mXQ==",
- "license": "BSD-3-Clause",
- "engines": {
- "node": ">= 18"
- },
- "funding": {
- "url": "https://github.com/sponsors/cyyynthia"
- }
- },
"node_modules/standardwebhooks": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/standardwebhooks/-/standardwebhooks-1.1.1.tgz",
diff --git a/examples/pi/package.json b/examples/pi/package.json
index 4fe7483ed..895a57280 100644
--- a/examples/pi/package.json
+++ b/examples/pi/package.json
@@ -12,8 +12,7 @@
"test": "node --experimental-strip-types --test test/*.test.mjs"
},
"dependencies": {
- "@switchyard/runner": "file:../../bindings/typescript",
- "smol-toml": "1.9.0"
+ "@switchyard/libsy": "file:../../bindings/typescript"
},
"peerDependencies": {
"@earendil-works/pi-ai": "*",
diff --git a/examples/pi/switchyard-router.ts b/examples/pi/switchyard-router.ts
index b9a0caeec..aa322e4fd 100644
--- a/examples/pi/switchyard-router.ts
+++ b/examples/pi/switchyard-router.ts
@@ -2,10 +2,30 @@
// SPDX-License-Identifier: Apache-2.0
import { readFileSync } from "node:fs";
-import { parse, stringify } from "smol-toml";
-import { Runner, type RoutingContent, type RoutingMessage } from "@switchyard/runner";
-import type { Message } from "@earendil-works/pi-ai";
-import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
+import { join } from "node:path";
+import {
+ CompositeRouter, type CompositeOptions, type CompositeState, type DecisionRequest,
+ type DecisionResponse, type RoutingContent, type RoutingMessage,
+} from "@switchyard/libsy";
+import type {
+ ClassifierApi, ClassifierContext, ClassifierModel, ClassifierResult, Message,
+} from "@earendil-works/pi-ai";
+import { getAgentDir, type ExtensionAPI } from "@earendil-works/pi-coding-agent";
+
+const POLICY = {
+ classifier: {
+ classify_trigger: "user_turn",
+ fail_open: false,
+ decision: {
+ cutoff: 0.4,
+ candidates: { a: "capable", b: "efficient" },
+ evidence: {
+ description: "Candidate a is the larger reasoning-oriented coding and thinking model. It is intended for complex problems. Candidate b is the smaller fast coding model. It is intended for simple questions, simple local reads and edits, routine tests and straightforward fixes. These are qualitative deployment roles, not empirical success rates.",
+ },
+ },
+ },
+ stage: { efficient_first_threshold: 0.5, capable_first_threshold: 0.2 },
+} satisfies Omit<CompositeOptions, "targets">;
function modelIdentity(reference: unknown) {
if (typeof reference !== "string") throw new Error("Expected a model in provider/model form");
@@ -15,19 +35,57 @@ function modelIdentity(reference: unknown) {
return { reference, provider: reference.slice(0, slash), id: reference.slice(slash + 1) };
}
-function loadRunner(): Runner {
- const algorithm = parse(readFileSync(process.env.SWITCHYARD_CONFIG ?? new URL("./switchyard.toml", import.meta.url), "utf8"));
- const targets = Object.fromEntries([algorithm.efficient_target, algorithm.capable_target].map((value) => {
- const { reference } = modelIdentity(value);
- return [reference, { id: reference, llm_client: "pi" }];
- }));
- // Keep the provider in the routing identity so equal model IDs on different providers stay distinct.
- return Runner.fromToml(stringify({
- schema_version: 1,
- llm_clients: { pi: { format: "openai_chat", base_url: "http://127.0.0.1:1/v1" } },
- targets,
- routes: { auto: { ...algorithm, id: "switchyard/auto" } },
+function loadRouter() {
+ const path = join(getAgentDir(), "switchyard-router.json");
+ let classifier, capable, efficient;
+ try {
+ const config = JSON.parse(readFileSync(path, "utf8"));
+ classifier = modelIdentity(config.decision_model);
+ capable = modelIdentity(config.capable_model);
+ efficient = modelIdentity(config.efficient_model);
+ } catch (cause) {
+ throw new Error(`Configure ${path} with decision_model, capable_model and efficient_model in provider/model form. See README for full example. ${(cause as Error).message}`, { cause });
+ }
+ return { classifier, router: CompositeRouter.create({
+ ...POLICY,
+ targets: { classifier: classifier.reference, capable: capable.reference, efficient: efficient.reference },
+ }) };
+}
+
+function text(value: unknown): string {
+ return typeof value === "string" ? value : JSON.stringify(value ?? null);
+}
+
+function classifierContext(request: DecisionRequest): ClassifierContext {
+ const questions: ClassifierContext["questions"] = {};
+ for (const [id, question] of Object.entries(request.questions)) {
+ if (question.kind.type !== "choice") throw new Error(`Expected a choice question: ${id}`);
+ questions[id] = {
+ type: "choice",
+ instructions: text(question.instructions),
+ criteria: Object.fromEntries(question.kind.data.options.map((option) => [option.id, text(option.description)])),
+ };
+ }
+ return { state: request.context as ClassifierContext["state"], questions };
+}
+
+function decisionResponse(request: DecisionRequest, result: ClassifierResult): DecisionResponse {
+ if (result.stopReason === "aborted") {
+ throw Object.assign(new Error(result.errorMessage ?? "Classification cancelled"), { name: "AbortError", code: "ABORT_ERR" });
+ }
+ if (result.stopReason === "error") throw new Error(result.errorMessage ?? "Classification failed");
+ const answers = Object.fromEntries(Object.entries(request.questions).map(([id, question]) => {
+ const answer = result.answers[id];
+ if (question.kind.type !== "choice" || answer?.type !== "choice"
+ || !question.kind.data.options.some((option) => option.id === answer.choice)) {
+ throw new Error(`Expected a declared choice for question: ${id}`);
+ }
+ return [id, {
+ value: { type: "choice" as const, data: { selected: answer.choice, probabilities: answer.probabilities } },
+ provider_confidence: answer.confidence,
+ }];
}));
+ return { model: `${result.provider}/${result.model}`, answers };
}
function routingMessages(messages: readonly Message[]): RoutingMessage[] {
@@ -57,19 +115,32 @@ function routingMessages(messages: readonly Message[]): RoutingMessage[] {
}
export default function (pi: ExtensionAPI) {
- let runner: Runner | undefined;
- pi.registerVirtualModel({
+ let loaded: ReturnType<typeof loadRouter> | undefined;
+ let classifierModel: ClassifierModel<ClassifierApi> | undefined;
+ pi.registerVirtualModel<CompositeState>({
provider: "switchyard",
id: "auto",
name: "Auto (Switchyard)",
thinkingLevels: ["medium"],
async route(request, ctx) {
- runner ??= loadRunner();
- const decision = await runner.decide("switchyard/auto", routingMessages(request.messages), { signal: request.signal });
- const identity = modelIdentity(decision.model);
+ const { router, classifier } = loaded ??= loadRouter();
+ const judge = classifierModel ??= ctx.modelRegistry.findOfType("classifier", classifier.provider, classifier.id);
+ if (!judge) throw new Error(`Unknown classifier: ${classifier.reference}`);
+ const selection = await router.route(routingMessages(request.messages), {
+ state: request.state,
+ signal: request.signal,
+ decide: async (decision) => decisionResponse(decision, await ctx.modelRegistry.classify(
+ judge, classifierContext(decision), { signal: request.signal },
+ )),
+ }).catch((error) => {
+ // The binding wraps classifier errors as routing failures; preserve cancellation for Pi.
+ if (error.cause?.name === "AbortError") throw error.cause;
+ throw error;
+ });
+ const identity = modelIdentity(selection.model);
const model = ctx.modelRegistry.find(identity.provider, identity.id);
if (!model) throw new Error(`Model ${identity.reference} is not in the catalog`);
- return { model, thinkingLevel: "medium" };
+ return { model, thinkingLevel: "medium", state: selection.state };
},
});
}
diff --git a/examples/pi/switchyard.toml b/examples/pi/switchyard.toml
deleted file mode 100644
index 7fbd91dc6..000000000
--- a/examples/pi/switchyard.toml
+++ /dev/null
@@ -1,6 +0,0 @@
-# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
-# SPDX-License-Identifier: Apache-2.0
-
-type = "auto"
-efficient_target = "openai-codex/gpt-6-luna"
-capable_target = "openai-codex/gpt-6.1-sol"
diff --git a/examples/pi/test/router.test.mjs b/examples/pi/test/router.test.mjs
index 77ca88a9a..1f51f5e58 100644
--- a/examples/pi/test/router.test.mjs
+++ b/examples/pi/test/router.test.mjs
@@ -2,20 +2,48 @@
// SPDX-License-Identifier: Apache-2.0
import assert from "node:assert/strict";
+import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
import { test } from "node:test";
-import { Runner } from "@switchyard/runner";
+import { CompositeRouter } from "@switchyard/libsy";
import extension from "../switchyard-router.ts";
-const context = { modelRegistry: { find: (provider, id) => ({ provider, id }) } };
+const config = {
+ decision_model: "openrouter/inception/mercury-decide:free",
+ capable_model: "openai-codex/gpt-6.1-sol",
+ efficient_model: "openai-codex/gpt-6-luna",
+};
const request = (fields = {}) => ({
reason: "user", messages: [{ role: "user", content: "task" }],
thinkingLevel: "high", signal: new AbortController().signal, ...fields,
});
+const result = (advantage = 0.6) => ({
+ provider: "openrouter", model: "inception/mercury-decide:free", stopReason: "stop",
+ answers: { route: {
+ type: "choice", choice: advantage > 0.5 ? "advantage" : "no_advantage", probabilities: { advantage, no_advantage: 1 - advantage }, confidence: 0.9,
+ } },
+});
-function setup() {
+function setup(t, answer = result()) {
+ const dir = mkdtempSync(join(tmpdir(), "switchyard-router-"));
+ const previous = process.env.PI_CODING_AGENT_DIR;
+ process.env.PI_CODING_AGENT_DIR = dir;
+ t.after(() => {
+ if (previous === undefined) delete process.env.PI_CODING_AGENT_DIR;
+ else process.env.PI_CODING_AGENT_DIR = previous;
+ rmSync(dir, { recursive: true, force: true });
+ });
+ const path = join(dir, "switchyard-router.json");
+ writeFileSync(path, JSON.stringify(config));
+ const registry = {
+ find: t.mock.fn((provider, id) => ({ provider, id })),
+ findOfType: t.mock.fn((type, provider, id) => ({ type, provider, id })),
+ classify: t.mock.fn(async () => answer),
+ };
let definition;
extension({ registerVirtualModel: (value) => { definition = value; } });
- return { definition, route: (req) => definition.route(req, context) };
+ return { definition, registry, path, route: (req) => definition.route(req, { modelRegistry: registry }) };
}
function tool(id, name, args, text, isError = false) {
@@ -25,25 +53,125 @@ function tool(id, name, args, text, isError = false) {
];
}
-test("auto uses Luna, escalates to Sol on tool failure, and returns to Luna on recovery", async () => {
- const { route, definition } = setup();
+test("Pi classifies the libsy question and retains the capable tier on a read continuation", async (t) => {
+ const { route, definition, registry } = setup(t);
assert.deepEqual(definition.thinkingLevels, ["medium"]);
- const messages = request().messages;
- const initial = await route(request({ messages }));
- assert.equal(initial.model.id, "gpt-6-luna");
+ const req = request();
+ const initial = await route(req);
+ assert.deepEqual(initial.model, { provider: "openai-codex", id: "gpt-6.1-sol" });
assert.equal(initial.thinkingLevel, "medium");
+ assert.ok(initial.state);
+ assert.equal(registry.findOfType.mock.callCount(), 1);
+ assert.deepEqual(registry.findOfType.mock.calls[0].arguments, ["classifier", "openrouter", "inception/mercury-decide:free"]);
+ assert.equal(registry.classify.mock.callCount(), 1);
+ const [judge, context, options] = registry.classify.mock.calls[0].arguments;
+ assert.equal(judge, registry.findOfType.mock.calls[0].result);
+ assert.deepEqual(options, { signal: req.signal });
+ assert.deepEqual(context.state.comparison, { capable: "a", efficient: "b" });
+ assert.equal(context.state.evidence.description, "Candidate a is the larger reasoning-oriented coding and thinking model. It is intended for complex problems. Candidate b is the smaller fast coding model. It is intended for simple questions, simple local reads and edits, routine tests and straightforward fixes. These are qualitative deployment roles, not empirical success rates.");
+ assert.deepEqual(context.state.task, [{ role: "user", content: [{ type: "text", text: "task" }] }]);
+ assert.equal(context.questions.route.type, "choice");
+ assert.equal(typeof context.questions.route.instructions, "string");
+ assert.deepEqual(JSON.parse(context.questions.route.instructions), JSON.parse(readFileSync(new URL(
+ "../../../crates/libsy/src/prompts/capability-classifier/relative_advantage.json", import.meta.url,
+ ), "utf8")));
+ assert.deepEqual(Object.keys(context.questions.route.criteria), ["advantage", "no_advantage"]);
+ const continued = await route(request({
+ reason: "continuation", state: initial.state,
+ messages: [...req.messages, ...tool("read", "read", { path: "parser.rs" }, "fn parse() {}")],
+ }));
+ assert.deepEqual(continued, { model: initial.model, thinkingLevel: "medium", state: undefined });
+ assert.equal(registry.findOfType.mock.callCount(), 1);
+ assert.equal(registry.classify.mock.callCount(), 1);
+});
+
+test("cutoff equality selects efficient, then a tool failure holds capable for two requests", async (t) => {
+ const { route, registry } = setup(t, result(0.4));
+ const messages = request().messages;
+ let selected = await route(request({ messages }));
+ assert.equal(selected.model.id, "gpt-6-luna");
messages.push(...tool("failed", "bash", { command: "pytest" }, "out of memory", true));
- const escalated = await route(request({ reason: "continuation", messages }));
- assert.equal(escalated.model.id, "gpt-6.1-sol");
+ selected = await route(request({ reason: "continuation", messages, state: selected.state }));
+ assert.equal(selected.model.id, "gpt-6.1-sol");
+ assert.deepEqual(selected.state.capable_hold_turns, { count: 2 });
for (let i = 0; i < 3; i++) messages.push(...tool(`write-${i}`, "write", { path: `file-${i}` }, "wrote file"));
- const recovered = await route(request({ reason: "retry", messages }));
- assert.equal(recovered.model.id, "gpt-6-luna");
+ let state = selected.state;
+ for (const expected of ["gpt-6.1-sol", "gpt-6.1-sol", "gpt-6-luna"]) {
+ selected = await route(request({ reason: "continuation", messages, state }));
+ state = selected.state ?? state;
+ assert.equal(selected.model.id, expected);
+ }
+ assert.equal(registry.classify.mock.callCount(), 1);
+});
+
+test("classifier lookup, answer validation, failures, and cancellation", async (t) => {
+ const { route, registry } = setup(t);
+ for (const [answer, expected] of [
+ [{ stopReason: "error", errorMessage: "classifier unavailable" }, /classifier unavailable/],
+ [{ stopReason: "aborted" }, { name: "AbortError", code: "ABORT_ERR" }],
+ ...[undefined, { type: "bool", probability: 0.6 }, { type: "choice", choice: "other" }].map((answer) => [
+ { ...result(), answers: { route: answer } }, /Expected a declared choice for question: route/,
+ ]),
+ ]) {
+ registry.classify.mock.mockImplementation(async () => answer);
+ await assert.rejects(route(request()), expected);
+ }
+ const calls = registry.classify.mock.callCount();
+ await assert.rejects(route(request({ signal: AbortSignal.abort() })), { name: "AbortError", code: "ABORT_ERR" });
+ assert.equal(registry.classify.mock.callCount(), calls);
+ assert.equal(registry.findOfType.mock.callCount(), 1);
+});
+
+test("a missing classifier fails until it appears, then the lookup is cached", async (t) => {
+ const { route, registry } = setup(t);
+ registry.findOfType.mock.mockImplementationOnce(() => undefined);
+ await assert.rejects(route(request()), { message: `Unknown classifier: ${config.decision_model}` });
+ assert.equal(registry.classify.mock.callCount(), 0);
+ await route(request());
+ await route(request());
+ assert.equal(registry.findOfType.mock.callCount(), 2);
+ assert.equal(registry.classify.mock.callCount(), 2);
+});
+
+test("config loads lazily, names missing fields, and is cached until reload", async (t) => {
+ const { route, path } = setup(t);
+ rmSync(path);
+ const check = (error) => {
+ for (const value of [path, ...Object.keys(config)]) assert.ok(error.message.includes(value));
+ return true;
+ };
+ await assert.rejects(route(request()), check);
+ for (const field of Object.keys(config)) {
+ const incomplete = { ...config };
+ delete incomplete[field];
+ writeFileSync(path, JSON.stringify(incomplete));
+ await assert.rejects(route(request()), check);
+ }
+ writeFileSync(path, JSON.stringify({ ...config, efficient_model: config.capable_model }));
+ await assert.rejects(route(request()), { code: "ERR_CONFIG", message: /^capable and efficient targets must differ/ });
+ writeFileSync(path, JSON.stringify(config));
+ const initial = await route(request());
+ rmSync(path);
+ assert.deepEqual(await route(request()), initial);
});
-test("passes text, tool arguments, results, and failure flags to Switchyard", async (t) => {
- const decide = t.mock.fn(async () => ({ target: "openrouter/vendor/model", model: "openrouter/vendor/model" }));
- t.mock.method(Runner, "fromToml", () => ({ decide }));
- const { route } = setup();
+test("message and classifier adapters preserve content and model identities", async (t) => {
+ const { route, registry } = setup(t);
+ const nativeRoute = t.mock.fn(async (_, { decide }) => {
+ const response = await decide({ context: { task: "task" }, questions: { route: {
+ instructions: "Choose a model",
+ kind: { type: "choice", data: { options: [
+ { id: "advantage", description: "Complex work" }, { id: "no_advantage", description: { task: "Simple work" } }, { id: "unspecified" },
+ ] } },
+ } } });
+ assert.deepEqual(response, { model: config.decision_model, answers: { route: {
+ value: { type: "choice", data: { selected: "advantage", probabilities: result().answers.route.probabilities } },
+ provider_confidence: 0.9,
+ } } });
+ await assert.rejects(decide({ context: {}, questions: { route: { kind: { type: "score" } } } }), /Expected a choice question/);
+ return { model: "openrouter/vendor/model" };
+ });
+ t.mock.method(CompositeRouter, "create", () => ({ route: nativeRoute }));
const req = request({ messages: [
{ role: "system", content: "rules", sections: { extra: "more rules" } },
{ role: "user", content: [{ type: "text", text: "task" }, { type: "image", data: "image" }] },
@@ -51,11 +179,19 @@ test("passes text, tool arguments, results, and failure flags to Switchyard", as
{ role: "assistant", content: [{ type: "thinking", thinking: "private" }] },
] });
assert.deepEqual((await route(req)).model, { provider: "openrouter", id: "vendor/model" });
- assert.deepEqual(decide.mock.calls[0].arguments, ["switchyard/auto", [
+ const [messages, options] = nativeRoute.mock.calls[0].arguments;
+ assert.deepEqual(messages, [
{ role: "system", content: [{ type: "text", text: "rules" }, { type: "text", text: "more rules" }] },
{ role: "user", content: [{ type: "text", text: "task" }] },
{ role: "assistant", content: [{ type: "tool_call", id: "call", name: "bash", arguments: { command: "pytest" } }] },
{ role: "tool", content: [{ type: "tool_result", tool_call_id: "call", content: [{ type: "text", text: "out of memory" }], is_error: true }] },
{ role: "assistant", content: [] },
- ], { signal: req.signal }]);
+ ]);
+ assert.equal(options.signal, req.signal);
+ assert.equal(options.state, req.state);
+ assert.deepEqual(registry.classify.mock.calls[0].arguments[1], {
+ state: { task: "task" }, questions: { route: {
+ type: "choice", instructions: "Choose a model", criteria: { advantage: "Complex work", no_advantage: '{"task":"Simple work"}', unspecified: "null" },
+ } },
+ });
});