Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
feat(pi): Update extension to Unity Router (#1247)
Graham King7060fc3f30ee
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 687d0658e..6b5848c98 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -36,6 +36,15 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).### Changed+- **TypeScript composite stage profiles** - `CompositeRouter` options take+ `stage.efficient_first` and `stage.capable_first` profiles in place of+ `efficient_first_threshold`, `capable_first_threshold`, and+ `efficient_hold_turns`. Each profile sets `threshold`, `recent_window`, holds,+ and `capable_system_prompt`. `classifier.cutoff` sets the judge cutoff.+ `Selection.systemPrompt` returns the routed prompt. Stage state keys gain a+ `composite.profile_<n>.` prefix. State saved earlier keeps its tier and drops+ its stage holds. The Pi example adds a planning prompt to capable-first turns.+ **API and state change (breaking).**- **Simpler decision requests and routing question** — `DecisionRequest.context`is renamed to `input`. Decision judges send the conversation messages as`input`, with string instructions and a description for every choice. Thediff --git a/bindings/typescript/README.md b/bindings/typescript/README.mdindex 9886ece79..e519da679 100644--- a/bindings/typescript/README.md+++ b/bindings/typescript/README.md@@ -91,9 +91,12 @@ its HTTP request alive until stream completion or cancellation.## CompositeRouter-Capable-first routing defaults to pinning the efficient tier after switching to-it. Set `stage.efficient_hold_turns: 0` to disable this hold. Efficient-first-defaults to no efficient hold. Persist the returned state to retain holds.+The classifier picks one of two stage profiles for each turn: `efficient_first`+or `capable_first`. Each profile keeps its own holds. The capable-first profile+defaults to pinning the efficient tier after switching to it. Set+`stage.capable_first.efficient_hold_turns: 0` to disable this hold. The+efficient-first profile defaults to no efficient hold. Persist the returned state+to retain holds.```tsimport { CompositeRouter, type DecisionRequest, type DecisionResponse } from "@switchyard/libsy";@@ -108,7 +111,10 @@ const router = CompositeRouter.create({classify_trigger: "user_turn",fail_open: false,},- stage: { efficient_first_threshold: 0.5, capable_first_threshold: 0.2 },+ stage: {+ efficient_first: { threshold: 0.5 },+ capable_first: { threshold: 0.2 },+ },});// Supply an adapter for your classifier provider.@@ -120,6 +126,11 @@ const selection = await router.route([// selection.model is a configured target model ID.```+Each profile accepts `threshold`, `recent_window`, `capable_hold_turns`,+`efficient_hold_turns`, and `capable_system_prompt`. When a profile routes to+the capable model with a `capable_system_prompt`, `selection.systemPrompt` holds+that prompt. Prepend it to the system prompt of the request you send.+`create` synchronously validates the options. Reuse the router for concurrentrequests. `route` takes `RoutingMessage` values with text, tool calls, and toolresults. Pass the current branch's conversation on each call.@@ -140,8 +151,9 @@ const response: DecisionResponse = {};```-The judge selects capable when `probabilities.capable` is above `0.6`.-Stage tool signals then select the final model. `user_turn` judges new user+The judge selects capable-first when `probabilities.capable` is above+`classifier.cutoff`, which defaults to `0.6`. Stage tool signals in the selected+profile then select the final model. `user_turn` judges new usermessages; `new_session` judges when carried state has no tier.Store `selection.state` when present and pass it as `options.state` on the next@@ -159,7 +171,7 @@ Treat `CompositeState` as opaque. It carries the judge's tier and stage holds.A rejected `decide` call follows `classifier.fail_open`. With `false`, routingfails with the original error as `cause`. With `true`, routing keeps the previous-tier or uses the stage default. An answer without a usable probability for the+tier or uses the efficient-first profile. An answer without a usable probability for the`apply_cutoff_to` choice (`capable` by default) follows the same policy.An aborted signal closes the Rust run and rejects promptly, including whilediff --git a/bindings/typescript/src/index.ts b/bindings/typescript/src/index.tsindex a75f16db1..500431bb9 100644--- a/bindings/typescript/src/index.ts+++ b/bindings/typescript/src/index.ts@@ -14,19 +14,32 @@ export interface CompositeOptions {classifier: {classify_trigger: "user_turn" | "new_session";fail_open: boolean;+ /** Selects capable-first when the `capable` probability is above this. Defaults to 0.6. */+ cutoff?: number;};- stage: {- efficient_first_threshold: number;- capable_first_threshold: number;- /** Defaults to -1 for capable-first, otherwise 0. Set 0 to disable the hold. */- efficient_hold_turns?: number;- };+ /** Stage profiles. The classifier's tier selects one for the turn. */+ stage: { efficient_first: StageProfile; capable_first: StageProfile };+}++export interface StageProfile {+ /** Confidence the tool signals need to switch tiers. */+ threshold: number;+ /** Trailing tool results scored. Defaults to 3. */+ recent_window?: number;+ /** Requests kept on capable after an escalation. Defaults to 2. Set 0 to disable, -1 to pin. */+ capable_hold_turns?: number;+ /** Defaults to -1 for capable-first, otherwise 0. Set 0 to disable the hold, -1 to pin. */+ efficient_hold_turns?: number;+ /** System prompt for requests routed to the capable model. */+ capable_system_prompt?: string;}export interface Selection {model: string;/** Present only when the state changed. */state?: CompositeState;+ /** Prepend this to the system prompt of the request sent to `model`. */+ systemPrompt?: string;}/** Text and tool activity used for routing, in Switchyard's message format. */@@ -78,7 +91,7 @@ export interface DecisionAnswer {type NativeStep =| { kind: "decision"; request: string }- | { kind: "done"; model: string; state: string | null };+ | { kind: "done"; model: string; state: string | null; systemPrompt: string | null };interface NativeRun {next(response: string | null, error: string | null): Promise<NativeStep>;@@ -158,9 +171,10 @@ export class CompositeRouter {const step = await Promise.race([run.next(response, failure), aborted]);if (signal?.aborted) throw abortError();if (step.kind === "done") {- return step.state === null- ? { model: step.model }- : { model: step.model, state: JSON.parse(step.state) };+ const selection: Selection = { model: step.model };+ if (step.state !== null) selection.state = JSON.parse(step.state);+ if (step.systemPrompt !== null) selection.systemPrompt = step.systemPrompt;+ return selection;}response = null;failure = null;diff --git a/bindings/typescript/test/composite.test.mjs b/bindings/typescript/test/composite.test.mjsindex eab68ccd0..635c630ee 100644--- a/bindings/typescript/test/composite.test.mjs+++ b/bindings/typescript/test/composite.test.mjs@@ -12,7 +12,7 @@ const options = {classify_trigger: "user_turn",fail_open: false,},- stage: { efficient_first_threshold: 0.5, capable_first_threshold: 0.2 },+ stage: { efficient_first: { threshold: 0.5 }, capable_first: { threshold: 0.2 } },};const messages = [{ role: "user", content: [{ type: "text", text: "Fix the parser" }] }];const continuation = [@@ -55,7 +55,7 @@ test("decision request, cutoff, and branch-owned state", async () => {}assert.deepEqual(await router.route(continuation, { decide: unexpectedDecision }), {model: options.targets.efficient,- state: { stage_previous_tier: { string: "weak" } },+ state: { "composite.profile_0.stage_previous_tier": { string: "weak" } },});const once = structuredClone(options);@@ -90,7 +90,7 @@ test("serialized state preserves capable holds until they expire", async () => {test("capable-first pins efficient by default and accepts an explicit opt-out", async () => {for (const turns of [undefined, 0]) {const config = structuredClone(options);- config.stage.efficient_hold_turns = turns;+ config.stage.capable_first.efficient_hold_turns = turns;const router = CompositeRouter.create(config);const initial = await router.route(messages, { decide: async () => response(0.61) });assert.equal(initial.model, options.targets.capable);@@ -111,12 +111,40 @@ test("capable-first pins efficient by default and accepts an explicit opt-out",}});+test("the cutoff selects a profile and the capable-first prompt follows the capable model", async () => {+ const config = structuredClone(options);+ config.classifier.cutoff = 0.9;+ config.stage.capable_first.capable_system_prompt = "Plan first.";+ const router = CompositeRouter.create(config);+ assert.equal(+ (await router.route(messages, { decide: async () => response(0.9) })).model, options.targets.efficient,+ );+ const initial = await router.route(messages, { decide: async () => response(0.91) });+ assert.equal(initial.model, options.targets.capable);+ assert.equal(initial.systemPrompt, "Plan first.");+ const write = structuredClone(continuation);+ write[1].content[0].name = "write";+ const selected = await router.route(write, { state: initial.state, decide: unexpectedDecision });+ assert.equal(selected.model, options.targets.efficient);+ assert.equal(selected.systemPrompt, undefined);++ const strict = structuredClone(config);+ strict.stage.capable_first.threshold = 1.0;+ const strictRouter = CompositeRouter.create(strict);+ assert.equal(+ (await strictRouter.route(write, { state: initial.state, decide: unexpectedDecision })).model,+ options.targets.capable,+ );+});+test("validation and decision failure policy", async () => {for (const change of [(config) => { config.targets.efficient = config.targets.capable; },(config) => { config.classifier.decision = { cutoff: 0.4 }; },+ (config) => { config.classifier.cutoff = 1.5; },+ (config) => { config.stage.efficient_first.picker = "capable_first"; },(config) => { config.classifier.classify_trigger = "every_request"; },- (config) => { config.stage.efficient_hold_turns = -2; },+ (config) => { config.stage.capable_first.efficient_hold_turns = -2; },]) {const config = structuredClone(options);change(config);@@ -144,7 +172,7 @@ test("validation and decision failure policy", async () => {assert.deepEqual(await fallback.route(messages, { state, decide }), { model: options.targets.capable });assert.deepEqual(await fallback.route(messages, { decide }), {model: options.targets.efficient,- state: { stage_previous_tier: { string: "weak" } },+ state: { "composite.profile_0.stage_previous_tier": { string: "weak" } },});for (const answer of [{ answers: {} },@@ -157,7 +185,7 @@ test("validation and decision failure policy", async () => {assert.deepEqual(await fallback.route(messages, { state, decide }), { model: options.targets.capable });assert.deepEqual(await fallback.route(messages, { decide }), {model: options.targets.efficient,- state: { stage_previous_tier: { string: "weak" } },+ state: { "composite.profile_0.stage_previous_tier": { string: "weak" } },});}});diff --git a/crates/switchyard-node/src/lib.rs b/crates/switchyard-node/src/lib.rsindex 32bf423c3..8288e3011 100644--- a/crates/switchyard-node/src/lib.rs+++ b/crates/switchyard-node/src/lib.rs@@ -16,11 +16,11 @@ use parking_lot::Mutex;use serde::Deserialize;use switchyard_libsy::{Algorithm, CallDecision, CapabilityJudgeConfig, ClassifyTrigger, CompositeJudgeConfig,- CompositeRouter, CompositeRouterConfig, LibsyError, PickerMode, RuntimeModels,- StageRouterConfig, State, Step, StepStream, TaskClassifierConfig,+ CompositeRouter, DecisionJudgeConfig, LibsyError, PickerMode, RuntimeModels, StageRouterConfig,+ State, Step, StepStream, TaskClassifierConfig,};use switchyard_protocol::{- Category, DecisionResponse, LlmClientError, LlmRequest, ModelId, Request,+ Category, ContentBlock, DecisionResponse, LlmClientError, LlmRequest, ModelId, Request,};// The JS wrapper turns these prefixes into public Error codes.@@ -57,14 +57,37 @@ struct Targets {struct ClassifierOptions {classify_trigger: ClassifyTrigger,fail_open: bool,+ cutoff: Option<f64>,}#[derive(Deserialize)]#[serde(deny_unknown_fields)]struct StageOptions {- efficient_first_threshold: f64,- capable_first_threshold: f64,+ efficient_first: StageProfile,+ capable_first: StageProfile,+}++#[derive(Deserialize)]+#[serde(deny_unknown_fields)]+struct StageProfile {+ threshold: f64,+ recent_window: Option<usize>,+ capable_hold_turns: Option<i64>,efficient_hold_turns: Option<i64>,+ capable_system_prompt: Option<String>,+}++impl StageProfile {+ fn config(self, mode: PickerMode) -> StageRouterConfig {+ let mut config = StageRouterConfig::new(mode, self.threshold);+ config.recent_window = self.recent_window;+ if let Some(turns) = self.capable_hold_turns {+ config.capable_hold_turns = turns;+ }+ config.efficient_hold_turns = self.efficient_hold_turns;+ config.capable_system_prompt = self.capable_system_prompt;+ config+ }}#[napi]@@ -88,19 +111,20 @@ impl NativeComposite {"capable and efficient targets must differ",));}- let mut stage_config =- StageRouterConfig::new(PickerMode::EfficientFirst, stage.efficient_first_threshold);- stage_config.capable_first_threshold = stage.capable_first_threshold;- stage_config.efficient_hold_turns = stage.efficient_hold_turns;- let router = CompositeRouter::new(CompositeRouterConfig {- judge: CompositeJudgeConfig::Capability(TaskClassifierConfig {- judge: CapabilityJudgeConfig::Decision(Default::default()),+ // The judge's efficient tier selects profile 0 and its capable tier profile 1.+ let router = CompositeRouter::new_with_profiles(+ CompositeJudgeConfig::Capability(TaskClassifierConfig {+ judge: CapabilityJudgeConfig::Decision(DecisionJudgeConfig {+ cutoff: classifier.cutoff,+ ..Default::default()+ }),classify_trigger: classifier.classify_trigger,fail_open: classifier.fail_open,..Default::default()}),- stage: stage_config,- })+ stage.efficient_first.config(PickerMode::EfficientFirst),+ stage.capable_first.config(PickerMode::CapableFirst),+ ).map_err(|err| error("ERR_CONFIG", err))?;let models = RuntimeModels::new([@@ -164,6 +188,7 @@ pub struct NativeStep {pub request: Option<String>,pub model: Option<String>,pub state: Option<String>,+ pub system_prompt: Option<String>,}struct Run {@@ -217,6 +242,20 @@ impl Run {.map_err(routing_error)?.to_string(),);+ // `start` sends no instructions, so a first instruction block is the+ // routing prompt that libsy prepended for the selected model.+ result.system_prompt = outcome+ .request+ .llm_request+ .instructions+ .into_iter()+ .next()+ .and_then(|block| {+ block.content.into_iter().find_map(|content| match content {+ ContentBlock::Text { text } => Some(text),+ _ => None,+ })+ });if outcome.state != self.initial_state {result.state =Some(serde_json::to_string(&outcome.state).map_err(routing_error)?);diff --git a/examples/pi/README.md b/examples/pi/README.mdindex 51c52b0e9..92e406949 100644--- a/examples/pi/README.md+++ b/examples/pi/README.md@@ -4,11 +4,18 @@ This extension registers `switchyard/auto`. Pi calls a Decision classifier onuser turns. Switchyard's composite algorithm uses that answer and recent toolactivity to select a completion model. Thinking stays `medium`.-The fixed policy selects the capable tier when the classifier's `capable`-probability exceeds 0.6. Exactly 0.6 selects efficient. Stage routing then picks-the model on every request, using the classifier's tier as its fallback. Its-efficient-first and capable-first thresholds are 0.5 and 0.2. A tool failure can-escalate to the capable model, with a two-request hold during recovery.+The fixed policy selects a capable-first profile when the classifier's `capable`+probability exceeds 0.6. Exactly 0.6 selects the efficient-first profile. Stage+routing then picks the model on every request from the last three tool results.++- Efficient-first uses a 0.4 threshold. A tool failure can escalate to the+ capable model, with a two-request hold during recovery.+- Capable-first starts on the capable model with a planning prompt. The prompt+ asks the model to plan with read-only tools, then make the first edit. After the+ switch to the efficient model, routing stays there until the next user turn.++The extension adds the planning prompt ahead of Pi's system prompt on each+request routed to the capable model in the capable-first profile.## Configuration@@ -71,7 +78,8 @@ classifier state is `{ "input": [...] }`, holding the routing messages.Pi stores routing state on the session branch. It follows forks and `/tree`, andsurvives compaction, resume, and extension reload. Tool continuations reuse thestored tier and capable hold while stage signals select the model. State is-returned only when it changes.+returned only when it changes. Sessions saved before stage profiles keep their+tier and drop any stage hold.Retries reuse stored state. Retrying a user-turn request calls the classifieragain. Direct requests, such as compaction summaries, start with fresh state;diff --git a/examples/pi/switchyard-router.ts b/examples/pi/switchyard-router.tsindex 09b7fafde..a67a96ea7 100644--- a/examples/pi/switchyard-router.ts+++ b/examples/pi/switchyard-router.ts@@ -10,14 +10,24 @@ import {import type {ClassifierApi, ClassifierContext, ClassifierModel, ClassifierResult, JsonValue, Message,} from "@earendil-works/pi-ai";-import { getAgentDir, type ExtensionAPI } from "@earendil-works/pi-coding-agent";+import { getAgentDir, type ContextWithSystemEvent, type ExtensionAPI } from "@earendil-works/pi-coding-agent";++const PLANNING_PROMPT = "You are in the planning phase. Inspect the task and the relevant code, then form a concrete implementation plan before modifying any files. Use read-only tools as needed. Do not make an edit until the plan is complete. After completing the plan, immediately make the first edit in the same task to hand execution off. Do not finish or return a final response with only the plan.";const POLICY = {classifier: {classify_trigger: "user_turn",fail_open: false,+ cutoff: 0.6,+ },+ stage: {+ efficient_first: { threshold: 0.4, recent_window: 3, capable_hold_turns: 2, efficient_hold_turns: 0 },+ // After the first switch to efficient, stay there until the next user turn.+ capable_first: {+ threshold: 0.0, recent_window: 3, capable_hold_turns: 0, efficient_hold_turns: -1,+ capable_system_prompt: PLANNING_PROMPT,+ },},- stage: { efficient_first_threshold: 0.5, capable_first_threshold: 0.2 },} satisfies Omit<CompositeOptions, "targets">;function modelIdentity(reference: unknown) {@@ -78,6 +88,17 @@ function decisionResponse(request: DecisionRequest, result: ClassifierResult): Dreturn { model: `${result.provider}/${result.model}`, answers };}+// Routing prompts go first, as libsy places them ahead of the request's own instructions.+function prependSystemPrompt(messages: ContextWithSystemEvent["messages"], prompt: string) {+ const [first, ...rest] = messages;+ if (first?.role !== "system") return [{ role: "system" as const, content: prompt, timestamp: Date.now() }, ...messages];+ // Pi's structured prompt keeps its text in sections and leaves content empty.+ const content = typeof first.content === "string"+ ? (first.content ? [{ type: "text" as const, text: first.content }] : [])+ : first.content;+ return [{ ...first, content: [{ type: "text" as const, text: prompt }, ...content] }, ...rest];+}+function routingMessages(messages: readonly Message[]): RoutingMessage[] {return messages.map((message) => {const blocks = typeof message.content === "string"@@ -107,12 +128,21 @@ function routingMessages(messages: readonly Message[]): RoutingMessage[] {export default function (pi: ExtensionAPI) {let loaded: ReturnType<typeof loadRouter> | undefined;let classifierModel: ClassifierModel<ClassifierApi> | undefined;+ // Pi builds the request context after routing, so the routed prompt waits for that event.+ let pendingPrompt: string | undefined;+ pi.on("context_with_system", (event) => {+ const prompt = pendingPrompt;+ pendingPrompt = undefined;+ return prompt === undefined ? undefined : { messages: prependSystemPrompt(event.messages, prompt) };+ });pi.registerVirtualModel<CompositeState>({provider: "switchyard",id: "auto",name: "Auto (Switchyard)",thinkingLevels: ["medium"],async route(request, ctx) {+ // Direct requests, such as compaction summaries, skip the agent loop's context event.+ if (request.reason !== "direct") pendingPrompt = undefined;const { router, classifier } = loaded ??= loadRouter();const judge = classifierModel ??= ctx.modelRegistry.findOfType("classifier", classifier.provider, classifier.id);if (!judge) throw new Error(`Unknown classifier: ${classifier.reference}`);@@ -130,6 +160,7 @@ export default function (pi: ExtensionAPI) {const identity = modelIdentity(selection.model);const model = ctx.modelRegistry.find(identity.provider, identity.id);if (!model) throw new Error(`Model ${identity.reference} is not in the catalog`);+ if (request.reason !== "direct") pendingPrompt = selection.systemPrompt;return { model, thinkingLevel: "medium", state: selection.state };},});diff --git a/examples/pi/test/router.test.mjs b/examples/pi/test/router.test.mjsindex 09c8f96be..e173fdf07 100644--- a/examples/pi/test/router.test.mjs+++ b/examples/pi/test/router.test.mjs@@ -42,8 +42,15 @@ function setup(t, answer = result()) {classify: t.mock.fn(async () => answer),};let definition;- extension({ registerVirtualModel: (value) => { definition = value; } });- return { definition, registry, path, route: (req) => definition.route(req, { modelRegistry: registry }) };+ const handlers = {};+ extension({+ registerVirtualModel: (value) => { definition = value; },+ on: (event, handler) => { handlers[event] = handler; },+ });+ return {+ definition, registry, path, route: (req) => definition.route(req, { modelRegistry: registry }),+ context: (messages) => handlers.context_with_system({ type: "context_with_system", messages }),+ };}function tool(id, name, args, text, isError = false) {@@ -89,7 +96,7 @@ test("cutoff equality selects efficient, then a tool failure holds capable for tmessages.push(...tool("failed", "bash", { command: "pytest" }, "out of memory", true));selected = await route(request({ reason: "continuation", messages, state: selected.state }));assert.equal(selected.model.id, "gpt-6.1-sol");- assert.deepEqual(selected.state.capable_hold_turns, { count: 2 });+ assert.deepEqual(selected.state["composite.profile_0.capable_hold_turns"], { count: 2 });for (let i = 0; i < 3; i++) messages.push(...tool(`write-${i}`, "write", { path: `file-${i}` }, "wrote file"));let state = selected.state;for (const expected of ["gpt-6.1-sol", "gpt-6.1-sol", "gpt-6-luna"]) {@@ -100,6 +107,46 @@ test("cutoff equality selects efficient, then a tool failure holds capable for tassert.equal(registry.classify.mock.callCount(), 1);});+test("capable-first adds the planning prompt until the first edit hands off to efficient", async (t) => {+ const { route, context } = setup(t);+ const system = { role: "system", content: "rules", sections: { tools: "tool rules" }, timestamp: 1 };+ const user = { role: "user", content: "task", timestamp: 2 };+ assert.equal(await context([system, user]), undefined);++ const messages = request().messages;+ let selected = await route(request({ messages }));+ assert.equal(selected.model.id, "gpt-6.1-sol");+ const { messages: [prompted, ...rest] } = await context([system, user]);+ assert.deepEqual(rest, [user]);+ assert.deepEqual(prompted.sections, system.sections);+ assert.equal(prompted.content.length, 2);+ assert.match(prompted.content[0].text, /^You are in the planning phase\./);+ assert.deepEqual(prompted.content[1], { type: "text", text: "rules" });+ // The prompt applies to one request only.+ assert.equal(await context([system, user]), undefined);++ // Pi's structured prompt has empty content, which gains no empty text block.+ await route(request({ messages }));+ const structured = { ...system, content: "" };+ const { messages: [only] } = await context([structured, user]);+ assert.deepEqual(only.content.map((block) => block.text), [prompted.content[0].text]);+ assert.deepEqual(only.sections, system.sections);++ // A direct request leaves the pending prompt alone.+ await route(request({ messages }));+ await route(request({ reason: "direct", messages }));+ assert.ok(await context([system, user]));++ messages.push(...tool("edit", "write", { path: "parser.rs" }, "wrote file"));+ const handoff = await route(request({ reason: "continuation", messages, state: selected.state }));+ assert.equal(handoff.model.id, "gpt-6-luna");+ assert.equal(await context([system, user]), undefined);+ // A failure after the handoff stays on efficient.+ messages.push(...tool("failed", "bash", { command: "pytest" }, "out of memory", true));+ selected = await route(request({ reason: "continuation", messages, state: handoff.state ?? selected.state }));+ assert.equal(selected.model.id, "gpt-6-luna");+});+test("classifier lookup, answer validation, failures, and cancellation", async (t) => {const { route, registry } = setup(t);for (const [answer, expected] of [