Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust
4.0 KB
Raw Blame

Pi virtual model using Switchyard composite routing

This extension registers switchyard/auto. Pi calls a Decision classifier on user turns. Switchyard's composite algorithm uses that answer and recent tool activity to select a completion model. Thinking stays medium.

The fixed policy selects a capable-first profile when the classifier's capable probability exceeds 0.6. Exactly 0.6 selects the efficient-first profile. Stage routing then picks the model on every request from the last three tool results.

  • Efficient-first uses a 0.4 threshold. A tool failure can escalate to the capable model, with a two-request hold during recovery.
  • Capable-first starts on the capable model with a planning prompt. The prompt asks the model to plan with read-only tools, then make the first edit. After the switch to the efficient model, routing stays there until the next user turn.

The extension adds the planning prompt ahead of Pi's system prompt on each request routed to the capable model in the capable-first profile.

Configuration

Create <agent-dir>/switchyard-router.json. The agent directory defaults to ~/.pi/agent; PI_CODING_AGENT_DIR overrides it.

{
  "decision_model": "openrouter/inception/mercury-decide:free",
  "capable_model": "openai-codex/gpt-6.1-sol",
  "efficient_model": "openai-codex/gpt-6-luna"
}

Use provider/model references from Pi's catalog. The first slash separates the provider from the model ID, so openrouter/vendor/model keeps vendor/model as the ID. The Decision model must be registered as a classifier in Pi.

The classifier chooses between starting with the more capable model and starting with the efficient model. Pick a capable model for complex work and an efficient model for routine work.

The extension reads the config on its first routing request and caches it. Reload the extension after editing the config. Pi owns credentials and all model calls, including classifier calls. Classifier and completion calls may cost money.

Local setup

Requires Node.js 22.19 or newer, the repository's Rust toolchain, and Pi with the virtual-model and classifier APIs. Development checks use Pi 1.0.4.

From the repository root:

npm --prefix bindings/typescript ci
npm --prefix bindings/typescript run build
npm --prefix examples/pi ci
npm --prefix examples/pi run typecheck
npm --prefix examples/pi test

Tests run the real composite algorithm with a local classifier mock, check message conversion, and load the extension through Pi's public resource loader.

Sign in to the configured providers through Pi's /login and confirm the models are in your catalog. Then:

pi -e ./examples/pi/switchyard-router.ts --model switchyard/auto

Conversation handling

The extension passes visible text, tool-call arguments, paired tool results, and failure flags to Switchyard. Pi keeps the original images and reasoning for the chosen completion model. The classifier sees the opening user task and latest user message. Its instructions come from libsy and are sent to Pi as text. The classifier state is { "input": [...] }, holding the routing messages.

Pi stores routing state on the session branch. It follows forks and /tree, and survives compaction, resume, and extension reload. Tool continuations reuse the stored tier and capable hold while stage signals select the model. State is returned only when it changes. Sessions saved before stage profiles keep their tier and drop any stage hold.

Retries reuse stored state. Retrying a user-turn request calls the classifier again. Direct requests, such as compaction summaries, start with fresh state; Pi discards their returned state. They call the classifier when their last message contains user text. Image-only user messages retain the previous tier because routing uses text and tool activity.

Classifier failures fail the request, and cancellation propagates to Pi. An answer without a usable capable probability also fails the request.

For the binding API, see bindings/typescript.