Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
feat: Move LLM classifier algo (#90)
Graham Kingc328f2aa44d0
diff --git a/crates/libsy-examples/Cargo.toml b/crates/libsy-examples/Cargo.tomlindex 1c83d1f6e..82643f8b6 100644--- a/crates/libsy-examples/Cargo.toml+++ b/crates/libsy-examples/Cargo.toml@@ -4,7 +4,7 @@[package]name = "libsy-examples"version = "0.1.0"-description = "Reference algorithms and runnable agents built on libsy"+description = "Example algorithms built on libsy"publish = falseauthors.workspace = trueedition.workspace = truediff --git a/crates/libsy-examples/src/lib.rs b/crates/libsy-examples/src/lib.rsindex 802964834..a3c9ff554 100644--- a/crates/libsy-examples/src/lib.rs+++ b/crates/libsy-examples/src/lib.rs@@ -7,10 +7,7 @@//! that drive them. The core crate provides [`libsy::RandomAlgo`] for uniform random//! routing.//!-//! - [`llm_class::LlmClassifierOrchAlgo`] — classify with one model, then route to a-//! strong/weak model (multi-step).//! - [`ensemble::EnsembleOrchAlgo`] — fan out to several models, judge, and commit//! (stateful).pub mod ensemble;-pub mod llm_class;diff --git a/crates/libsy/README.md b/crates/libsy/README.mdindex 36f28495b..3282bc240 100644--- a/crates/libsy/README.md+++ b/crates/libsy/README.md@@ -12,7 +12,7 @@ Build a target set, pick an algorithm, run a request:```rustuse libsy::{Algorithm, Context, RoutedLlmClient, LlmTarget, LlmTargetSet, Request};-use libsy_examples::llm_class::LlmClassifierOrchAlgo;+use libsy::LlmClassifierOrchAlgo;use switchyard_protocol::{completion_text, text_request};use std::sync::Arc;@@ -38,7 +38,7 @@ println!("routed to {}", trace.last().unwrap().selected_model());println!("answer: {}", completion_text(&response.llm_response.aggregate().await?));```-Runnable: [`research_agent`](../libsy-examples/examples/research_agent.rs) (in the `libsy-examples` crate).+Runnable: [`research_agent`](examples/research_agent.rs)## Requests & responses@@ -163,7 +163,7 @@ println!("{}", completion_text(&agg));An `LlmResponseChunk` is the provider-neutral streaming event (`MessageStart`, `TextDelta`,`ReasoningDelta`, `ToolCallDelta`, `Usage`, `MessageStop`, `Error`); `ResponseAccumulator`is the fold behind `aggregate()` if you need to assemble one yourself. Runnable end-to-end-demo: [`streaming_agent`](../libsy-examples/examples/streaming_agent.rs).+demo: [`streaming_agent`](examples/streaming_agent.rs).## Driving the calls yourself (`run_stream`)@@ -171,7 +171,7 @@ demo: [`streaming_agent`](../libsy-examples/examples/streaming_agent.rs).`default_client`), streams each `Decision` as it happens, and ends with `ReturnToAgent`.This is the *step* stream (one `Step` at a time); it is orthogonal to whether any singleresponse is itself streamed — see [Streaming responses](#streaming-responses).-Runnable: [`research_agent_core`](../libsy-examples/examples/research_agent_core.rs) (in the `libsy-examples` crate).+Runnable: [`research_agent_core`](examples/research_agent_core.rs).```rustlet stream = algo.clone().run_stream(Context::default(), req);@@ -217,7 +217,7 @@ pub trait Decision: Send + Sync {Give it a `new(config.., target_set)` constructor and `Arc`-wrap it — there is no builder.Example — the LLM classifier (classify, then route; full version in-[`libsy-examples/src/llm_class.rs`](../libsy-examples/src/llm_class.rs)):+[`src/algorithms/llm_class.rs`](src/algorithms/llm_class.rs)):```rust#[async_trait]@@ -251,26 +251,25 @@ impl Algorithm for LlmClassifierOrchAlgo {## Explore-The core crate includes uniform random routing. More reference algorithms and runnable-agents live in the sibling [`libsy-examples`](../libsy-examples) crate (compiled and tested-with `cargo test -p libsy-examples`).+The core crate includes uniform random routing and naive LLM classifier. Runnable+agents live in [`examples`](examples/) folder.**Reference algorithms** — implementations to read and route with:- [`RandomAlgo`](src/algorithms/rand.rs) — uniform random over the set(one call).-- [`LlmClassifierOrchAlgo`](../libsy-examples/src/llm_class.rs) — classify, then route+- [`LlmClassifierOrchAlgo`](src/algorithms/llm_class.rs) — classify, then routestrong/weak; fail open to strong.- [`EnsembleOrchAlgo`](../libsy-examples/src/ensemble.rs) — stateful: fan out tocandidates, judge the best, commit to the winner after N exploration turns.-**Runnable agents** (`cargo run -p libsy-examples --example <name>`):+**Runnable agents** (`cargo run -p libsy --example <name>`):-- [`research_agent`](../libsy-examples/examples/research_agent.rs) — client-backed+- [`research_agent`](examples/research_agent.rs) — client-backedtargets, `run` (libsy makes the calls).-- [`research_agent_core`](../libsy-examples/examples/research_agent_core.rs) — client-less+- [`research_agent_core`](examples/research_agent_core.rs) — client-lesstargets, `run_stream` (the agent makes the calls).-- [`streaming_agent`](../libsy-examples/examples/streaming_agent.rs) — a target that streams+- [`streaming_agent`](examples/streaming_agent.rs) — a target that streamsits response; the agent forwards each `LlmResponseChunk` to the caller token-by-token.## Not yet builtdiff --git a/crates/libsy-examples/examples/research_agent.rs b/crates/libsy/examples/research_agent.rssimilarity index 96%rename from crates/libsy-examples/examples/research_agent.rsrename to crates/libsy/examples/research_agent.rsindex cba976b59..eabfb9bda 100644--- a/crates/libsy-examples/examples/research_agent.rs+++ b/crates/libsy/examples/research_agent.rs@@ -9,17 +9,17 @@//! multi-step routing (classify -> route) happens inside the classifier algorithm; the//! agent never sees it. To drive the step stream yourself instead, use//! `Algorithm::run_stream`. Run with:-//! cargo run -p libsy-examples --example research_agent+//! cargo run -p libsy --example research_agentuse std::error::Error;use std::sync::Arc;use async_trait::async_trait;+use libsy::LlmClassifierOrchAlgo;use libsy::{Algorithm, Context, Decision, LlmResponse, LlmTarget, LlmTargetSet, Request, Response,RoutedLlmClient,};-use libsy_examples::llm_class::LlmClassifierOrchAlgo;use switchyard_protocol::{completion_text, text_request, text_response};const CLASSIFIER: &str = "classifier/model";diff --git a/crates/libsy-examples/examples/research_agent_core.rs b/crates/libsy/examples/research_agent_core.rssimilarity index 97%rename from crates/libsy-examples/examples/research_agent_core.rsrename to crates/libsy/examples/research_agent_core.rsindex 16fa959af..0199728bb 100644--- a/crates/libsy-examples/examples/research_agent_core.rs+++ b/crates/libsy/examples/research_agent_core.rs@@ -7,15 +7,15 @@//! surfaces as a `CallLlm` step. The agent makes the "real" model call itself and//! fulfills the promise — this is the offload/streaming path ("ask, don't call").//! The classifier's two steps show up as two `model call:` lines. Run with:-//! cargo run -p libsy-examples --example research_agent_core+//! cargo run -p libsy --example research_agent_coreuse std::error::Error;use std::sync::Arc;+use libsy::LlmClassifierOrchAlgo;use libsy::{Algorithm, Context, Decision, LlmResponse, LlmTarget, LlmTargetSet, Request, Response, Step,};-use libsy_examples::llm_class::LlmClassifierOrchAlgo;use switchyard_protocol::{completion_text, text_request, text_response};use tokio_stream::StreamExt;diff --git a/crates/libsy-examples/examples/streaming_agent.rs b/crates/libsy/examples/streaming_agent.rssimilarity index 98%rename from crates/libsy-examples/examples/streaming_agent.rsrename to crates/libsy/examples/streaming_agent.rsindex f871a4e07..5c245a8b9 100644--- a/crates/libsy-examples/examples/streaming_agent.rs+++ b/crates/libsy/examples/streaming_agent.rs@@ -10,7 +10,7 @@//! and prints each [`LlmResponseChunk`] as it arrives — true token streaming end to end.//! Contrast [`Algorithm::run`], which would aggregate the same stream into one buffered//! answer. Run with:-//! cargo run -p libsy-examples --example streaming_agent+//! cargo run -p libsy --example streaming_agentuse std::error::Error;use std::io::Write;diff --git a/crates/libsy/src/algorithms.rs b/crates/libsy/src/algorithms.rsindex 6f88a702d..ce17da02b 100644--- a/crates/libsy/src/algorithms.rs+++ b/crates/libsy/src/algorithms.rs@@ -1,5 +1,6 @@// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.// SPDX-License-Identifier: Apache-2.0+pub mod llm_class;pub mod noop;pub mod rand;diff --git a/crates/libsy-examples/src/llm_class.rs b/crates/libsy/src/algorithms/llm_class.rssimilarity index 82%rename from crates/libsy-examples/src/llm_class.rsrename to crates/libsy/src/algorithms/llm_class.rsindex 67e9336a4..e3fe7ebf0 100644--- a/crates/libsy-examples/src/llm_class.rs+++ b/crates/libsy/src/algorithms/llm_class.rs@@ -15,13 +15,11 @@ use std::sync::Arc;use async_trait::async_trait;-use libsy::{Algorithm, Context, Decision, Driver, LlmTargetSet, Request, Response};-use switchyard_protocol::{completion_text, prompt_text, text_request};+use crate::{Algorithm, Context, Decision, Driver, LlmTargetSet, Request, Response};+use switchyard_protocol::{completion_text, LlmRequest, Message, Role};-/// Preamble prepended to the user prompt when asking the classifier target for a-/// strong-win-rate score.-const CLASSIFIER_PROMPT_PREAMBLE: &str = "Rate how strongly this request needs a frontier model. \- Reply with a single strong-win-rate score in [0, 1]:\n";+/// The system prompt tells the classifier what to do+const CLASSIFIER_SYSTEM_PROMPT: &str = "Rate how strongly this request needs a frontier model. Reply with a single strong-win-rate score in [0, 1].";/// The tier a classifier score selected.#[derive(Clone, Copy, Debug, PartialEq, Eq)]@@ -96,6 +94,49 @@ impl LlmClassifierOrchAlgo {}}+/// The first User message as well as the last <recent_turn_window> turns (User + Assistant).+/// If fewer than 5 turns have happened, include the whole message.+fn trim_messages(messages: &[Message], recent_turn_window: usize) -> Vec<Message> {+ let mut system = Vec::new();+ let mut first_user = None;+ let mut first_user_idx = None;+ for (idx, message) in messages.iter().enumerate() {+ match message.role {+ Role::System | Role::Developer => system.push(message.clone()),+ Role::User if first_user.is_none() => {+ first_user = Some(message.clone());+ first_user_idx = Some(idx);+ }+ _ => {}+ }+ }+ let Some(first_user) = first_user else {+ return system;+ };+ let tail = messages+ .iter()+ .enumerate()+ .filter(|(idx, message)| {+ *idx > first_user_idx.unwrap_or(0)+ && !matches!(message.role, Role::System | Role::Developer)+ })+ .map(|(_, message)| message.clone())+ .collect::<Vec<_>>();+ if recent_turn_window == 0 {+ let mut out = system;+ out.push(first_user);+ if let Some(last_user) = tail.iter().rev().find(|message| message.role == Role::User) {+ out.push(last_user.clone());+ }+ return out;+ }+ let mut out = system;+ out.push(first_user);+ let start = tail.len().saturating_sub(recent_turn_window);+ out.extend_from_slice(&tail[start..]);+ out+}+#[async_trait]impl Algorithm for LlmClassifierOrchAlgo {async fn create_run_task(@@ -104,18 +145,22 @@ impl Algorithm for LlmClassifierOrchAlgo {driver: Driver,request: Request,) -> Result<Response, Box<dyn Error + Send + Sync>> {- let user_prompt = prompt_text(&request.llm_request);// The agent's inbound name rides through unchanged on every sub-call; the// model each sub-call actually hits is carried by its decision instead.let inbound = request.llm_request.model.clone();+ let mut just_the_key_messages = trim_messages(&request.llm_request.messages, 5);+ just_the_key_messages.insert(0, Message::text(Role::System, CLASSIFIER_SYSTEM_PROMPT));+// 1. Classify: call the classifier target with the score-eliciting prompt.let classifier_target = self.target_set.get_target(&self.classifier_model)?;+ let llm_request = LlmRequest {+ model: inbound.clone(),+ messages: just_the_key_messages,+ ..LlmRequest::default()+ };let classify_request = Request {- llm_request: text_request(- inbound.clone(),- format!("{CLASSIFIER_PROMPT_PREAMBLE}{user_prompt}"),- ),+ llm_request,raw_request: request.raw_request.clone(),metadata: request.metadata.clone(),};@@ -160,14 +205,9 @@ impl Algorithm for LlmClassifierOrchAlgo {score,tier: Some(tier),});- let routed_request = Request {- llm_request: text_request(inbound, user_prompt),- raw_request: request.raw_request,- metadata: request.metadata,- };driver.info(ctx.clone(), route_decision.clone()).await?;driver- .call_llm_target(ctx, &routed_target, routed_request, route_decision)+ .call_llm_target(ctx, &routed_target, request, route_decision).await}}@@ -175,7 +215,7 @@ impl Algorithm for LlmClassifierOrchAlgo {#[cfg(test)]mod tests {use super::*;- use libsy::{LlmResponse, LlmTarget, Response, RoutedLlmClient};+ use crate::{LlmResponse, LlmTarget, Response, RoutedLlmClient};use std::sync::Mutex;use switchyard_protocol::text_response;@@ -239,7 +279,7 @@ mod tests {fn request(prompt: &str) -> Request {Request {- llm_request: text_request(Some("auto".to_string()), prompt),+ llm_request: switchyard_protocol::text_request(Some("auto".to_string()), prompt),raw_request: None,metadata: None,}@@ -338,20 +378,4 @@ mod tests {assert_eq!(routed.score, None);Ok(())}-- #[tokio::test]- async fn classifier_prompt_includes_the_user_text() -> Result<(), Box<dyn Error + Send + Sync>>- {- let (algo, seen) = algo(0.5, "0.9");- orch(algo)- .run(Context::default(), request("prove it"))- .await?;- let seen = seen.lock().map_err(|_| "lock poisoned")?;- // Two calls: the classifier (preamble + user text), then the routed model.- assert_eq!(seen.len(), 2);- assert!(prompt_text(&seen[0].llm_request).contains("prove it"));- assert!(prompt_text(&seen[0].llm_request).contains("frontier model"));- assert_eq!(prompt_text(&seen[1].llm_request), "prove it");- Ok(())- }}diff --git a/crates/libsy/src/lib.rs b/crates/libsy/src/lib.rsindex 9d107233f..2cf6ecf3b 100644--- a/crates/libsy/src/lib.rs+++ b/crates/libsy/src/lib.rs@@ -54,11 +54,13 @@//!//! ## Algorithms//!-//! [`RandomAlgo`] provides uniform random routing in the core crate. Worked-//! implementations of an LLM classifier and a stateful ensemble, plus runnable-//! agents, live in the `libsy-examples` crate.+//! [`RandomAlgo`] provides uniform random routing in the core crate.+//!+//! [`LlmClassifierOrchAlgo`] classifies with one model, then routes to a+//! strong/weak model depending on the classifier's choice.mod algorithms;+pub use algorithms::llm_class::{ClassifierDecision, ClassifierTier, LlmClassifierOrchAlgo};pub use algorithms::noop::{NoopAlgo, NoopDecision};pub use algorithms::rand::{RandomAlgo, RandomDecision};