Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

feat: Move LLM classifier algo (#90)

Graham Kingc328f2aa44d0
diff --git a/crates/libsy-examples/Cargo.toml b/crates/libsy-examples/Cargo.toml
index 1c83d1f6e..82643f8b6 100644
--- a/crates/libsy-examples/Cargo.toml
+++ b/crates/libsy-examples/Cargo.toml
@@ -4,7 +4,7 @@
[package]
name = "libsy-examples"
version = "0.1.0"
-description = "Reference algorithms and runnable agents built on libsy"
+description = "Example algorithms built on libsy"
publish = false
authors.workspace = true
edition.workspace = true
diff --git a/crates/libsy-examples/src/lib.rs b/crates/libsy-examples/src/lib.rs
index 802964834..a3c9ff554 100644
--- a/crates/libsy-examples/src/lib.rs
+++ b/crates/libsy-examples/src/lib.rs
@@ -7,10 +7,7 @@
//! that drive them. The core crate provides [`libsy::RandomAlgo`] for uniform random
//! routing.
//!
-//! - [`llm_class::LlmClassifierOrchAlgo`] — classify with one model, then route to a
-//! strong/weak model (multi-step).
//! - [`ensemble::EnsembleOrchAlgo`] — fan out to several models, judge, and commit
//! (stateful).
pub mod ensemble;
-pub mod llm_class;
diff --git a/crates/libsy/README.md b/crates/libsy/README.md
index 36f28495b..3282bc240 100644
--- a/crates/libsy/README.md
+++ b/crates/libsy/README.md
@@ -12,7 +12,7 @@ Build a target set, pick an algorithm, run a request:
```rust
use libsy::{Algorithm, Context, RoutedLlmClient, LlmTarget, LlmTargetSet, Request};
-use libsy_examples::llm_class::LlmClassifierOrchAlgo;
+use libsy::LlmClassifierOrchAlgo;
use switchyard_protocol::{completion_text, text_request};
use std::sync::Arc;
@@ -38,7 +38,7 @@ println!("routed to {}", trace.last().unwrap().selected_model());
println!("answer: {}", completion_text(&response.llm_response.aggregate().await?));
```
-Runnable: [`research_agent`](../libsy-examples/examples/research_agent.rs) (in the `libsy-examples` crate).
+Runnable: [`research_agent`](examples/research_agent.rs)
## Requests & responses
@@ -163,7 +163,7 @@ println!("{}", completion_text(&agg));
An `LlmResponseChunk` is the provider-neutral streaming event (`MessageStart`, `TextDelta`,
`ReasoningDelta`, `ToolCallDelta`, `Usage`, `MessageStop`, `Error`); `ResponseAccumulator`
is the fold behind `aggregate()` if you need to assemble one yourself. Runnable end-to-end
-demo: [`streaming_agent`](../libsy-examples/examples/streaming_agent.rs).
+demo: [`streaming_agent`](examples/streaming_agent.rs).
## Driving the calls yourself (`run_stream`)
@@ -171,7 +171,7 @@ demo: [`streaming_agent`](../libsy-examples/examples/streaming_agent.rs).
`default_client`), streams each `Decision` as it happens, and ends with `ReturnToAgent`.
This is the *step* stream (one `Step` at a time); it is orthogonal to whether any single
response is itself streamed — see [Streaming responses](#streaming-responses).
-Runnable: [`research_agent_core`](../libsy-examples/examples/research_agent_core.rs) (in the `libsy-examples` crate).
+Runnable: [`research_agent_core`](examples/research_agent_core.rs).
```rust
let stream = algo.clone().run_stream(Context::default(), req);
@@ -217,7 +217,7 @@ pub trait Decision: Send + Sync {
Give it a `new(config.., target_set)` constructor and `Arc`-wrap it — there is no builder.
Example — the LLM classifier (classify, then route; full version in
-[`libsy-examples/src/llm_class.rs`](../libsy-examples/src/llm_class.rs)):
+[`src/algorithms/llm_class.rs`](src/algorithms/llm_class.rs)):
```rust
#[async_trait]
@@ -251,26 +251,25 @@ impl Algorithm for LlmClassifierOrchAlgo {
## Explore
-The core crate includes uniform random routing. More reference algorithms and runnable
-agents live in the sibling [`libsy-examples`](../libsy-examples) crate (compiled and tested
-with `cargo test -p libsy-examples`).
+The core crate includes uniform random routing and naive LLM classifier. Runnable
+agents live in [`examples`](examples/) folder.
**Reference algorithms** — implementations to read and route with:
- [`RandomAlgo`](src/algorithms/rand.rs) — uniform random over the set
(one call).
-- [`LlmClassifierOrchAlgo`](../libsy-examples/src/llm_class.rs) — classify, then route
+- [`LlmClassifierOrchAlgo`](src/algorithms/llm_class.rs) — classify, then route
strong/weak; fail open to strong.
- [`EnsembleOrchAlgo`](../libsy-examples/src/ensemble.rs) — stateful: fan out to
candidates, judge the best, commit to the winner after N exploration turns.
-**Runnable agents** (`cargo run -p libsy-examples --example <name>`):
+**Runnable agents** (`cargo run -p libsy --example <name>`):
-- [`research_agent`](../libsy-examples/examples/research_agent.rs) — client-backed
+- [`research_agent`](examples/research_agent.rs) — client-backed
targets, `run` (libsy makes the calls).
-- [`research_agent_core`](../libsy-examples/examples/research_agent_core.rs) — client-less
+- [`research_agent_core`](examples/research_agent_core.rs) — client-less
targets, `run_stream` (the agent makes the calls).
-- [`streaming_agent`](../libsy-examples/examples/streaming_agent.rs) — a target that streams
+- [`streaming_agent`](examples/streaming_agent.rs) — a target that streams
its response; the agent forwards each `LlmResponseChunk` to the caller token-by-token.
## Not yet built
diff --git a/crates/libsy-examples/examples/research_agent.rs b/crates/libsy/examples/research_agent.rs
similarity index 96%
rename from crates/libsy-examples/examples/research_agent.rs
rename to crates/libsy/examples/research_agent.rs
index cba976b59..eabfb9bda 100644
--- a/crates/libsy-examples/examples/research_agent.rs
+++ b/crates/libsy/examples/research_agent.rs
@@ -9,17 +9,17 @@
//! multi-step routing (classify -> route) happens inside the classifier algorithm; the
//! agent never sees it. To drive the step stream yourself instead, use
//! `Algorithm::run_stream`. Run with:
-//! cargo run -p libsy-examples --example research_agent
+//! cargo run -p libsy --example research_agent
use std::error::Error;
use std::sync::Arc;
use async_trait::async_trait;
+use libsy::LlmClassifierOrchAlgo;
use libsy::{
Algorithm, Context, Decision, LlmResponse, LlmTarget, LlmTargetSet, Request, Response,
RoutedLlmClient,
};
-use libsy_examples::llm_class::LlmClassifierOrchAlgo;
use switchyard_protocol::{completion_text, text_request, text_response};
const CLASSIFIER: &str = "classifier/model";
diff --git a/crates/libsy-examples/examples/research_agent_core.rs b/crates/libsy/examples/research_agent_core.rs
similarity index 97%
rename from crates/libsy-examples/examples/research_agent_core.rs
rename to crates/libsy/examples/research_agent_core.rs
index 16fa959af..0199728bb 100644
--- a/crates/libsy-examples/examples/research_agent_core.rs
+++ b/crates/libsy/examples/research_agent_core.rs
@@ -7,15 +7,15 @@
//! surfaces as a `CallLlm` step. The agent makes the "real" model call itself and
//! fulfills the promise — this is the offload/streaming path ("ask, don't call").
//! The classifier's two steps show up as two `model call:` lines. Run with:
-//! cargo run -p libsy-examples --example research_agent_core
+//! cargo run -p libsy --example research_agent_core
use std::error::Error;
use std::sync::Arc;
+use libsy::LlmClassifierOrchAlgo;
use libsy::{
Algorithm, Context, Decision, LlmResponse, LlmTarget, LlmTargetSet, Request, Response, Step,
};
-use libsy_examples::llm_class::LlmClassifierOrchAlgo;
use switchyard_protocol::{completion_text, text_request, text_response};
use tokio_stream::StreamExt;
diff --git a/crates/libsy-examples/examples/streaming_agent.rs b/crates/libsy/examples/streaming_agent.rs
similarity index 98%
rename from crates/libsy-examples/examples/streaming_agent.rs
rename to crates/libsy/examples/streaming_agent.rs
index f871a4e07..5c245a8b9 100644
--- a/crates/libsy-examples/examples/streaming_agent.rs
+++ b/crates/libsy/examples/streaming_agent.rs
@@ -10,7 +10,7 @@
//! and prints each [`LlmResponseChunk`] as it arrives — true token streaming end to end.
//! Contrast [`Algorithm::run`], which would aggregate the same stream into one buffered
//! answer. Run with:
-//! cargo run -p libsy-examples --example streaming_agent
+//! cargo run -p libsy --example streaming_agent
use std::error::Error;
use std::io::Write;
diff --git a/crates/libsy/src/algorithms.rs b/crates/libsy/src/algorithms.rs
index 6f88a702d..ce17da02b 100644
--- a/crates/libsy/src/algorithms.rs
+++ b/crates/libsy/src/algorithms.rs
@@ -1,5 +1,6 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
+pub mod llm_class;
pub mod noop;
pub mod rand;
diff --git a/crates/libsy-examples/src/llm_class.rs b/crates/libsy/src/algorithms/llm_class.rs
similarity index 82%
rename from crates/libsy-examples/src/llm_class.rs
rename to crates/libsy/src/algorithms/llm_class.rs
index 67e9336a4..e3fe7ebf0 100644
--- a/crates/libsy-examples/src/llm_class.rs
+++ b/crates/libsy/src/algorithms/llm_class.rs
@@ -15,13 +15,11 @@ use std::sync::Arc;
use async_trait::async_trait;
-use libsy::{Algorithm, Context, Decision, Driver, LlmTargetSet, Request, Response};
-use switchyard_protocol::{completion_text, prompt_text, text_request};
+use crate::{Algorithm, Context, Decision, Driver, LlmTargetSet, Request, Response};
+use switchyard_protocol::{completion_text, LlmRequest, Message, Role};
-/// Preamble prepended to the user prompt when asking the classifier target for a
-/// strong-win-rate score.
-const CLASSIFIER_PROMPT_PREAMBLE: &str = "Rate how strongly this request needs a frontier model. \
- Reply with a single strong-win-rate score in [0, 1]:\n";
+/// The system prompt tells the classifier what to do
+const CLASSIFIER_SYSTEM_PROMPT: &str = "Rate how strongly this request needs a frontier model. Reply with a single strong-win-rate score in [0, 1].";
/// The tier a classifier score selected.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -96,6 +94,49 @@ impl LlmClassifierOrchAlgo {
}
}
+/// The first User message as well as the last <recent_turn_window> turns (User + Assistant).
+/// If fewer than 5 turns have happened, include the whole message.
+fn trim_messages(messages: &[Message], recent_turn_window: usize) -> Vec<Message> {
+ let mut system = Vec::new();
+ let mut first_user = None;
+ let mut first_user_idx = None;
+ for (idx, message) in messages.iter().enumerate() {
+ match message.role {
+ Role::System | Role::Developer => system.push(message.clone()),
+ Role::User if first_user.is_none() => {
+ first_user = Some(message.clone());
+ first_user_idx = Some(idx);
+ }
+ _ => {}
+ }
+ }
+ let Some(first_user) = first_user else {
+ return system;
+ };
+ let tail = messages
+ .iter()
+ .enumerate()
+ .filter(|(idx, message)| {
+ *idx > first_user_idx.unwrap_or(0)
+ && !matches!(message.role, Role::System | Role::Developer)
+ })
+ .map(|(_, message)| message.clone())
+ .collect::<Vec<_>>();
+ if recent_turn_window == 0 {
+ let mut out = system;
+ out.push(first_user);
+ if let Some(last_user) = tail.iter().rev().find(|message| message.role == Role::User) {
+ out.push(last_user.clone());
+ }
+ return out;
+ }
+ let mut out = system;
+ out.push(first_user);
+ let start = tail.len().saturating_sub(recent_turn_window);
+ out.extend_from_slice(&tail[start..]);
+ out
+}
+
#[async_trait]
impl Algorithm for LlmClassifierOrchAlgo {
async fn create_run_task(
@@ -104,18 +145,22 @@ impl Algorithm for LlmClassifierOrchAlgo {
driver: Driver,
request: Request,
) -> Result<Response, Box<dyn Error + Send + Sync>> {
- let user_prompt = prompt_text(&request.llm_request);
// The agent's inbound name rides through unchanged on every sub-call; the
// model each sub-call actually hits is carried by its decision instead.
let inbound = request.llm_request.model.clone();
+ let mut just_the_key_messages = trim_messages(&request.llm_request.messages, 5);
+ just_the_key_messages.insert(0, Message::text(Role::System, CLASSIFIER_SYSTEM_PROMPT));
+
// 1. Classify: call the classifier target with the score-eliciting prompt.
let classifier_target = self.target_set.get_target(&self.classifier_model)?;
+ let llm_request = LlmRequest {
+ model: inbound.clone(),
+ messages: just_the_key_messages,
+ ..LlmRequest::default()
+ };
let classify_request = Request {
- llm_request: text_request(
- inbound.clone(),
- format!("{CLASSIFIER_PROMPT_PREAMBLE}{user_prompt}"),
- ),
+ llm_request,
raw_request: request.raw_request.clone(),
metadata: request.metadata.clone(),
};
@@ -160,14 +205,9 @@ impl Algorithm for LlmClassifierOrchAlgo {
score,
tier: Some(tier),
});
- let routed_request = Request {
- llm_request: text_request(inbound, user_prompt),
- raw_request: request.raw_request,
- metadata: request.metadata,
- };
driver.info(ctx.clone(), route_decision.clone()).await?;
driver
- .call_llm_target(ctx, &routed_target, routed_request, route_decision)
+ .call_llm_target(ctx, &routed_target, request, route_decision)
.await
}
}
@@ -175,7 +215,7 @@ impl Algorithm for LlmClassifierOrchAlgo {
#[cfg(test)]
mod tests {
use super::*;
- use libsy::{LlmResponse, LlmTarget, Response, RoutedLlmClient};
+ use crate::{LlmResponse, LlmTarget, Response, RoutedLlmClient};
use std::sync::Mutex;
use switchyard_protocol::text_response;
@@ -239,7 +279,7 @@ mod tests {
fn request(prompt: &str) -> Request {
Request {
- llm_request: text_request(Some("auto".to_string()), prompt),
+ llm_request: switchyard_protocol::text_request(Some("auto".to_string()), prompt),
raw_request: None,
metadata: None,
}
@@ -338,20 +378,4 @@ mod tests {
assert_eq!(routed.score, None);
Ok(())
}
-
- #[tokio::test]
- async fn classifier_prompt_includes_the_user_text() -> Result<(), Box<dyn Error + Send + Sync>>
- {
- let (algo, seen) = algo(0.5, "0.9");
- orch(algo)
- .run(Context::default(), request("prove it"))
- .await?;
- let seen = seen.lock().map_err(|_| "lock poisoned")?;
- // Two calls: the classifier (preamble + user text), then the routed model.
- assert_eq!(seen.len(), 2);
- assert!(prompt_text(&seen[0].llm_request).contains("prove it"));
- assert!(prompt_text(&seen[0].llm_request).contains("frontier model"));
- assert_eq!(prompt_text(&seen[1].llm_request), "prove it");
- Ok(())
- }
}
diff --git a/crates/libsy/src/lib.rs b/crates/libsy/src/lib.rs
index 9d107233f..2cf6ecf3b 100644
--- a/crates/libsy/src/lib.rs
+++ b/crates/libsy/src/lib.rs
@@ -54,11 +54,13 @@
//!
//! ## Algorithms
//!
-//! [`RandomAlgo`] provides uniform random routing in the core crate. Worked
-//! implementations of an LLM classifier and a stateful ensemble, plus runnable
-//! agents, live in the `libsy-examples` crate.
+//! [`RandomAlgo`] provides uniform random routing in the core crate.
+//!
+//! [`LlmClassifierOrchAlgo`] classifies with one model, then routes to a
+//! strong/weak model depending on the classifier's choice.
mod algorithms;
+pub use algorithms::llm_class::{ClassifierDecision, ClassifierTier, LlmClassifierOrchAlgo};
pub use algorithms::noop::{NoopAlgo, NoopDecision};
pub use algorithms::rand::{RandomAlgo, RandomDecision};