Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust
1.8 KB
Raw Blame
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

//! Request-scoped observations emitted while serving an algorithm run.

use std::sync::Arc;
use std::time::Duration;

use switchyard_libsy::OutcomeMetadata;
use switchyard_protocol::{ModelId, Usage};

/// One completed model call observed while serving an algorithm run.
#[derive(Clone, Debug)]
pub struct ModelCallObservation {
    /// Model selected for the completed call.
    pub selected_model: ModelId,
    /// Whether the call completed successfully.
    pub is_success: bool,
    /// Time spent waiting for the model call to resolve.
    pub duration: Duration,
    /// Normalized usage for a buffered successful response.
    pub usage: Option<Usage>,
}

/// Events emitted inline while [`crate::run()`] serves a routing request.
#[derive(Clone, Debug)]
pub enum RunObservation {
    /// Metadata attached to the completed routing outcome.
    Outcome(OutcomeMetadata),
    /// A completed model call requested by the algorithm for routing work.
    LlmCall(ModelCallObservation),
    /// A completed decision call requested by the algorithm for routing work.
    DecisionCall(ModelCallObservation),
    /// A completed terminal model call made from the routing outcome.
    AnswerCall(ModelCallObservation),
    /// Routing time recorded by the `switchyard.routing_overhead_ms` metric.
    RoutingOverhead(Duration),
}

/// Request-scoped callback for algorithm-run observations.
///
/// The runner invokes this callback inline while resolving observations. Several
/// runs may invoke the same observer concurrently, so implementations must be
/// thread-safe, fast, and non-blocking.
pub type RunObserver = Arc<dyn Fn(RunObservation) + Send + Sync>;