Skip to content

The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]

2.4 KB
Raw Blame

Route boundary

These invariants apply to lifecycle-bearing public calls. Value-oriented APIs that return an already running Future retain their explicit contract

Route modules own public argument projection, route-specific host operations, response and error construction, and composition of the core route with the Python host. Provider dispatch, transport execution and normalization belong to core and provider crates. Runtime waiting, cancellation mechanics and execution state validation belong to litellm-host-python

Before execution starts, perform only admission checks needed to select native execution or legacy fallback. Do not fully project a request just to decide admission. Keep effectful settings reads, HTTP client acquisition, secret-source construction and caller context capture inside execution, after argument preparation and SDK preflight. Configure resources from that prepared view, never the original entrypoint kwargs

The host driver owns sequencing and terminal events. The bridge supplies fallible resource composition without exposing route types to the driver. An unstarted async call performs no resource setup. Setup errors after start follow the terminal failure contract and never authorize fallback or provider replay

Use the shared run_public_call boundary with hooks supplied by bridge composition. callbacks-legacy-python owns legacy argument sharing and Logging dispatch behind PublicCall and LegacyLogging. Route bindings supply callbacks-legacy-python::LoggingOperation when composing legacy logging and may retain the request needed for projection, but must not duplicate the legacy callback contract

Regression tests must observe that an unstarted call does no setup, hook and preflight rewrites affect resource configuration, setup failures reach the selected failure handler once, and provider work is not replayed. Retain existing read-point and object-identity guarantees while changing setup timing

Layout

messages/ is the reference shape. Each lifecycle route is a folder: mod.rs holds run_<route> and the thin sync and async #[pyfunction] entrypoints that take NativeCall and call it, host.rs holds the PythonBinding host, and anything else route-specific (projection, error mapping, extra pyclasses) gets its own sibling file. Scaffolded routes without a lifecycle (embeddings, audio transcription) stay a single <route>.rs