Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

feat(ai-gateway): route models by connection slug (#18628)

Tobias Wochinger6c1fd54fa718
diff --git a/ai-gateway/README.md b/ai-gateway/README.md
index 09029b52fc..524226c1b6 100644
--- a/ai-gateway/README.md
+++ b/ai-gateway/README.md
@@ -78,8 +78,8 @@ do not load dotenv files:
| `LANGFUSE_LOG_FORMAT` | `text` | `text`, `json` |
| `LANGFUSE_LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error`, `fatal` |
| `LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS` | `10` | Integer from 1 to 300 |
-| `LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS` | `128` | Positive integer up to Tokio's semaphore capacity; authenticated requests per instance |
-| `LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS` | `128` | Positive integer up to Tokio's semaphore capacity; concurrent Web resolutions per instance |
+| `LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS` | `128` | Positive integer up to Tokio's semaphore capacity; buffered and active requests per instance |
+| `LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS` | `128` | Positive integer up to Tokio's semaphore capacity; separate model-catalog and Web-resolution budgets per instance |
| `LANGFUSE_AI_GATEWAY_TELEMETRY_BUFFER_BYTES` | `67108864` (64 MiB) | Integer from 4194304 (4 MiB) to 4294967296 (4 GiB); span and credential bytes an instance holds for upload before dropping new records |
The gateway shares `LANGFUSE_LOG_LEVEL` with Web and worker. Values are lowercase;
@@ -172,11 +172,10 @@ does not set CORS headers, so no `Access-Control-Expose-Headers` is involved.
### Resolution context
Once Web resolution succeeds, the request also carries its tenant, and once it is
-sent upstream, the provider connection it was sent to. Requests rejected
-before resolution carry none of them. A resolved request that fails before reaching
-the provider, such as a 413 body or 503 execution capacity rejection, carries only
-the tenant. Provider failures before response headers (502/504) still name the
-connection that was tried.
+sent upstream, the provider connection it was sent to. Requests rejected before
+resolution, including invalid or oversized bodies, carry none of them. A resolved
+request rejected by execution admission carries only the tenant. Provider failures
+before response headers (502/504) still name the connection that was tried.
| Response header | Server span field | JSON log key | Value |
| --- | --- | --- | --- |
@@ -225,7 +224,7 @@ dimensions. The histogram's count supplies request volume and error rates,
including responses rejected before provider execution. Metrics are independent
of trace sampling. Health probes are excluded.
-Gateway-specific metrics include `gateway.active` (resolution/execution),
+Gateway-specific metrics include `gateway.active` (model_catalog/resolution/execution),
`gateway.phase.duration` (execution/provider.first_byte), `gateway.executions`,
`gateway.admission.rejected`, and `gateway.telemetry.records`. Durations use seconds;
first-byte timing measures provider bytes arriving at the gateway, not first-token
@@ -372,14 +371,18 @@ error type follows the status (`authentication_error`, `permission_error`,
`not_found_error`, `request_too_large`, `invalid_request_error`, `overloaded_error`,
`api_error`).
-The gateway authenticates through Web before reserving execution capacity or
-reading the request body. Web resolution uses a separate concurrency budget,
-released after resolution finishes; authenticated requests then reserve execution
-capacity through body reading and response completion. Either budget returns 503
-immediately when full, without a waiting queue. Both defaults of 128 are provisional
-guardrails, not measured capacity: tune them independently using load tests for the
-instance resources, request sizes and stream durations. These limits bound work;
-they do not guarantee fairness between clients or tenants.
+After validating the credential header, the gateway buffers and validates the request
+under the active-request admission budget, then resolves the credential and
+model-aware route through Web under a separate budget. It holds the body slot through
+the provider request. Model catalogs use another budget so slow catalog discovery
+cannot block inference resolution.
+Execution capacity remains reserved through response completion. An exhausted budget
+returns 503 immediately without a waiting queue. Request-body and execution budgets
+use the active-request limit; model-catalog and Web-call budgets use the resolution
+limit. Their defaults of 128 are provisional
+guardrails, not measured capacity: tune them using load tests for the instance
+resources, request sizes and stream durations. These limits bound work; they do not
+guarantee fairness between clients or tenants.
Other limits are 10 MiB request bodies in both namespaces, 30 seconds to read a request, 5 seconds to
connect, 120 seconds for provider response headers or an individual upstream read,
@@ -818,15 +821,19 @@ async fn example(web_base_url: &str, service_key: &str, gateway_key: &str)
-> Result<(), ResolutionError>
{
let control_plane = ControlPlaneClient::new(ControlPlaneConfig::new(web_base_url, service_key)?)?;
- let context = control_plane.resolve(gateway_key, ApiFormat::OpenAiResponses).await?;
+ let request_id = "018f47a2-4d6e-7a61-b495-5e5f665c1e12";
+ let context = control_plane
+ .resolve(gateway_key, ApiFormat::OpenAiResponses, "gpt-4.1", request_id)
+ .await?;
assert_eq!(context.connection().base_url(), "https://api.openai.com/v1");
Ok(())
}
```
-Each call sends `POST <base path>/api/internal/ai-gateway/v1/resolve` with
+After extracting the model, inference sends
+`POST <base path>/api/internal/ai-gateway/v1/resolve` with
`Authorization: Bearer <gateway key>`, the Web v1 HMAC header, and exactly
-`{"apiFormat":"<format>"}`, where the format is the one the public route serves
+`{"apiFormat":"<format>","model":"<model>"}`, where the format is the one the public route serves
(`openai.chat-completions` for `/openai/v1/chat/completions`, `openai.responses` for
the other `/openai/v1` routes). The client neither receives nor
parses an inference request body. Reuse the resolver across requests: its connection
@@ -896,8 +903,9 @@ cargo test --locked
- `http.rs`: credential extraction, bounded body reads and gateway error envelopes.
- `inference.rs`: `InferenceService` coordinates resolution, admission and forwarding.
- `providers/mod.rs`: `ProviderTransport` with shared admission, the `Route` enum of
- official operations (method, upstream path, capture, forwarded query) and credential
+ official operations (method, upstream path and capture), model rewriting and credential
placement.
+- `providers/model_list`: provider-native OpenAI and Anthropic model catalog responses.
- `transport/mod.rs`: per-format header policies, bounded byte relay and response lifetime.
- `resolution/mod.rs`: trusted base URL configuration and bounded Web HTTP client.
- `resolution/contracts.rs`: strict Web response validation and immutable execution context.
diff --git a/ai-gateway/src/capture/anthropic_messages.rs b/ai-gateway/src/capture/anthropic_messages.rs
index 3d158a1210..7a7f44f450 100644
--- a/ai-gateway/src/capture/anthropic_messages.rs
+++ b/ai-gateway/src/capture/anthropic_messages.rs
@@ -280,10 +280,6 @@ impl AnthropicMessagesCapture {
self.facts.capture_complete = self.request_complete && self.facts.output_complete;
}
- pub fn requested_model(&self) -> Option<&str> {
- self.facts.requested_model.as_deref()
- }
-
pub fn into_facts(mut self) -> ProviderFacts {
if matches!(self.body, ResponseBody::Sse(_)) {
self.end_body();
diff --git a/ai-gateway/src/capture/anthropic_messages/tests.rs b/ai-gateway/src/capture/anthropic_messages/tests.rs
index cd6017d354..b29e40b1d2 100644
--- a/ai-gateway/src/capture/anthropic_messages/tests.rs
+++ b/ai-gateway/src/capture/anthropic_messages/tests.rs
@@ -558,8 +558,8 @@ async fn streamed_messages_upload_one_generation_with_native_usage() {
context,
&headers,
Bytes::from_static(REQUEST),
+ crate::providers::model_range(REQUEST).unwrap(),
Route::AnthropicMessages,
- None,
&mut crate::correlation::RequestCorrelation::default(),
)
.await
diff --git a/ai-gateway/src/capture/mod.rs b/ai-gateway/src/capture/mod.rs
index 70337a9813..ca95bc8349 100644
--- a/ai-gateway/src/capture/mod.rs
+++ b/ai-gateway/src/capture/mod.rs
@@ -82,14 +82,6 @@ impl ProtocolCapture {
}
}
- fn requested_model(&self) -> Option<&str> {
- match self {
- Self::OpenAiResponses(capture) => capture.requested_model(),
- Self::OpenAiChatCompletions(capture) => capture.requested_model(),
- Self::AnthropicMessages(capture) => capture.requested_model(),
- }
- }
-
fn client_metadata(&self) -> Option<&Map<String, Value>> {
match self {
Self::OpenAiResponses(capture) => capture.client_metadata(),
@@ -109,6 +101,7 @@ pub(crate) struct ExecutionCapture {
completion_start_ms: Option<u128>,
http_status: Option<u16>,
metadata: Value,
+ provider_model: Option<String>,
delivery: Option<(telemetry::Telemetry, telemetry::DeliveryContext)>,
}
@@ -126,6 +119,7 @@ impl ExecutionCapture {
completion_start_ms: None,
http_status: None,
metadata: json!({}),
+ provider_model: None,
delivery: None,
}
}
@@ -181,6 +175,7 @@ impl ExecutionCapture {
"key_metadata": key_metadata,
"ingestion_mode": if full { "full" } else { "usage" },
}),
+ provider_model: None,
delivery: None,
}
}
@@ -205,11 +200,8 @@ impl ExecutionCapture {
self.delivery = Some((telemetry, delivery));
}
- /// The request's `model`, if the capture could parse it.
- pub fn requested_model(&self) -> Option<&str> {
- self.protocol
- .as_ref()
- .and_then(ProtocolCapture::requested_model)
+ pub fn set_provider_model(&mut self, model: &str) {
+ self.provider_model = bounded_string(model);
}
pub fn record_response(&mut self, status: u16, headers: &HeaderMap) {
@@ -244,7 +236,13 @@ impl ExecutionCapture {
let Some(protocol) = self.protocol.take() else {
return;
};
- let (api_format, inference) = protocol.into_facts();
+ let (api_format, mut inference) = protocol.into_facts();
+ if let Some(provider_model) = self.provider_model.take() {
+ if inference.model == inference.requested_model {
+ inference.model = Some(provider_model.clone());
+ }
+ inference.requested_model = Some(provider_model);
+ }
let facts = InferenceFacts {
api_format,
start_time_unix_ms: self.start_time_unix_ms,
diff --git a/ai-gateway/src/capture/openai_chat_completions.rs b/ai-gateway/src/capture/openai_chat_completions.rs
index a7ce3fdef1..bb29946883 100644
--- a/ai-gateway/src/capture/openai_chat_completions.rs
+++ b/ai-gateway/src/capture/openai_chat_completions.rs
@@ -341,10 +341,6 @@ impl OpenAiChatCompletionsCapture {
self.facts.capture_complete = self.request_complete && self.facts.output_complete;
}
- pub fn requested_model(&self) -> Option<&str> {
- self.facts.requested_model.as_deref()
- }
-
pub fn into_facts(mut self) -> ProviderFacts {
if matches!(self.body, ResponseBody::Sse(_)) {
self.end_body();
diff --git a/ai-gateway/src/capture/openai_responses.rs b/ai-gateway/src/capture/openai_responses.rs
index 34cb05ba62..5270cf489e 100644
--- a/ai-gateway/src/capture/openai_responses.rs
+++ b/ai-gateway/src/capture/openai_responses.rs
@@ -100,10 +100,6 @@ impl OpenAiResponsesCapture {
self.facts.capture_complete = self.request_complete && self.facts.output_complete;
}
- pub fn requested_model(&self) -> Option<&str> {
- self.facts.requested_model.as_deref()
- }
-
pub fn into_facts(mut self) -> ProviderFacts {
if matches!(self.body, ResponseBody::Sse(_)) {
self.end_body();
diff --git a/ai-gateway/src/capture/openai_responses/tests.rs b/ai-gateway/src/capture/openai_responses/tests.rs
index 19fe30c9a9..173fd4dc6e 100644
--- a/ai-gateway/src/capture/openai_responses/tests.rs
+++ b/ai-gateway/src/capture/openai_responses/tests.rs
@@ -529,7 +529,7 @@ async fn upstream_read_timeout_is_logged_as_timeout() {
provider.try_admit().unwrap(),
context,
&HeaderMap::new(),
- Bytes::from_static(b"{}"),
+ Bytes::from_static(br#"{"model":"requested"}"#),
)
.await
.unwrap();
@@ -584,7 +584,7 @@ async fn native_http_relay_logs_once_on_eof_drop_and_unpolled_deadline() {
provider.try_admit().unwrap(),
context,
&HeaderMap::new(),
- Bytes::from_static(br#"{"input":"request-content"}"#),
+ Bytes::from_static(br#"{"model":"requested","input":"request-content"}"#),
)
.await
.unwrap();
@@ -648,7 +648,7 @@ async fn provider_future_finalizes_on_timeout_and_cancellation_before_headers()
provider.try_admit().unwrap(),
context,
&headers,
- Bytes::from_static(b"{}"),
+ Bytes::from_static(br#"{"model":"requested"}"#),
);
if cancel {
tokio::select! {
@@ -745,7 +745,9 @@ async fn client_compression_preferences_do_not_disable_capture() {
provider.try_admit().unwrap(),
context,
&headers,
- Bytes::from(json!({"input":"hello","stream":streaming}).to_string()),
+ Bytes::from(
+ json!({"model":"requested","input":"hello","stream":streaming}).to_string(),
+ ),
)
.await
.unwrap();
diff --git a/ai-gateway/src/http.rs b/ai-gateway/src/http.rs
index 261b929a60..1582536360 100644
--- a/ai-gateway/src/http.rs
+++ b/ai-gateway/src/http.rs
@@ -1,4 +1,4 @@
-use std::{error::Error, sync::Arc, time::Duration};
+use std::{error::Error, ops::Range, sync::Arc, time::Duration};
use axum::{
Json, Router,
@@ -8,13 +8,14 @@ use axum::{
response::{IntoResponse, Response},
routing::{get, post},
};
-use serde::Serialize;
+use serde::{Deserialize, Serialize};
+use serde_json::value::RawValue;
use tracing::Instrument;
use crate::{
correlation::{RequestCorrelation, assign_request_id},
inference::{InferenceService, RequestPreparationError},
- providers::{ProviderError, Route, forwarded_query},
+ providers::{ProviderError, Route},
resolution::{ApiFormat, ResolutionError},
server::GatewayLifecycleState,
};
@@ -53,102 +54,203 @@ pub fn router(inference: Option<InferenceService>, lifecycle: GatewayLifecycleSt
}
async fn handle_responses(State(state): State<InferenceRouteState>, request: Request) -> Response {
- handle(state, request, Route::OpenAiResponses).await
+ handle_inference(state, request, Route::OpenAiResponses).await
}
async fn handle_responses_compact(
State(state): State<InferenceRouteState>,
request: Request,
) -> Response {
- handle(state, request, Route::OpenAiResponsesCompact).await
+ handle_inference(state, request, Route::OpenAiResponsesCompact).await
}
async fn handle_chat_completions(
State(state): State<InferenceRouteState>,
request: Request,
) -> Response {
- handle(state, request, Route::OpenAiChatCompletions).await
+ handle_inference(state, request, Route::OpenAiChatCompletions).await
}
async fn handle_openai_models(
State(state): State<InferenceRouteState>,
request: Request,
) -> Response {
- handle(state, request, Route::OpenAiModels).await
+ handle_models(state, request, ApiFormat::OpenAiResponses).await
}
async fn handle_messages(State(state): State<InferenceRouteState>, request: Request) -> Response {
- handle(state, request, Route::AnthropicMessages).await
+ handle_inference(state, request, Route::AnthropicMessages).await
}
async fn handle_count_tokens(
State(state): State<InferenceRouteState>,
request: Request,
) -> Response {
- handle(state, request, Route::AnthropicCountTokens).await
+ handle_inference(state, request, Route::AnthropicCountTokens).await
}
async fn handle_anthropic_models(
State(state): State<InferenceRouteState>,
request: Request,
) -> Response {
- handle(state, request, Route::AnthropicModels).await
+ handle_models(state, request, ApiFormat::AnthropicMessages).await
}
-async fn handle(state: InferenceRouteState, mut request: Request, route: Route) -> Response {
+async fn handle_inference(
+ state: InferenceRouteState,
+ mut request: Request,
+ route: Route,
+) -> Response {
let mut correlation = request
.extensions_mut()
.remove::<RequestCorrelation>()
.unwrap_or_default();
- let mut response = match relay(state, request, route, &mut correlation).await {
+ let result = match prepare_request(&state, request.headers(), route.api_format()) {
+ Ok((inference, gateway_key)) => {
+ relay(&inference, &gateway_key, request, route, &mut correlation).await
+ }
+ Err(error) => Err(error),
+ };
+ finish_response(result, route.api_format(), &correlation)
+}
+
+async fn handle_models(
+ state: InferenceRouteState,
+ mut request: Request,
+ api_format: ApiFormat,
+) -> Response {
+ let correlation = request
+ .extensions_mut()
+ .remove::<RequestCorrelation>()
+ .unwrap_or_default();
+ let result = match prepare_request(&state, request.headers(), api_format) {
+ Ok((inference, gateway_key)) => {
+ models(&inference, &gateway_key, api_format, &correlation).await
+ }
+ Err(error) => Err(error),
+ };
+ finish_response(result, api_format, &correlation)
+}
+
+fn prepare_request(
+ state: &InferenceRouteState,
+ headers: &HeaderMap,
+ api_format: ApiFormat,
+) -> Result<(Arc<InferenceService>, String), InferenceHttpError> {
+ let inference = state
+ .inference
+ .as_ref()
+ .filter(|_| state.lifecycle.is_ready())
+ .cloned()
+ .ok_or(InferenceHttpError::Unavailable)?;
+ let gateway_key = gateway_key(headers, api_format)?.to_owned();
+ Ok((inference, gateway_key))
+}
+
+fn finish_response(
+ result: Result<Response, InferenceHttpError>,
+ api_format: ApiFormat,
+ correlation: &RequestCorrelation,
+) -> Response {
+ let mut response = match result {
Ok(response) => response,
- Err(error) => error.into_native_response(route.api_format(), correlation.id()),
+ Err(error) => error.into_native_response(api_format, correlation.id()),
};
correlation.apply_generation_headers(response.headers_mut());
correlation.apply_resolution_headers(response.headers_mut());
response
}
+async fn models(
+ inference: &InferenceService,
+ gateway_key: &str,
+ api_format: ApiFormat,
+ correlation: &RequestCorrelation,
+) -> Result<Response, InferenceHttpError> {
+ let body = inference
+ .fetch_model_catalog(gateway_key, api_format, correlation)
+ .await
+ .map_err(InferenceHttpError::Resolution)?;
+ Ok(([(header::CONTENT_TYPE, "application/json")], body).into_response())
+}
+
async fn relay(
- state: InferenceRouteState,
+ inference: &InferenceService,
+ gateway_key: &str,
request: Request,
route: Route,
correlation: &mut RequestCorrelation,
) -> Result<Response, InferenceHttpError> {
- let inference = state
- .inference
- .as_ref()
- .filter(|_| state.lifecycle.is_ready())
- .ok_or(InferenceHttpError::Unavailable)?;
- let gateway_key = gateway_key(request.headers(), route.api_format())?.to_owned();
+ // Model extraction and upstream rewriting require an uncompressed JSON body.
+ if !crate::transport::identity_encoding(request.headers()) {
+ return Err(InferenceHttpError::UnsupportedEncoding);
+ }
+ let _body_permit = inference
+ .try_reserve_request_body()
+ .map_err(InferenceHttpError::from)?;
+ let (parts, body) = request.into_parts();
+ let request_body = request_body_and_model(body).await?;
let (permit, context) = inference
- .resolve_and_admit(&gateway_key, route.api_format(), correlation)
+ .resolve_and_admit(
+ gateway_key,
+ route.api_format(),
+ &request_body.model,
+ correlation,
+ )
.await
- .map_err(|error| match error {
- RequestPreparationError::Resolution(error) => InferenceHttpError::Resolution(error),
- RequestPreparationError::Provider(error) => InferenceHttpError::Provider(error),
- })?;
- let query = forwarded_query(route, request.uri().query());
- let (parts, body) = request.into_parts();
- let bytes = if route.method() == axum::http::Method::GET {
- Bytes::new()
- } else {
- read_request_body(body).await?
- };
+ .map_err(InferenceHttpError::from)?;
inference
.forward(
permit,
context,
&parts.headers,
- bytes,
+ request_body.bytes,
+ request_body.model_range,
route,
- query.as_deref(),
correlation,
)
.await
.map_err(InferenceHttpError::Provider)
}
+struct InferenceRequestBody {
+ bytes: Bytes,
+ model: String,
+ model_range: Range<usize>,
+}
+
+#[derive(Deserialize)]
+struct InferenceRequest<'a> {
+ #[serde(borrow)]
+ model: &'a RawValue,
+}
+
+async fn request_body_and_model(body: Body) -> Result<InferenceRequestBody, InferenceHttpError> {
+ let bytes = read_request_body(body).await?;
+ let request = serde_json::from_slice::<InferenceRequest<'_>>(&bytes)
+ .map_err(|_| InferenceHttpError::InvalidBody)?;
+ let raw_model = request.model.get();
+ let model =
+ serde_json::from_str::<String>(raw_model).map_err(|_| InferenceHttpError::InvalidBody)?;
+ if model.is_empty() {
+ return Err(InferenceHttpError::InvalidBody);
+ }
+ let start = raw_model
+ .as_ptr()
+ .addr()
+ .checked_sub(bytes.as_ptr().addr())
+ .ok_or(InferenceHttpError::InvalidBody)?;
+ let end = start
+ .checked_add(raw_model.len())
+ .filter(|end| *end <= bytes.len())
+ .ok_or(InferenceHttpError::InvalidBody)?;
+ Ok(InferenceRequestBody {
+ bytes,
+ model,
+ model_range: start..end,
+ })
+}
+
/// Buffer the client body. The wait is mostly the caller's upload, so it gets its
/// own span and the measured size lands on the server span for aggregation.
async fn read_request_body(body: Body) -> Result<Bytes, InferenceHttpError> {
@@ -240,11 +342,21 @@ enum InferenceHttpError {
AmbiguousCredential,
TooLarge,
RequestTimeout,
+ UnsupportedEncoding,
InvalidBody,
Resolution(ResolutionError),
Provider(ProviderError),
}
+impl From<RequestPreparationError> for InferenceHttpError {
+ fn from(error: RequestPreparationError) -> Self {
+ match error {
+ RequestPreparationError::Resolution(error) => Self::Resolution(error),
+ RequestPreparationError::Provider(error) => Self::Provider(error),
+ }
+ }
+}
+
#[derive(Serialize)]
struct OpenAiErrorResponse<'a> {
error: OpenAiErrorDetail,
@@ -282,11 +394,13 @@ impl InferenceHttpError {
Self::AmbiguousCredential => ("authentication", "ambiguous_credential"),
Self::TooLarge => ("request", "body_too_large"),
Self::RequestTimeout => ("request", "body_timeout"),
+ Self::UnsupportedEncoding => ("request", "unsupported_content_encoding"),
Self::InvalidBody => ("request", "invalid_body"),
Self::Resolution(error) => (
"resolution",
match error {
R::InvalidCredential | R::Authentication => "authentication",
+ R::InvalidRequest => "invalid_request",
R::Forbidden => "forbidden",
R::NoRoute => "no_route",
R::Timeout => "timeout",
@@ -348,12 +462,24 @@ impl InferenceHttpError {
"invalid_request_error",
"request_timeout",
),
+ Self::UnsupportedEncoding => (
+ StatusCode::UNSUPPORTED_MEDIA_TYPE,
+ "Unsupported request Content-Encoding",
+ "invalid_request_error",
+ "unsupported_content_encoding",
+ ),
Self::InvalidBody => (
StatusCode::BAD_REQUEST,
"Invalid request body",
"invalid_request_error",
"invalid_body",
),
+ Self::Resolution(R::InvalidRequest) => (
+ StatusCode::BAD_REQUEST,
+ "Invalid gateway model",
+ "invalid_request_error",
+ "invalid_model",
+ ),
Self::Unavailable
| Self::Resolution(R::Unavailable | R::Configuration)
| Self::Provider(ProviderError::Configuration | ProviderError::Busy) => (
diff --git a/ai-gateway/src/http/tests.rs b/ai-gateway/src/http/tests.rs
index 267b4f2358..b205f75f2e 100644
--- a/ai-gateway/src/http/tests.rs
+++ b/ai-gateway/src/http/tests.rs
@@ -5,12 +5,16 @@ use crate::{
test_support::{FakeServer, resolution_response, resolution_response_for},
};
use futures_util::{StreamExt, stream};
-use std::{convert::Infallible, task::Poll};
+use std::{
+ convert::Infallible,
+ io::Write,
+ sync::atomic::{AtomicBool, Ordering},
+};
use tokio::{net::TcpListener, sync::oneshot, task::JoinHandle};
use tower::ServiceExt;
#[tokio::test]
-async fn invalid_credentials_are_rejected_before_admission_or_body_reads() {
+async fn control_plane_rejections_do_not_consume_provider_capacity() {
let web = FakeServer::start(|_| async {
Response::builder()
.status(StatusCode::UNAUTHORIZED)
@@ -35,17 +39,12 @@ async fn invalid_credentials_are_rejected_before_admission_or_body_reads() {
.await;
// Exercise both an exhausted and an available execution pool.
for occupied in [Some(permit), None] {
- let body = Body::from_stream(stream::poll_fn(
- |_| -> Poll<Option<Result<&'static str, Infallible>>> {
- panic!("an invalid credential must be rejected before polling the body");
- },
- ));
let response = tokio::time::timeout(
Duration::from_secs(2),
gateway.app.clone().oneshot(
Request::post("/openai/v1/responses")
.header(header::AUTHORIZATION, "Bearer invalid-key")
- .body(body)
+ .body(Body::from(r#"{"model":"Production key:test"}"#))
.unwrap(),
),
)
@@ -97,7 +96,7 @@ async fn resolution_capacity_is_bounded_and_released_on_cancellation_and_failure
let request = |key| {
Request::post("/openai/v1/responses")
.header(header::AUTHORIZATION, key)
- .body(Body::from("{}"))
+ .body(Body::from(r#"{"model":"Production key:test"}"#))
.unwrap()
};
let mut stalled = Box::pin(gateway.app.clone().oneshot(request("Bearer stalled")));
@@ -109,14 +108,26 @@ async fn resolution_capacity_is_bounded_and_released_on_cancellation_and_failure
})
.await
.unwrap();
+ let body_polled = Arc::new(AtomicBool::new(false));
+ let observed = body_polled.clone();
+ let body = Body::from_stream(stream::once(async move {
+ observed.store(true, Ordering::SeqCst);
+ Ok::<_, Infallible>(Bytes::from_static(br#"{"model":"Production key:test"}"#))
+ }));
let busy = tokio::time::timeout(
Duration::from_secs(2),
- gateway.app.clone().oneshot(request("Bearer valid")),
+ gateway.app.clone().oneshot(
+ Request::post("/openai/v1/responses")
+ .header(header::AUTHORIZATION, "Bearer valid")
+ .body(body)
+ .unwrap(),
+ ),
)
.await
.unwrap()
.unwrap();
assert_eq!(busy.status(), StatusCode::SERVICE_UNAVAILABLE);
+ assert!(!body_polled.load(Ordering::SeqCst));
assert_eq!(web.calls(), 1);
assert_eq!(provider.calls(), 0);
@@ -138,6 +149,164 @@ async fn resolution_capacity_is_bounded_and_released_on_cancellation_and_failure
assert_eq!(provider.calls(), 1);
}
+#[tokio::test]
+async fn slow_body_reads_do_not_consume_resolution_capacity() {
+ let web = FakeServer::start(|request| async move {
+ if request.uri().path() == "/api/internal/ai-gateway/v1/models" {
+ return Response::new(Body::from(
+ r#"{"data":[{"id":"Production key:gpt-4.1","provider":"OPENAI","canonicalSlug":"gpt-4.1","displayName":"GPT-4.1","createdAt":"1970-01-01T00:00:10.000Z"}]}"#,
+ ));
+ }
+ resolution_response("provider-secret")
+ })
+ .await;
+ let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;
+ let gateway = Gateway::start(Some(InferenceService::for_test(
+ web.control_plane(),
+ ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),
+ 1,
+ )))
+ .await;
+ let body_entered = Arc::new(tokio::sync::Notify::new());
+ let entered = body_entered.clone();
+ let held_body = Body::from_stream(stream::once(async move {
+ entered.notify_one();
+ std::future::pending::<Result<Bytes, Infallible>>().await
+ }));
+ let mut held = Box::pin(
+ gateway.app.clone().oneshot(
+ Request::post("/openai/v1/responses")
+ .header(header::AUTHORIZATION, "Bearer held")
+ .body(held_body)
+ .unwrap(),
+ ),
+ );
+ tokio::time::timeout(Duration::from_secs(2), async {
+ tokio::select! {
+ () = body_entered.notified() => {},
+ _ = &mut held => panic!("body read should remain pending"),
+ }
+ })
+ .await
+ .unwrap();
+ let body_polled = Arc::new(AtomicBool::new(false));
+ let observed = body_polled.clone();
+ let body = Body::from_stream(stream::once(async move {
+ observed.store(true, Ordering::SeqCst);
+ Ok::<_, Infallible>(Bytes::from_static(br#"{"model":"connection:model"}"#))
+ }));
+ let busy = gateway
+ .app
+ .clone()
+ .oneshot(
+ Request::post("/openai/v1/responses")
+ .header(header::AUTHORIZATION, "Bearer busy")
+ .body(body)
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(busy.status(), StatusCode::SERVICE_UNAVAILABLE);
+ assert!(!body_polled.load(Ordering::SeqCst));
+ let models = gateway
+ .app
+ .clone()
+ .oneshot(
+ Request::get("/openai/v1/models")
+ .header(header::AUTHORIZATION, "Bearer models")
+ .body(Body::empty())
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(models.status(), StatusCode::OK);
+ assert_eq!(web.calls(), 1);
+ assert_eq!(provider.calls(), 0);
+ drop(held);
+ let response = gateway
+ .app
+ .clone()
+ .oneshot(
+ Request::post("/openai/v1/responses")
+ .header(header::AUTHORIZATION, "Bearer released")
+ .body(Body::from(r#"{"model":"connection:model"}"#))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(response.status(), StatusCode::OK);
+ assert_eq!(web.calls(), 2);
+ assert_eq!(provider.calls(), 1);
+}
+
+#[tokio::test]
+async fn slow_model_catalogs_do_not_consume_resolution_capacity() {
+ let entered = Arc::new(tokio::sync::Notify::new());
+ let started = entered.clone();
+ let web = FakeServer::start(move |request| {
+ let started = started.clone();
+ async move {
+ if request.uri().path() == "/api/internal/ai-gateway/v1/models" {
+ started.notify_one();
+ return std::future::pending().await;
+ }
+ resolution_response("provider-secret")
+ }
+ })
+ .await;
+ let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;
+ let gateway = Gateway::start(Some(InferenceService::for_test(
+ web.control_plane(),
+ ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),
+ 1,
+ )))
+ .await;
+ let mut held = Box::pin(
+ gateway.app.clone().oneshot(
+ Request::get("/openai/v1/models")
+ .header(header::AUTHORIZATION, "Bearer models-held")
+ .body(Body::empty())
+ .unwrap(),
+ ),
+ );
+ tokio::time::timeout(Duration::from_secs(2), async {
+ tokio::select! {
+ () = entered.notified() => {},
+ _ = &mut held => panic!("model catalog should remain pending"),
+ }
+ })
+ .await
+ .unwrap();
+
+ let inference = gateway
+ .app
+ .clone()
+ .oneshot(
+ Request::post("/openai/v1/responses")
+ .header(header::AUTHORIZATION, "Bearer inference")
+ .body(Body::from(r#"{"model":"connection:model"}"#))
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(inference.status(), StatusCode::OK);
+ let busy = gateway
+ .app
+ .clone()
+ .oneshot(
+ Request::get("/openai/v1/models")
+ .header(header::AUTHORIZATION, "Bearer models-busy")
+ .body(Body::empty())
+ .unwrap(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(busy.status(), StatusCode::SERVICE_UNAVAILABLE);
+ assert_eq!(web.calls(), 2);
+ assert_eq!(provider.calls(), 1);
+ drop(held);
+}
+
impl Gateway {
async fn start(inference: Option<InferenceService>) -> Self {
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
@@ -234,6 +403,10 @@ fn assert_pre_upstream_headers(headers: &reqwest::header::HeaderMap, resolved: b
}
#[tokio::test]
+#[expect(
+ clippy::too_many_lines,
+ reason = "the table keeps native error envelope coverage in one place"
+)]
async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body() {
let web = FakeServer::start(|request| async move {
if request.headers().get(header::AUTHORIZATION)
@@ -278,7 +451,10 @@ async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body()
false,
),
(
- configured.chat_completions().bearer_auth("rejected"),
+ configured
+ .chat_completions()
+ .bearer_auth("rejected")
+ .body(r#"{"model":"Production key:test"}"#),
StatusCode::UNAUTHORIZED,
false,
false,
@@ -290,16 +466,22 @@ async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body()
.body(vec![b'x'; MAX_REQUEST_BYTES + 1]),
StatusCode::PAYLOAD_TOO_LARGE,
false,
- true,
+ false,
),
(
- configured.post().bearer_auth("rejected"),
+ configured
+ .post()
+ .bearer_auth("rejected")
+ .body(r#"{"model":"Production key:test"}"#),
StatusCode::UNAUTHORIZED,
false,
false,
),
(
- configured.messages().header("x-api-key", "rejected"),
+ configured
+ .messages()
+ .header("x-api-key", "rejected")
+ .body(r#"{"model":"Production key:test"}"#),
StatusCode::UNAUTHORIZED,
true,
false,
@@ -311,7 +493,7 @@ async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body()
.body(vec![b'x'; MAX_REQUEST_BYTES + 1]),
StatusCode::PAYLOAD_TOO_LARGE,
false,
- true,
+ false,
),
] {
let response = request
@@ -368,13 +550,15 @@ async fn resolved_requests_name_their_tenant_and_the_provider_connection_sent_to
(
anthropic(
"messages/count_tokens",
- r#"{"messages":[],"model":"claude-opus-4-1"}"#,
+ r#"{"messages":[],"model":"Production key:claude-opus-4-1"}"#,
),
"anthropic",
),
- (gateway.models().bearer_auth("gateway-openai"), "openai"),
(
- anthropic("messages", r#"{"model":"claude-opus-4-1","messages":[]}"#),
+ anthropic(
+ "messages",
+ r#"{"model":"Production key:claude-opus-4-1","messages":[]}"#,
+ ),
"anthropic",
),
] {
@@ -392,7 +576,7 @@ async fn resolved_requests_name_their_tenant_and_the_provider_connection_sent_to
// The model is an operator-side span and log field only.
assert_eq!(header("langfuse-model"), None);
}
- assert_eq!(provider.calls(), 3);
+ assert_eq!(provider.calls(), 2);
}
impl Drop for Gateway {
@@ -422,10 +606,13 @@ async fn native_json_and_sse_traverse_resolution_and_relay() {
.to_str()
.unwrap()
.to_owned();
- assert_eq!(
- to_bytes(request.into_body(), 1024).await.unwrap(),
- r#"{"apiFormat":"openai.responses"}"#
- );
+ let body: serde_json::Value =
+ serde_json::from_slice(&to_bytes(request.into_body(), 1024).await.unwrap()).unwrap();
+ assert_eq!(body["apiFormat"], "openai.responses");
+ assert!(matches!(
+ body["model"].as_str(),
+ Some("Production key:unchanged" | "Production key:anything")
+ ));
resolution_response(if key == "Bearer gateway-json" {
"provider-json"
} else {
@@ -447,14 +634,14 @@ async fn native_json_and_sse_traverse_resolution_and_relay() {
.to_owned();
let body = to_bytes(request.into_body(), 4096).await.unwrap();
if key == "Bearer provider-json" {
- assert_eq!(body, "{ \"model\":\"unchanged\", \"unknown\":true }");
+ assert_eq!(body, r#"{ "model":"provider-model", "unknown":true }"#);
Response::builder()
.header("content-type", "application/json")
.body(Body::from("{\"output\":[]}"))
.unwrap()
} else {
assert_eq!(key, "Bearer provider-sse");
- assert_eq!(body, r#"{"model":"anything","stream":true}"#);
+ assert_eq!(body, r#"{"model":"provider-model","stream":true}"#);
Response::builder()
.header("content-type", "text/event-stream")
.body(Body::from_stream(stream::iter([
@@ -470,12 +657,12 @@ async fn native_json_and_sse_traverse_resolution_and_relay() {
.post()
.bearer_auth("gateway-json")
.header("x-langfuse-project-id", "attacker-project")
- .body("{ \"model\":\"unchanged\", \"unknown\":true }")
+ .body("{ \"model\":\"Production key:unchanged\", \"unknown\":true }")
.send();
let sse = gateway
.post()
.bearer_auth("gateway-sse")
- .body(r#"{"model":"anything","stream":true}"#)
+ .body(r#"{"model":"Production key:anything","stream":true}"#)
.send();
let (json, sse) = tokio::join!(json, sse);
let json = json.unwrap();
@@ -492,8 +679,20 @@ async fn native_json_and_sse_traverse_resolution_and_relay() {
}
#[tokio::test]
-async fn compact_relays_opaque_json_and_models_proxies_openai_list_without_query() {
- let web = FakeServer::start(|_| async { resolution_response("provider-secret") }).await;
+async fn compact_rewrites_model_and_models_come_from_the_control_plane() {
+ let web = FakeServer::start(|request| async move {
+ if request.uri().path() == "/api/internal/ai-gateway/v1/models" {
+ assert_eq!(request.uri().query(), Some("api_format=openai.responses"));
+ return Response::builder()
+ .header("content-type", "application/json")
+ .body(Body::from(
+ r#"{"data":[{"id":"Production key:gpt-4.1","provider":"OPENAI","canonicalSlug":"gpt-4.1","displayName":"GPT-4.1","createdAt":"1970-01-01T00:00:10.000Z"}]}"#,
+ ))
+ .unwrap();
+ }
+ resolution_response("provider-secret")
+ })
+ .await;
let provider = FakeServer::start(|request| async move {
let method = request.method().clone();
let uri = request.uri().to_string();
@@ -502,7 +701,7 @@ async fn compact_relays_opaque_json_and_models_proxies_openai_list_without_query
("POST", "/v1/responses/compact") => {
assert_eq!(
body,
- r#"{"model":"gpt-4.1","encrypted_content":"opaque-ciphertext"}"#
+ r#"{"model":"provider-model","encrypted_content":"opaque-ciphertext"}"#
);
Response::builder()
.header("content-type", "application/json")
@@ -511,15 +710,6 @@ async fn compact_relays_opaque_json_and_models_proxies_openai_list_without_query
))
.unwrap()
}
- ("GET", "/v1/models") => {
- assert!(body.is_empty());
- Response::builder()
- .header("content-type", "application/json")
- .body(Body::from(
- r#"{"object":"list","data":[{"id":"gpt-4.1","object":"model"}]}"#,
- ))
- .unwrap()
- }
other => panic!("unexpected provider request {other:?}"),
}
})
@@ -528,7 +718,7 @@ async fn compact_relays_opaque_json_and_models_proxies_openai_list_without_query
let compact = gateway
.compact()
.bearer_auth("gateway-key")
- .body(r#"{"model":"gpt-4.1","encrypted_content":"opaque-ciphertext"}"#)
+ .body(r#"{"model":"Production key:gpt-4.1","encrypted_content":"opaque-ciphertext"}"#)
.send()
.await
.unwrap();
@@ -549,10 +739,65 @@ async fn compact_relays_opaque_json_and_models_proxies_openai_list_without_query
assert_eq!(models.status(), StatusCode::OK);
assert_eq!(
models.text().await.unwrap(),
- r#"{"object":"list","data":[{"id":"gpt-4.1","object":"model"}]}"#
+ r#"{"object":"list","data":[{"id":"Production key:gpt-4.1","object":"model","created":10,"owned_by":"openai"}]}"#
);
assert_eq!(web.calls(), 2);
- assert_eq!(provider.calls(), 2);
+ assert_eq!(provider.calls(), 1);
+}
+
+#[tokio::test]
+async fn model_routes_request_the_matching_control_plane_catalog() {
+ let web = FakeServer::start(|request| async move {
+ assert_eq!(request.uri().path(), "/api/internal/ai-gateway/v1/models");
+ let api_format = request
+ .uri()
+ .query()
+ .unwrap()
+ .strip_prefix("api_format=")
+ .unwrap();
+ let body = match api_format {
+ "openai.responses" => {
+ r#"{"data":[{"id":"Production key:gpt-4.1","provider":"OPENAI","canonicalSlug":"gpt-4.1","displayName":"GPT-4.1","createdAt":"1970-01-01T00:00:10.000Z"}]}"#
+ }
+ "anthropic.messages" => {
+ r#"{"data":[{"id":"Production key:claude-sonnet-4-5","provider":"ANTHROPIC","canonicalSlug":"claude-sonnet-4-5","displayName":"Claude Sonnet 4.5","createdAt":"2026-01-01T00:00:00.000Z"}]}"#
+ }
+ other => panic!("unexpected API format {other}"),
+ };
+ Response::builder()
+ .header("content-type", "application/json")
+ .body(Body::from(body))
+ .unwrap()
+ })
+ .await;
+ let provider = FakeServer::start(|request| async move {
+ panic!("model catalog must not reach a provider: {}", request.uri())
+ })
+ .await;
+ let gateway = Gateway::start(Some(inference(&web, &provider))).await;
+ for (path, credential, expected) in [
+ (
+ "openai/v1/models",
+ ("authorization", "Bearer gateway-openai"),
+ r#"{"object":"list","data":[{"id":"Production key:gpt-4.1","object":"model","created":10,"owned_by":"openai"}]}"#,
+ ),
+ (
+ "anthropic/v1/models",
+ ("x-api-key", "gateway-anthropic"),
+ r#"{"data":[{"type":"model","id":"Production key:claude-sonnet-4-5","display_name":"Claude Sonnet 4.5","created_at":"2026-01-01T00:00:00.000Z"}],"has_more":false,"first_id":"Production key:claude-sonnet-4-5","last_id":"Production key:claude-sonnet-4-5"}"#,
+ ),
+ ] {
+ let response = reqwest::Client::new()
+ .get(format!("{}/{path}", gateway.url))
+ .header(credential.0, credential.1)
+ .send()
+ .await
+ .unwrap();
+ assert_eq!(response.status(), StatusCode::OK);
+ assert_eq!(response.text().await.unwrap(), expected);
+ }
+ assert_eq!(web.calls(), 2);
+ assert_eq!(provider.calls(), 0);
}
#[tokio::test]
@@ -610,19 +855,76 @@ async fn malformed_credentials_skip_resolution_and_oversized_body_releases_capac
.await
.unwrap();
assert_eq!(response.status(), StatusCode::PAYLOAD_TOO_LARGE);
- assert_eq!(web.calls(), 1);
+ assert_eq!(web.calls(), 0);
assert_eq!(provider.calls(), 0);
let next = gateway
.post()
.bearer_auth("valid-key")
- .body("{}")
+ .body(r#"{"model":"Production key:test"}"#)
.send()
.await
.unwrap();
assert_eq!(next.status(), StatusCode::OK);
+ assert_eq!(web.calls(), 1);
assert_eq!(provider.calls(), 1);
}
+#[tokio::test]
+async fn compressed_request_bodies_are_explicitly_rejected() {
+ let web = FakeServer::start(|_| async { resolution_response("provider-secret") }).await;
+ let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;
+ let gateway = Gateway::start(Some(inference(&web, &provider))).await;
+ let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default());
+ encoder
+ .write_all(br#"{"model":"connection:model","input":"hello"}"#)
+ .unwrap();
+ for (encoding, body) in [
+ ("gzip", encoder.finish().unwrap()),
+ (
+ "zstd",
+ b"encoded bytes are rejected before parsing".to_vec(),
+ ),
+ ] {
+ let response = gateway
+ .post()
+ .bearer_auth("valid-key")
+ .header(header::CONTENT_ENCODING, encoding)
+ .header(header::CONTENT_TYPE, "application/json")
+ .body(body)
+ .send()
+ .await
+ .unwrap();
+ assert_eq!(response.status(), StatusCode::UNSUPPORTED_MEDIA_TYPE);
+ let body: serde_json::Value =
+ serde_json::from_slice(&response.bytes().await.unwrap()).unwrap();
+ assert_eq!(body["error"]["code"], "unsupported_content_encoding");
+ }
+ assert_eq!(web.calls(), 0);
+ assert_eq!(provider.calls(), 0);
+}
+
+#[tokio::test]
+async fn requests_without_a_model_are_rejected_before_resolution() {
+ let web = FakeServer::start(|_| async { resolution_response("provider-secret") }).await;
+ let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;
+ let gateway = Gateway::start(Some(inference(&web, &provider))).await;
+
+ let response = gateway
+ .post()
+ .bearer_auth("valid-key")
+ .header(header::CONTENT_TYPE, "application/json")
+ .body(r#"{"prompt":{"id":"pmpt_123"}}"#)
+ .send()
+ .await
+ .unwrap();
+
+ assert_eq!(response.status(), StatusCode::BAD_REQUEST);
+ let body: serde_json::Value = serde_json::from_slice(&response.bytes().await.unwrap()).unwrap();
+ assert_eq!(body["error"]["code"], "invalid_body");
+ assert_eq!(web.calls(), 0);
+ assert_eq!(provider.calls(), 0);
+}
+
#[tokio::test]
async fn resolution_failures_are_sanitized_and_never_call_provider() {
let web = FakeServer::start(|request| async move {
@@ -650,6 +952,7 @@ async fn resolution_failures_are_sanitized_and_never_call_provider() {
let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;
let gateway = Gateway::start(Some(inference(&web, &provider))).await;
for (key, status) in [
+ ("400", 400),
("401", 401),
("403", 403),
("404", 404),
@@ -660,7 +963,7 @@ async fn resolution_failures_are_sanitized_and_never_call_provider() {
let response = gateway
.post()
.bearer_auth(key)
- .body("{}")
+ .body(r#"{"model":"Production key:test"}"#)
.send()
.await
.unwrap();
@@ -687,7 +990,7 @@ async fn slow_resolution_has_a_bounded_wait() {
gateway
.post()
.bearer_auth("slow-resolution")
- .body("{}")
+ .body(r#"{"model":"Production key:test"}"#)
.send(),
)
.await
@@ -825,18 +1128,7 @@ async fn chat_completions_streams_carry_the_request_and_generation_ids() {
let telemetry =
Telemetry::for_test(&ControlPlaneConfig::new(&sink.url, "service-key").unwrap());
let (resolved, mut resolution_ids) = tokio::sync::mpsc::channel(1);
- let web = FakeServer::start(move |request| {
- let resolved = resolved.clone();
- async move {
- let id = request.headers()["langfuse-gateway-request-id"]
- .to_str()
- .unwrap()
- .to_owned();
- resolved.send(id).await.unwrap();
- resolution_response_for(ApiFormat::OpenAiChatCompletions, "provider-chat")
- }
- })
- .await;
+ let web = chat_completion_resolution_server(resolved).await;
let provider = FakeServer::start(|request| async move {
assert_eq!(request.uri(), "/v1/chat/completions");
Response::builder()
@@ -908,6 +1200,23 @@ async fn chat_completions_streams_carry_the_request_and_generation_ids() {
.await;
}
+async fn chat_completion_resolution_server(
+ resolved: tokio::sync::mpsc::Sender<String>,
+) -> FakeServer {
+ FakeServer::start(move |request| {
+ let resolved = resolved.clone();
+ async move {
+ let id = request.headers()["langfuse-gateway-request-id"]
+ .to_str()
+ .unwrap()
+ .to_owned();
+ resolved.send(id).await.unwrap();
+ resolution_response_for(ApiFormat::OpenAiChatCompletions, "provider-chat")
+ }
+ })
+ .await
+}
+
#[tokio::test]
async fn unconfigured_gateway_is_live_but_not_ready_and_inference_is_unavailable() {
let gateway = Gateway::start(None).await;
@@ -975,7 +1284,7 @@ async fn active_admission_is_held_through_stream_and_shutdown_drains_it() {
let mut response = gateway
.post()
.bearer_auth("key")
- .body("{}")
+ .body(r#"{"model":"Production key:test"}"#)
.send()
.await
.unwrap();
@@ -986,7 +1295,7 @@ async fn active_admission_is_held_through_stream_and_shutdown_drains_it() {
gateway
.post()
.bearer_auth("key")
- .body("{}")
+ .body(r#"{"model":"Production key:test"}"#)
.send()
.await
.unwrap()
@@ -1033,7 +1342,7 @@ async fn client_disconnect_releases_capacity_for_the_next_request() {
let mut first = gateway
.post()
.bearer_auth("key")
- .body("{}")
+ .body(r#"{"model":"Production key:test"}"#)
.send()
.await
.unwrap();
@@ -1044,7 +1353,7 @@ async fn client_disconnect_releases_capacity_for_the_next_request() {
let response = gateway
.post()
.bearer_auth("key")
- .body("{}")
+ .body(r#"{"model":"Production key:test"}"#)
.send()
.await
.unwrap();
@@ -1076,7 +1385,7 @@ async fn anthropic_messages_accept_either_credential_header_and_relay_native_sse
let key = anthropic_gateway_key(&request).unwrap();
assert_eq!(
to_bytes(request.into_body(), 1024).await.unwrap(),
- r#"{"apiFormat":"anthropic.messages"}"#
+ r#"{"apiFormat":"anthropic.messages","model":"Production key:claude-sonnet-4-5"}"#
);
if key == "gateway-anthropic" {
resolution_response_for(ApiFormat::AnthropicMessages, "provider-anthropic")
@@ -1093,6 +1402,10 @@ async fn anthropic_messages_accept_either_credential_header_and_relay_native_sse
assert_eq!(request.headers()["x-api-key"], "provider-anthropic");
assert_eq!(request.headers()["anthropic-version"], "2023-06-01");
assert!(!request.headers().contains_key(header::AUTHORIZATION));
+ assert_eq!(
+ to_bytes(request.into_body(), 4096).await.unwrap(),
+ r#"{"model":"provider-model","stream":true,"messages":[]}"#
+ );
Response::builder()
.header("content-type", "text/event-stream")
.header("request-id", "req_upstream")
@@ -1120,7 +1433,7 @@ async fn anthropic_messages_accept_either_credential_header_and_relay_native_sse
.post(format!("{}/anthropic/v1/messages?beta=true", gateway.url))
.header("anthropic-version", "2023-06-01")
.header("content-type", "application/json")
- .body(r#"{"model":"claude-sonnet-4-5","stream":true,"messages":[]}"#);
+ .body(r#"{"model":"Production key:claude-sonnet-4-5","stream":true,"messages":[]}"#);
for (name, value) in credential_headers {
request = request.header(name, value);
}
@@ -1198,7 +1511,7 @@ async fn anthropic_credential_conflicts_and_failures_use_the_native_error_envelo
for (credential_headers, status, kind, message) in cases {
let mut request = client
.post(format!("{}/anthropic/v1/messages", gateway.url))
- .body("{}");
+ .body(r#"{"model":"Production key:test"}"#);
for (name, value) in credential_headers {
request = request.header(*name, *value);
}
@@ -1243,7 +1556,7 @@ async fn anthropic_capacity_rejections_use_the_native_envelope() {
let response = reqwest::Client::new()
.post(format!("{}/anthropic/v1/messages", gateway.url))
.header("x-api-key", "gateway-anthropic")
- .body("{}")
+ .body(r#"{"model":"Production key:test"}"#)
.send()
.await
.unwrap();
@@ -1283,11 +1596,26 @@ async fn openai_routes_ignore_x_api_key_and_keep_their_envelope() {
#[tokio::test]
async fn chat_completions_resolve_their_own_format_and_relay_native_json_and_errors() {
let web = FakeServer::start(|request| async move {
- assert_eq!(
- to_bytes(request.into_body(), 1024).await.unwrap(),
- r#"{"apiFormat":"openai.chat-completions"}"#
- );
- resolution_response_for(ApiFormat::OpenAiChatCompletions, "provider-chat")
+ let request: serde_json::Value =
+ serde_json::from_slice(&to_bytes(request.into_body(), 1024).await.unwrap()).unwrap();
+ assert_eq!(request["apiFormat"], "openai.chat-completions");
+ let provider_model = request["model"]
+ .as_str()
+ .and_then(|model| model.split_once(':'))
+ .map(|(_, provider_model)| provider_model)
+ .unwrap();
+ let bytes = to_bytes(
+ resolution_response_for(ApiFormat::OpenAiChatCompletions, "provider-chat").into_body(),
+ 4096,
+ )
+ .await
+ .unwrap();
+ let mut response: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
+ response["connection"]["provider_model"] = provider_model.into();
+ Response::builder()
+ .header("content-type", "application/json")
+ .body(Body::from(response.to_string()))
+ .unwrap()
})
.await;
let provider = FakeServer::start(|request| async move {
@@ -1314,12 +1642,12 @@ async fn chat_completions_resolve_their_own_format_and_relay_native_json_and_err
let gateway = Gateway::start(Some(inference(&web, &provider))).await;
for (body, status, expected) in [
(
- r#"{ "model":"gpt-4.1-mini", "messages":[] }"#,
+ r#"{ "model":"Production key:gpt-4.1-mini", "messages":[] }"#,
StatusCode::OK,
r#"{"id":"chatcmpl-1","choices":[]}"#,
),
(
- r#"{"model":"unknown-model","messages":[]}"#,
+ r#"{"model":"Production key:unknown-model","messages":[]}"#,
StatusCode::NOT_FOUND,
r#"{"error":{"code":"model_not_found"}}"#,
),
@@ -1348,22 +1676,18 @@ async fn anthropic_routes_share_the_request_body_limit() {
resolution_response_for(ApiFormat::AnthropicMessages, "provider-anthropic")
})
.await;
- let provider = FakeServer::start(|request| async move {
- let body = to_bytes(request.into_body(), MAX_REQUEST_BYTES)
- .await
- .unwrap();
- assert_eq!(body.len(), MAX_REQUEST_BYTES);
- Response::new(Body::empty())
- })
- .await;
+ let provider = FakeServer::start(|_| async move { Response::new(Body::empty()) }).await;
let gateway = Gateway::start(Some(inference(&web, &provider))).await;
let client = reqwest::Client::new();
for path in ["messages", "messages/count_tokens"] {
let send = |size| {
+ let prefix = r#"{"model":"Production key:test","padding":""#;
+ let suffix = r#""}"#;
+ let padding = "x".repeat(size - prefix.len() - suffix.len());
client
.post(format!("{}/anthropic/v1/{path}", gateway.url))
.header("x-api-key", "gateway-anthropic")
- .body(vec![b'x'; size])
+ .body(format!("{prefix}{padding}{suffix}"))
.send()
};
assert_eq!(
diff --git a/ai-gateway/src/inference.rs b/ai-gateway/src/inference.rs
index 4c13ca773f..23282f2dda 100644
--- a/ai-gateway/src/inference.rs
+++ b/ai-gateway/src/inference.rs
@@ -1,9 +1,11 @@
//! Resolve the caller's credential, then execute using only the trusted context.
+use std::ops::Range;
+
use axum::{
body::{Body, Bytes},
http::{HeaderMap, Response},
};
-use tokio::sync::Semaphore;
+use tokio::sync::{Semaphore, SemaphorePermit};
use tracing::Instrument;
use crate::{
@@ -17,6 +19,8 @@ use crate::{
pub struct InferenceService {
control_plane: ControlPlaneClient,
provider: ProviderTransport,
+ request_body_capacity: Semaphore,
+ model_catalog_capacity: Semaphore,
resolution_capacity: Semaphore,
telemetry: Option<crate::telemetry::Telemetry>,
}
@@ -26,6 +30,10 @@ pub(crate) enum RequestPreparationError {
Provider(ProviderError),
}
+pub(crate) struct RequestBodyPermit<'a> {
+ _permit: SemaphorePermit<'a>,
+}
+
impl InferenceService {
/// Initialize pooled clients for resolution and provider requests.
///
@@ -46,6 +54,8 @@ impl InferenceService {
provider: ProviderTransport::new(max_active_requests)
.map_err(|_| ResolutionError::Configuration)?
.with_telemetry(telemetry.clone()),
+ request_body_capacity: Semaphore::new(max_active_requests),
+ model_catalog_capacity: Semaphore::new(max_concurrent_resolutions),
resolution_capacity: Semaphore::new(max_concurrent_resolutions),
telemetry: Some(telemetry),
})
@@ -55,13 +65,49 @@ impl InferenceService {
self.telemetry.clone()
}
- /// Authenticate with a separate bounded budget before reserving execution capacity.
- /// The API format comes from the public route, so Web selects a compatible connection.
- /// A successful resolution is recorded even if execution admission then fails.
+ fn try_reserve_resolution(&self) -> Result<SemaphorePermit<'_>, ResolutionError> {
+ self.resolution_capacity.try_acquire().map_err(|_| {
+ crate::observability::rejected("resolution");
+ ResolutionError::Unavailable
+ })
+ }
+
+ pub(crate) fn try_reserve_request_body(
+ &self,
+ ) -> Result<RequestBodyPermit<'_>, RequestPreparationError> {
+ let permit = self.request_body_capacity.try_acquire().map_err(|_| {
+ crate::observability::rejected("request_body");
+ RequestPreparationError::Resolution(ResolutionError::Unavailable)
+ })?;
+ Ok(RequestBodyPermit { _permit: permit })
+ }
+
+ pub(crate) async fn fetch_model_catalog(
+ &self,
+ gateway_key: &str,
+ api_format: ApiFormat,
+ correlation: &RequestCorrelation,
+ ) -> Result<Vec<u8>, ResolutionError> {
+ let _permit = self.model_catalog_capacity.try_acquire().map_err(|_| {
+ crate::observability::rejected("model_catalog");
+ ResolutionError::Unavailable
+ })?;
+ let _active = crate::observability::Active::new("model_catalog");
+ let body = self
+ .control_plane
+ .models(gateway_key, api_format, correlation.id())
+ .await?;
+ crate::providers::model_list::provider_response(&body, api_format)
+ }
+
+ /// Authenticate, then reserve execution capacity. The API format comes from the public route,
+ /// so Web selects a compatible connection. A successful resolution is recorded even if
+ /// execution admission then fails.
pub(crate) async fn resolve_and_admit(
&self,
gateway_key: &str,
api_format: ApiFormat,
+ model: &str,
correlation: &mut RequestCorrelation,
) -> Result<(RequestPermit, ResolvedRequestContext), RequestPreparationError> {
let span = tracing::info_span!(
@@ -70,7 +116,9 @@ impl InferenceService {
gateway.outcome = tracing::field::Empty
);
async {
- let prepared = self.prepare(gateway_key, api_format, correlation).await;
+ let prepared = self
+ .prepare(gateway_key, api_format, model, correlation)
+ .await;
tracing::Span::current().record(
"gateway.outcome",
match &prepared {
@@ -89,19 +137,20 @@ impl InferenceService {
&self,
gateway_key: &str,
api_format: ApiFormat,
+ model: &str,
correlation: &mut RequestCorrelation,
) -> Result<(RequestPermit, ResolvedRequestContext), RequestPreparationError> {
+ let resolution_permit = self
+ .try_reserve_resolution()
+ .map_err(RequestPreparationError::Resolution)?;
let context = {
- let _permit = self.resolution_capacity.try_acquire().map_err(|_| {
- crate::observability::rejected("resolution");
- RequestPreparationError::Resolution(ResolutionError::Unavailable)
- })?;
let _active = crate::observability::Active::new("resolution");
self.control_plane
- .resolve(gateway_key, api_format, correlation.id())
+ .resolve(gateway_key, api_format, model, correlation.id())
.await
.map_err(RequestPreparationError::Resolution)?
};
+ drop(resolution_permit);
correlation.record_resolution(&context);
let permit = self
.provider
@@ -120,12 +169,20 @@ impl InferenceService {
context: ResolvedRequestContext,
headers: &HeaderMap,
body: Bytes,
+ model_range: Range<usize>,
route: Route,
- query: Option<&str>,
correlation: &mut RequestCorrelation,
) -> Result<Response<Body>, ProviderError> {
self.provider
- .forward_route(permit, context, headers, body, route, query, correlation)
+ .forward_route(
+ permit,
+ context,
+ headers,
+ body,
+ model_range,
+ route,
+ correlation,
+ )
.await
}
@@ -138,6 +195,8 @@ impl InferenceService {
Self {
control_plane,
provider,
+ request_body_capacity: Semaphore::new(resolutions),
+ model_catalog_capacity: Semaphore::new(resolutions),
resolution_capacity: Semaphore::new(resolutions),
telemetry: None,
}
diff --git a/ai-gateway/src/observability/http/tests.rs b/ai-gateway/src/observability/http/tests.rs
index 25fd158371..79ab5832ab 100644
--- a/ai-gateway/src/observability/http/tests.rs
+++ b/ai-gateway/src/observability/http/tests.rs
@@ -205,7 +205,7 @@ async fn provider_http_errors_are_traced_without_changing_the_response() {
provider.try_admit().unwrap(),
context,
&HeaderMap::new(),
- Bytes::from_static(b"{}"),
+ Bytes::from_static(br#"{"model":"requested"}"#),
)
.await
.unwrap();
@@ -330,7 +330,7 @@ async fn generation_context_is_isolated_from_operational_spans_and_outbound_head
let observed = observed.lock().unwrap();
assert_eq!(observed.len(), 3);
for (name, headers, payload) in &*observed {
- let child = spans.iter().find(|span| span.name == *name).unwrap();
+ let child = outbound_span(&spans, name);
assert_outbound_context(headers, (*name != "provider.headers").then_some(child));
if *name == "ingestion" {
assert_generation_context(payload);
@@ -341,6 +341,17 @@ async fn generation_context_is_isolated_from_operational_spans_and_outbound_head
type SpanData = opentelemetry_sdk::trace::SpanData;
+fn outbound_span<'a>(spans: &'a [SpanData], name: &str) -> &'a SpanData {
+ let resolution = spans.iter().find(|span| span.name == "resolution").unwrap();
+ spans
+ .iter()
+ .find(|span| {
+ span.name == name
+ && (name != "resolver" || span.parent_span_id == resolution.span_context.span_id())
+ })
+ .unwrap()
+}
+
/// Every phase between the caller's headers and the last body byte has a span in
/// the server trace, so a waterfall shows no unattributed wall time. Batched
/// ingestion runs in its own trace, linked back to each request it carries.
@@ -349,7 +360,6 @@ fn assert_phase_spans(spans: &[SpanData], server: &SpanData) {
assert_eq!(spans.len(), 9);
for (name, parent) in [
("resolution", server),
- ("resolver", named("resolution")),
("request.body", server),
("request.capture", server),
("provider.headers", server),
@@ -368,6 +378,15 @@ fn assert_phase_spans(spans: &[SpanData], server: &SpanData) {
"{name}"
);
}
+ let resolver_parents = spans
+ .iter()
+ .filter(|span| span.name == "resolver")
+ .map(|span| span.parent_span_id)
+ .collect::<Vec<_>>();
+ assert_eq!(
+ resolver_parents,
+ [named("resolution").span_context.span_id()]
+ );
let batch = named("telemetry.batch");
assert_eq!(batch.parent_span_id, opentelemetry::trace::SpanId::INVALID);
assert_ne!(
@@ -430,7 +449,7 @@ fn assert_attribute(span: &SpanData, key: &str, expected: impl Into<opentelemetr
assert_eq!(actual.value, expected.into(), "{} {key}", span.name);
}
-const GENERATION_REQUEST: &str = r#"{"model":"test-model"}"#;
+const GENERATION_REQUEST: &str = r#"{"model":"Production key:test-model"}"#;
const PROVIDER_RESPONSE: &str = r#"{"id":"resp_1","output":[]}"#;
fn generation_request() -> HttpRequest<Body> {
@@ -524,7 +543,7 @@ fn assert_request_correlation(
"provider_connection_name",
"Production key",
),
- ("gateway.upstream.model", "upstream_model", "test-model"),
+ ("gateway.upstream.model", "upstream_model", "provider-model"),
];
for (attribute, _, expected) in resolution {
assert_attribute(server, attribute, expected);
diff --git a/ai-gateway/src/providers/chat_completions.rs b/ai-gateway/src/providers/chat_completions.rs
index f1ad94e254..7ef460468a 100644
--- a/ai-gateway/src/providers/chat_completions.rs
+++ b/ai-gateway/src/providers/chat_completions.rs
@@ -2,25 +2,23 @@
//! gateway opts in on the caller's behalf. The resulting usage chunk is relayed.
use std::fmt;
-use axum::{body::Bytes, http::HeaderMap};
+use axum::body::Bytes;
use serde::{
Deserialize, Deserializer,
de::{MapAccess, Visitor},
};
use serde_json::{Value, value::RawValue};
-use crate::transport::identity_encoding;
-
/// The rewritten body for a streamed request that did not ask for usage, or
/// `None` when the original bytes must be forwarded unchanged. Every other
/// top-level member keeps its original bytes and position.
-pub(super) fn request_stream_usage(headers: &HeaderMap, body: &[u8]) -> Option<Bytes> {
- if !identity_encoding(headers) {
- return None;
- }
+pub(super) fn request_stream_usage(body: &[u8], provider_model: &str) -> Option<Bytes> {
let Members(mut members) = serde_json::from_slice(body).ok()?;
// Duplicate members are resolved differently by different parsers.
- if count(&members, "stream") != 1 || count(&members, "stream_options") > 1 {
+ if count(&members, "model") != 1
+ || count(&members, "stream") != 1
+ || count(&members, "stream_options") > 1
+ {
return None;
}
if member(&members, "stream")? != "true" {
@@ -40,6 +38,8 @@ pub(super) fn request_stream_usage(headers: &HeaderMap, body: &[u8]) -> Option<B
let mut options = options;
options.insert("include_usage".to_owned(), Value::Bool(true));
let options = RawValue::from_string(Value::Object(options).to_string()).ok()?;
+ let model = RawValue::from_string(serde_json::to_string(provider_model).ok()?).ok()?;
+ members.iter_mut().find(|(key, _)| key == "model")?.1 = model;
match members.iter_mut().find(|(key, _)| key == "stream_options") {
Some((_, raw)) => *raw = options,
None => members.push(("stream_options".to_owned(), options)),
@@ -95,10 +95,9 @@ impl<'de> Deserialize<'de> for Members {
#[cfg(test)]
mod tests {
use super::*;
- use axum::http::{HeaderValue, header};
fn rewrite(body: &str) -> Option<String> {
- request_stream_usage(&HeaderMap::new(), body.as_bytes())
+ request_stream_usage(body.as_bytes(), "provider-model")
.map(|bytes| String::from_utf8(bytes.to_vec()).unwrap())
}
@@ -108,19 +107,21 @@ mod tests {
rewrite(r#"{"model":"gpt-4.1", "messages":[{"role":"user","content":"1.0e2 \u00e9"}],"stream":true}"#)
.as_deref(),
Some(
- r#"{"model":"gpt-4.1","messages":[{"role":"user","content":"1.0e2 \u00e9"}],"stream":true,"stream_options":{"include_usage":true}}"#
+ r#"{"model":"provider-model","messages":[{"role":"user","content":"1.0e2 \u00e9"}],"stream":true,"stream_options":{"include_usage":true}}"#
)
);
assert_eq!(
rewrite(r#"{"stream_options":{"include_obfuscation":false,"include_usage":false},"stream":true,"model":"m"}"#)
.as_deref(),
Some(
- r#"{"stream_options":{"include_obfuscation":false,"include_usage":true},"stream":true,"model":"m"}"#
+ r#"{"stream_options":{"include_obfuscation":false,"include_usage":true},"stream":true,"model":"provider-model"}"#
)
);
assert_eq!(
- rewrite(r#"{"stream":true,"stream_options":null}"#).as_deref(),
- Some(r#"{"stream":true,"stream_options":{"include_usage":true}}"#)
+ rewrite(r#"{"model":"m","stream":true,"stream_options":null}"#).as_deref(),
+ Some(
+ r#"{"model":"provider-model","stream":true,"stream_options":{"include_usage":true}}"#
+ )
);
}
@@ -140,8 +141,5 @@ mod tests {
] {
assert_eq!(rewrite(body), None, "{body}");
}
- let mut encoded = HeaderMap::new();
- encoded.insert(header::CONTENT_ENCODING, HeaderValue::from_static("gzip"));
- assert!(request_stream_usage(&encoded, br#"{"stream":true}"#).is_none());
}
}
diff --git a/ai-gateway/src/providers/mod.rs b/ai-gateway/src/providers/mod.rs
index 3c0202f94b..cda1a77e27 100644
--- a/ai-gateway/src/providers/mod.rs
+++ b/ai-gateway/src/providers/mod.rs
@@ -1,10 +1,11 @@
mod chat_completions;
+pub(crate) mod model_list;
-use std::{sync::Arc, time::Duration};
+use std::{ops::Range, sync::Arc, time::Duration};
use axum::{
body::{Body, Bytes},
- http::{HeaderMap, HeaderName, HeaderValue, Method, Response, header},
+ http::{HeaderMap, HeaderName, HeaderValue, Response, header},
};
use reqwest::Client;
use reqwest_middleware::ClientWithMiddleware;
@@ -27,22 +28,17 @@ pub(crate) enum Route {
OpenAiResponses,
OpenAiResponsesCompact,
OpenAiChatCompletions,
- OpenAiModels,
AnthropicMessages,
AnthropicCountTokens,
- AnthropicModels,
}
impl Route {
pub(crate) fn provider(self) -> Provider {
match self {
- Self::OpenAiResponses
- | Self::OpenAiResponsesCompact
- | Self::OpenAiChatCompletions
- | Self::OpenAiModels => Provider::OpenAi,
- Self::AnthropicMessages | Self::AnthropicCountTokens | Self::AnthropicModels => {
- Provider::Anthropic
+ Self::OpenAiResponses | Self::OpenAiResponsesCompact | Self::OpenAiChatCompletions => {
+ Provider::OpenAi
}
+ Self::AnthropicMessages | Self::AnthropicCountTokens => Provider::Anthropic,
}
}
@@ -56,19 +52,11 @@ impl Route {
}
}
- pub(crate) fn method(self) -> Method {
- match self {
- Self::OpenAiModels | Self::AnthropicModels => Method::GET,
- _ => Method::POST,
- }
- }
-
fn path(self) -> &'static str {
match self {
Self::OpenAiResponses => "/responses",
Self::OpenAiResponsesCompact => "/responses/compact",
Self::OpenAiChatCompletions => "/chat/completions",
- Self::OpenAiModels | Self::AnthropicModels => "/models",
Self::AnthropicMessages => "/messages",
Self::AnthropicCountTokens => "/messages/count_tokens",
}
@@ -83,13 +71,6 @@ impl Route {
| Self::AnthropicMessages
)
}
-
- pub(crate) fn forwarded_query_parameters(self) -> &'static [&'static str] {
- match self {
- Self::AnthropicModels => &["limit", "after_id", "before_id"],
- _ => &[],
- }
- }
}
pub(crate) struct ProviderLimits {
@@ -181,6 +162,7 @@ impl ProviderTransport {
/// # Errors
/// See [`Self::forward_route`].
+ #[cfg(test)]
pub async fn forward(
&self,
permit: RequestPermit,
@@ -188,13 +170,14 @@ impl ProviderTransport {
headers: &HeaderMap,
body: Bytes,
) -> Result<Response<Body>, ProviderError> {
+ let model_range = model_range(&body)?;
self.forward_route(
permit,
context,
headers,
body,
+ model_range,
Route::OpenAiResponses,
- None,
&mut RequestCorrelation::from_headers(headers),
)
.await
@@ -210,13 +193,14 @@ impl ProviderTransport {
context: ResolvedRequestContext,
headers: &HeaderMap,
body: Bytes,
+ model_range: Range<usize>,
route: Route,
- query: Option<&str>,
correlation: &mut RequestCorrelation,
) -> Result<Response<Body>, ProviderError> {
let api_format = route.api_format();
let (credential_name, mut credential) = provider_credential(&context)?;
credential.set_sensitive(true);
+ let provider_model = context.connection().provider_model();
let mut capture = if route.captures_generation() {
let mut capture = ExecutionCapture::for_request(api_format, &context, headers, &body);
if let Some(telemetry) = &self.telemetry {
@@ -226,26 +210,32 @@ impl ProviderTransport {
} else {
ExecutionCapture::unobserved()
};
- // The capture keeps the caller's own request; only the upstream copy asks for usage.
+ // Capture the caller's body shape, but keep upstream-only changes out of the input.
let body = match route {
Route::OpenAiChatCompletions => {
- chat_completions::request_stream_usage(headers, &body).unwrap_or(body)
+ match chat_completions::request_stream_usage(&body, provider_model) {
+ Some(rewritten) => {
+ drop(body);
+ rewritten
+ }
+ None => rewrite_model(body, model_range, provider_model)?,
+ }
}
- _ => body,
+ _ => rewrite_model(body, model_range, provider_model)?,
};
- // The relay forwards the caller's `model` unchanged; only a captured route has parsed it.
- let model = capture.requested_model().map(str::to_owned);
- correlation.record_upstream(UpstreamTarget::for_connection(context.connection(), model));
+ capture.set_provider_model(provider_model);
+ correlation.record_upstream(UpstreamTarget::for_connection(
+ context.connection(),
+ Some(provider_model.to_owned()),
+ ));
let mut upstream = self
.client
- .request(route.method(), self.request_url(route, query))
+ .post(self.request_url(route))
.with_extension(DisableOtelPropagation)
.headers(transport::request_headers(headers, api_format))
.header(header::ACCEPT_ENCODING, "identity")
.header(credential_name, credential);
- if route.method() != Method::GET {
- upstream = upstream.body(body);
- }
+ upstream = upstream.body(body);
let response = tokio::time::timeout_at(
permit
.deadline
@@ -286,7 +276,7 @@ impl ProviderTransport {
Ok(downstream)
}
- fn request_url(&self, route: Route, query: Option<&str>) -> String {
+ fn request_url(&self, route: Route) -> String {
let base = self
.origin_override
.as_deref()
@@ -295,10 +285,6 @@ impl ProviderTransport {
let mut url = String::with_capacity(base.len() + route.path().len());
url.push_str(base);
url.push_str(route.path());
- if let Some(query) = query.filter(|query| !query.is_empty()) {
- url.push('?');
- url.push_str(query);
- }
url
}
@@ -310,6 +296,50 @@ impl ProviderTransport {
}
}
+#[expect(
+ clippy::needless_pass_by_value,
+ reason = "ownership releases the caller body when rewriting returns, before the upstream wait"
+)]
+fn rewrite_model(
+ body: Bytes,
+ model_range: Range<usize>,
+ provider_model: &str,
+) -> Result<Bytes, ProviderError> {
+ if model_range.start > model_range.end || model_range.end > body.len() {
+ return Err(ProviderError::Configuration);
+ }
+ let model = serde_json::to_vec(provider_model).map_err(|_| ProviderError::Configuration)?;
+ let capacity = body.len() - model_range.len() + model.len();
+ let mut rewritten = Vec::with_capacity(capacity);
+ rewritten.extend_from_slice(&body[..model_range.start]);
+ rewritten.extend_from_slice(&model);
+ rewritten.extend_from_slice(&body[model_range.end..]);
+ Ok(Bytes::from(rewritten))
+}
+
+#[cfg(test)]
+pub(crate) fn model_range(body: &[u8]) -> Result<Range<usize>, ProviderError> {
+ #[derive(serde::Deserialize)]
+ struct ModelMember<'a> {
+ #[serde(borrow)]
+ model: &'a serde_json::value::RawValue,
+ }
+
+ let request = serde_json::from_slice::<ModelMember<'_>>(body)
+ .map_err(|_| ProviderError::Configuration)?;
+ let model = request.model.get();
+ let start = model
+ .as_ptr()
+ .addr()
+ .checked_sub(body.as_ptr().addr())
+ .ok_or(ProviderError::Configuration)?;
+ let end = start
+ .checked_add(model.len())
+ .filter(|end| *end <= body.len())
+ .ok_or(ProviderError::Configuration)?;
+ Ok(start..end)
+}
+
fn provider_credential(
context: &ResolvedRequestContext,
) -> Result<(HeaderName, HeaderValue), ProviderError> {
@@ -321,18 +351,5 @@ fn provider_credential(
Ok((name, value))
}
-pub(crate) fn forwarded_query(route: Route, query: Option<&str>) -> Option<String> {
- let allowed = route.forwarded_query_parameters();
- let query = query?;
- let kept: Vec<&str> = query
- .split('&')
- .filter(|pair| {
- let key = pair.split('=').next().unwrap_or("");
- allowed.contains(&key)
- })
- .collect();
- (!kept.is_empty()).then(|| kept.join("&"))
-}
-
#[cfg(test)]
mod tests;
diff --git a/ai-gateway/src/providers/model_list/anthropic.rs b/ai-gateway/src/providers/model_list/anthropic.rs
new file mode 100644
index 0000000000..87573d85a9
--- /dev/null
+++ b/ai-gateway/src/providers/model_list/anthropic.rs
@@ -0,0 +1,46 @@
+use chrono::DateTime;
+use serde::Serialize;
+
+use super::CatalogEntry;
+use crate::resolution::ResolutionError;
+
+// API contract: https://platform.claude.com/docs/en/api/models/list
+#[derive(Serialize)]
+struct Response<'a> {
+ data: Vec<Model<'a>>,
+ has_more: bool,
+ first_id: Option<&'a str>,
+ last_id: Option<&'a str>,
+}
+
+#[derive(Serialize)]
+struct Model<'a> {
+ #[serde(rename = "type")]
+ kind: &'static str,
+ id: &'a str,
+ display_name: &'a str,
+ created_at: &'a str,
+}
+
+pub(super) fn response(entries: &[CatalogEntry]) -> Result<Vec<u8>, ResolutionError> {
+ let data = entries
+ .iter()
+ .map(|entry| {
+ DateTime::parse_from_rfc3339(&entry.created_at)
+ .map_err(|_| ResolutionError::InvalidResponse)?;
+ Ok(Model {
+ kind: "model",
+ id: &entry.id,
+ display_name: &entry.display_name,
+ created_at: &entry.created_at,
+ })
+ })
+ .collect::<Result<Vec<_>, _>>()?;
+ serde_json::to_vec(&Response {
+ data,
+ has_more: false,
+ first_id: entries.first().map(|entry| entry.id.as_str()),
+ last_id: entries.last().map(|entry| entry.id.as_str()),
+ })
+ .map_err(|_| ResolutionError::InvalidResponse)
+}
diff --git a/ai-gateway/src/providers/model_list/mod.rs b/ai-gateway/src/providers/model_list/mod.rs
new file mode 100644
index 0000000000..a1c0128da2
--- /dev/null
+++ b/ai-gateway/src/providers/model_list/mod.rs
@@ -0,0 +1,49 @@
+mod anthropic;
+mod openai;
+
+use serde::Deserialize;
+
+use crate::resolution::{ApiFormat, ResolutionError};
+
+#[derive(Deserialize)]
+struct Catalog {
+ data: Vec<CatalogEntry>,
+}
+
+#[derive(Deserialize)]
+#[serde(rename_all = "camelCase")]
+struct CatalogEntry {
+ id: String,
+ provider: CatalogProvider,
+ #[serde(rename = "canonicalSlug")]
+ _canonical_slug: String,
+ display_name: String,
+ created_at: String,
+}
+
+#[derive(Clone, Copy, Deserialize, PartialEq, Eq)]
+enum CatalogProvider {
+ #[serde(rename = "OPENAI")]
+ OpenAi,
+ #[serde(rename = "ANTHROPIC")]
+ Anthropic,
+}
+
+pub(crate) fn provider_response(
+ body: &[u8],
+ api_format: ApiFormat,
+) -> Result<Vec<u8>, ResolutionError> {
+ let catalog: Catalog =
+ serde_json::from_slice(body).map_err(|_| ResolutionError::InvalidResponse)?;
+ let provider = match api_format {
+ ApiFormat::OpenAiResponses | ApiFormat::OpenAiChatCompletions => CatalogProvider::OpenAi,
+ ApiFormat::AnthropicMessages => CatalogProvider::Anthropic,
+ };
+ if catalog.data.iter().any(|entry| entry.provider != provider) {
+ return Err(ResolutionError::InvalidResponse);
+ }
+ match provider {
+ CatalogProvider::OpenAi => openai::response(&catalog.data),
+ CatalogProvider::Anthropic => anthropic::response(&catalog.data),
+ }
+}
diff --git a/ai-gateway/src/providers/model_list/openai.rs b/ai-gateway/src/providers/model_list/openai.rs
new file mode 100644
index 0000000000..63b02976cb
--- /dev/null
+++ b/ai-gateway/src/providers/model_list/openai.rs
@@ -0,0 +1,45 @@
+use chrono::DateTime;
+use serde::Serialize;
+
+use super::CatalogEntry;
+use crate::resolution::ResolutionError;
+
+// API contract: https://developers.openai.com/api/reference/resources/models/methods/list
+#[derive(Serialize)]
+struct Response<'a> {
+ object: &'static str,
+ data: Vec<Model<'a>>,
+}
+
+#[derive(Serialize)]
+struct Model<'a> {
+ id: &'a str,
+ object: &'static str,
+ created: i64,
+ owned_by: &'static str,
+}
+
+pub(super) fn response(entries: &[CatalogEntry]) -> Result<Vec<u8>, ResolutionError> {
+ let data = entries
+ .iter()
+ .map(|entry| {
+ let created = DateTime::parse_from_rfc3339(&entry.created_at)
+ .map_err(|_| ResolutionError::InvalidResponse)?
+ .timestamp();
+ if created < 0 {
+ return Err(ResolutionError::InvalidResponse);
+ }
+ Ok(Model {
+ id: &entry.id,
+ object: "model",
+ created,
+ owned_by: "openai",
+ })
+ })
+ .collect::<Result<Vec<_>, _>>()?;
+ serde_json::to_vec(&Response {
+ object: "list",
+ data,
+ })
+ .map_err(|_| ResolutionError::InvalidResponse)
+}
diff --git a/ai-gateway/src/providers/tests.rs b/ai-gateway/src/providers/tests.rs
index 85038269a0..5699269b16 100644
--- a/ai-gateway/src/providers/tests.rs
+++ b/ai-gateway/src/providers/tests.rs
@@ -36,9 +36,99 @@ const CALLER_CONTEXT_HEADERS: [(&str, &str); 9] = [
("langfuse-metadata", "private:value"),
];
+#[test]
+fn model_rewrite_changes_only_the_top_level_model_bytes() {
+ let original = Bytes::from_static(
+ br#"{ "z": {"second":2, "first":1}, "model" : "connection:model", "number": 1.0e2, "escaped":"\u00e9" }"#,
+ );
+ let expected = Bytes::from_static(
+ br#"{ "z": {"second":2, "first":1}, "model" : "provider-model", "number": 1.0e2, "escaped":"\u00e9" }"#,
+ );
+ let model_range = model_range(&original).unwrap();
+ assert_eq!(
+ rewrite_model(original, model_range, "provider-model").unwrap(),
+ expected
+ );
+}
+
+#[test]
+fn model_rewrite_fails_closed_for_an_invalid_model_range() {
+ let missing = Bytes::from_static(br#"{"input":{"model":"nested-model"}}"#);
+ assert!(matches!(
+ rewrite_model(missing, 100..110, "provider-model"),
+ Err(ProviderError::Configuration)
+ ));
+}
+
#[tokio::test]
-async fn preserves_opaque_bytes_and_isolates_request_and_response_headers() {
- const REQUEST: &[u8] = b"{ \"model\": \"opaque\", \"future_field\": [true, 3] }\n";
+async fn invalid_model_rewrite_never_reaches_the_provider() {
+ let upstream = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;
+ let relay = provider(&upstream, 1);
+ let result = relay
+ .forward_route(
+ relay.try_admit().unwrap(),
+ resolved_request_context("provider-secret").await,
+ &HeaderMap::new(),
+ Bytes::from_static(br#"{"model":"requested"}"#),
+ 100..110,
+ Route::OpenAiResponses,
+ &mut crate::correlation::RequestCorrelation::default(),
+ )
+ .await;
+
+ assert!(matches!(result, Err(ProviderError::Configuration)));
+ assert_eq!(upstream.calls(), 0);
+}
+
+#[test]
+fn model_rewrite_accepts_a_shorter_provider_model() {
+ let original = Bytes::from_static(br#"{"model":"connection:model","input":"large"}"#);
+ let model_range = model_range(&original).unwrap();
+ assert_eq!(
+ rewrite_model(original, model_range, "m").unwrap(),
+ r#"{"model":"m","input":"large"}"#
+ );
+}
+
+#[tokio::test]
+async fn missing_provider_model_fails_resolution() {
+ let web = FakeServer::start(|_| async {
+ let bytes = to_bytes(
+ crate::test_support::resolution_response("provider-secret").into_body(),
+ 4096,
+ )
+ .await
+ .unwrap();
+ let mut body: serde_json::Value = serde_json::from_slice(&bytes).unwrap();
+ body["connection"]
+ .as_object_mut()
+ .unwrap()
+ .remove("provider_model");
+ Response::new(Body::from(body.to_string()))
+ })
+ .await;
+ let result = web
+ .control_plane()
+ .resolve(
+ "gateway-secret",
+ ApiFormat::OpenAiResponses,
+ "Production key:gpt-4o",
+ "test-request-id",
+ )
+ .await;
+
+ assert!(matches!(
+ result,
+ Err(crate::resolution::ResolutionError::InvalidResponse)
+ ));
+}
+
+#[tokio::test]
+async fn rewrites_the_model_and_isolates_request_and_response_headers() {
+ const REQUEST: &[u8] =
+ br#"{ "model": "opaque", "future_field": {"model":"opaque","values":[true,3]} }"#;
+ const UPSTREAM_REQUEST: &[u8] =
+ br#"{ "model": "provider-model", "future_field": {"model":"opaque","values":[true,3]} }"#;
const RESPONSE: &[u8] = b"\x1f\x8b\x08\x00opaque-encoded-response";
let upstream = FakeServer::start(|request| async move {
assert_eq!(request.uri(), "/v1/responses");
@@ -49,6 +139,10 @@ async fn preserves_opaque_bytes_and_isolates_request_and_response_headers() {
);
assert_eq!(request.headers()[header::CONTENT_TYPE], "application/json");
assert_eq!(request.headers()[header::ACCEPT_ENCODING], "identity");
+ assert_eq!(
+ request.headers()[header::CONTENT_LENGTH].to_str().unwrap(),
+ UPSTREAM_REQUEST.len().to_string()
+ );
for name in [
"cookie",
"langfuse-gateway-authorization",
@@ -62,7 +156,10 @@ async fn preserves_opaque_bytes_and_isolates_request_and_response_headers() {
{
assert!(!request.headers().contains_key(name), "forwarded {name}");
}
- assert_eq!(to_bytes(request.into_body(), 1024).await.unwrap(), REQUEST);
+ assert_eq!(
+ to_bytes(request.into_body(), 1024).await.unwrap(),
+ UPSTREAM_REQUEST
+ );
Response::builder()
.status(201)
.header("content-type", "application/json")
@@ -149,7 +246,7 @@ async fn preserves_provider_failures_without_retries_or_redirects() {
relay.try_admit().unwrap(),
resolved_request_context("provider-secret").await,
&HeaderMap::new(),
- Bytes::new(),
+ Bytes::from_static(br#"{"model":"requested"}"#),
)
.await
.unwrap();
@@ -191,7 +288,7 @@ async fn streams_sse_incrementally_and_holds_admission_until_eof() {
relay.try_admit().unwrap(),
resolved_request_context("provider-secret").await,
&HeaderMap::new(),
- Bytes::new(),
+ Bytes::from_static(br#"{"model":"requested"}"#),
)
.await
.unwrap();
@@ -238,22 +335,22 @@ async fn simultaneous_requests_keep_provider_credentials_and_bodies_isolated() {
relay.try_admit().unwrap(),
alice,
&headers,
- Bytes::from_static(b"alice-body")
+ Bytes::from_static(br#"{"model":"requested","owner":"alice"}"#)
),
relay.forward(
relay.try_admit().unwrap(),
bob,
&headers,
- Bytes::from_static(b"bob-body")
+ Bytes::from_static(br#"{"model":"requested","owner":"bob"}"#)
),
);
assert_eq!(
to_bytes(alice.unwrap().into_body(), 1024).await.unwrap(),
- "Bearer alice-secret:alice-body"
+ r#"Bearer alice-secret:{"model":"provider-model","owner":"alice"}"#
);
assert_eq!(
to_bytes(bob.unwrap().into_body(), 1024).await.unwrap(),
- "Bearer bob-secret:bob-body"
+ r#"Bearer bob-secret:{"model":"provider-model","owner":"bob"}"#
);
assert_eq!(upstream.calls(), 2);
}
@@ -291,7 +388,7 @@ async fn dropping_downstream_cancels_upstream_and_releases_admission() {
relay.try_admit().unwrap(),
resolved_request_context("provider-secret").await,
&HeaderMap::new(),
- Bytes::new(),
+ Bytes::from_static(br#"{"model":"requested"}"#),
)
.await
.unwrap();
@@ -324,7 +421,7 @@ async fn deadlines_bound_headers_and_stalled_bodies_without_exposing_transport_d
relay.try_admit().unwrap(),
resolved_request_context("provider-secret").await,
&HeaderMap::new(),
- Bytes::new(),
+ Bytes::from_static(br#"{"model":"requested"}"#),
)
.await;
assert!(matches!(response, Err(ProviderError::Timeout)));
@@ -345,7 +442,7 @@ async fn deadlines_bound_headers_and_stalled_bodies_without_exposing_transport_d
relay.try_admit().unwrap(),
context,
&HeaderMap::new(),
- Bytes::new(),
+ Bytes::from_static(br#"{"model":"requested"}"#),
)
.await
.unwrap();
@@ -543,6 +640,13 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rel
.unwrap()["value"]["stringValue"],
if cancelled { "WARNING" } else { "ERROR" }
);
+ assert_eq!(
+ attrs
+ .iter()
+ .find(|a| a["key"] == "langfuse.observation.model.name")
+ .unwrap()["value"]["stringValue"],
+ "provider-model"
+ );
assert_eq!(metadata["langfuse.gateway.response.status_code"], 200);
assert!(metadata.get("relay_outcome").is_none());
telemetry
@@ -554,15 +658,15 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rel
}
#[tokio::test]
-async fn compact_posts_compact_path_and_models_get_skips_ingestion() {
+async fn compact_posts_compact_path_and_captures_ingestion() {
use crate::{resolution::ControlPlaneConfig, telemetry::Telemetry};
const COMPACT: &[u8] =
br#"{"model":"gpt-4.1","input":[{"encrypted_content":"opaque-ciphertext"}]}"#;
+ const UPSTREAM_COMPACT: &[u8] =
+ br#"{"model":"provider-model","input":[{"encrypted_content":"opaque-ciphertext"}]}"#;
const COMPACT_RESPONSE: &str =
r#"{"id":"comp_1","output":[{"encrypted_content":"opaque-ciphertext"}]}"#;
- const MODELS: &str = r#"{"object":"list","data":[{"id":"gpt-4.1","object":"model"}]}"#;
-
let (sent, mut received) = tokio::sync::mpsc::channel(2);
let sink = FakeServer::start(move |request| {
let sent = sent.clone();
@@ -578,26 +682,14 @@ async fn compact_posts_compact_path_and_models_get_skips_ingestion() {
let telemetry =
Telemetry::for_test(&ControlPlaneConfig::new(&sink.url, "service-key").unwrap());
let upstream = FakeServer::start(|request| async move {
- let uri = request.uri().to_string();
- let method = request.method().clone();
+ assert_eq!(request.uri(), "/v1/responses/compact");
+ assert_eq!(request.method(), "POST");
let body = to_bytes(request.into_body(), 4096).await.unwrap();
- match (method.as_str(), uri.as_str()) {
- ("POST", "/v1/responses/compact") => {
- assert_eq!(body, COMPACT);
- Response::builder()
- .header("content-type", "application/json")
- .body(Body::from(COMPACT_RESPONSE))
- .unwrap()
- }
- ("GET", "/v1/models") => {
- assert!(body.is_empty());
- Response::builder()
- .header("content-type", "application/json")
- .body(Body::from(MODELS))
- .unwrap()
- }
- other => panic!("unexpected provider request {other:?}"),
- }
+ assert_eq!(body, UPSTREAM_COMPACT);
+ Response::builder()
+ .header("content-type", "application/json")
+ .body(Body::from(COMPACT_RESPONSE))
+ .unwrap()
})
.await;
let relay = provider(&upstream, 1).with_telemetry(telemetry.clone());
@@ -607,8 +699,8 @@ async fn compact_posts_compact_path_and_models_get_skips_ingestion() {
resolved_request_context("provider-secret").await,
&HeaderMap::new(),
Bytes::from_static(COMPACT),
+ model_range(COMPACT).unwrap(),
Route::OpenAiResponsesCompact,
- None,
&mut crate::correlation::RequestCorrelation::default(),
)
.await
@@ -625,26 +717,12 @@ async fn compact_posts_compact_path_and_models_get_skips_ingestion() {
payload["resourceSpans"][0]["scopeSpans"][0]["spans"][0]["name"],
"openai.responses"
);
- let models = relay
- .forward_route(
- relay.try_admit().unwrap(),
- resolved_request_context("provider-secret").await,
- &HeaderMap::new(),
- Bytes::new(),
- Route::OpenAiModels,
- None,
- &mut crate::correlation::RequestCorrelation::default(),
- )
- .await
- .unwrap();
- assert_eq!(to_bytes(models.into_body(), 4096).await.unwrap(), MODELS);
- tokio::time::sleep(Duration::from_millis(50)).await;
assert!(received.try_recv().is_err());
telemetry
.shutdown(Instant::now() + Duration::from_secs(1))
.await;
assert_eq!(sink.calls(), 1);
- assert_eq!(upstream.calls(), 2);
+ assert_eq!(upstream.calls(), 1);
}
fn assert_completed_upload(payload: &serde_json::Value, streaming: bool) {
@@ -764,6 +842,7 @@ fn headers(pairs: &[(&'static str, &'static str)]) -> HeaderMap {
#[tokio::test]
async fn sends_the_x_api_key_credential_and_forwards_the_open_anthropic_header_family() {
const REQUEST: &[u8] = br#"{"model":"claude-sonnet-4-5","system":[{"type":"text","text":"attribution","cache_control":{"type":"ephemeral"}}],"future_field":true}"#;
+ const UPSTREAM_REQUEST: &[u8] = br#"{"model":"provider-model","system":[{"type":"text","text":"attribution","cache_control":{"type":"ephemeral"}}],"future_field":true}"#;
let upstream = FakeServer::start(|request| async move {
assert_eq!(request.uri(), "/v1/messages");
assert_eq!(request.method(), "POST");
@@ -779,7 +858,10 @@ async fn sends_the_x_api_key_credential_and_forwards_the_open_anthropic_header_f
for (name, _) in CONSUMED_HEADERS {
assert!(!request.headers().contains_key(*name), "forwarded {name}");
}
- assert_eq!(to_bytes(request.into_body(), 1024).await.unwrap(), REQUEST);
+ assert_eq!(
+ to_bytes(request.into_body(), 1024).await.unwrap(),
+ UPSTREAM_REQUEST
+ );
Response::builder()
.status(StatusCode::OK)
.header("content-type", "application/json")
@@ -822,8 +904,8 @@ async fn sends_the_x_api_key_credential_and_forwards_the_open_anthropic_header_f
.concat(),
),
Bytes::from_static(REQUEST),
+ model_range(REQUEST).unwrap(),
Route::AnthropicMessages,
- None,
&mut crate::correlation::RequestCorrelation::default(),
)
.await
@@ -852,96 +934,42 @@ async fn sends_the_x_api_key_credential_and_forwards_the_open_anthropic_header_f
}
#[tokio::test]
-async fn count_tokens_relays_the_body_and_models_forwards_only_pagination() {
+async fn count_tokens_relays_the_body() {
let upstream = FakeServer::start(|request| async move {
- let path = request.uri().path().to_owned();
- let query = request.uri().query().map(str::to_owned);
- match path.as_str() {
- "/v1/messages/count_tokens" => {
- assert_eq!(request.method(), "POST");
- assert_eq!(query, None);
- assert_eq!(
- to_bytes(request.into_body(), 1024).await.unwrap(),
- r#"{"model":"claude-sonnet-4-5","messages":[]}"#
- );
- Response::builder()
- .header("content-type", "application/json")
- .body(Body::from(r#"{"input_tokens":42}"#))
- .unwrap()
- }
- "/v1/models" => {
- assert_eq!(request.method(), "GET");
- assert_eq!(query.as_deref(), Some("limit=1000&after_id=model_x"));
- assert!(request.headers().get(header::CONTENT_LENGTH).is_none());
- Response::builder()
- .header("content-type", "application/json")
- .body(Body::from(r#"{"data":[{"id":"claude-sonnet-4-5"}]}"#))
- .unwrap()
- }
- other => panic!("unexpected upstream path {other}"),
- }
+ assert_eq!(request.uri(), "/v1/messages/count_tokens");
+ assert_eq!(request.method(), "POST");
+ assert_eq!(
+ to_bytes(request.into_body(), 1024).await.unwrap(),
+ r#"{"model":"provider-model","messages":[]}"#
+ );
+ Response::builder()
+ .header("content-type", "application/json")
+ .body(Body::from(r#"{"input_tokens":42}"#))
+ .unwrap()
})
.await;
let relay = transport(&upstream);
- for (route, query, body, expected) in [
- (
+ let context =
+ resolved_request_context_for(ApiFormat::AnthropicMessages, "provider-secret", "usage")
+ .await;
+ let response = relay
+ .forward_route(
+ relay.try_admit().unwrap(),
+ context,
+ &headers(&[("content-type", "application/json")]),
+ Bytes::from_static(br#"{"model":"claude-sonnet-4-5","messages":[]}"#),
+ model_range(br#"{"model":"claude-sonnet-4-5","messages":[]}"#).unwrap(),
Route::AnthropicCountTokens,
- Some("beta=true"),
- r#"{"model":"claude-sonnet-4-5","messages":[]}"#,
- r#"{"input_tokens":42}"#,
- ),
- (
- Route::AnthropicModels,
- Some("limit=1000&evil=1&after_id=model_x"),
- "",
- r#"{"data":[{"id":"claude-sonnet-4-5"}]}"#,
- ),
- ] {
- let context =
- resolved_request_context_for(ApiFormat::AnthropicMessages, "provider-secret", "usage")
- .await;
- let query = forwarded_query(route, query);
- let response = relay
- .forward_route(
- relay.try_admit().unwrap(),
- context,
- &headers(&[("content-type", "application/json")]),
- Bytes::from(body),
- route,
- query.as_deref(),
- &mut crate::correlation::RequestCorrelation::default(),
- )
- .await
- .unwrap();
- assert_eq!(response.status(), StatusCode::OK);
- assert_eq!(
- to_bytes(response.into_body(), 1024).await.unwrap(),
- expected
- );
- }
- assert_eq!(upstream.calls(), 2);
-}
-
-#[test]
-fn only_inference_routes_capture_generations_and_only_models_forwards_query() {
- assert!(Route::AnthropicMessages.captures_generation());
- assert!(!Route::AnthropicCountTokens.captures_generation());
- assert!(!Route::AnthropicModels.captures_generation());
- assert_eq!(
- forwarded_query(Route::AnthropicMessages, Some("beta=true&limit=5")),
- None
- );
- assert_eq!(
- forwarded_query(
- Route::AnthropicModels,
- Some("before_id=a&limit=&evil&after_id=b%20c")
- ),
- Some("before_id=a&limit=&after_id=b%20c".to_owned())
- );
+ &mut crate::correlation::RequestCorrelation::default(),
+ )
+ .await
+ .unwrap();
+ assert_eq!(response.status(), StatusCode::OK);
assert_eq!(
- forwarded_query(Route::AnthropicModels, Some("evil=1")),
- None
+ to_bytes(response.into_body(), 1024).await.unwrap(),
+ r#"{"input_tokens":42}"#
);
+ assert_eq!(upstream.calls(), 1);
}
#[test]
@@ -989,13 +1017,7 @@ async fn chat_completions_streams_request_usage_and_relay_the_usage_chunk() {
const REQUESTED: &str = r#"{"model":"gpt-4.1-mini","messages":[{"role":"user","content":"Say hello"}],"stream":true,"stream_options":{"include_usage":true}}"#;
// Callers receive the usage chunk whether or not they asked for it.
let relayed = [CHAT_CONTENT_CHUNK, CHAT_USAGE_CHUNK, CHAT_DONE].concat();
- for (request, forwarded) in [
- (
- UNREQUESTED,
- r#"{"model":"gpt-4.1-mini","messages":[{"role":"user","content":"Say hello"}],"stream":true,"stream_options":{"include_usage":true}}"#,
- ),
- (REQUESTED, REQUESTED),
- ] {
+ for request in [UNREQUESTED, REQUESTED] {
let (sent, mut received) = tokio::sync::mpsc::channel(2);
let sink = FakeServer::start(move |request| {
let sent = sent.clone();
@@ -1021,7 +1043,7 @@ async fn chat_completions_streams_request_usage_and_relay_the_usage_chunk() {
}
assert_eq!(
to_bytes(request.into_body(), 4096).await.unwrap(),
- forwarded
+ r#"{"model":"provider-model","messages":[{"role":"user","content":"Say hello"}],"stream":true,"stream_options":{"include_usage":true}}"#
);
// Arrive split mid-event, as a provider stream may.
let native = [CHAT_CONTENT_CHUNK, CHAT_USAGE_CHUNK, CHAT_DONE].concat();
@@ -1054,8 +1076,8 @@ async fn chat_completions_streams_request_usage_and_relay_the_usage_chunk() {
.concat(),
),
Bytes::from_static(request.as_bytes()),
+ model_range(request.as_bytes()).unwrap(),
Route::OpenAiChatCompletions,
- None,
&mut crate::correlation::RequestCorrelation::default(),
)
.await
@@ -1077,12 +1099,12 @@ async fn chat_completions_streams_request_usage_and_relay_the_usage_chunk() {
}
#[tokio::test]
-async fn chat_completions_forward_non_streamed_bodies_unchanged() {
+async fn chat_completions_rewrite_non_streamed_models() {
const JSON_REQUEST: &str = r#"{"model":"gpt-4.1-mini", "messages":[]}"#;
let upstream = FakeServer::start(|request| async move {
assert_eq!(
to_bytes(request.into_body(), 4096).await.unwrap(),
- JSON_REQUEST
+ r#"{"model":"provider-model", "messages":[]}"#
);
Response::builder()
.header("content-type", "application/json")
@@ -1102,8 +1124,8 @@ async fn chat_completions_forward_non_streamed_bodies_unchanged() {
.await,
&headers(&[("content-type", "application/json")]),
Bytes::from_static(JSON_REQUEST.as_bytes()),
+ model_range(JSON_REQUEST.as_bytes()).unwrap(),
Route::OpenAiChatCompletions,
- None,
&mut crate::correlation::RequestCorrelation::default(),
)
.await
diff --git a/ai-gateway/src/resolution/contracts.rs b/ai-gateway/src/resolution/contracts.rs
index 5e3ec18975..da7424fbb6 100644
--- a/ai-gateway/src/resolution/contracts.rs
+++ b/ai-gateway/src/resolution/contracts.rs
@@ -16,6 +16,16 @@ pub enum ApiFormat {
AnthropicMessages,
}
+impl ApiFormat {
+ pub fn as_str(self) -> &'static str {
+ match self {
+ Self::OpenAiResponses => "openai.responses",
+ Self::OpenAiChatCompletions => "openai.chat-completions",
+ Self::AnthropicMessages => "anthropic.messages",
+ }
+ }
+}
+
#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Provider {
@@ -143,6 +153,7 @@ pub struct ProviderConnection {
name: Option<String>,
#[serde(deserialize_with = "string_enum")]
provider: Provider,
+ provider_model: String,
#[serde(deserialize_with = "string_enum")]
api_format: ApiFormat,
base_url: String,
@@ -160,6 +171,9 @@ impl ProviderConnection {
pub fn provider(&self) -> Provider {
self.provider
}
+ pub fn provider_model(&self) -> &str {
+ &self.provider_model
+ }
pub fn api_format(&self) -> ApiFormat {
self.api_format
}
@@ -314,6 +328,7 @@ pub(super) fn decode(
]
.iter()
.any(|value| value.trim().is_empty())
+ || connection.provider_model.trim().is_empty()
{
return Err(ResolutionError::InvalidResponse);
}
diff --git a/ai-gateway/src/resolution/mod.rs b/ai-gateway/src/resolution/mod.rs
index 98993a867d..fd58217ce5 100644
--- a/ai-gateway/src/resolution/mod.rs
+++ b/ai-gateway/src/resolution/mod.rs
@@ -7,10 +7,10 @@ pub use contracts::{
ProviderCredential, RequestAttribution, ResolvedRequestContext,
};
use reqwest::{
- Client, Url,
+ Client, Method, Url,
header::{AUTHORIZATION, CONTENT_TYPE, HeaderValue},
};
-use reqwest_middleware::ClientWithMiddleware;
+use reqwest_middleware::{ClientWithMiddleware, RequestBuilder};
use std::{
fmt,
net::IpAddr,
@@ -18,6 +18,7 @@ use std::{
};
const RESOLVE_PATH: &str = "/api/internal/ai-gateway/v1/resolve";
+const MODELS_PATH: &str = "/api/internal/ai-gateway/v1/models";
const REQUEST_ID_HEADER: &str = "langfuse-gateway-request-id";
// Web sits behind load balancers that drop idle connections after 60s by default
// (AWS ALB). Evicting pooled connections first keeps the resolver from sending on a
@@ -25,7 +26,8 @@ const REQUEST_ID_HEADER: &str = "langfuse-gateway-request-id";
const POOL_IDLE_TIMEOUT: Duration = Duration::from_secs(50);
const TCP_KEEPALIVE: Duration = Duration::from_secs(30);
-struct ResolutionLimits {
+#[derive(Clone, Copy)]
+struct ControlPlaneLimits {
timeout: Duration,
max_response_bytes: usize,
}
@@ -34,7 +36,8 @@ struct ResolutionLimits {
pub struct ControlPlaneConfig {
web_url: Url,
service_key: String,
- limits: ResolutionLimits,
+ resolve_limits: ControlPlaneLimits,
+ models_limits: ControlPlaneLimits,
}
impl ControlPlaneConfig {
@@ -66,10 +69,15 @@ impl ControlPlaneConfig {
Ok(Self {
web_url: url,
service_key: service_key.to_owned(),
- limits: ResolutionLimits {
+ resolve_limits: ControlPlaneLimits {
timeout: Duration::from_secs(5),
max_response_bytes: 256 * 1024,
},
+ // Model catalogs are unpaginated and can require cold provider discovery.
+ models_limits: ControlPlaneLimits {
+ timeout: Duration::from_secs(30),
+ max_response_bytes: 10 * 1024 * 1024,
+ },
})
}
@@ -125,32 +133,68 @@ impl ControlPlaneClient {
&self,
gateway_key: &str,
api_format: ApiFormat,
+ model: &str,
request_id: &str,
) -> Result<ResolvedRequestContext, ResolutionError> {
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map_err(|_| ResolutionError::Unavailable)?;
- self.resolve_at(gateway_key, api_format, timestamp, request_id)
+ self.resolve_at(gateway_key, api_format, model, timestamp, request_id)
.await
}
+ /// Fetch the caller-visible model catalog from trusted Web.
+ ///
+ /// # Errors
+ /// Returns the same sanitized errors as [`Self::resolve`].
+ pub async fn models(
+ &self,
+ gateway_key: &str,
+ api_format: ApiFormat,
+ request_id: &str,
+ ) -> Result<Vec<u8>, ResolutionError> {
+ let timestamp = SystemTime::now()
+ .duration_since(UNIX_EPOCH)
+ .map_err(|_| ResolutionError::Unavailable)?;
+ let mut endpoint = self.config.endpoint(MODELS_PATH);
+ endpoint
+ .query_pairs_mut()
+ .append_pair("api_format", api_format.as_str());
+ let request = self.authenticated_request(
+ Method::GET,
+ endpoint,
+ gateway_key,
+ timestamp.as_secs(),
+ request_id,
+ )?;
+ self.execute(request, self.config.models_limits).await
+ }
+
async fn resolve_at(
&self,
gateway_key: &str,
api_format: ApiFormat,
+ model: &str,
timestamp: Duration,
request_id: &str,
) -> Result<ResolvedRequestContext, ResolutionError> {
- if gateway_key.len() > 8192 || !valid_token(gateway_key) {
- return Err(ResolutionError::InvalidCredential);
- }
let started = Instant::now();
- let body = tokio::time::timeout(
- self.config.limits.timeout,
- self.fetch(gateway_key, api_format, timestamp.as_secs(), request_id),
- )
- .await
- .map_err(|_| ResolutionError::Timeout)??;
+ let body = serde_json::to_vec(&serde_json::json!({
+ "apiFormat": api_format,
+ "model": model,
+ }))
+ .map_err(|_| ResolutionError::Configuration)?;
+ let request = self
+ .authenticated_request(
+ Method::POST,
+ self.config.endpoint(RESOLVE_PATH),
+ gateway_key,
+ timestamp.as_secs(),
+ request_id,
+ )?
+ .header(CONTENT_TYPE, "application/json")
+ .body(body);
+ let body = self.execute(request, self.config.resolve_limits).await?;
contracts::decode(
&body,
api_format,
@@ -158,13 +202,17 @@ impl ControlPlaneClient {
)
}
- async fn fetch(
+ fn authenticated_request(
&self,
+ method: Method,
+ endpoint: Url,
gateway_key: &str,
- api_format: ApiFormat,
timestamp: u64,
request_id: &str,
- ) -> Result<Vec<u8>, ResolutionError> {
+ ) -> Result<RequestBuilder, ResolutionError> {
+ if gateway_key.len() > 8192 || !valid_token(gateway_key) {
+ return Err(ResolutionError::InvalidCredential);
+ }
let mut credential = HeaderValue::from_str(&format!("Bearer {gateway_key}"))
.map_err(|_| ResolutionError::InvalidCredential)?;
credential.set_sensitive(true);
@@ -175,47 +223,62 @@ impl ControlPlaneClient {
))
.map_err(|_| ResolutionError::Configuration)?;
signature.set_sensitive(true);
- let body = serde_json::to_vec(&serde_json::json!({ "apiFormat": api_format }))
- .map_err(|_| ResolutionError::Configuration)?;
- let mut response = self
+ Ok(self
.client
- .post(self.config.endpoint(RESOLVE_PATH))
+ .request(method, endpoint)
.header(AUTHORIZATION, credential)
.header("langfuse-gateway-authorization", signature)
- .header(REQUEST_ID_HEADER, request_id)
- .header(CONTENT_TYPE, "application/json")
- .body(body)
- .send()
- .await
- .map_err(|_| ResolutionError::Transport)?;
- match response.status().as_u16() {
- 200 => {}
- 401 => return Err(ResolutionError::Authentication),
- 403 => return Err(ResolutionError::Forbidden),
- 404 => return Err(ResolutionError::NoRoute),
- 429 | 500..=599 => return Err(ResolutionError::Unavailable),
- _ => return Err(ResolutionError::InvalidResponse),
- }
- let limit = self.config.limits.max_response_bytes;
- if response
- .content_length()
- .is_some_and(|size| size > limit as u64)
- {
+ .header(REQUEST_ID_HEADER, request_id))
+ }
+
+ async fn execute(
+ &self,
+ request: RequestBuilder,
+ limits: ControlPlaneLimits,
+ ) -> Result<Vec<u8>, ResolutionError> {
+ tokio::time::timeout(limits.timeout, async {
+ let response = request
+ .send()
+ .await
+ .map_err(|_| ResolutionError::Transport)?;
+ read_response(response, limits.max_response_bytes).await
+ })
+ .await
+ .map_err(|_| ResolutionError::Timeout)?
+ }
+}
+
+async fn read_response(
+ mut response: reqwest::Response,
+ limit: usize,
+) -> Result<Vec<u8>, ResolutionError> {
+ match response.status().as_u16() {
+ 200 => {}
+ 400 => return Err(ResolutionError::InvalidRequest),
+ 401 => return Err(ResolutionError::Authentication),
+ 403 => return Err(ResolutionError::Forbidden),
+ 404 => return Err(ResolutionError::NoRoute),
+ 429 | 500..=599 => return Err(ResolutionError::Unavailable),
+ _ => return Err(ResolutionError::InvalidResponse),
+ }
+ if response
+ .content_length()
+ .is_some_and(|size| size > limit as u64)
+ {
+ return Err(ResolutionError::ResponseTooLarge);
+ }
+ let mut bytes = Vec::new();
+ while let Some(chunk) = response
+ .chunk()
+ .await
+ .map_err(|_| ResolutionError::Transport)?
+ {
+ if chunk.len() > limit - bytes.len() {
return Err(ResolutionError::ResponseTooLarge);
}
- let mut bytes = Vec::new();
- while let Some(chunk) = response
- .chunk()
- .await
- .map_err(|_| ResolutionError::Transport)?
- {
- if chunk.len() > limit - bytes.len() {
- return Err(ResolutionError::ResponseTooLarge);
- }
- bytes.extend_from_slice(&chunk);
- }
- Ok(bytes)
+ bytes.extend_from_slice(&chunk);
}
+ Ok(bytes)
}
fn valid_token(value: &str) -> bool {
@@ -227,6 +290,7 @@ fn valid_token(value: &str) -> bool {
pub enum ResolutionError {
Configuration,
InvalidCredential,
+ InvalidRequest,
Authentication,
Forbidden,
NoRoute,
@@ -242,6 +306,7 @@ impl fmt::Display for ResolutionError {
f.write_str(match self {
Self::Configuration => "invalid resolver configuration",
Self::InvalidCredential => "invalid gateway credential",
+ Self::InvalidRequest => "invalid gateway request",
Self::Authentication => "gateway authentication failed",
Self::Forbidden => "gateway access forbidden",
Self::NoRoute => "no eligible connection",
diff --git a/ai-gateway/src/resolution/tests.rs b/ai-gateway/src/resolution/tests.rs
index 009cc19513..4949754b0a 100644
--- a/ai-gateway/src/resolution/tests.rs
+++ b/ai-gateway/src/resolution/tests.rs
@@ -58,16 +58,16 @@ impl FakeWeb {
}
fn control_plane(&self) -> ControlPlaneClient {
- self.control_plane_with_limits(Duration::from_secs(2), 256 * 1024)
+ self.control_plane_with_resolve_limits(Duration::from_secs(2), 256 * 1024)
}
- fn control_plane_with_limits(
+ fn control_plane_with_resolve_limits(
&self,
timeout: Duration,
max_response_bytes: usize,
) -> ControlPlaneClient {
let mut config = ControlPlaneConfig::new(&self.url, SERVICE_KEY).unwrap();
- config.limits = ResolutionLimits {
+ config.resolve_limits = ControlPlaneLimits {
timeout,
max_response_bytes,
};
@@ -97,6 +97,7 @@ fn success(project: &str, provider_token: &str) -> Value {
"id": "connection-1",
"name": "Primary OpenAI",
"provider": "openai",
+ "provider_model": "gpt-4.1",
"api_format": "openai.responses",
"base_url": "https://api.openai.com/v1",
"auth": {"type": "Bearer", "token": provider_token}
@@ -114,7 +115,7 @@ fn success(project: &str, provider_token: &str) -> Value {
}
#[tokio::test]
-async fn signs_the_exact_key_and_sends_only_the_api_format() {
+async fn signs_the_exact_key_and_sends_api_format_and_model() {
let web = FakeWeb::start(|request| async move {
assert_eq!(request.method(), "POST");
assert_eq!(request.uri(), "/api/internal/ai-gateway/v1/resolve");
@@ -124,7 +125,7 @@ async fn signs_the_exact_key_and_sends_only_the_api_format() {
assert_eq!(request.headers()["langfuse-gateway-authorization"],
"HMAC timestamp=1800000000,signature=e104b6baa50d9b766b036e27876683aa1f4bff7f31bb6af66dce6b636a53b2bd");
assert_eq!(to_bytes(request.into_body(), 1024).await.unwrap(),
- r#"{"apiFormat":"openai.responses"}"#);
+ r#"{"apiFormat":"openai.responses","model":"primary:gpt-4.1"}"#);
response(success("project-1", "private-provider-token"))
}).await;
@@ -133,6 +134,7 @@ async fn signs_the_exact_key_and_sends_only_the_api_format() {
.resolve_at(
GATEWAY_KEY,
ApiFormat::OpenAiResponses,
+ "primary:gpt-4.1",
NOW_TIME,
REQUEST_ID,
)
@@ -142,6 +144,7 @@ async fn signs_the_exact_key_and_sends_only_the_api_format() {
assert_eq!(context.connection().name(), Some("Primary OpenAI"));
assert_eq!(context.connection().provider(), Provider::OpenAi);
assert_eq!(context.connection().provider().as_str(), "openai");
+ assert_eq!(context.connection().provider_model(), "gpt-4.1");
assert!(matches!(
context.connection().credential(),
ProviderCredential::Bearer("private-provider-token")
@@ -171,9 +174,81 @@ async fn signs_the_exact_key_and_sends_only_the_api_format() {
assert_eq!(web.calls(), 1);
}
+#[tokio::test]
+async fn fetches_models_from_the_control_plane() {
+ let web = FakeWeb::start(|request| async move {
+ assert_eq!(request.method(), "GET");
+ assert_eq!(
+ request.uri(),
+ "/api/internal/ai-gateway/v1/models?api_format=openai.responses"
+ );
+ assert_eq!(
+ request.headers()[header::AUTHORIZATION],
+ "Bearer gw_test_alice"
+ );
+ assert!(
+ request
+ .headers()
+ .contains_key("langfuse-gateway-authorization")
+ );
+ assert_eq!(request.headers()["langfuse-gateway-request-id"], REQUEST_ID);
+ assert!(
+ to_bytes(request.into_body(), 1024)
+ .await
+ .unwrap()
+ .is_empty()
+ );
+ response(json!({
+ "data": [{
+ "id": "Primary OpenAI:gpt-4.1",
+ "provider": "OPENAI"
+ }]
+ }))
+ })
+ .await;
+
+ let body = web
+ .control_plane()
+ .models(GATEWAY_KEY, ApiFormat::OpenAiResponses, REQUEST_ID)
+ .await
+ .unwrap();
+ assert_eq!(
+ serde_json::from_slice::<Value>(&body).unwrap()["data"][0]["id"],
+ "Primary OpenAI:gpt-4.1"
+ );
+}
+
+#[tokio::test]
+async fn model_catalogs_have_a_larger_response_limit_than_resolution() {
+ let body = "x".repeat(300 * 1024);
+ let web = FakeWeb::start(move |_| {
+ let body = body.clone();
+ async move { Response::new(Body::from(body)) }
+ })
+ .await;
+ let control_plane = web.control_plane();
+
+ let resolution = control_plane
+ .resolve_at(
+ GATEWAY_KEY,
+ ApiFormat::OpenAiResponses,
+ "primary:gpt-4.1",
+ NOW_TIME,
+ REQUEST_ID,
+ )
+ .await;
+ assert!(matches!(resolution, Err(ResolutionError::ResponseTooLarge)));
+
+ let models = control_plane
+ .models(GATEWAY_KEY, ApiFormat::OpenAiResponses, REQUEST_ID)
+ .await
+ .unwrap();
+ assert_eq!(models.len(), 300 * 1024);
+}
+
#[tokio::test]
async fn denied_and_failed_resolutions_are_classified_without_retries_or_body_leaks() {
- for status in [401, 403, 404, 500, 503] {
+ for status in [400, 401, 403, 404, 500, 503] {
let web = FakeWeb::start(move |_| async move {
Response::builder()
.status(status)
@@ -186,12 +261,14 @@ async fn denied_and_failed_resolutions_are_classified_without_retries_or_body_le
.resolve_at(
GATEWAY_KEY,
ApiFormat::OpenAiResponses,
+ "primary:gpt-4.1",
NOW_TIME,
REQUEST_ID,
)
.await
.unwrap_err();
assert!(match status {
+ 400 => matches!(error, ResolutionError::InvalidRequest),
401 => matches!(error, ResolutionError::Authentication),
403 => matches!(error, ResolutionError::Forbidden),
404 => matches!(error, ResolutionError::NoRoute),
@@ -223,6 +300,7 @@ async fn never_follows_redirects_with_credentials() {
.resolve_at(
GATEWAY_KEY,
ApiFormat::OpenAiResponses,
+ "primary:gpt-4.1",
NOW_TIME,
REQUEST_ID
)
@@ -250,10 +328,11 @@ async fn rejects_oversized_declared_and_chunked_responses() {
})
.await;
let result = web
- .control_plane_with_limits(Duration::from_secs(2), 128)
+ .control_plane_with_resolve_limits(Duration::from_secs(2), 128)
.resolve_at(
GATEWAY_KEY,
ApiFormat::OpenAiResponses,
+ "primary:gpt-4.1",
NOW_TIME,
REQUEST_ID,
)
@@ -282,10 +361,11 @@ async fn deadline_covers_response_headers_and_body() {
.await;
let result = tokio::time::timeout(
Duration::from_millis(500),
- web.control_plane_with_limits(Duration::from_millis(30), 256 * 1024)
+ web.control_plane_with_resolve_limits(Duration::from_millis(30), 256 * 1024)
.resolve_at(
GATEWAY_KEY,
ApiFormat::OpenAiResponses,
+ "primary:gpt-4.1",
NOW_TIME,
REQUEST_ID,
),
@@ -303,7 +383,13 @@ async fn malformed_credentials_never_reach_web() {
for key in ["", "has space", "has\r\nheader", "has\ttab"] {
let result = web
.control_plane()
- .resolve_at(key, ApiFormat::OpenAiResponses, NOW_TIME, REQUEST_ID)
+ .resolve_at(
+ key,
+ ApiFormat::OpenAiResponses,
+ "primary:gpt-4.1",
+ NOW_TIME,
+ REQUEST_ID,
+ )
.await;
assert!(matches!(result, Err(ResolutionError::InvalidCredential)));
}
@@ -324,6 +410,7 @@ async fn rejects_a_grant_that_expires_while_resolving_across_a_second_boundary()
.resolve_at(
GATEWAY_KEY,
ApiFormat::OpenAiResponses,
+ "primary:gpt-4.1",
NOW_TIME + Duration::from_millis(900),
REQUEST_ID,
)
@@ -350,12 +437,14 @@ async fn concurrent_resolutions_keep_credentials_and_contexts_isolated() {
control_plane.resolve_at(
"gw_test_alice",
ApiFormat::OpenAiResponses,
+ "primary:gpt-4.1",
NOW_TIME,
REQUEST_ID
),
control_plane.resolve_at(
"gw_test_bob",
ApiFormat::OpenAiResponses,
+ "primary:gpt-4.1",
NOW_TIME,
REQUEST_ID
),
@@ -380,6 +469,7 @@ fn anthropic_success(provider_key: &str) -> Value {
body["connection"] = json!({
"id": "connection-anthropic",
"provider": "anthropic",
+ "provider_model": "claude-sonnet-4-5",
"api_format": "anthropic.messages",
"base_url": "https://api.anthropic.com/v1",
"auth": {"type": "x-api-key", "header": "x-api-key", "value": provider_key}
@@ -393,7 +483,7 @@ async fn resolves_anthropic_messages_with_the_native_credential_scheme() {
let web = FakeWeb::start(|request| async move {
assert_eq!(
to_bytes(request.into_body(), 1024).await.unwrap(),
- r#"{"apiFormat":"anthropic.messages"}"#
+ r#"{"apiFormat":"anthropic.messages","model":"primary:claude-sonnet-4-5"}"#
);
response(anthropic_success("private-anthropic-key"))
})
@@ -404,6 +494,7 @@ async fn resolves_anthropic_messages_with_the_native_credential_scheme() {
.resolve_at(
GATEWAY_KEY,
ApiFormat::AnthropicMessages,
+ "primary:claude-sonnet-4-5",
NOW_TIME,
REQUEST_ID,
)
@@ -443,7 +534,7 @@ async fn chat_completions_resolve_only_an_openai_connection_for_that_format() {
async move {
assert_eq!(
to_bytes(request.into_body(), 1024).await.unwrap(),
- r#"{"apiFormat":"openai.chat-completions"}"#
+ r#"{"apiFormat":"openai.chat-completions","model":"primary:gpt-4.1"}"#
);
response(body)
}
@@ -454,6 +545,7 @@ async fn chat_completions_resolve_only_an_openai_connection_for_that_format() {
.resolve_at(
GATEWAY_KEY,
ApiFormat::OpenAiChatCompletions,
+ "primary:gpt-4.1",
NOW_TIME,
REQUEST_ID,
)
@@ -566,6 +658,7 @@ async fn rejects_incompatible_or_incomplete_execution_contexts() {
("/connection/auth/token", json!("")),
("/connection/auth/token", json!("bad\r\nheader")),
("/connection/id", json!("")),
+ ("/connection/provider_model", json!("")),
("/attribution/project_id", json!("")),
("/attribution/provider_connection_id", json!("")),
(
@@ -640,6 +733,7 @@ async fn rejects_ingestion_tokens_that_authorize_a_different_tenant() {
.resolve_at(
GATEWAY_KEY,
ApiFormat::OpenAiResponses,
+ "primary:gpt-4.1",
NOW_TIME,
REQUEST_ID
)
@@ -660,7 +754,13 @@ async fn assert_invalid_context_for(body: Value, api_format: ApiFormat) {
.await;
let result = web
.control_plane()
- .resolve_at(GATEWAY_KEY, api_format, NOW_TIME, REQUEST_ID)
+ .resolve_at(
+ GATEWAY_KEY,
+ api_format,
+ "primary:provider-model",
+ NOW_TIME,
+ REQUEST_ID,
+ )
.await;
assert!(matches!(result, Err(ResolutionError::InvalidResponse)));
assert_eq!(web.calls(), 1);
@@ -677,6 +777,7 @@ async fn rejects_malformed_json_without_exposing_response_contents() {
.resolve_at(
GATEWAY_KEY,
ApiFormat::OpenAiResponses,
+ "primary:gpt-4.1",
NOW_TIME,
REQUEST_ID,
)
diff --git a/ai-gateway/src/test_support.rs b/ai-gateway/src/test_support.rs
index 37b59075bb..6cefe779b7 100644
--- a/ai-gateway/src/test_support.rs
+++ b/ai-gateway/src/test_support.rs
@@ -100,16 +100,19 @@ pub(crate) fn resolution_response_for(
let connection = match api_format {
ApiFormat::OpenAiResponses => json!({
"id": "connection-1", "name": "Production key", "provider": "openai", "api_format": "openai.responses",
+ "provider_model": "provider-model",
"base_url": "https://api.openai.com/v1",
"auth": {"type": "Bearer", "token": provider_secret}
}),
ApiFormat::OpenAiChatCompletions => json!({
"id": "connection-1", "name": "Production key", "provider": "openai", "api_format": "openai.chat-completions",
+ "provider_model": "provider-model",
"base_url": "https://api.openai.com/v1",
"auth": {"type": "Bearer", "token": provider_secret}
}),
ApiFormat::AnthropicMessages => json!({
"id": "connection-1", "name": "Production key", "provider": "anthropic", "api_format": "anthropic.messages",
+ "provider_model": "provider-model",
"base_url": "https://api.anthropic.com/v1",
"auth": {"type": "x-api-key", "header": "x-api-key", "value": provider_secret}
}),
@@ -170,7 +173,12 @@ pub(crate) async fn resolved_request_context_for(
})
.await;
web.control_plane()
- .resolve("gateway-secret", api_format, "test-request-id")
+ .resolve(
+ "gateway-secret",
+ api_format,
+ "Production key:provider-model",
+ "test-request-id",
+ )
.await
.unwrap()
}
diff --git a/packages/shared/prisma/migrations/20261012134000_add_unique_gateway_connection_names/migration.sql b/packages/shared/prisma/migrations/20261012134000_add_unique_gateway_connection_names/migration.sql
new file mode 100644
index 0000000000..9fbb371274
--- /dev/null
+++ b/packages/shared/prisma/migrations/20261012134000_add_unique_gateway_connection_names/migration.sql
@@ -0,0 +1,2 @@
+CREATE UNIQUE INDEX gateway_ai_connections_organization_id_name_key
+ON gateway_ai_connections(organization_id, name);
diff --git a/packages/shared/prisma/schema.prisma b/packages/shared/prisma/schema.prisma
index e2c431c3b2..2597b1c145 100644
--- a/packages/shared/prisma/schema.prisma
+++ b/packages/shared/prisma/schema.prisma
@@ -361,6 +361,7 @@ model GatewayAiConnection {
createdAt DateTime @default(now()) @map("created_at")
updatedAt DateTime @default(now()) @updatedAt @map("updated_at")
+ @@unique([organizationId, name])
@@unique([organizationId, routingPriority])
@@index([organizationId, status, routingPriority])
@@map("gateway_ai_connections")
diff --git a/web/src/__e2e__/ai-gateway.gatewaye2e.ts b/web/src/__e2e__/ai-gateway.gatewaye2e.ts
index 43950ef48c..df7af2c53e 100644
--- a/web/src/__e2e__/ai-gateway.gatewaye2e.ts
+++ b/web/src/__e2e__/ai-gateway.gatewaye2e.ts
@@ -313,11 +313,17 @@ describe("AI gateway live end-to-end", () => {
async ({ provider, apiFormat, baseUrl, authType }) => {
await withGatewayApiKey(
{ provider, apiFormat },
- async ({ secretKey, id: keyId, providerConnectionId }) => {
+ async ({
+ secretKey,
+ id: keyId,
+ providerConnectionId,
+ providerConnectionName,
+ }) => {
const resolveBody = await gatewayControlPlaneRequest({
path: RESOLVE_PATH,
apiFormat,
virtualSecretKey: secretKey,
+ model: `${providerConnectionName}:ai-gateway-e2e`,
});
const resolved = GatewayResolveResponseSchema.parse(resolveBody);
@@ -377,6 +383,7 @@ async function withGatewayApiKey(
secretKey: string;
id: string;
providerConnectionId: string;
+ providerConnectionName: string;
}) => Promise<void>,
) {
const targetConnectionId = connectionIdsByProvider.get(input.provider);
@@ -386,6 +393,10 @@ async function withGatewayApiKey(
);
}
await selectOnlyConnection(targetConnectionId);
+ const targetConnection = await prisma.gatewayAiConnection.findUniqueOrThrow({
+ where: { id: targetConnectionId },
+ select: { name: true },
+ });
const gatewayKey = await admin.caller.aiGateway.createApiKey({
orgId: ORGANIZATION_ID,
@@ -410,6 +421,7 @@ async function withGatewayApiKey(
secretKey: gatewayKey.secretKey,
id: gatewayKey.id,
providerConnectionId: targetConnectionId,
+ providerConnectionName: targetConnection.name,
});
} finally {
await admin.caller.aiGateway.revokeApiKey({
@@ -547,6 +559,7 @@ async function gatewayControlPlaneRequest(input: {
path: string;
apiFormat: GatewayApiFormat;
virtualSecretKey: string;
+ model?: string;
}): Promise<unknown> {
const timestamp = Math.floor(Date.now() / 1000);
const canonicalMessage = [
@@ -571,7 +584,7 @@ async function gatewayControlPlaneRequest(input: {
},
body: isModelsRequest
? undefined
- : JSON.stringify({ apiFormat: input.apiFormat }),
+ : JSON.stringify({ apiFormat: input.apiFormat, model: input.model }),
});
return readJsonResponse(response, 200, input.path);
}
diff --git a/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts b/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts
index 050ea08197..f7359bcb6b 100644
--- a/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts
+++ b/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts
@@ -527,20 +527,20 @@ describe("AI gateway control plane", () => {
).rejects.toThrow();
});
- it("routes by format and priority and issues a verifiable ingestion token", async () => {
+ it("routes by connection slug and issues a verifiable ingestion token", async () => {
const { caller, org, project } = await prepare();
await caller.aiGateway.updateConfig({
orgId: org.id,
defaultIngestionProjectId: project.id,
ingestionMode: "FULL",
});
- const openAiPrimary = await caller.aiGateway.createConnection({
+ await caller.aiGateway.createConnection({
orgId: org.id,
name: "OpenAI first",
provider: "OPENAI",
credential: "sk-test-openai-primary",
});
- await caller.aiGateway.createConnection({
+ const openAiSecondary = await caller.aiGateway.createConnection({
orgId: org.id,
name: "OpenAI second",
provider: "OPENAI",
@@ -563,16 +563,17 @@ describe("AI gateway control plane", () => {
const result = await authenticateRequest({
fastHashedSecretKey: createShaHash(gatewayKey.secretKey, env.SALT),
apiFormat,
+ model: "OpenAI second:openai/gpt-4o",
});
- expect(openAiPrimary.routingPriority).toBe(0);
expect(result.connection).toEqual({
- id: openAiPrimary.id,
- name: "OpenAI first",
+ id: openAiSecondary.id,
+ name: "OpenAI second",
provider: "openai",
+ provider_model: "openai/gpt-4o",
api_format: apiFormat,
base_url: "https://api.openai.com/v1",
- auth: { type: "Bearer", token: "sk-test-openai-primary" },
+ auth: { type: "Bearer", token: "sk-test-openai-secondary" },
});
const ingestionToken = result.ingestion!.access_token;
const ingestionClaims = jwtVerifier.verify({ token: ingestionToken });
@@ -597,6 +598,132 @@ describe("AI gateway control plane", () => {
expect(result.ingestion?.expires_at).toBe(ingestionClaims.exp);
});
+ it("routes unprefixed models by provider and rejects unknown models", async () => {
+ configureGatewayJwtSigning();
+ const { caller, org, project } = await prepare();
+ await caller.aiGateway.updateConfig({
+ orgId: org.id,
+ defaultIngestionProjectId: project.id,
+ ingestionMode: "FULL",
+ });
+ const openAiPrimary = await caller.aiGateway.createConnection({
+ orgId: org.id,
+ name: "OpenAI primary",
+ provider: "OPENAI",
+ credential: "sk-test-openai-primary",
+ });
+ const openAiSecondary = await caller.aiGateway.createConnection({
+ orgId: org.id,
+ name: "OpenAI secondary",
+ provider: "OPENAI",
+ credential: "sk-test-openai-secondary",
+ });
+ const anthropic = await caller.aiGateway.createConnection({
+ orgId: org.id,
+ name: "Anthropic",
+ provider: "ANTHROPIC",
+ credential: "sk-test-anthropic",
+ });
+ const gatewayKey = await caller.aiGateway.createApiKey({
+ orgId: org.id,
+ metadata: {},
+ });
+ const otherOrganization = await prepare();
+ await otherOrganization.caller.aiGateway.createConnection({
+ orgId: otherOrganization.org.id,
+ name: "Unknown",
+ provider: "OPENAI",
+ credential: "sk-other-organization",
+ });
+ const params = {
+ fastHashedSecretKey: createShaHash(gatewayKey.secretKey, env.SALT),
+ apiFormat: "openai.responses" as const,
+ };
+
+ await expect(
+ authenticateRequest({ ...params, model: "gpt-4o" }),
+ ).resolves.toMatchObject({
+ connection: {
+ id: openAiPrimary.id,
+ provider: "openai",
+ provider_model: "gpt-4o",
+ },
+ });
+ await expect(
+ authenticateRequest({
+ ...params,
+ apiFormat: "anthropic.messages",
+ model: "claude-haiku-4-5-20251001",
+ }),
+ ).resolves.toMatchObject({
+ connection: {
+ id: anthropic.id,
+ provider: "anthropic",
+ provider_model: "claude-haiku-4-5-20251001",
+ },
+ });
+ await expect(
+ authenticateRequest({
+ ...params,
+ model: "OpenAI secondary:gpt-4o",
+ }),
+ ).resolves.toMatchObject({
+ connection: {
+ id: openAiSecondary.id,
+ provider: "openai",
+ provider_model: "gpt-4o",
+ },
+ });
+ await expect(
+ authenticateRequest({ ...params, model: "random-model" }),
+ ).rejects.toMatchObject({ status: 400 });
+ await expect(
+ authenticateRequest({ ...params, model: "Unknown:gpt-4o" }),
+ ).rejects.toMatchObject({ status: 404 });
+ });
+
+ it("requires connection names to be unique within an organization", async () => {
+ const { caller, org } = await prepare();
+ await expect(
+ caller.aiGateway.createConnection({
+ orgId: org.id,
+ name: "Production:Primary",
+ provider: "OPENAI",
+ credential: "sk-openai",
+ }),
+ ).rejects.toThrow("Connection names cannot contain a colon.");
+
+ await caller.aiGateway.createConnection({
+ orgId: org.id,
+ name: "Production",
+ provider: "OPENAI",
+ credential: "sk-openai",
+ });
+
+ await expect(
+ caller.aiGateway.createConnection({
+ orgId: org.id,
+ name: "Production",
+ provider: "ANTHROPIC",
+ credential: "sk-anthropic",
+ }),
+ ).rejects.toThrow("A provider credential with this name already exists.");
+
+ const backup = await caller.aiGateway.createConnection({
+ orgId: org.id,
+ name: "Backup",
+ provider: "ANTHROPIC",
+ credential: "sk-anthropic",
+ });
+ await expect(
+ caller.aiGateway.updateConnection({
+ orgId: org.id,
+ id: backup.id,
+ name: "Production",
+ }),
+ ).rejects.toThrow("A provider credential with this name already exists.");
+ });
+
it("changes ERROR only for credential auth failures and explicit recovery", async () => {
const { caller, org, session } = await prepare();
const connection = await caller.aiGateway.createConnection({
@@ -787,6 +914,12 @@ describe("AI gateway control plane", () => {
provider: "OPENAI",
credential: "sk-test",
});
+ await caller.aiGateway.createConnection({
+ orgId: org.id,
+ name: "OpenAI backup",
+ provider: "OPENAI",
+ credential: "sk-test-backup",
+ });
const key = await caller.aiGateway.createApiKey({
orgId: org.id,
metadata: {},
@@ -794,6 +927,7 @@ describe("AI gateway control plane", () => {
const resolveParams = {
fastHashedSecretKey: createShaHash(key.secretKey, env.SALT),
apiFormat: "openai.responses" as const,
+ model: "OpenAI:gpt-4o",
};
const lookup = vi.spyOn(prisma.gatewayApiKeyAssociation, "findFirst");
@@ -802,20 +936,37 @@ describe("AI gateway control plane", () => {
connection: { auth: { type: "Bearer", token: "sk-test" } },
});
}
+ await expect(
+ authenticateRequest({
+ ...resolveParams,
+ model: "OpenAI backup:gpt-4o",
+ }),
+ ).resolves.toMatchObject({
+ connection: { auth: { type: "Bearer", token: "sk-test-backup" } },
+ });
// The second call must not reach Postgres: this endpoint runs on every LLM
// request through the gateway.
expect(lookup).toHaveBeenCalledOnce();
- // Disabling a connection has to take effect now, not when the TTL expires.
+ // Renaming a connection invalidates its old model slugs immediately.
await caller.aiGateway.updateConnection({
orgId: org.id,
id: connection.id,
- status: "DISABLED",
+ name: "OpenAI renamed",
});
await expect(authenticateRequest(resolveParams)).rejects.toEqual(
expect.objectContaining<Partial<GatewayResolveError>>({ status: 404 }),
);
expect(lookup).toHaveBeenCalledTimes(2);
+ await expect(
+ authenticateRequest({
+ ...resolveParams,
+ model: "OpenAI renamed:gpt-4o",
+ }),
+ ).resolves.toMatchObject({
+ connection: { name: "OpenAI renamed" },
+ });
+ expect(lookup).toHaveBeenCalledTimes(2);
lookup.mockRestore();
});
@@ -823,6 +974,7 @@ describe("AI gateway control plane", () => {
const resolveParams = {
fastHashedSecretKey: createShaHash(`sk-lf-${randomUUID()}`, env.SALT),
apiFormat: "openai.responses" as const,
+ model: "gpt-4o",
};
const lookup = vi.spyOn(prisma.gatewayApiKeyAssociation, "findFirst");
@@ -855,7 +1007,7 @@ describe("AI gateway control plane", () => {
Object.assign(env, { API_AUTH_MIGRATION: "enforce" });
await expect(
- authenticator.authenticateRequest(request),
+ authenticator.authenticateModelsRequest(request),
).resolves.toMatchObject({
organizationId: org.id,
});
@@ -865,7 +1017,7 @@ describe("AI gateway control plane", () => {
data: { systemRole: "INGEST", roleId: "system/INGEST" },
});
await expect(
- authenticator.authenticateRequest(request),
+ authenticator.authenticateModelsRequest(request),
).rejects.toMatchObject({
status: 403,
message: "API key is not allowed to invoke the gateway",
@@ -873,7 +1025,7 @@ describe("AI gateway control plane", () => {
Object.assign(env, { API_AUTH_MIGRATION: "legacy" });
await expect(
- authenticator.authenticateRequest(request),
+ authenticator.authenticateModelsRequest(request),
).resolves.toMatchObject({
organizationId: org.id,
});
@@ -930,20 +1082,27 @@ describe("AI gateway control plane", () => {
authenticateRequest({
fastHashedSecretKey: createShaHash(key.secretKey, env.SALT),
apiFormat,
+ model: "OpenAI:gpt-4o",
}),
).rejects.toEqual(
expect.objectContaining<Partial<GatewayResolveError>>({ status: 403 }),
);
});
- it("returns a canonical model catalog across enabled compatible connections", async () => {
+ it("returns connection-prefixed models without consolidating connections", async () => {
const { caller, org } = await prepare();
await caller.aiGateway.createConnection({
orgId: org.id,
- name: "OpenAI",
+ name: "OpenAI primary",
provider: "OPENAI",
credential: "sk-openai",
});
+ await caller.aiGateway.createConnection({
+ orgId: org.id,
+ name: "OpenAI backup",
+ provider: "OPENAI",
+ credential: "sk-openai-backup",
+ });
await caller.aiGateway.createConnection({
orgId: org.id,
name: "Anthropic",
@@ -1001,7 +1160,7 @@ describe("AI gateway control plane", () => {
const fastHashedSecretKey = createShaHash(key.secretKey, env.SALT);
const openAiContext = await new GatewayApiKeyAuthenticator(
prisma,
- ).authenticateRequest({
+ ).authenticateModelsRequest({
fastHashedSecretKey,
apiFormat: "openai.responses",
});
@@ -1013,14 +1172,28 @@ describe("AI gateway control plane", () => {
expect(result).toEqual({
data: [
{
- id: "a-model",
+ id: "OpenAI backup:a-model",
provider: "OPENAI",
canonicalSlug: "a-model",
displayName: "a-model",
createdAt: "1970-01-01T00:00:10.000Z",
},
{
- id: "shared-model",
+ id: "OpenAI backup:shared-model",
+ provider: "OPENAI",
+ canonicalSlug: "shared-model",
+ displayName: "shared-model",
+ createdAt: "1970-01-01T00:00:15.000Z",
+ },
+ {
+ id: "OpenAI primary:a-model",
+ provider: "OPENAI",
+ canonicalSlug: "a-model",
+ displayName: "a-model",
+ createdAt: "1970-01-01T00:00:10.000Z",
+ },
+ {
+ id: "OpenAI primary:shared-model",
provider: "OPENAI",
canonicalSlug: "shared-model",
displayName: "shared-model",
@@ -1030,7 +1203,7 @@ describe("AI gateway control plane", () => {
});
const anthropicContext = await new GatewayApiKeyAuthenticator(
prisma,
- ).authenticateRequest({
+ ).authenticateModelsRequest({
fastHashedSecretKey,
apiFormat: "anthropic.messages",
});
@@ -1041,14 +1214,14 @@ describe("AI gateway control plane", () => {
expect(anthropicResult).toEqual({
data: [
{
- id: "claude-a",
+ id: "Anthropic:claude-a",
provider: "ANTHROPIC",
canonicalSlug: "claude-a",
displayName: "Claude A",
createdAt: "2026-01-01T00:00:00.000Z",
},
{
- id: "claude-b",
+ id: "Anthropic:claude-b",
provider: "ANTHROPIC",
canonicalSlug: "claude-b",
displayName: "Claude B",
@@ -1056,10 +1229,10 @@ describe("AI gateway control plane", () => {
},
],
});
- expect(fetcher).toHaveBeenCalledTimes(2);
+ expect(fetcher).toHaveBeenCalledTimes(3);
});
- it("fails model discovery instead of returning a partial catalog", async () => {
+ it("returns partial catalogs and fails only when every connection fails", async () => {
const { caller, org } = await prepare();
await caller.aiGateway.createConnection({
orgId: org.id,
@@ -1088,7 +1261,7 @@ describe("AI gateway control plane", () => {
const context = await new GatewayApiKeyAuthenticator(
prisma,
- ).authenticateRequest({
+ ).authenticateModelsRequest({
fastHashedSecretKey: createShaHash(key.secretKey, env.SALT),
apiFormat: "openai.chat-completions",
});
@@ -1097,6 +1270,26 @@ describe("AI gateway control plane", () => {
context,
apiFormat: "openai.chat-completions",
}),
+ ).resolves.toEqual({
+ data: [
+ {
+ id: "OpenAI fallback:available-model",
+ provider: "OPENAI",
+ canonicalSlug: "available-model",
+ displayName: "available-model",
+ createdAt: "1970-01-01T00:00:10.000Z",
+ },
+ ],
+ });
+
+ const unavailableFetcher = vi
+ .fn<typeof fetch>()
+ .mockResolvedValue(new Response(null, { status: 500 }));
+ await expect(
+ new GatewayModelsService(prisma, unavailableFetcher, null).list({
+ context,
+ apiFormat: "openai.chat-completions",
+ }),
).rejects.toEqual(
expect.objectContaining<Partial<GatewayResolveError>>({ status: 503 }),
);
diff --git a/web/src/__tests__/server/unit/ai-gateway-resolve-route.servertest.ts b/web/src/__tests__/server/unit/ai-gateway-resolve-route.servertest.ts
index d4783b8ba7..e5ce62da0d 100644
--- a/web/src/__tests__/server/unit/ai-gateway-resolve-route.servertest.ts
+++ b/web/src/__tests__/server/unit/ai-gateway-resolve-route.servertest.ts
@@ -62,20 +62,49 @@ describe("POST /api/internal/ai-gateway/v1/resolve", () => {
])("accepts api format %s and advances to authorization", async (format) => {
// 401 rather than 400 proves the body passed the schema: the route only
// reaches signature verification after parsing succeeds.
- expect((await post({ apiFormat: format })).statusCode).toBe(401);
- });
-
- it("accepts a model without using it for resolution", async () => {
expect(
(
await post({
- apiFormat: "openai.responses",
- model: "gpt-4o-mini",
+ apiFormat: format,
+ model: "production:model-with:separator",
})
).statusCode,
).toBe(401);
});
+ it.each([
+ { model: "gpt-4o-mini", apiFormat: "openai.responses" },
+ { model: "claude-haiku-4-5-20251001", apiFormat: "anthropic.messages" },
+ ])(
+ "accepts inferred provider model $model and advances to authorization",
+ async ({ model, apiFormat }) => {
+ expect((await post({ apiFormat, model })).statusCode).toBe(401);
+ },
+ );
+
+ it("rejects a missing model before authorization", async () => {
+ expect((await post({ apiFormat: "openai.responses" })).statusCode).toBe(
+ 400,
+ );
+ });
+
+ it.each([
+ {
+ case: "unknown unprefixed model",
+ body: { apiFormat: "openai.responses", model: "random-model" },
+ },
+ {
+ case: "empty connection name",
+ body: { apiFormat: "openai.responses", model: ":gpt-4o-mini" },
+ },
+ {
+ case: "empty provider model",
+ body: { apiFormat: "openai.responses", model: "production:" },
+ },
+ ])("rejects an invalid model slug: $case", async ({ body }) => {
+ expect((await post(body)).statusCode).toBe(400);
+ });
+
it("rejects a body that is not valid JSON", async () => {
const { req, res } = createMocks<NextApiRequest, NextApiResponse>({
method: "POST",
diff --git a/web/src/__tests__/server/unit/gatewayModelSlug.servertest.ts b/web/src/__tests__/server/unit/gatewayModelSlug.servertest.ts
new file mode 100644
index 0000000000..6973e94660
--- /dev/null
+++ b/web/src/__tests__/server/unit/gatewayModelSlug.servertest.ts
@@ -0,0 +1,86 @@
+import { describe, expect, it } from "vitest";
+
+import { resolveGatewayModelRoute } from "@/src/features/ai-gateway/fns/gatewayModelSlug";
+
+const connections = [
+ { id: "openai-primary", name: "OpenAI primary", provider: "OPENAI" },
+ { id: "anthropic", name: "Anthropic", provider: "ANTHROPIC" },
+ { id: "openai-secondary", name: "OpenAI secondary", provider: "OPENAI" },
+] as const;
+
+describe("resolveGatewayModelRoute", () => {
+ it.each([
+ {
+ model: "OpenAI secondary:gpt-4o",
+ connectionId: "openai-secondary",
+ providerModel: "gpt-4o",
+ },
+ {
+ model: "OpenAI secondary:model:with:colon",
+ connectionId: "openai-secondary",
+ providerModel: "model:with:colon",
+ },
+ {
+ model: "gpt-6-sol",
+ connectionId: "openai-primary",
+ providerModel: "gpt-6-sol",
+ },
+ {
+ model: "ft:gpt-4o-mini-2024-07-18:acme::abc123",
+ connectionId: "openai-primary",
+ providerModel: "ft:gpt-4o-mini-2024-07-18:acme::abc123",
+ },
+ {
+ model: "claude-sonnet-4-5",
+ connectionId: "anthropic",
+ providerModel: "claude-sonnet-4-5",
+ },
+ ])(
+ "routes $model to $connectionId",
+ ({ model, connectionId, providerModel }) => {
+ const route = resolveGatewayModelRoute(model, connections);
+
+ expect(route).not.toBeNull();
+ expect(route?.connection?.id).toBe(connectionId);
+ expect(route?.providerModel).toBe(providerModel);
+ },
+ );
+
+ it.each(["gpt", "claude", "random-model", ":model", "OpenAI primary:"])(
+ "rejects unsupported model %s",
+ (model) => {
+ expect(resolveGatewayModelRoute(model, connections)).toBeNull();
+ },
+ );
+
+ it("returns a valid unresolved route for a missing connection", () => {
+ expect(resolveGatewayModelRoute("Unknown:gpt-4o", connections)).toEqual({
+ connection: null,
+ providerModel: "gpt-4o",
+ });
+ expect(
+ resolveGatewayModelRoute("claude-sonnet-4-5", connections.slice(0, 1)),
+ ).toEqual({
+ connection: null,
+ providerModel: "claude-sonnet-4-5",
+ });
+ });
+
+ it("prefers an explicit ft connection over the OpenAI fine-tune fallback", () => {
+ const ftConnection = {
+ id: "ft-connection",
+ name: "ft",
+ provider: "ANTHROPIC",
+ } as const;
+
+ expect(
+ resolveGatewayModelRoute("ft:claude-sonnet-4-5", [
+ ...connections,
+ ftConnection,
+ ]),
+ ).toEqual({
+ connection: ftConnection,
+ providerModel: "claude-sonnet-4-5",
+ });
+ });
+});
diff --git a/web/src/__tests__/server/unit/gatewayResolveCache.servertest.ts b/web/src/__tests__/server/unit/gatewayResolveCache.servertest.ts
index c935496ebc..fd3f0b49b6 100644
--- a/web/src/__tests__/server/unit/gatewayResolveCache.servertest.ts
+++ b/web/src/__tests__/server/unit/gatewayResolveCache.servertest.ts
@@ -19,7 +19,7 @@ const context: CachedResolveContext = {
keyMetadata: {},
ingestionProjectId: null,
ingestionMode: null,
- connection: null,
+ connections: [],
};
describe("GatewayResolveCache", () => {
diff --git a/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.clienttest.ts b/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.clienttest.ts
index 9500ec50b8..6e8f55d593 100644
--- a/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.clienttest.ts
+++ b/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.clienttest.ts
@@ -17,17 +17,32 @@ describe("aggregateModels", () => {
provider: "ANTHROPIC",
} as Connection;
const results = [
- { connectionId: "second", success: true, models: ["a-model"] },
- { connectionId: "first", success: true, models: ["z-model", "b-model"] },
+ {
+ connectionId: "second",
+ success: true,
+ models: ["a-model", "shared-model"],
+ },
+ {
+ connectionId: "first",
+ success: true,
+ models: ["z-model", "b-model", "shared-model"],
+ },
] as Result[];
expect(aggregateModels(results, [first]).map((model) => model.id)).toEqual([
- "b-model",
- "z-model",
+ "First:b-model",
+ "First:shared-model",
+ "First:z-model",
]);
expect(
aggregateModels(results, [first, second]).map((model) => model.id),
- ).toEqual(["b-model", "z-model", "a-model"]);
+ ).toEqual([
+ "First:b-model",
+ "First:shared-model",
+ "First:z-model",
+ "Second:a-model",
+ "Second:shared-model",
+ ]);
});
it("does not duplicate badges for connections repeated across pages", () => {
diff --git a/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.ts b/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.ts
index 38433b41f1..fa819278e5 100644
--- a/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.ts
+++ b/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.ts
@@ -1,4 +1,5 @@
import type { GatewayProvider } from "@/src/features/ai-gateway/types/gatewayProvider";
+import { formatGatewayModelSlug } from "@/src/features/ai-gateway/fns/gatewayModelSlug";
import type { RouterOutputs } from "@/src/utils/api";
type Connection = RouterOutputs["aiGateway"]["listConnections"]["data"][number];
@@ -28,7 +29,7 @@ export function aggregateModels(
const resultsByConnectionId = new Map(
results.map((result) => [result.connectionId, result]),
);
- const models = new Map<string, ModelRow>();
+ const models: ModelRow[] = [];
const seenConnectionIds = new Set<string>();
for (const connection of connections) {
@@ -39,29 +40,18 @@ export function aggregateModels(
for (const modelId of result.models.toSorted((left, right) =>
left.localeCompare(right),
)) {
- const existing = models.get(modelId);
const availableVia = {
connectionId: connection.id,
connectionName: connection.name,
provider: connection.provider,
};
- if (existing) {
- existing.availableVia.push(availableVia);
- existing.apiFormats = [
- ...new Set([
- ...existing.apiFormats,
- ...providerFormats[connection.provider],
- ]),
- ];
- } else {
- models.set(modelId, {
- id: modelId,
- availableVia: [availableVia],
- apiFormats: [...providerFormats[connection.provider]],
- });
- }
+ models.push({
+ id: formatGatewayModelSlug(connection.name, modelId),
+ availableVia: [availableVia],
+ apiFormats: [...providerFormats[connection.provider]],
+ });
}
}
- return [...models.values()];
+ return models;
}
diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/GatewayProvidersPage.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/GatewayProvidersPage.tsx
index 8184e95a64..8b404f5423 100644
--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/GatewayProvidersPage.tsx
+++ b/web/src/features/ai-gateway/components/GatewayProvidersPage/GatewayProvidersPage.tsx
@@ -1,4 +1,4 @@
-import { useRef, useState } from "react";
+import { useState } from "react";
import { Plus } from "lucide-react";
import Header from "@/src/components/layouts/header";
@@ -8,7 +8,7 @@ import { PaginationBar } from "@/src/components/design-system/PaginationBar/Pagi
import { Button } from "@/src/components/ui/button";
import { Skeleton } from "@/src/components/ui/skeleton";
import { ProviderDialogController } from "@/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderDialogController/ProviderDialogController";
-import { ConnectedGatewayProvidersTable } from "@/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/ConnectedGatewayProvidersTable";
+import { GatewayProvidersTable } from "@/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable";
import { buildGatewayModelsUrl } from "@/src/features/ai-gateway/fns/gatewayUrls/buildGatewayModelsUrl";
import type { GatewayConnection } from "@/src/features/ai-gateway/types/gatewayProvider";
import { api, reportNonTrpcError } from "@/src/utils/api";
@@ -30,10 +30,7 @@ export function GatewayProvidersPage({
Record<string, number>
>({});
const [pageIndex, setPageIndex] = useState(0);
- const [isReordering, setIsReordering] = useState(false);
- const reorderLock = useRef(false);
const utils = api.useUtils();
- const reorder = api.aiGateway.reorderConnections.useMutation();
const retry = api.aiGateway.retryConnection.useMutation();
const remove = api.aiGateway.deleteConnection.useMutation();
@@ -101,9 +98,6 @@ export function GatewayProvidersPage({
const visiblePageIndex = Math.min(pageIndex, Math.max(0, pages.length - 1));
const allConnections = pages.flatMap((page) => page.data);
const connections = pages[visiblePageIndex]?.data ?? [];
- const pageOffset = pages
- .slice(0, visiblePageIndex)
- .reduce((count, page) => count + page.data.length, 0);
const modelCounts: Record<string, number | "loading"> = modelsQuery.isPending
? Object.fromEntries(
allConnections
@@ -133,9 +127,8 @@ export function GatewayProvidersPage({
}
/>
<p className="text-muted-foreground text-sm">
- Requests use the first compatible enabled credential in routing priority
- order. Credentials are validated against the provider when they are
- saved.
+ Model names are prefixed with their connection name. Credentials are
+ validated against the provider when they are saved.
</p>
<ConfirmationDialogController<GatewayConnection>
title="Delete provider credential"
@@ -151,58 +144,12 @@ export function GatewayProvidersPage({
<ProviderDialogController organizationId={organizationId}>
{({ openEditDialog }) => (
<div className="flex max-h-[60dvh] flex-col overflow-hidden">
- <ConnectedGatewayProvidersTable
+ <GatewayProvidersTable
connections={connections}
- pageOffset={pageOffset}
- previousConnectionId={allConnections[pageOffset - 1]?.id}
- nextConnectionId={
- allConnections[pageOffset + connections.length]?.id
- }
modelCounts={modelCounts}
getModelsUrl={(connection) =>
buildGatewayModelsUrl(organizationId, connection.id)
}
- canReorder={!connectionsQuery.hasNextPage && !isReordering}
- onReorder={async (sourceId, targetId) => {
- if (reorderLock.current) return false;
- const sourceIndex = allConnections.findIndex(
- (connection) => connection.id === sourceId,
- );
- const targetIndex = allConnections.findIndex(
- (connection) => connection.id === targetId,
- );
- if (sourceIndex < 0 || targetIndex < 0) return false;
- reorderLock.current = true;
- setIsReordering(true);
- const connectionIds = allConnections.map(
- (connection) => connection.id,
- );
- const [movedId] = connectionIds.splice(sourceIndex, 1);
- if (!movedId) return false;
- connectionIds.splice(targetIndex, 0, movedId);
- try {
- await reorder.mutateAsync({
- orgId: organizationId,
- connectionIds,
- });
- await utils.aiGateway.listConnections.invalidate({
- orgId: organizationId,
- });
- if (
- Math.floor(sourceIndex / 50) !==
- Math.floor(targetIndex / 50)
- ) {
- setPageIndex(Math.floor(targetIndex / 50));
- }
- return true;
- } catch (error) {
- reportNonTrpcError(error, "ai-gateway-providers");
- return false;
- } finally {
- reorderLock.current = false;
- setIsReordering(false);
- }
- }}
onRetry={retryConnection}
retryDisabled={
retry.isPending
diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.stories.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.stories.tsx
new file mode 100644
index 0000000000..1c19ea26f8
--- /dev/null
+++ b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.stories.tsx
@@ -0,0 +1,31 @@
+import { fn } from "storybook/test";
+
+import preview from "@/.storybook/preview";
+import { CredentialFields } from "./CredentialFields";
+
+const meta = preview.meta({ component: CredentialFields });
+
+const callbacks = {
+ onNameChange: fn(),
+ onCredentialChange: fn(),
+};
+
+export const Default = meta.story({
+ args: {
+ name: "Production",
+ nameError: null,
+ credential: "",
+ credentialPlaceholder: "Enter secret key",
+ ...callbacks,
+ },
+});
+
+export const DuplicateNameError = meta.story({
+ args: {
+ name: "Production",
+ nameError: "A provider credential with this name already exists.",
+ credential: "",
+ credentialPlaceholder: "Enter secret key",
+ ...callbacks,
+ },
+});
diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.tsx
index d93855616d..c2f336990a 100644
--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.tsx
+++ b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.tsx
@@ -1,14 +1,16 @@
-import { Input } from "@/src/components/ui/input";
+import { Input } from "@/src/components/design-system/Input/Input";
import { Label } from "@/src/components/ui/label";
export function CredentialFields({
name,
+ nameError,
credential,
credentialPlaceholder,
onNameChange,
onCredentialChange,
}: {
name: string;
+ nameError: string | null;
credential: string;
credentialPlaceholder: string;
onNameChange: (name: string) => void;
@@ -18,25 +20,41 @@ export function CredentialFields({
<>
<div>
<Label htmlFor="gateway-credential-name">Name</Label>
- <Input
- id="gateway-credential-name"
- className="mt-1.5"
- placeholder="Production"
- value={name}
- onChange={(event) => onNameChange(event.target.value)}
- />
+ <div className="mt-1.5 space-y-1">
+ <Input
+ id="gateway-credential-name"
+ placeholder="Production"
+ value={name}
+ aria-invalid={Boolean(nameError)}
+ aria-describedby={
+ nameError ? "gateway-credential-name-error" : undefined
+ }
+ error={Boolean(nameError)}
+ onChange={(event) => onNameChange(event.target.value)}
+ />
+ {nameError ? (
+ <p
+ id="gateway-credential-name-error"
+ role="alert"
+ className="text-destructive text-sm"
+ >
+ {nameError}
+ </p>
+ ) : null}
+ </div>
</div>
<div className="ph-no-capture">
<Label htmlFor="gateway-secret-key">Secret key</Label>
- <Input
- id="gateway-secret-key"
- className="mt-1.5"
- type="password"
- autoComplete="new-password"
- placeholder={credentialPlaceholder}
- value={credential}
- onChange={(event) => onCredentialChange(event.target.value)}
- />
+ <div className="mt-1.5">
+ <Input
+ id="gateway-secret-key"
+ type="password"
+ autoComplete="new-password"
+ placeholder={credentialPlaceholder}
+ value={credential}
+ onChange={(event) => onCredentialChange(event.target.value)}
+ />
+ </div>
</div>
</>
);
diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/ConnectedGatewayProvidersTable.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/ConnectedGatewayProvidersTable.tsx
deleted file mode 100644
index 6dfc1a5b4c..0000000000
--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/ConnectedGatewayProvidersTable.tsx
+++ /dev/null
@@ -1,59 +0,0 @@
-import { useState } from "react";
-
-import { reorderProviderIds } from "@/src/features/ai-gateway/fns/providerReorder/reorderProviderIds";
-import {
- GatewayProvidersTable,
- type GatewayProvidersTableProps,
-} from "./GatewayProvidersTable";
-
-type Props = Omit<GatewayProvidersTableProps, "onMove"> & {
- onReorder: (sourceId: string, targetId: string) => Promise<boolean>;
-};
-
-export function ConnectedGatewayProvidersTable(props: Props) {
- const { onReorder, ...tableProps } = props;
- const serverOrderKey = JSON.stringify(
- props.connections.map((connection) => connection.id),
- );
- const [optimisticOrder, setOptimisticOrder] = useState<{
- base: string;
- ids: string[];
- } | null>(null);
- const serverIds = props.connections.map((connection) => connection.id);
- const sameMembers =
- optimisticOrder &&
- JSON.stringify([...optimisticOrder.ids].sort()) ===
- JSON.stringify([...serverIds].sort());
- const orderedIds =
- optimisticOrder &&
- sameMembers &&
- (serverOrderKey === optimisticOrder.base ||
- serverOrderKey === JSON.stringify(optimisticOrder.ids))
- ? optimisticOrder.ids
- : serverIds;
- const connections = props.connections.toSorted(
- (left, right) => orderedIds.indexOf(left.id) - orderedIds.indexOf(right.id),
- );
-
- const onMove = async (sourceId: string, targetId: string) => {
- const previousIds = orderedIds;
- const nextIds = reorderProviderIds(previousIds, sourceId, targetId);
- if (nextIds === previousIds) {
- if (sourceId !== targetId) await onReorder(sourceId, targetId);
- return;
- }
-
- setOptimisticOrder({ base: serverOrderKey, ids: nextIds });
- if (!(await onReorder(sourceId, targetId))) {
- setOptimisticOrder(null);
- }
- };
-
- return (
- <GatewayProvidersTable
- {...tableProps}
- connections={connections}
- onMove={onMove}
- />
- );
-}
diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.clienttest.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.clienttest.tsx
deleted file mode 100644
index 415404767a..0000000000
--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.clienttest.tsx
+++ /dev/null
@@ -1,148 +0,0 @@
-/* eslint-disable @repo/prefer-stories-over-client-tests */
-import { fireEvent, render, screen, waitFor } from "@testing-library/react";
-import { describe, expect, it, vi } from "vitest";
-
-import type { GatewayConnection } from "@/src/features/ai-gateway/types/gatewayProvider";
-import { ConnectedGatewayProvidersTable } from "./ConnectedGatewayProvidersTable";
-
-const connections = ["Alpha", "Beta"].map((name, index) => ({
- id: name.toLowerCase(),
- name,
- provider: "OPENAI" as const,
- displaySecret: "sk-...test",
- status: "ENABLED" as const,
- organizationId: "org-1",
- createdById: "user-1",
- routingPriority: index,
- createdAt: new Date("2026-09-08T12:00:00.000Z"),
- updatedAt: new Date("2026-09-08T12:00:00.000Z"),
-})) satisfies GatewayConnection[];
-
-describe("gateway providers table", () => {
- it("renders priority first and exposes a table actions menu", () => {
- render(
- <ConnectedGatewayProvidersTable
- connections={connections}
- pageOffset={0}
- modelCounts={{}}
- getModelsUrl={(connection) => `/models/${connection.id}`}
- onRetry={() => {}}
- onEdit={() => {}}
- onDelete={() => {}}
- canReorder
- onReorder={async () => true}
- />,
- );
-
- expect(screen.getAllByRole("columnheader")[0]).toHaveTextContent(
- "Priority",
- );
- expect(
- screen.getAllByRole("button", { name: "Open actions menu" }),
- ).toHaveLength(2);
- });
-
- it("keeps a successful move through a stale refresh and shows absolute priorities", async () => {
- const onReorder = vi.fn(async () => true);
- const props = {
- connections,
- pageOffset: 50,
- modelCounts: {},
- getModelsUrl: (connection: GatewayConnection) =>
- `/models/${connection.id}`,
- onRetry: () => {},
- onEdit: () => {},
- onDelete: () => {},
- canReorder: true,
- onReorder,
- };
- const { rerender } = render(<ConnectedGatewayProvidersTable {...props} />);
-
- expect(screen.getAllByRole("row")[1]).toHaveTextContent("51");
- fireEvent.click(
- screen.getAllByRole("button", { name: "Move credential down" })[0]!,
- );
- await waitFor(() => expect(onReorder).toHaveBeenCalledOnce());
- rerender(
- <ConnectedGatewayProvidersTable
- {...props}
- connections={[...connections]}
- />,
- );
- expect(screen.getAllByRole("row")[1]).toHaveTextContent("Beta");
- expect(screen.getAllByRole("row")[1]).toHaveTextContent("51");
-
- rerender(
- <ConnectedGatewayProvidersTable
- {...props}
- connections={[...connections].reverse()}
- />,
- );
- expect(screen.getAllByRole("row")[1]).toHaveTextContent("Beta");
- });
-
- it("allows a boundary credential to move to the adjacent page", async () => {
- const onReorder = vi.fn(async () => true);
- render(
- <ConnectedGatewayProvidersTable
- connections={connections}
- pageOffset={50}
- previousConnectionId="previous-page"
- nextConnectionId="next-page"
- modelCounts={{}}
- getModelsUrl={(connection) => `/models/${connection.id}`}
- onRetry={() => {}}
- onEdit={() => {}}
- onDelete={() => {}}
- canReorder
- onReorder={onReorder}
- />,
- );
-
- fireEvent.click(
- screen.getAllByRole("button", { name: "Move credential up" })[0]!,
- );
- await waitFor(() =>
- expect(onReorder).toHaveBeenCalledWith("alpha", "previous-page"),
- );
- fireEvent.click(
- screen.getAllByRole("button", { name: "Move credential down" })[1]!,
- );
- await waitFor(() =>
- expect(onReorder).toHaveBeenCalledWith("beta", "next-page"),
- );
- });
-
- it("adopts a new server order that differs from an optimistic move", async () => {
- const gamma = { ...connections[0]!, id: "gamma", name: "Gamma" };
- const onReorder = vi.fn(async () => true);
- const props = {
- pageOffset: 0,
- modelCounts: {},
- getModelsUrl: (connection: GatewayConnection) =>
- `/models/${connection.id}`,
- onRetry: () => {},
- onEdit: () => {},
- onDelete: () => {},
- canReorder: true,
- onReorder,
- };
- const { rerender } = render(
- <ConnectedGatewayProvidersTable
- {...props}
- connections={[...connections, gamma]}
- />,
- );
- fireEvent.click(
- screen.getAllByRole("button", { name: "Move credential down" })[0]!,
- );
- await waitFor(() => expect(onReorder).toHaveBeenCalledOnce());
- rerender(
- <ConnectedGatewayProvidersTable
- {...props}
- connections={[gamma, ...connections]}
- />,
- );
- expect(screen.getAllByRole("row")[1]).toHaveTextContent("Gamma");
- });
-});
diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.stories.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.stories.tsx
index ba38edbecb..392cdf8e18 100644
--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.stories.tsx
+++ b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.stories.tsx
@@ -1,15 +1,12 @@
-import { useState, type ComponentProps } from "react";
-import { Button } from "@/src/components/ui/button";
-import { expect, fn, userEvent } from "storybook/test";
+import type { ComponentProps } from "react";
+import { fn } from "storybook/test";
import preview from "@/.storybook/preview";
import { GatewayProvidersTable } from "./GatewayProvidersTable";
-import { ConnectedGatewayProvidersTable } from "./ConnectedGatewayProvidersTable";
const meta = preview.meta({ component: GatewayProvidersTable });
const onCredentialAction = fn();
-const onReorder = fn(async () => true);
const connections = [
{
@@ -63,18 +60,9 @@ const actions = {
onRetry: (connection) => onCredentialAction(connection.id),
onEdit: (connection) => onCredentialAction(connection.id),
onDelete: (connection) => onCredentialAction(connection.id),
- canReorder: true,
- pageOffset: 0,
- onMove: fn(),
} satisfies Pick<
ComponentProps<typeof GatewayProvidersTable>,
- | "getModelsUrl"
- | "onRetry"
- | "onEdit"
- | "onDelete"
- | "canReorder"
- | "onMove"
- | "pageOffset"
+ "getModelsUrl" | "onRetry" | "onEdit" | "onDelete"
>;
export const OrderedCredentials = meta.story({
@@ -103,79 +91,3 @@ export const Empty = meta.story({
...actions,
},
});
-
-export const MovesImmediately = meta.story({
- name: "(Test) Moves Immediately",
- args: {
- connections,
- modelCounts: {},
- ...actions,
- },
- render: (args) => (
- <ConnectedGatewayProvidersTable {...args} onReorder={onReorder} />
- ),
- play: async ({ canvas }) => {
- onReorder.mockClear();
- await userEvent.click(
- canvas.getAllByRole("button", { name: "Move credential down" })[0]!,
- );
-
- const rows = canvas.getAllByRole("row").slice(1);
- await expect(rows[0]).toHaveTextContent("Fallback");
- await expect(rows[1]).toHaveTextContent("Primary");
- await expect(onReorder).toHaveBeenCalledWith(
- "connection-openai",
- "connection-anthropic",
- );
- },
-});
-
-export const ResetsWhenServerOrderChanges = meta.story({
- name: "(Test) Resets When Server Order Changes",
- args: {
- connections: [
- ...connections,
- {
- ...connections[0]!,
- id: "connection-third",
- name: "Third",
- routingPriority: 2,
- },
- ],
- modelCounts: {},
- ...actions,
- },
- render: (args) => <ServerOrderExample {...args} />,
- play: async ({ canvas }) => {
- await userEvent.click(
- canvas.getAllByRole("button", { name: "Move credential down" })[0]!,
- );
- await expect(canvas.getAllByRole("row")[1]).toHaveTextContent("Fallback");
- await userEvent.click(
- canvas.getByRole("button", { name: "Update server order" }),
- );
- await expect(canvas.getAllByRole("row")[1]).toHaveTextContent("Third");
- await expect(canvas.getAllByRole("row")[2]).toHaveTextContent("Fallback");
- await expect(canvas.getAllByRole("row")[3]).toHaveTextContent("Primary");
- },
-});
-
-function ServerOrderExample(
- args: ComponentProps<typeof GatewayProvidersTable>,
-) {
- const [serverConnections, setServerConnections] = useState(args.connections);
- return (
- <div>
- <Button
- onClick={() => setServerConnections([...args.connections].reverse())}
- >
- Update server order
- </Button>
- <ConnectedGatewayProvidersTable
- {...args}
- connections={serverConnections}
- onReorder={async () => true}
- />
- </div>
- );
-}
diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.tsx
index 453cf542e9..0b1ed89960 100644
--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.tsx
+++ b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.tsx
@@ -1,13 +1,6 @@
/* eslint-disable no-nested-ternary */
import { useMemo } from "react";
-import {
- ArrowDown,
- ArrowUp,
- Pencil,
- RefreshCw,
- Route,
- Trash2,
-} from "lucide-react";
+import { Pencil, RefreshCw, Route, Trash2 } from "lucide-react";
import Link from "next/link";
import { SiAnthropic, SiOpenai } from "react-icons/si";
@@ -15,7 +8,6 @@ import { SettingsTable } from "@/src/components/SettingsTable/SettingsTable";
import { createStatusTableColumn } from "@/src/components/design-system/table/columns/createStatusTableColumn";
import { createTextTableColumn } from "@/src/components/design-system/table/columns/createTextTableColumn";
import type { LangfuseColumnDef } from "@/src/components/table/types";
-import { Button } from "@/src/components/ui/button";
import { providerLabels } from "@/src/features/ai-gateway/constants/providerLabels";
import type {
GatewayConnection,
@@ -24,9 +16,6 @@ import type {
export type GatewayProvidersTableProps = {
connections: GatewayConnection[];
- pageOffset: number;
- previousConnectionId?: string;
- nextConnectionId?: string;
modelCounts: Record<string, number | "loading">;
getModelsUrl: (connection: GatewayConnection) => string;
onRetry: (connection: GatewayConnection) => void;
@@ -35,8 +24,6 @@ export type GatewayProvidersTableProps = {
retryDisabled?: { reason: string };
editDisabled?: { reason: string };
deleteDisabled?: { reason: string };
- canReorder: boolean;
- onMove: (sourceId: string, targetId: string) => void;
};
const connectionStatus = {
@@ -47,9 +34,6 @@ const connectionStatus = {
export function GatewayProvidersTable({
connections,
- pageOffset,
- previousConnectionId,
- nextConnectionId,
modelCounts,
getModelsUrl,
onRetry,
@@ -58,67 +42,20 @@ export function GatewayProvidersTable({
retryDisabled,
editDisabled,
deleteDisabled,
- canReorder,
- onMove,
}: GatewayProvidersTableProps) {
const columns = useMemo<LangfuseColumnDef<GatewayConnection>[]>(
() => [
- {
- accessorKey: "routingPriority",
- header: "Priority",
- size: 110,
- cell: ({ row }) => (
- <div className="group flex items-center gap-1">
- <span className="flex size-6 shrink-0 items-center justify-center font-mono">
- {pageOffset + row.index + 1}
- </span>
- <div className="flex items-center opacity-40 transition-opacity group-hover:opacity-100 focus-within:opacity-100">
- <Button
- size="icon-xs"
- variant="ghost"
- disabled={
- !canReorder || (row.index === 0 && !previousConnectionId)
- }
- aria-label="Move credential up"
- onClick={() => {
- const targetId =
- connections[row.index - 1]?.id ?? previousConnectionId;
- if (targetId) onMove(row.original.id, targetId);
- }}
- >
- <ArrowUp className="icon-sm text-icon-foreground" />
- </Button>
- <Button
- size="icon-xs"
- variant="ghost"
- disabled={
- !canReorder ||
- (row.index === connections.length - 1 && !nextConnectionId)
- }
- aria-label="Move credential down"
- onClick={() => {
- const targetId =
- connections[row.index + 1]?.id ?? nextConnectionId;
- if (targetId) onMove(row.original.id, targetId);
- }}
- >
- <ArrowDown className="icon-sm text-icon-foreground" />
- </Button>
- </div>
- </div>
- ),
- },
+ createTextTableColumn<GatewayConnection>({
+ accessorKey: "name",
+ header: "Name",
+ size: 180,
+ }),
{
accessorKey: "provider",
header: "Provider",
size: 170,
cell: ({ row }) => <ProviderName provider={row.original.provider} />,
},
- createTextTableColumn<GatewayConnection>({
- accessorKey: "name",
- header: "Name",
- size: 180,
- }),
createTextTableColumn<GatewayConnection>({
accessorKey: "displaySecret",
header: "Credential",
@@ -145,16 +82,7 @@ export function GatewayProvidersTable({
),
},
],
- [
- canReorder,
- connections,
- pageOffset,
- previousConnectionId,
- nextConnectionId,
- getModelsUrl,
- modelCounts,
- onMove,
- ],
+ [getModelsUrl, modelCounts],
);
return (
diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderDialogController/ProviderDialogController.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderDialogController/ProviderDialogController.tsx
index 46522ceefa..b25946e57e 100644
--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderDialogController/ProviderDialogController.tsx
+++ b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderDialogController/ProviderDialogController.tsx
@@ -1,5 +1,6 @@
/* eslint-disable no-nested-ternary */
import { useState, type ReactNode } from "react";
+import { TriangleAlert } from "lucide-react";
import { Alert } from "@/src/components/design-system/Alert/Alert";
import { Button } from "@/src/components/ui/button";
@@ -14,6 +15,7 @@ import { Input } from "@/src/components/ui/input";
import { Label } from "@/src/components/ui/label";
import { CredentialFields } from "@/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields";
import { ProviderSelect } from "@/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderSelect";
+import { DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR } from "@/src/features/ai-gateway/constants/gatewayConnection";
import { providerLabels } from "@/src/features/ai-gateway/constants/providerLabels";
import type {
GatewayConnection,
@@ -25,6 +27,9 @@ import {
reportTrpcErrorWithoutToast,
} from "@/src/utils/api";
+const INVALID_CONNECTION_NAME_ERROR =
+ "Connection names cannot contain a colon.";
+
export function ProviderDialogController({
organizationId,
connection,
@@ -41,6 +46,7 @@ export function ProviderDialogController({
connection?.provider ?? "OPENAI",
);
const [name, setName] = useState(connection?.name ?? "");
+ const [nameError, setNameError] = useState<string | null>(null);
const [credential, setCredential] = useState("");
const utils = api.useUtils();
const test = api.aiGateway.testConnection.useMutation({
@@ -57,14 +63,28 @@ export function ProviderDialogController({
});
const isSaving = create.isPending || update.isPending;
const isPending = test.isPending || isSaving;
- const isError = test.isError || create.isError || update.isError;
+ const saveError = create.error ?? update.error;
+ const isNameMutationError =
+ saveError?.message === DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR;
+ const isError =
+ test.isError ||
+ ((create.isError || update.isError) && !isNameMutationError);
+ const errorTitle = test.isError
+ ? "Provider validation failed"
+ : "Provider credential could not be saved";
const reset = () => {
setProvider("OPENAI");
setName("");
+ setNameError(null);
setCredential("");
};
+ const updateName = (value: string) => {
+ setName(value);
+ setNameError(null);
+ };
+
const testCredential = async (
selectedConnection: GatewayConnection | null,
) => {
@@ -83,6 +103,12 @@ export function ProviderDialogController({
selectedConnection: GatewayConnection | null,
closeDialog: () => void,
) => {
+ if (name.includes(":")) {
+ setNameError(INVALID_CONNECTION_NAME_ERROR);
+ return;
+ }
+ setNameError(null);
+
try {
if (selectedConnection) {
await update.mutateAsync({
@@ -107,6 +133,13 @@ export function ProviderDialogController({
closeDialog();
reset();
} catch (error) {
+ if (
+ error instanceof Error &&
+ error.message === DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR
+ ) {
+ setNameError(DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR);
+ return;
+ }
reportNonTrpcError(error, "ai-gateway-providers");
}
};
@@ -141,18 +174,28 @@ export function ProviderDialogController({
)}
<CredentialFields
name={name}
+ nameError={nameError}
credential={credential}
credentialPlaceholder={
selectedConnection
? "Leave blank to keep current key"
: "Enter secret key"
}
- onNameChange={setName}
+ onNameChange={updateName}
onCredentialChange={setCredential}
/>
+ {selectedConnection && name.trim() !== selectedConnection.name ? (
+ <Alert variant="warning" icon={TriangleAlert}>
+ <Alert.Title>Renaming changes model slugs</Alert.Title>
+ <Alert.Description>
+ Requests using model slugs with the old connection name will
+ stop resolving.
+ </Alert.Description>
+ </Alert>
+ ) : null}
{isError ? (
<Alert variant="destructive">
- <Alert.Title>Provider validation failed</Alert.Title>
+ <Alert.Title>{errorTitle}</Alert.Title>
<Alert.Description>
{test.error?.message ??
create.error?.message ??
@@ -196,12 +239,14 @@ export function ProviderDialogController({
openAddDialog: () => {
setProvider(connection?.provider ?? "OPENAI");
setName(connection?.name ?? "");
+ setNameError(null);
setCredential("");
openDialog(connection ?? null);
},
openEditDialog: (selectedConnection) => {
setProvider(selectedConnection.provider);
setName(selectedConnection.name);
+ setNameError(null);
setCredential("");
openDialog(selectedConnection);
},
diff --git a/web/src/features/ai-gateway/components/gatewayMutationErrors.clienttest.tsx b/web/src/features/ai-gateway/components/gatewayMutationErrors.clienttest.tsx
deleted file mode 100644
index 684c64dbfb..0000000000
--- a/web/src/features/ai-gateway/components/gatewayMutationErrors.clienttest.tsx
+++ /dev/null
@@ -1,379 +0,0 @@
-import { TRPCClientError } from "@trpc/client";
-import { fireEvent, render, screen, waitFor } from "@testing-library/react";
-
-import { Button } from "@/src/components/ui/button";
-import { TooltipProvider } from "@/src/components/ui/tooltip";
-import type { GatewayConnection } from "@/src/features/ai-gateway/types/gatewayProvider";
-
-const {
- reportTrpcErrorWithoutToastMock,
- reportNonTrpcErrorMock,
- showErrorToastMock,
- testConnectionMutateAsync,
- createApiKeyMutateAsync,
- mutationStates,
-} = vi.hoisted(() => {
- const mutationStates = {
- testConnection: {
- isError: false,
- isPending: false,
- error: null as Error | null,
- },
- createConnection: {
- isError: false,
- isPending: false,
- error: null as Error | null,
- },
- updateConnection: {
- isError: false,
- isPending: false,
- error: null as Error | null,
- },
- createApiKey: {
- isError: false,
- isPending: false,
- error: null as Error | null,
- },
- };
-
- return {
- reportTrpcErrorWithoutToastMock: vi.fn(),
- reportNonTrpcErrorMock: vi.fn(),
- showErrorToastMock: vi.fn(),
- testConnectionMutateAsync: vi.fn(),
- createApiKeyMutateAsync: vi.fn(),
- mutationStates,
- };
-});
-
-const providerValidationError = () =>
- new TRPCClientError("Provider credential validation failed", {
- result: {
- error: {
- message: "Provider credential validation failed",
- code: -32600,
- data: {
- code: "BAD_REQUEST",
- httpStatus: 400,
- path: "aiGateway.createConnection",
- },
- },
- },
- });
-
-const gatewayApiKeyError = () =>
- new TRPCClientError("Gateway key limit reached", {
- result: {
- error: {
- message: "Gateway key limit reached",
- code: -32600,
- data: {
- code: "BAD_REQUEST",
- httpStatus: 400,
- path: "aiGateway.createApiKey",
- },
- },
- },
- });
-
-vi.mock("@/src/utils/api", () => {
- const React = require("react");
-
- const createStatefulMutation = <TArgs extends unknown[]>(
- mutateImpl: (...args: TArgs) => Promise<unknown>,
- stateKey: keyof typeof mutationStates,
- ) => {
- return (options?: { onError?: (error: unknown) => void }) => {
- const [mutationState, setMutationState] = React.useState({
- isError: false,
- isPending: false,
- error: null as Error | null,
- });
-
- return {
- ...mutationState,
- mutateAsync: async (...args: TArgs) => {
- setMutationState(
- (current: {
- isError: boolean;
- isPending: boolean;
- error: Error | null;
- }) => ({ ...current, isPending: true }),
- );
- try {
- return await mutateImpl(...args);
- } catch (error) {
- const normalizedError =
- error instanceof Error ? error : new Error(String(error));
- mutationStates[stateKey] = {
- isError: true,
- isPending: false,
- error: normalizedError,
- };
- options?.onError?.(error);
- setMutationState({
- isError: true,
- isPending: false,
- error: normalizedError,
- });
- throw error;
- }
- },
- };
- };
- };
-
- return {
- api: {
- useUtils: () => ({
- aiGateway: {
- listConnections: { invalidate: vi.fn() },
- listApiKeys: { invalidate: vi.fn() },
- },
- }),
- aiGateway: {
- testConnection: {
- useMutation: createStatefulMutation(
- async (input: {
- orgId: string;
- provider: string;
- credential: string;
- }) => testConnectionMutateAsync(input),
- "testConnection",
- ),
- },
- createConnection: {
- useMutation: createStatefulMutation(async () => {
- throw providerValidationError();
- }, "createConnection"),
- },
- updateConnection: {
- useMutation: createStatefulMutation(async () => {
- throw providerValidationError();
- }, "updateConnection"),
- },
- createApiKey: {
- useMutation: createStatefulMutation(
- async (input: {
- orgId: string;
- note?: string;
- metadata: Record<string, string>;
- }) => {
- createApiKeyMutateAsync(input);
- throw gatewayApiKeyError();
- },
- "createApiKey",
- ),
- },
- },
- },
- reportTrpcErrorWithoutToast: reportTrpcErrorWithoutToastMock,
- reportNonTrpcError: reportNonTrpcErrorMock,
- };
-});
-
-vi.mock("@/src/features/notifications/showErrorToast", () => ({
- showErrorToast: showErrorToastMock,
-}));
-
-vi.mock("next/router", () => ({
- useRouter: () => ({ query: {}, push: vi.fn() }),
-}));
-
-import { CreateGatewayApiKeyDialogController } from "@/src/features/ai-gateway/components/GatewayApiKeysPage/components/CreateGatewayApiKeyDialogController/CreateGatewayApiKeyDialogController";
-import { ProviderDialogController } from "@/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderDialogController/ProviderDialogController";
-
-const testConnection: GatewayConnection = {
- id: "conn-1",
- organizationId: "org-1",
- name: "Production",
- provider: "OPENAI",
- status: "ENABLED",
- displaySecret: "sk-...abcd",
- createdById: "user-1",
- routingPriority: 0,
- createdAt: new Date("2026-09-04T12:00:00.000Z"),
- updatedAt: new Date("2026-09-04T12:00:00.000Z"),
-};
-
-describe("gateway mutation local error handling", () => {
- beforeAll(() => {
- vi.stubGlobal(
- "ResizeObserver",
- class {
- observe() {}
- unobserve() {}
- disconnect() {}
- },
- );
- Element.prototype.scrollIntoView = vi.fn();
- Element.prototype.hasPointerCapture = vi.fn();
- });
-
- beforeEach(() => {
- reportTrpcErrorWithoutToastMock.mockClear();
- reportNonTrpcErrorMock.mockClear();
- showErrorToastMock.mockClear();
- testConnectionMutateAsync.mockClear();
- createApiKeyMutateAsync.mockClear();
- mutationStates.testConnection = {
- isError: false,
- isPending: false,
- error: null,
- };
- mutationStates.createConnection = {
- isError: false,
- isPending: false,
- error: null,
- };
- mutationStates.updateConnection = {
- isError: false,
- isPending: false,
- error: null,
- };
- mutationStates.createApiKey = {
- isError: false,
- isPending: false,
- error: null,
- };
- });
-
- it("tests a provider credential without saving", async () => {
- render(
- <ProviderDialogController organizationId="org-1">
- {({ openAddDialog }) => (
- <Button onClick={openAddDialog}>Add provider</Button>
- )}
- </ProviderDialogController>,
- );
-
- fireEvent.click(screen.getByRole("button", { name: "Add provider" }));
- fireEvent.change(screen.getByLabelText("Secret key"), {
- target: { value: "sk-test-only" },
- });
- fireEvent.click(screen.getByRole("button", { name: "Test" }));
-
- await waitFor(() => {
- expect(testConnectionMutateAsync).toHaveBeenCalledWith({
- orgId: "org-1",
- provider: "OPENAI",
- credential: "sk-test-only",
- });
- });
- expect(
- screen.getByRole("heading", { name: "Add provider credential" }),
- ).toBeInTheDocument();
- });
-
- it("create provider dialog routes tRPC failures locally without a global toast", async () => {
- render(
- <ProviderDialogController organizationId="org-1">
- {({ openAddDialog }) => (
- <Button onClick={openAddDialog}>Add provider</Button>
- )}
- </ProviderDialogController>,
- );
-
- fireEvent.click(screen.getByRole("button", { name: "Add provider" }));
- fireEvent.change(screen.getByLabelText("Name"), {
- target: { value: "Production" },
- });
- fireEvent.change(screen.getByLabelText("Secret key"), {
- target: { value: "sk-invalid" },
- });
- fireEvent.click(screen.getByRole("button", { name: "Test and save" }));
-
- await waitFor(() => {
- expect(reportTrpcErrorWithoutToastMock).toHaveBeenCalledWith(
- expect.any(TRPCClientError),
- "ai-gateway-providers",
- );
- });
- expect(reportNonTrpcErrorMock).toHaveBeenCalledWith(
- expect.any(TRPCClientError),
- "ai-gateway-providers",
- );
- expect(showErrorToastMock).not.toHaveBeenCalled();
- expect(
- await screen.findByText("Provider credential validation failed"),
- ).toBeInTheDocument();
- });
-
- it("opens the selected credential through the dialog controller", () => {
- render(
- <ProviderDialogController organizationId="org-1">
- {({ openEditDialog }) => (
- <Button onClick={() => openEditDialog(testConnection)}>
- Edit provider
- </Button>
- )}
- </ProviderDialogController>,
- );
-
- fireEvent.click(screen.getByRole("button", { name: "Edit provider" }));
- expect(
- screen.getByRole("heading", { name: "Edit provider credential" }),
- ).toBeInTheDocument();
- expect(screen.getByLabelText("Name")).toHaveValue(testConnection.name);
- });
-
- it("edit provider dialog routes tRPC failures locally without a global toast", async () => {
- render(
- <ProviderDialogController
- organizationId="org-1"
- connection={testConnection}
- >
- {({ openAddDialog }) => (
- <Button onClick={openAddDialog}>Edit provider</Button>
- )}
- </ProviderDialogController>,
- );
-
- fireEvent.click(screen.getByRole("button", { name: "Edit provider" }));
- fireEvent.change(screen.getByLabelText("Secret key"), {
- target: { value: "sk-invalid" },
- });
- fireEvent.click(screen.getByRole("button", { name: "Test and save" }));
-
- await waitFor(() => {
- expect(reportTrpcErrorWithoutToastMock).toHaveBeenCalledWith(
- expect.any(TRPCClientError),
- "ai-gateway-providers",
- );
- });
- expect(showErrorToastMock).not.toHaveBeenCalled();
- expect(
- await screen.findByText("Provider credential validation failed"),
- ).toBeInTheDocument();
- });
-
- it("create gateway api key dialog routes tRPC failures locally and ignores blank metadata rows", async () => {
- render(
- <TooltipProvider>
- <CreateGatewayApiKeyDialogController organizationId="org-1">
- {({ openDialog }) => <Button onClick={openDialog}>Issue key</Button>}
- </CreateGatewayApiKeyDialogController>
- </TooltipProvider>,
- );
-
- fireEvent.click(screen.getByRole("button", { name: "Issue key" }));
- fireEvent.click(screen.getByRole("button", { name: "Create key" }));
-
- await waitFor(() => {
- expect(reportTrpcErrorWithoutToastMock).toHaveBeenCalledWith(
- expect.any(TRPCClientError),
- "ai-gateway-api-keys",
- );
- });
- expect(showErrorToastMock).not.toHaveBeenCalled();
- expect(
- await screen.findByText("Gateway key limit reached"),
- ).toBeInTheDocument();
- expect(createApiKeyMutateAsync).toHaveBeenCalledWith({
- orgId: "org-1",
- name: undefined,
- metadata: {},
- expiresAt: null,
- });
- });
-});
diff --git a/web/src/features/ai-gateway/constants/gatewayConnection.ts b/web/src/features/ai-gateway/constants/gatewayConnection.ts
new file mode 100644
index 0000000000..5757e9b255
--- /dev/null
+++ b/web/src/features/ai-gateway/constants/gatewayConnection.ts
@@ -0,0 +1,2 @@
+export const DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR =
+ "A provider credential with this name already exists.";
diff --git a/web/src/features/ai-gateway/fns/gatewayModelSlug.ts b/web/src/features/ai-gateway/fns/gatewayModelSlug.ts
new file mode 100644
index 0000000000..74ac65a482
--- /dev/null
+++ b/web/src/features/ai-gateway/fns/gatewayModelSlug.ts
@@ -0,0 +1,75 @@
+export function formatGatewayModelSlug(
+ connectionName: string,
+ providerModel: string,
+) {
+ return `${connectionName}:${providerModel}`;
+}
+
+export function resolveGatewayModelRoute<
+ Connection extends GatewayRouteConnection,
+>(
+ model: string,
+ connections: readonly Connection[],
+): { connection: Connection | null; providerModel: string } | null {
+ const modelSlug = parseGatewayModelSlug(model);
+ const explicitConnection = modelSlug
+ ? (connections.find(
+ (connection) => connection.name === modelSlug.connectionName,
+ ) ?? null)
+ : null;
+ if (modelSlug && explicitConnection) {
+ return {
+ connection: explicitConnection,
+ providerModel: modelSlug.providerModel,
+ };
+ }
+
+ if (model.startsWith("ft:")) {
+ return {
+ connection:
+ connections.find((connection) => connection.provider === "OPENAI") ??
+ null,
+ providerModel: model,
+ };
+ }
+
+ if (modelSlug) {
+ return {
+ connection: null,
+ providerModel: modelSlug.providerModel,
+ };
+ }
+
+ if (model.includes(":")) return null;
+
+ let provider: "OPENAI" | "ANTHROPIC" | null = null;
+ if (model.startsWith("gpt-")) provider = "OPENAI";
+ if (model.startsWith("claude-")) provider = "ANTHROPIC";
+ if (!provider) return null;
+
+ return {
+ connection:
+ connections.find((connection) => connection.provider === provider) ??
+ null,
+ providerModel: model,
+ };
+}
+
+function parseGatewayModelSlug(model: string) {
+ const separatorIndex = model.indexOf(":");
+ if (separatorIndex <= 0) {
+ return null;
+ }
+ const providerModel = model.slice(separatorIndex + 1);
+ if (!providerModel.trim()) return null;
+
+ return {
+ connectionName: model.slice(0, separatorIndex),
+ providerModel,
+ };
+}
+
+type GatewayRouteConnection = {
+ name: string;
+ provider: string;
+};
diff --git a/web/src/features/ai-gateway/fns/providerReorder/reorderProviderIds.ts b/web/src/features/ai-gateway/fns/providerReorder/reorderProviderIds.ts
deleted file mode 100644
index 910d77946a..0000000000
--- a/web/src/features/ai-gateway/fns/providerReorder/reorderProviderIds.ts
+++ /dev/null
@@ -1,17 +0,0 @@
-export function reorderProviderIds(
- ids: string[],
- sourceId: string,
- targetId: string,
-) {
- const sourceIndex = ids.indexOf(sourceId);
- const targetIndex = ids.indexOf(targetId);
- if (sourceIndex < 0 || targetIndex < 0 || sourceIndex === targetIndex) {
- return ids;
- }
-
- const nextIds = [...ids];
- const [movedId] = nextIds.splice(sourceIndex, 1);
- if (!movedId) return ids;
- nextIds.splice(targetIndex, 0, movedId);
- return nextIds;
-}
diff --git a/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts b/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts
index 48cbd0bfc7..a9d61bf8f7 100644
--- a/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts
+++ b/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts
@@ -13,6 +13,7 @@ import {
gatewayProviders,
providerSupportsApiFormat,
} from "@/src/features/ai-gateway/server/provider/registry";
+import { resolveGatewayModelRoute } from "@/src/features/ai-gateway/fns/gatewayModelSlug";
import {
type CachedResolveContext,
GATEWAY_RESOLVE_KEY_NON_EXISTENT,
@@ -20,7 +21,13 @@ import {
} from "@/src/features/ai-gateway/server/resolve/gatewayResolveCache";
import { GatewayResolveRepository } from "@/src/features/ai-gateway/server/resolve/gatewayResolveRepository";
-export type GatewayApiKeyAuthContext = CachedResolveContext;
+export type GatewayApiKeyAuthContext = Omit<
+ CachedResolveContext,
+ "connections"
+> & {
+ connection: CachedResolveContext["connections"][number] | null;
+ providerModel: string | null;
+};
type GatewayApiKeyAuthenticatorConfig = {
resolveCache?: GatewayResolveCache;
@@ -41,13 +48,47 @@ export class GatewayApiKeyAuthenticator {
async authenticateRequest(params: {
fastHashedSecretKey: string;
apiFormat: GatewayApiFormat;
+ model: string;
+ }): Promise<GatewayApiKeyAuthContext> {
+ const context = await this.authenticateKey(params);
+ const route = resolveGatewayModelRoute(params.model, context.connections);
+ if (!route) {
+ throw new GatewayControlPlaneError("Invalid gateway model", 400);
+ }
+
+ return {
+ ...context,
+ providerModel: route.providerModel,
+ connection: route.connection,
+ };
+ }
+
+ async authenticateModelsRequest(params: {
+ fastHashedSecretKey: string;
+ apiFormat: GatewayApiFormat;
}): Promise<GatewayApiKeyAuthContext> {
- const cached = await this.cache.get(params);
+ const context = await this.authenticateKey(params);
+ return {
+ ...context,
+ providerModel: null,
+ connection: null,
+ };
+ }
+
+ private async authenticateKey(params: {
+ fastHashedSecretKey: string;
+ apiFormat: GatewayApiFormat;
+ }): Promise<CachedResolveContext> {
+ const cacheParams = {
+ fastHashedSecretKey: params.fastHashedSecretKey,
+ apiFormat: params.apiFormat,
+ };
+ const cached = await this.cache.get(cacheParams);
if (cached === GATEWAY_RESOLVE_KEY_NON_EXISTENT) {
throw new GatewayControlPlaneError("Invalid gateway key", 401);
}
- const context = cached ?? (await this.loadAndCacheContext(params));
+ const context = cached ?? (await this.loadAndCacheContext(cacheParams));
// Re-check on cache hits so removing an organization from the allowlist
// takes effect without waiting for the cache TTL.
@@ -63,7 +104,7 @@ export class GatewayApiKeyAuthenticator {
private async loadAndCacheContext(params: {
fastHashedSecretKey: string;
apiFormat: GatewayApiFormat;
- }): Promise<GatewayApiKeyAuthContext> {
+ }): Promise<CachedResolveContext> {
const row = await this.repository.resolveContext({
fastHashedSecretKey: params.fastHashedSecretKey,
providers: supportedProviders(params.apiFormat),
@@ -73,7 +114,8 @@ export class GatewayApiKeyAuthenticator {
const organization = row?.apiKey.organization;
if (!row || !organizationId || !organization) {
await this.cache.set({
- ...params,
+ fastHashedSecretKey: params.fastHashedSecretKey,
+ apiFormat: params.apiFormat,
context: GATEWAY_RESOLVE_KEY_NON_EXISTENT,
});
throw new GatewayControlPlaneError("Invalid gateway key", 401);
@@ -85,9 +127,7 @@ export class GatewayApiKeyAuthenticator {
Boolean(project) &&
!project?.deletedAt &&
project?.orgId === organizationId;
- const connection = organization.gatewayAiConnections[0];
-
- const context: GatewayApiKeyAuthContext = {
+ const context: CachedResolveContext = {
organizationId,
apiKeyId: row.apiKeyId,
keyMetadata: toKeyMetadata(row.metadata),
@@ -95,14 +135,7 @@ export class GatewayApiKeyAuthenticator {
? (gatewayConfig?.defaultIngestionProjectId ?? null)
: null,
ingestionMode: gatewayConfig?.ingestionMode ?? null,
- connection: connection
- ? {
- id: connection.id,
- name: connection.name,
- provider: connection.provider,
- encryptedCredential: connection.encryptedCredential,
- }
- : null,
+ connections: organization.gatewayAiConnections,
};
if (env.API_AUTH_MIGRATION === "enforce") {
@@ -110,7 +143,8 @@ export class GatewayApiKeyAuthenticator {
}
await this.cache.set({
- ...params,
+ fastHashedSecretKey: params.fastHashedSecretKey,
+ apiFormat: params.apiFormat,
context,
expiresAt: row.apiKey.expiresAt,
});
diff --git a/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.servertest.ts b/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.servertest.ts
index 69e6b83e6d..4727ab68ff 100644
--- a/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.servertest.ts
+++ b/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.servertest.ts
@@ -108,7 +108,7 @@ describe("withGatewayResolveSignatureVerification", () => {
async (secret) => {
vi.useFakeTimers();
vi.setSystemTime(now);
- const body = '{ "apiFormat": "openai.responses" }\n';
+ const body = '{ "apiFormat": "openai.responses", "model": "gpt-5" }\n';
const req = request({
authorization: " Bearer sk-gateway ",
gatewayAuthorization: gatewayAuthorization({ secret }),
@@ -124,10 +124,34 @@ describe("withGatewayResolveSignatureVerification", () => {
res,
fastHashedSecretKey: createShaHash("sk-gateway", "test-salt"),
apiFormat: "openai.responses",
+ model: "gpt-5",
});
},
);
+ it.each(["production: ", "production:\t"])(
+ "rejects a whitespace-only provider model in %j",
+ async (model) => {
+ const handler = vi.fn();
+ const res = response();
+
+ await withGatewayResolveSignatureVerification(handler)(
+ request({
+ authorization: "Bearer sk-gateway",
+ body: JSON.stringify({
+ apiFormat: "openai.responses",
+ model,
+ }),
+ }),
+ res,
+ );
+
+ expect(res.status).toHaveBeenCalledWith(400);
+ expect(res.json).toHaveBeenCalledWith({ error: "Invalid request" });
+ expect(handler).not.toHaveBeenCalled();
+ },
+ );
+
it("rejects a missing Bearer token on the models endpoint", async () => {
const handler = vi.fn();
const res = response();
@@ -179,7 +203,7 @@ describe("withGatewayResolveSignatureVerification", () => {
secret: "current-service-secret",
key: "sk-other-gateway",
}),
- body: '{"apiFormat":"openai.responses"}',
+ body: '{"apiFormat":"openai.responses","model":"gpt-5"}',
}),
res,
);
@@ -191,7 +215,7 @@ describe("withGatewayResolveSignatureVerification", () => {
it("rejects stale signatures and headers containing a key id", async () => {
vi.useFakeTimers();
vi.setSystemTime(now);
- const body = '{"apiFormat":"openai.responses"}';
+ const body = '{"apiFormat":"openai.responses","model":"gpt-5"}';
const staleHeader = gatewayAuthorization({
secret: "current-service-secret",
timestamp: Math.floor(now.getTime() / 1000) - 301,
diff --git a/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.ts b/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.ts
index 92fbaf9fd6..6637e47ffb 100644
--- a/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.ts
+++ b/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.ts
@@ -8,6 +8,7 @@ import { env } from "@/src/env.mjs";
import { verifyHmacSha256 } from "@/src/server/utils/hmac";
import { createShaHash } from "@langfuse/shared/src/server/auth/apiKeys";
+import { resolveGatewayModelRoute } from "@/src/features/ai-gateway/fns/gatewayModelSlug";
import { GatewayApiFormatSchema, type GatewayApiFormat } from "../provider";
import { GatewayControlPlaneError } from "@/src/features/ai-gateway/server/gatewayControlPlaneError";
@@ -18,7 +19,9 @@ const SIGNATURE_MAX_AGE_SECONDS = 5 * 60;
const resolveBodySchema = z
.object({
apiFormat: GatewayApiFormatSchema,
- model: z.string().min(1).optional(),
+ model: z
+ .string()
+ .refine((model) => resolveGatewayModelRoute(model, []) !== null),
})
.strict();
const modelsQuerySchema = z
@@ -38,6 +41,7 @@ type GatewayControlPlaneHandlerParams = {
type GatewayResolveHandler = (
params: GatewayControlPlaneHandlerParams & {
apiFormat: GatewayApiFormat;
+ model: string;
},
) => Promise<unknown>;
@@ -104,7 +108,12 @@ export function withGatewayResolveSignatureVerification(
handler: GatewayResolveHandler,
) {
return withGatewayControlPlaneAuth(
- ({ body, ...params }) => handler({ ...params, apiFormat: body.apiFormat }),
+ ({ body, ...params }) =>
+ handler({
+ ...params,
+ apiFormat: body.apiFormat,
+ model: body.model,
+ }),
resolveBodySchema,
);
}
diff --git a/web/src/features/ai-gateway/server/provider/connection/gatewayProviderService.ts b/web/src/features/ai-gateway/server/provider/connection/gatewayProviderService.ts
index a84e4e8e0c..d16a077fc4 100644
--- a/web/src/features/ai-gateway/server/provider/connection/gatewayProviderService.ts
+++ b/web/src/features/ai-gateway/server/provider/connection/gatewayProviderService.ts
@@ -1,7 +1,8 @@
-import type {
- GatewayConnectionStatus,
- GatewayProvider,
- PrismaClient,
+import {
+ Prisma,
+ type GatewayConnectionStatus,
+ type GatewayProvider,
+ type PrismaClient,
} from "@langfuse/shared/src/db";
import { InvalidRequestError, LangfuseNotFoundError } from "@langfuse/shared";
import { encrypt } from "@langfuse/shared/encryption";
@@ -9,6 +10,7 @@ import { LLMAdapter, testModelCall } from "@langfuse/shared/src/server";
import { getDisplaySecretKey } from "@langfuse/shared/src/server/auth/apiKeys";
import { auditLog } from "@/src/features/audit-logs/server";
+import { DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR } from "@/src/features/ai-gateway/constants/gatewayConnection";
import type { OrgAuthedContext } from "@/src/server/api/trpc";
import {
type GatewayProviderName,
@@ -18,11 +20,6 @@ import { invalidateGatewayResolveCacheForOrganization } from "@/src/features/ai-
import { GatewayProviderRepository } from "./gatewayProviderRepository";
import { GatewayModelCatalogService } from "../models/gatewayModelCatalogService";
-type CredentialValidator = (params: {
- provider: GatewayProviderName;
- credential: string;
-}) => Promise<void>;
-
export class GatewayProviderService {
private readonly repository: GatewayProviderRepository;
private readonly modelCatalogService: GatewayModelCatalogService;
@@ -72,15 +69,20 @@ export class GatewayProviderService {
provider: params.provider,
credential: params.credential,
});
- const connection = await this.repository.createConnection({
- organizationId: params.organizationId,
- name: params.name,
- provider: params.provider,
- encryptedCredential: encrypt(params.credential),
- displaySecret: getDisplaySecretKey(params.credential),
- createdById: params.session.user.id,
- status: "ENABLED",
- });
+ let connection;
+ try {
+ connection = await this.repository.createConnection({
+ organizationId: params.organizationId,
+ name: params.name,
+ provider: params.provider,
+ encryptedCredential: encrypt(params.credential),
+ displaySecret: getDisplaySecretKey(params.credential),
+ createdById: params.session.user.id,
+ status: "ENABLED",
+ });
+ } catch (error) {
+ rethrowConnectionMutationError(error);
+ }
await invalidateGatewayResolveCacheForOrganization(params.organizationId);
await auditLog(
{
@@ -124,18 +126,23 @@ export class GatewayProviderService {
});
}
- const updated = await this.repository.updateConnection({
- organizationId: params.organizationId,
- id: params.id,
- name: params.name,
- status: params.credential ? "ENABLED" : params.status,
- encryptedCredential: params.credential
- ? encrypt(params.credential)
- : undefined,
- displaySecret: params.credential
- ? getDisplaySecretKey(params.credential)
- : undefined,
- });
+ let updated;
+ try {
+ updated = await this.repository.updateConnection({
+ organizationId: params.organizationId,
+ id: params.id,
+ name: params.name,
+ status: params.credential ? "ENABLED" : params.status,
+ encryptedCredential: params.credential
+ ? encrypt(params.credential)
+ : undefined,
+ displaySecret: params.credential
+ ? getDisplaySecretKey(params.credential)
+ : undefined,
+ });
+ } catch (error) {
+ rethrowConnectionMutationError(error);
+ }
await this.modelCatalogService.clearModelCache(
params.organizationId,
params.id,
@@ -222,6 +229,16 @@ export class GatewayProviderService {
}
}
+function rethrowConnectionMutationError(error: unknown): never {
+ if (
+ error instanceof Prisma.PrismaClientKnownRequestError &&
+ error.code === "P2002"
+ ) {
+ throw new InvalidRequestError(DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR);
+ }
+ throw error;
+}
+
async function validateGatewayCredential(params: {
provider: GatewayProviderName;
credential: string;
@@ -257,3 +274,8 @@ async function validateGatewayCredential(params: {
throw new InvalidRequestError("Provider credential validation failed");
}
}
+
+type CredentialValidator = (params: {
+ provider: GatewayProviderName;
+ credential: string;
+}) => Promise<void>;
diff --git a/web/src/features/ai-gateway/server/provider/index.ts b/web/src/features/ai-gateway/server/provider/index.ts
index f3c53b8921..2c012a00cc 100644
--- a/web/src/features/ai-gateway/server/provider/index.ts
+++ b/web/src/features/ai-gateway/server/provider/index.ts
@@ -10,4 +10,5 @@ export {
type GatewayApiFormat,
type GatewayMetadata,
type GatewayProviderId,
+ type GatewayResolveResponse,
} from "./registry";
diff --git a/web/src/features/ai-gateway/server/provider/models/gatewayModelsApiHandler.ts b/web/src/features/ai-gateway/server/provider/models/gatewayModelsApiHandler.ts
index 5a22db936a..5cb9d1553d 100644
--- a/web/src/features/ai-gateway/server/provider/models/gatewayModelsApiHandler.ts
+++ b/web/src/features/ai-gateway/server/provider/models/gatewayModelsApiHandler.ts
@@ -23,7 +23,7 @@ export async function gatewayModelsApiHandler({
try {
const context = await new GatewayApiKeyAuthenticator(
prisma,
- ).authenticateRequest({ fastHashedSecretKey, apiFormat });
+ ).authenticateModelsRequest({ fastHashedSecretKey, apiFormat });
const result = await new GatewayModelsService(prisma).list({
context,
apiFormat,
diff --git a/web/src/features/ai-gateway/server/provider/models/gatewayModelsRepository.ts b/web/src/features/ai-gateway/server/provider/models/gatewayModelsRepository.ts
index a914b7211e..aaa89c0ad4 100644
--- a/web/src/features/ai-gateway/server/provider/models/gatewayModelsRepository.ts
+++ b/web/src/features/ai-gateway/server/provider/models/gatewayModelsRepository.ts
@@ -14,7 +14,7 @@ export class GatewayModelsRepository {
status: "ENABLED",
},
orderBy: [{ routingPriority: "asc" }, { id: "asc" }],
- select: { id: true },
+ select: { id: true, name: true },
});
}
}
diff --git a/web/src/features/ai-gateway/server/provider/models/gatewayModelsService.ts b/web/src/features/ai-gateway/server/provider/models/gatewayModelsService.ts
index 4964157366..dc1855b14d 100644
--- a/web/src/features/ai-gateway/server/provider/models/gatewayModelsService.ts
+++ b/web/src/features/ai-gateway/server/provider/models/gatewayModelsService.ts
@@ -1,8 +1,9 @@
import type { GatewayProvider, PrismaClient } from "@langfuse/shared/src/db";
-import { redis as defaultRedis } from "@langfuse/shared/src/server";
+import { logger, redis as defaultRedis } from "@langfuse/shared/src/server";
import type { Cluster, Redis } from "ioredis";
import type { GatewayApiKeyAuthContext } from "@/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator";
+import { formatGatewayModelSlug } from "@/src/features/ai-gateway/fns/gatewayModelSlug";
import { GatewayControlPlaneError } from "@/src/features/ai-gateway/server/gatewayControlPlaneError";
import {
type GatewayApiFormat,
@@ -43,29 +44,45 @@ export class GatewayModelsService {
) as GatewayProvider[],
});
const results = await Promise.all(
- connections.map((connection) =>
- this.modelCatalogService.getModelCatalog({
+ connections.map(async (connection) => ({
+ connection,
+ catalog: await this.modelCatalogService.getModelCatalog({
organizationId: params.context.organizationId,
connectionId: connection.id,
}),
- ),
+ })),
);
- if (results.some((result) => !result.success)) {
+ const successfulResults = results.filter(
+ (result) => result.catalog.success,
+ );
+ for (const { connection, catalog } of results) {
+ if (!catalog.success) {
+ logger.warn("Failed to discover models for AI Gateway connection", {
+ organizationId: params.context.organizationId,
+ connectionId: connection.id,
+ connectionName: connection.name,
+ error: catalog.error,
+ });
+ }
+ }
+ if (results.length > 0 && successfulResults.length === 0) {
throw new GatewayControlPlaneError(
"Gateway model discovery is temporarily unavailable",
503,
);
}
- const modelsById = new Map<string, GatewayModelCatalogEntry>();
- for (const result of results) {
- if (!result.success) continue;
- for (const model of result.models) {
- if (!modelsById.has(model.id)) modelsById.set(model.id, model);
- }
- }
return {
- data: [...modelsById.values()].sort(compareModels),
+ data: successfulResults
+ .flatMap(({ connection, catalog }) =>
+ catalog.success
+ ? catalog.models.map((model) => ({
+ ...model,
+ id: formatGatewayModelSlug(connection.name, model.id),
+ }))
+ : [],
+ )
+ .sort(compareModels),
};
}
}
diff --git a/web/src/features/ai-gateway/server/provider/registry.ts b/web/src/features/ai-gateway/server/provider/registry.ts
index c9bb3b20df..8cebca0319 100644
--- a/web/src/features/ai-gateway/server/provider/registry.ts
+++ b/web/src/features/ai-gateway/server/provider/registry.ts
@@ -27,6 +27,7 @@ export const GatewayResolveResponseSchema = z
id: z.string(),
name: z.string(),
provider: z.enum(gatewayProviderIds),
+ provider_model: z.string(),
api_format: GatewayApiFormatSchema,
base_url: z.url(),
auth: z.union([
@@ -60,6 +61,9 @@ export const GatewayResolveResponseSchema = z
.strict(),
})
.strict();
+export type GatewayResolveResponse = z.infer<
+ typeof GatewayResolveResponseSchema
+>;
export const GatewayModelsResponseSchema = z
.object({
diff --git a/web/src/features/ai-gateway/server/resolve/gatewayResolveApiHandler.ts b/web/src/features/ai-gateway/server/resolve/gatewayResolveApiHandler.ts
index 7f7b08d7b1..374b6f6528 100644
--- a/web/src/features/ai-gateway/server/resolve/gatewayResolveApiHandler.ts
+++ b/web/src/features/ai-gateway/server/resolve/gatewayResolveApiHandler.ts
@@ -13,16 +13,18 @@ export async function gatewayResolveApiHandler({
res,
fastHashedSecretKey,
apiFormat,
+ model,
}: {
req: NextApiRequest;
res: NextApiResponse;
fastHashedSecretKey: string;
apiFormat: GatewayApiFormat;
+ model: string;
}) {
try {
const context = await new GatewayApiKeyAuthenticator(
prisma,
- ).authenticateRequest({ fastHashedSecretKey, apiFormat });
+ ).authenticateRequest({ fastHashedSecretKey, apiFormat, model });
const result = await new GatewayResolveService().resolve({
context,
apiFormat,
diff --git a/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts b/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts
index f905c1be27..1d18b5dfe7 100644
--- a/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts
+++ b/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts
@@ -29,14 +29,14 @@ export const CachedResolveContext = z.object({
keyMetadata: GatewayMetadataSchema,
ingestionProjectId: z.string().nullable(),
ingestionMode: z.enum(["USAGE", "FULL"]).nullable(),
- connection: z
- .object({
+ connections: z.array(
+ z.object({
id: z.string(),
name: z.string(),
provider: z.enum(["OPENAI", "ANTHROPIC"]),
encryptedCredential: z.string(),
- })
- .nullable(),
+ }),
+ ),
});
export type CachedResolveContext = z.infer<typeof CachedResolveContext>;
@@ -172,11 +172,11 @@ export class GatewayResolveCache {
if (!this.enabled) return;
try {
- await Promise.all(
- gatewayApiFormats.map((apiFormat) =>
+ await Promise.all([
+ ...gatewayApiFormats.map((apiFormat) =>
this.redis!.del(contextCacheKey(fastHashedSecretKey, apiFormat)),
),
- );
+ ]);
} catch (error) {
logger.error("Error invalidating gateway resolve cache", error);
}
diff --git a/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts b/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts
index 22360c2897..31dcdeee8e 100644
--- a/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts
+++ b/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts
@@ -42,7 +42,6 @@ export class GatewayResolveRepository {
status: "ENABLED",
},
orderBy: { routingPriority: "asc" },
- take: 1,
select: {
id: true,
name: true,
diff --git a/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts b/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts
index faf755d079..322b2644fd 100644
--- a/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts
+++ b/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts
@@ -42,7 +42,10 @@ const serviceWith = (findFirst: () => Promise<unknown>) => {
fastHashedSecretKey: string;
apiFormat: GatewayApiFormat;
}) => {
- const context = await contextResolver.authenticateRequest(params);
+ const context = await contextResolver.authenticateRequest({
+ ...params,
+ model: "Production OpenAI:gpt-4o",
+ });
return service.resolve({ context, apiFormat: params.apiFormat });
},
};
@@ -98,6 +101,7 @@ describe("GatewayResolveService", () => {
id: "connection-1",
name: "Production OpenAI",
provider: "openai",
+ provider_model: "gpt-4o",
api_format: "openai.responses",
base_url: "https://api.openai.com/v1",
auth: { type: "Bearer", token: "sk-test" },
diff --git a/web/src/features/ai-gateway/server/resolve/resolveService.ts b/web/src/features/ai-gateway/server/resolve/resolveService.ts
index 6c774d0d3a..b59f0b784f 100644
--- a/web/src/features/ai-gateway/server/resolve/resolveService.ts
+++ b/web/src/features/ai-gateway/server/resolve/resolveService.ts
@@ -14,6 +14,7 @@ import { GatewayControlPlaneError as GatewayResolveError } from "@/src/features/
import {
type GatewayApiFormat,
type GatewayProviderId,
+ type GatewayResolveResponse,
getGatewayProviderDefinition,
} from "@/src/features/ai-gateway/server/provider";
@@ -66,7 +67,7 @@ export class GatewayResolveService {
async resolve(params: {
context: GatewayApiKeyAuthContext;
apiFormat: GatewayApiFormat;
- }) {
+ }): Promise<GatewayResolveResponse> {
return instrumentAsync({ name: "gateway-resolve" }, async (span) => {
span.setAttribute(
"langfuse.organization.id",
@@ -80,13 +81,12 @@ export class GatewayResolveService {
403,
);
}
- if (!context.connection) {
+ if (!context.connection || !context.providerModel) {
throw new GatewayResolveError(
"No enabled gateway connection supports this API format",
404,
);
}
-
const provider = getGatewayProviderDefinition(
context.connection.provider,
);
@@ -98,6 +98,7 @@ export class GatewayResolveService {
name: context.connection.name,
provider:
context.connection.provider.toLowerCase() as GatewayProviderId,
+ provider_model: context.providerModel,
api_format: params.apiFormat,
base_url: provider.baseUrl,
auth:
diff --git a/web/src/features/ai-gateway/server/router.ts b/web/src/features/ai-gateway/server/router.ts
index fd40068614..17633a0e30 100644
--- a/web/src/features/ai-gateway/server/router.ts
+++ b/web/src/features/ai-gateway/server/router.ts
@@ -26,6 +26,14 @@ import {
} from "./provider";
const organizationInput = z.object({ orgId: z.string() });
+const connectionNameSchema = z
+ .string()
+ .trim()
+ .min(1)
+ .max(200)
+ .refine((name) => !name.includes(":"), {
+ message: "Connection names cannot contain a colon.",
+ });
const paginatedOrganizationInput = organizationInput.extend({
cursor: z.string().optional(),
limit: z.number().int().min(1).max(100).default(50),
@@ -100,7 +108,7 @@ export const aiGatewayRouter = createTRPCRouter({
createConnection: protectedOrganizationProcedureWithoutTracing
.input(
organizationInput.extend({
- name: z.string().trim().min(1).max(200),
+ name: connectionNameSchema,
provider: z.enum(GatewayProvider),
credential: z.string().min(1),
}),
@@ -121,7 +129,7 @@ export const aiGatewayRouter = createTRPCRouter({
.input(
organizationInput.extend({
id: z.string(),
- name: z.string().trim().min(1).max(200).optional(),
+ name: connectionNameSchema.optional(),
credential: z.string().min(1).optional(),
status: z
.enum([