🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
feat(ai-gateway): route models by connection slug (#18628)
Tobias Wochinger6c1fd54fa718
diff --git a/ai-gateway/README.md b/ai-gateway/README.mdindex 09029b52fc..524226c1b6 100644--- a/ai-gateway/README.md+++ b/ai-gateway/README.md@@ -78,8 +78,8 @@ do not load dotenv files:| `LANGFUSE_LOG_FORMAT` | `text` | `text`, `json` || `LANGFUSE_LOG_LEVEL` | `info` | `debug`, `info`, `warn`, `error`, `fatal` || `LANGFUSE_AI_GATEWAY_SHUTDOWN_TIMEOUT_SECONDS` | `10` | Integer from 1 to 300 |-| `LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS` | `128` | Positive integer up to Tokio's semaphore capacity; authenticated requests per instance |-| `LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS` | `128` | Positive integer up to Tokio's semaphore capacity; concurrent Web resolutions per instance |+| `LANGFUSE_AI_GATEWAY_MAX_ACTIVE_REQUESTS` | `128` | Positive integer up to Tokio's semaphore capacity; buffered and active requests per instance |+| `LANGFUSE_AI_GATEWAY_MAX_CONCURRENT_RESOLUTIONS` | `128` | Positive integer up to Tokio's semaphore capacity; separate model-catalog and Web-resolution budgets per instance || `LANGFUSE_AI_GATEWAY_TELEMETRY_BUFFER_BYTES` | `67108864` (64 MiB) | Integer from 4194304 (4 MiB) to 4294967296 (4 GiB); span and credential bytes an instance holds for upload before dropping new records |The gateway shares `LANGFUSE_LOG_LEVEL` with Web and worker. Values are lowercase;@@ -172,11 +172,10 @@ does not set CORS headers, so no `Access-Control-Expose-Headers` is involved.### Resolution contextOnce Web resolution succeeds, the request also carries its tenant, and once it is-sent upstream, the provider connection it was sent to. Requests rejected-before resolution carry none of them. A resolved request that fails before reaching-the provider, such as a 413 body or 503 execution capacity rejection, carries only-the tenant. Provider failures before response headers (502/504) still name the-connection that was tried.+sent upstream, the provider connection it was sent to. Requests rejected before+resolution, including invalid or oversized bodies, carry none of them. A resolved+request rejected by execution admission carries only the tenant. Provider failures+before response headers (502/504) still name the connection that was tried.| Response header | Server span field | JSON log key | Value || --- | --- | --- | --- |@@ -225,7 +224,7 @@ dimensions. The histogram's count supplies request volume and error rates,including responses rejected before provider execution. Metrics are independentof trace sampling. Health probes are excluded.-Gateway-specific metrics include `gateway.active` (resolution/execution),+Gateway-specific metrics include `gateway.active` (model_catalog/resolution/execution),`gateway.phase.duration` (execution/provider.first_byte), `gateway.executions`,`gateway.admission.rejected`, and `gateway.telemetry.records`. Durations use seconds;first-byte timing measures provider bytes arriving at the gateway, not first-token@@ -372,14 +371,18 @@ error type follows the status (`authentication_error`, `permission_error`,`not_found_error`, `request_too_large`, `invalid_request_error`, `overloaded_error`,`api_error`).-The gateway authenticates through Web before reserving execution capacity or-reading the request body. Web resolution uses a separate concurrency budget,-released after resolution finishes; authenticated requests then reserve execution-capacity through body reading and response completion. Either budget returns 503-immediately when full, without a waiting queue. Both defaults of 128 are provisional-guardrails, not measured capacity: tune them independently using load tests for the-instance resources, request sizes and stream durations. These limits bound work;-they do not guarantee fairness between clients or tenants.+After validating the credential header, the gateway buffers and validates the request+under the active-request admission budget, then resolves the credential and+model-aware route through Web under a separate budget. It holds the body slot through+the provider request. Model catalogs use another budget so slow catalog discovery+cannot block inference resolution.+Execution capacity remains reserved through response completion. An exhausted budget+returns 503 immediately without a waiting queue. Request-body and execution budgets+use the active-request limit; model-catalog and Web-call budgets use the resolution+limit. Their defaults of 128 are provisional+guardrails, not measured capacity: tune them using load tests for the instance+resources, request sizes and stream durations. These limits bound work; they do not+guarantee fairness between clients or tenants.Other limits are 10 MiB request bodies in both namespaces, 30 seconds to read a request, 5 seconds toconnect, 120 seconds for provider response headers or an individual upstream read,@@ -818,15 +821,19 @@ async fn example(web_base_url: &str, service_key: &str, gateway_key: &str)-> Result<(), ResolutionError>{let control_plane = ControlPlaneClient::new(ControlPlaneConfig::new(web_base_url, service_key)?)?;- let context = control_plane.resolve(gateway_key, ApiFormat::OpenAiResponses).await?;+ let request_id = "018f47a2-4d6e-7a61-b495-5e5f665c1e12";+ let context = control_plane+ .resolve(gateway_key, ApiFormat::OpenAiResponses, "gpt-4.1", request_id)+ .await?;assert_eq!(context.connection().base_url(), "https://api.openai.com/v1");Ok(())}```-Each call sends `POST <base path>/api/internal/ai-gateway/v1/resolve` with+After extracting the model, inference sends+`POST <base path>/api/internal/ai-gateway/v1/resolve` with`Authorization: Bearer <gateway key>`, the Web v1 HMAC header, and exactly-`{"apiFormat":"<format>"}`, where the format is the one the public route serves+`{"apiFormat":"<format>","model":"<model>"}`, where the format is the one the public route serves(`openai.chat-completions` for `/openai/v1/chat/completions`, `openai.responses` forthe other `/openai/v1` routes). The client neither receives norparses an inference request body. Reuse the resolver across requests: its connection@@ -896,8 +903,9 @@ cargo test --locked- `http.rs`: credential extraction, bounded body reads and gateway error envelopes.- `inference.rs`: `InferenceService` coordinates resolution, admission and forwarding.- `providers/mod.rs`: `ProviderTransport` with shared admission, the `Route` enum of- official operations (method, upstream path, capture, forwarded query) and credential+ official operations (method, upstream path and capture), model rewriting and credentialplacement.+- `providers/model_list`: provider-native OpenAI and Anthropic model catalog responses.- `transport/mod.rs`: per-format header policies, bounded byte relay and response lifetime.- `resolution/mod.rs`: trusted base URL configuration and bounded Web HTTP client.- `resolution/contracts.rs`: strict Web response validation and immutable execution context.diff --git a/ai-gateway/src/capture/anthropic_messages.rs b/ai-gateway/src/capture/anthropic_messages.rsindex 3d158a1210..7a7f44f450 100644--- a/ai-gateway/src/capture/anthropic_messages.rs+++ b/ai-gateway/src/capture/anthropic_messages.rs@@ -280,10 +280,6 @@ impl AnthropicMessagesCapture {self.facts.capture_complete = self.request_complete && self.facts.output_complete;}- pub fn requested_model(&self) -> Option<&str> {- self.facts.requested_model.as_deref()- }-pub fn into_facts(mut self) -> ProviderFacts {if matches!(self.body, ResponseBody::Sse(_)) {self.end_body();diff --git a/ai-gateway/src/capture/anthropic_messages/tests.rs b/ai-gateway/src/capture/anthropic_messages/tests.rsindex cd6017d354..b29e40b1d2 100644--- a/ai-gateway/src/capture/anthropic_messages/tests.rs+++ b/ai-gateway/src/capture/anthropic_messages/tests.rs@@ -558,8 +558,8 @@ async fn streamed_messages_upload_one_generation_with_native_usage() {context,&headers,Bytes::from_static(REQUEST),+ crate::providers::model_range(REQUEST).unwrap(),Route::AnthropicMessages,- None,&mut crate::correlation::RequestCorrelation::default(),).awaitdiff --git a/ai-gateway/src/capture/mod.rs b/ai-gateway/src/capture/mod.rsindex 70337a9813..ca95bc8349 100644--- a/ai-gateway/src/capture/mod.rs+++ b/ai-gateway/src/capture/mod.rs@@ -82,14 +82,6 @@ impl ProtocolCapture {}}- fn requested_model(&self) -> Option<&str> {- match self {- Self::OpenAiResponses(capture) => capture.requested_model(),- Self::OpenAiChatCompletions(capture) => capture.requested_model(),- Self::AnthropicMessages(capture) => capture.requested_model(),- }- }-fn client_metadata(&self) -> Option<&Map<String, Value>> {match self {Self::OpenAiResponses(capture) => capture.client_metadata(),@@ -109,6 +101,7 @@ pub(crate) struct ExecutionCapture {completion_start_ms: Option<u128>,http_status: Option<u16>,metadata: Value,+ provider_model: Option<String>,delivery: Option<(telemetry::Telemetry, telemetry::DeliveryContext)>,}@@ -126,6 +119,7 @@ impl ExecutionCapture {completion_start_ms: None,http_status: None,metadata: json!({}),+ provider_model: None,delivery: None,}}@@ -181,6 +175,7 @@ impl ExecutionCapture {"key_metadata": key_metadata,"ingestion_mode": if full { "full" } else { "usage" },}),+ provider_model: None,delivery: None,}}@@ -205,11 +200,8 @@ impl ExecutionCapture {self.delivery = Some((telemetry, delivery));}- /// The request's `model`, if the capture could parse it.- pub fn requested_model(&self) -> Option<&str> {- self.protocol- .as_ref()- .and_then(ProtocolCapture::requested_model)+ pub fn set_provider_model(&mut self, model: &str) {+ self.provider_model = bounded_string(model);}pub fn record_response(&mut self, status: u16, headers: &HeaderMap) {@@ -244,7 +236,13 @@ impl ExecutionCapture {let Some(protocol) = self.protocol.take() else {return;};- let (api_format, inference) = protocol.into_facts();+ let (api_format, mut inference) = protocol.into_facts();+ if let Some(provider_model) = self.provider_model.take() {+ if inference.model == inference.requested_model {+ inference.model = Some(provider_model.clone());+ }+ inference.requested_model = Some(provider_model);+ }let facts = InferenceFacts {api_format,start_time_unix_ms: self.start_time_unix_ms,diff --git a/ai-gateway/src/capture/openai_chat_completions.rs b/ai-gateway/src/capture/openai_chat_completions.rsindex a7ce3fdef1..bb29946883 100644--- a/ai-gateway/src/capture/openai_chat_completions.rs+++ b/ai-gateway/src/capture/openai_chat_completions.rs@@ -341,10 +341,6 @@ impl OpenAiChatCompletionsCapture {self.facts.capture_complete = self.request_complete && self.facts.output_complete;}- pub fn requested_model(&self) -> Option<&str> {- self.facts.requested_model.as_deref()- }-pub fn into_facts(mut self) -> ProviderFacts {if matches!(self.body, ResponseBody::Sse(_)) {self.end_body();diff --git a/ai-gateway/src/capture/openai_responses.rs b/ai-gateway/src/capture/openai_responses.rsindex 34cb05ba62..5270cf489e 100644--- a/ai-gateway/src/capture/openai_responses.rs+++ b/ai-gateway/src/capture/openai_responses.rs@@ -100,10 +100,6 @@ impl OpenAiResponsesCapture {self.facts.capture_complete = self.request_complete && self.facts.output_complete;}- pub fn requested_model(&self) -> Option<&str> {- self.facts.requested_model.as_deref()- }-pub fn into_facts(mut self) -> ProviderFacts {if matches!(self.body, ResponseBody::Sse(_)) {self.end_body();diff --git a/ai-gateway/src/capture/openai_responses/tests.rs b/ai-gateway/src/capture/openai_responses/tests.rsindex 19fe30c9a9..173fd4dc6e 100644--- a/ai-gateway/src/capture/openai_responses/tests.rs+++ b/ai-gateway/src/capture/openai_responses/tests.rs@@ -529,7 +529,7 @@ async fn upstream_read_timeout_is_logged_as_timeout() {provider.try_admit().unwrap(),context,&HeaderMap::new(),- Bytes::from_static(b"{}"),+ Bytes::from_static(br#"{"model":"requested"}"#),).await.unwrap();@@ -584,7 +584,7 @@ async fn native_http_relay_logs_once_on_eof_drop_and_unpolled_deadline() {provider.try_admit().unwrap(),context,&HeaderMap::new(),- Bytes::from_static(br#"{"input":"request-content"}"#),+ Bytes::from_static(br#"{"model":"requested","input":"request-content"}"#),).await.unwrap();@@ -648,7 +648,7 @@ async fn provider_future_finalizes_on_timeout_and_cancellation_before_headers()provider.try_admit().unwrap(),context,&headers,- Bytes::from_static(b"{}"),+ Bytes::from_static(br#"{"model":"requested"}"#),);if cancel {tokio::select! {@@ -745,7 +745,9 @@ async fn client_compression_preferences_do_not_disable_capture() {provider.try_admit().unwrap(),context,&headers,- Bytes::from(json!({"input":"hello","stream":streaming}).to_string()),+ Bytes::from(+ json!({"model":"requested","input":"hello","stream":streaming}).to_string(),+ ),).await.unwrap();diff --git a/ai-gateway/src/http.rs b/ai-gateway/src/http.rsindex 261b929a60..1582536360 100644--- a/ai-gateway/src/http.rs+++ b/ai-gateway/src/http.rs@@ -1,4 +1,4 @@-use std::{error::Error, sync::Arc, time::Duration};+use std::{error::Error, ops::Range, sync::Arc, time::Duration};use axum::{Json, Router,@@ -8,13 +8,14 @@ use axum::{response::{IntoResponse, Response},routing::{get, post},};-use serde::Serialize;+use serde::{Deserialize, Serialize};+use serde_json::value::RawValue;use tracing::Instrument;use crate::{correlation::{RequestCorrelation, assign_request_id},inference::{InferenceService, RequestPreparationError},- providers::{ProviderError, Route, forwarded_query},+ providers::{ProviderError, Route},resolution::{ApiFormat, ResolutionError},server::GatewayLifecycleState,};@@ -53,102 +54,203 @@ pub fn router(inference: Option<InferenceService>, lifecycle: GatewayLifecycleSt}async fn handle_responses(State(state): State<InferenceRouteState>, request: Request) -> Response {- handle(state, request, Route::OpenAiResponses).await+ handle_inference(state, request, Route::OpenAiResponses).await}async fn handle_responses_compact(State(state): State<InferenceRouteState>,request: Request,) -> Response {- handle(state, request, Route::OpenAiResponsesCompact).await+ handle_inference(state, request, Route::OpenAiResponsesCompact).await}async fn handle_chat_completions(State(state): State<InferenceRouteState>,request: Request,) -> Response {- handle(state, request, Route::OpenAiChatCompletions).await+ handle_inference(state, request, Route::OpenAiChatCompletions).await}async fn handle_openai_models(State(state): State<InferenceRouteState>,request: Request,) -> Response {- handle(state, request, Route::OpenAiModels).await+ handle_models(state, request, ApiFormat::OpenAiResponses).await}async fn handle_messages(State(state): State<InferenceRouteState>, request: Request) -> Response {- handle(state, request, Route::AnthropicMessages).await+ handle_inference(state, request, Route::AnthropicMessages).await}async fn handle_count_tokens(State(state): State<InferenceRouteState>,request: Request,) -> Response {- handle(state, request, Route::AnthropicCountTokens).await+ handle_inference(state, request, Route::AnthropicCountTokens).await}async fn handle_anthropic_models(State(state): State<InferenceRouteState>,request: Request,) -> Response {- handle(state, request, Route::AnthropicModels).await+ handle_models(state, request, ApiFormat::AnthropicMessages).await}-async fn handle(state: InferenceRouteState, mut request: Request, route: Route) -> Response {+async fn handle_inference(+ state: InferenceRouteState,+ mut request: Request,+ route: Route,+) -> Response {let mut correlation = request.extensions_mut().remove::<RequestCorrelation>().unwrap_or_default();- let mut response = match relay(state, request, route, &mut correlation).await {+ let result = match prepare_request(&state, request.headers(), route.api_format()) {+ Ok((inference, gateway_key)) => {+ relay(&inference, &gateway_key, request, route, &mut correlation).await+ }+ Err(error) => Err(error),+ };+ finish_response(result, route.api_format(), &correlation)+}++async fn handle_models(+ state: InferenceRouteState,+ mut request: Request,+ api_format: ApiFormat,+) -> Response {+ let correlation = request+ .extensions_mut()+ .remove::<RequestCorrelation>()+ .unwrap_or_default();+ let result = match prepare_request(&state, request.headers(), api_format) {+ Ok((inference, gateway_key)) => {+ models(&inference, &gateway_key, api_format, &correlation).await+ }+ Err(error) => Err(error),+ };+ finish_response(result, api_format, &correlation)+}++fn prepare_request(+ state: &InferenceRouteState,+ headers: &HeaderMap,+ api_format: ApiFormat,+) -> Result<(Arc<InferenceService>, String), InferenceHttpError> {+ let inference = state+ .inference+ .as_ref()+ .filter(|_| state.lifecycle.is_ready())+ .cloned()+ .ok_or(InferenceHttpError::Unavailable)?;+ let gateway_key = gateway_key(headers, api_format)?.to_owned();+ Ok((inference, gateway_key))+}++fn finish_response(+ result: Result<Response, InferenceHttpError>,+ api_format: ApiFormat,+ correlation: &RequestCorrelation,+) -> Response {+ let mut response = match result {Ok(response) => response,- Err(error) => error.into_native_response(route.api_format(), correlation.id()),+ Err(error) => error.into_native_response(api_format, correlation.id()),};correlation.apply_generation_headers(response.headers_mut());correlation.apply_resolution_headers(response.headers_mut());response}+async fn models(+ inference: &InferenceService,+ gateway_key: &str,+ api_format: ApiFormat,+ correlation: &RequestCorrelation,+) -> Result<Response, InferenceHttpError> {+ let body = inference+ .fetch_model_catalog(gateway_key, api_format, correlation)+ .await+ .map_err(InferenceHttpError::Resolution)?;+ Ok(([(header::CONTENT_TYPE, "application/json")], body).into_response())+}+async fn relay(- state: InferenceRouteState,+ inference: &InferenceService,+ gateway_key: &str,request: Request,route: Route,correlation: &mut RequestCorrelation,) -> Result<Response, InferenceHttpError> {- let inference = state- .inference- .as_ref()- .filter(|_| state.lifecycle.is_ready())- .ok_or(InferenceHttpError::Unavailable)?;- let gateway_key = gateway_key(request.headers(), route.api_format())?.to_owned();+ // Model extraction and upstream rewriting require an uncompressed JSON body.+ if !crate::transport::identity_encoding(request.headers()) {+ return Err(InferenceHttpError::UnsupportedEncoding);+ }+ let _body_permit = inference+ .try_reserve_request_body()+ .map_err(InferenceHttpError::from)?;+ let (parts, body) = request.into_parts();+ let request_body = request_body_and_model(body).await?;let (permit, context) = inference- .resolve_and_admit(&gateway_key, route.api_format(), correlation)+ .resolve_and_admit(+ gateway_key,+ route.api_format(),+ &request_body.model,+ correlation,+ ).await- .map_err(|error| match error {- RequestPreparationError::Resolution(error) => InferenceHttpError::Resolution(error),- RequestPreparationError::Provider(error) => InferenceHttpError::Provider(error),- })?;- let query = forwarded_query(route, request.uri().query());- let (parts, body) = request.into_parts();- let bytes = if route.method() == axum::http::Method::GET {- Bytes::new()- } else {- read_request_body(body).await?- };+ .map_err(InferenceHttpError::from)?;inference.forward(permit,context,&parts.headers,- bytes,+ request_body.bytes,+ request_body.model_range,route,- query.as_deref(),correlation,).await.map_err(InferenceHttpError::Provider)}+struct InferenceRequestBody {+ bytes: Bytes,+ model: String,+ model_range: Range<usize>,+}++#[derive(Deserialize)]+struct InferenceRequest<'a> {+ #[serde(borrow)]+ model: &'a RawValue,+}++async fn request_body_and_model(body: Body) -> Result<InferenceRequestBody, InferenceHttpError> {+ let bytes = read_request_body(body).await?;+ let request = serde_json::from_slice::<InferenceRequest<'_>>(&bytes)+ .map_err(|_| InferenceHttpError::InvalidBody)?;+ let raw_model = request.model.get();+ let model =+ serde_json::from_str::<String>(raw_model).map_err(|_| InferenceHttpError::InvalidBody)?;+ if model.is_empty() {+ return Err(InferenceHttpError::InvalidBody);+ }+ let start = raw_model+ .as_ptr()+ .addr()+ .checked_sub(bytes.as_ptr().addr())+ .ok_or(InferenceHttpError::InvalidBody)?;+ let end = start+ .checked_add(raw_model.len())+ .filter(|end| *end <= bytes.len())+ .ok_or(InferenceHttpError::InvalidBody)?;+ Ok(InferenceRequestBody {+ bytes,+ model,+ model_range: start..end,+ })+}+/// Buffer the client body. The wait is mostly the caller's upload, so it gets its/// own span and the measured size lands on the server span for aggregation.async fn read_request_body(body: Body) -> Result<Bytes, InferenceHttpError> {@@ -240,11 +342,21 @@ enum InferenceHttpError {AmbiguousCredential,TooLarge,RequestTimeout,+ UnsupportedEncoding,InvalidBody,Resolution(ResolutionError),Provider(ProviderError),}+impl From<RequestPreparationError> for InferenceHttpError {+ fn from(error: RequestPreparationError) -> Self {+ match error {+ RequestPreparationError::Resolution(error) => Self::Resolution(error),+ RequestPreparationError::Provider(error) => Self::Provider(error),+ }+ }+}+#[derive(Serialize)]struct OpenAiErrorResponse<'a> {error: OpenAiErrorDetail,@@ -282,11 +394,13 @@ impl InferenceHttpError {Self::AmbiguousCredential => ("authentication", "ambiguous_credential"),Self::TooLarge => ("request", "body_too_large"),Self::RequestTimeout => ("request", "body_timeout"),+ Self::UnsupportedEncoding => ("request", "unsupported_content_encoding"),Self::InvalidBody => ("request", "invalid_body"),Self::Resolution(error) => ("resolution",match error {R::InvalidCredential | R::Authentication => "authentication",+ R::InvalidRequest => "invalid_request",R::Forbidden => "forbidden",R::NoRoute => "no_route",R::Timeout => "timeout",@@ -348,12 +462,24 @@ impl InferenceHttpError {"invalid_request_error","request_timeout",),+ Self::UnsupportedEncoding => (+ StatusCode::UNSUPPORTED_MEDIA_TYPE,+ "Unsupported request Content-Encoding",+ "invalid_request_error",+ "unsupported_content_encoding",+ ),Self::InvalidBody => (StatusCode::BAD_REQUEST,"Invalid request body","invalid_request_error","invalid_body",),+ Self::Resolution(R::InvalidRequest) => (+ StatusCode::BAD_REQUEST,+ "Invalid gateway model",+ "invalid_request_error",+ "invalid_model",+ ),Self::Unavailable| Self::Resolution(R::Unavailable | R::Configuration)| Self::Provider(ProviderError::Configuration | ProviderError::Busy) => (diff --git a/ai-gateway/src/http/tests.rs b/ai-gateway/src/http/tests.rsindex 267b4f2358..b205f75f2e 100644--- a/ai-gateway/src/http/tests.rs+++ b/ai-gateway/src/http/tests.rs@@ -5,12 +5,16 @@ use crate::{test_support::{FakeServer, resolution_response, resolution_response_for},};use futures_util::{StreamExt, stream};-use std::{convert::Infallible, task::Poll};+use std::{+ convert::Infallible,+ io::Write,+ sync::atomic::{AtomicBool, Ordering},+};use tokio::{net::TcpListener, sync::oneshot, task::JoinHandle};use tower::ServiceExt;#[tokio::test]-async fn invalid_credentials_are_rejected_before_admission_or_body_reads() {+async fn control_plane_rejections_do_not_consume_provider_capacity() {let web = FakeServer::start(|_| async {Response::builder().status(StatusCode::UNAUTHORIZED)@@ -35,17 +39,12 @@ async fn invalid_credentials_are_rejected_before_admission_or_body_reads() {.await;// Exercise both an exhausted and an available execution pool.for occupied in [Some(permit), None] {- let body = Body::from_stream(stream::poll_fn(- |_| -> Poll<Option<Result<&'static str, Infallible>>> {- panic!("an invalid credential must be rejected before polling the body");- },- ));let response = tokio::time::timeout(Duration::from_secs(2),gateway.app.clone().oneshot(Request::post("/openai/v1/responses").header(header::AUTHORIZATION, "Bearer invalid-key")- .body(body)+ .body(Body::from(r#"{"model":"Production key:test"}"#)).unwrap(),),)@@ -97,7 +96,7 @@ async fn resolution_capacity_is_bounded_and_released_on_cancellation_and_failurelet request = |key| {Request::post("/openai/v1/responses").header(header::AUTHORIZATION, key)- .body(Body::from("{}"))+ .body(Body::from(r#"{"model":"Production key:test"}"#)).unwrap()};let mut stalled = Box::pin(gateway.app.clone().oneshot(request("Bearer stalled")));@@ -109,14 +108,26 @@ async fn resolution_capacity_is_bounded_and_released_on_cancellation_and_failure}).await.unwrap();+ let body_polled = Arc::new(AtomicBool::new(false));+ let observed = body_polled.clone();+ let body = Body::from_stream(stream::once(async move {+ observed.store(true, Ordering::SeqCst);+ Ok::<_, Infallible>(Bytes::from_static(br#"{"model":"Production key:test"}"#))+ }));let busy = tokio::time::timeout(Duration::from_secs(2),- gateway.app.clone().oneshot(request("Bearer valid")),+ gateway.app.clone().oneshot(+ Request::post("/openai/v1/responses")+ .header(header::AUTHORIZATION, "Bearer valid")+ .body(body)+ .unwrap(),+ ),).await.unwrap().unwrap();assert_eq!(busy.status(), StatusCode::SERVICE_UNAVAILABLE);+ assert!(!body_polled.load(Ordering::SeqCst));assert_eq!(web.calls(), 1);assert_eq!(provider.calls(), 0);@@ -138,6 +149,164 @@ async fn resolution_capacity_is_bounded_and_released_on_cancellation_and_failureassert_eq!(provider.calls(), 1);}+#[tokio::test]+async fn slow_body_reads_do_not_consume_resolution_capacity() {+ let web = FakeServer::start(|request| async move {+ if request.uri().path() == "/api/internal/ai-gateway/v1/models" {+ return Response::new(Body::from(+ r#"{"data":[{"id":"Production key:gpt-4.1","provider":"OPENAI","canonicalSlug":"gpt-4.1","displayName":"GPT-4.1","createdAt":"1970-01-01T00:00:10.000Z"}]}"#,+ ));+ }+ resolution_response("provider-secret")+ })+ .await;+ let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;+ let gateway = Gateway::start(Some(InferenceService::for_test(+ web.control_plane(),+ ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),+ 1,+ )))+ .await;+ let body_entered = Arc::new(tokio::sync::Notify::new());+ let entered = body_entered.clone();+ let held_body = Body::from_stream(stream::once(async move {+ entered.notify_one();+ std::future::pending::<Result<Bytes, Infallible>>().await+ }));+ let mut held = Box::pin(+ gateway.app.clone().oneshot(+ Request::post("/openai/v1/responses")+ .header(header::AUTHORIZATION, "Bearer held")+ .body(held_body)+ .unwrap(),+ ),+ );+ tokio::time::timeout(Duration::from_secs(2), async {+ tokio::select! {+ () = body_entered.notified() => {},+ _ = &mut held => panic!("body read should remain pending"),+ }+ })+ .await+ .unwrap();+ let body_polled = Arc::new(AtomicBool::new(false));+ let observed = body_polled.clone();+ let body = Body::from_stream(stream::once(async move {+ observed.store(true, Ordering::SeqCst);+ Ok::<_, Infallible>(Bytes::from_static(br#"{"model":"connection:model"}"#))+ }));+ let busy = gateway+ .app+ .clone()+ .oneshot(+ Request::post("/openai/v1/responses")+ .header(header::AUTHORIZATION, "Bearer busy")+ .body(body)+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(busy.status(), StatusCode::SERVICE_UNAVAILABLE);+ assert!(!body_polled.load(Ordering::SeqCst));+ let models = gateway+ .app+ .clone()+ .oneshot(+ Request::get("/openai/v1/models")+ .header(header::AUTHORIZATION, "Bearer models")+ .body(Body::empty())+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(models.status(), StatusCode::OK);+ assert_eq!(web.calls(), 1);+ assert_eq!(provider.calls(), 0);+ drop(held);+ let response = gateway+ .app+ .clone()+ .oneshot(+ Request::post("/openai/v1/responses")+ .header(header::AUTHORIZATION, "Bearer released")+ .body(Body::from(r#"{"model":"connection:model"}"#))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(response.status(), StatusCode::OK);+ assert_eq!(web.calls(), 2);+ assert_eq!(provider.calls(), 1);+}++#[tokio::test]+async fn slow_model_catalogs_do_not_consume_resolution_capacity() {+ let entered = Arc::new(tokio::sync::Notify::new());+ let started = entered.clone();+ let web = FakeServer::start(move |request| {+ let started = started.clone();+ async move {+ if request.uri().path() == "/api/internal/ai-gateway/v1/models" {+ started.notify_one();+ return std::future::pending().await;+ }+ resolution_response("provider-secret")+ }+ })+ .await;+ let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;+ let gateway = Gateway::start(Some(InferenceService::for_test(+ web.control_plane(),+ ProviderTransport::for_test(provider.url.clone(), ProviderLimits::default()),+ 1,+ )))+ .await;+ let mut held = Box::pin(+ gateway.app.clone().oneshot(+ Request::get("/openai/v1/models")+ .header(header::AUTHORIZATION, "Bearer models-held")+ .body(Body::empty())+ .unwrap(),+ ),+ );+ tokio::time::timeout(Duration::from_secs(2), async {+ tokio::select! {+ () = entered.notified() => {},+ _ = &mut held => panic!("model catalog should remain pending"),+ }+ })+ .await+ .unwrap();++ let inference = gateway+ .app+ .clone()+ .oneshot(+ Request::post("/openai/v1/responses")+ .header(header::AUTHORIZATION, "Bearer inference")+ .body(Body::from(r#"{"model":"connection:model"}"#))+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(inference.status(), StatusCode::OK);+ let busy = gateway+ .app+ .clone()+ .oneshot(+ Request::get("/openai/v1/models")+ .header(header::AUTHORIZATION, "Bearer models-busy")+ .body(Body::empty())+ .unwrap(),+ )+ .await+ .unwrap();+ assert_eq!(busy.status(), StatusCode::SERVICE_UNAVAILABLE);+ assert_eq!(web.calls(), 2);+ assert_eq!(provider.calls(), 1);+ drop(held);+}+impl Gateway {async fn start(inference: Option<InferenceService>) -> Self {let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();@@ -234,6 +403,10 @@ fn assert_pre_upstream_headers(headers: &reqwest::header::HeaderMap, resolved: b}#[tokio::test]+#[expect(+ clippy::too_many_lines,+ reason = "the table keeps native error envelope coverage in one place"+)]async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body() {let web = FakeServer::start(|request| async move {if request.headers().get(header::AUTHORIZATION)@@ -278,7 +451,10 @@ async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body()false,),(- configured.chat_completions().bearer_auth("rejected"),+ configured+ .chat_completions()+ .bearer_auth("rejected")+ .body(r#"{"model":"Production key:test"}"#),StatusCode::UNAUTHORIZED,false,false,@@ -290,16 +466,22 @@ async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body().body(vec![b'x'; MAX_REQUEST_BYTES + 1]),StatusCode::PAYLOAD_TOO_LARGE,false,- true,+ false,),(- configured.post().bearer_auth("rejected"),+ configured+ .post()+ .bearer_auth("rejected")+ .body(r#"{"model":"Production key:test"}"#),StatusCode::UNAUTHORIZED,false,false,),(- configured.messages().header("x-api-key", "rejected"),+ configured+ .messages()+ .header("x-api-key", "rejected")+ .body(r#"{"model":"Production key:test"}"#),StatusCode::UNAUTHORIZED,true,false,@@ -311,7 +493,7 @@ async fn every_error_carries_a_fresh_request_id_in_its_header_and_native_body().body(vec![b'x'; MAX_REQUEST_BYTES + 1]),StatusCode::PAYLOAD_TOO_LARGE,false,- true,+ false,),] {let response = request@@ -368,13 +550,15 @@ async fn resolved_requests_name_their_tenant_and_the_provider_connection_sent_to(anthropic("messages/count_tokens",- r#"{"messages":[],"model":"claude-opus-4-1"}"#,+ r#"{"messages":[],"model":"Production key:claude-opus-4-1"}"#,),"anthropic",),- (gateway.models().bearer_auth("gateway-openai"), "openai"),(- anthropic("messages", r#"{"model":"claude-opus-4-1","messages":[]}"#),+ anthropic(+ "messages",+ r#"{"model":"Production key:claude-opus-4-1","messages":[]}"#,+ ),"anthropic",),] {@@ -392,7 +576,7 @@ async fn resolved_requests_name_their_tenant_and_the_provider_connection_sent_to// The model is an operator-side span and log field only.assert_eq!(header("langfuse-model"), None);}- assert_eq!(provider.calls(), 3);+ assert_eq!(provider.calls(), 2);}impl Drop for Gateway {@@ -422,10 +606,13 @@ async fn native_json_and_sse_traverse_resolution_and_relay() {.to_str().unwrap().to_owned();- assert_eq!(- to_bytes(request.into_body(), 1024).await.unwrap(),- r#"{"apiFormat":"openai.responses"}"#- );+ let body: serde_json::Value =+ serde_json::from_slice(&to_bytes(request.into_body(), 1024).await.unwrap()).unwrap();+ assert_eq!(body["apiFormat"], "openai.responses");+ assert!(matches!(+ body["model"].as_str(),+ Some("Production key:unchanged" | "Production key:anything")+ ));resolution_response(if key == "Bearer gateway-json" {"provider-json"} else {@@ -447,14 +634,14 @@ async fn native_json_and_sse_traverse_resolution_and_relay() {.to_owned();let body = to_bytes(request.into_body(), 4096).await.unwrap();if key == "Bearer provider-json" {- assert_eq!(body, "{ \"model\":\"unchanged\", \"unknown\":true }");+ assert_eq!(body, r#"{ "model":"provider-model", "unknown":true }"#);Response::builder().header("content-type", "application/json").body(Body::from("{\"output\":[]}")).unwrap()} else {assert_eq!(key, "Bearer provider-sse");- assert_eq!(body, r#"{"model":"anything","stream":true}"#);+ assert_eq!(body, r#"{"model":"provider-model","stream":true}"#);Response::builder().header("content-type", "text/event-stream").body(Body::from_stream(stream::iter([@@ -470,12 +657,12 @@ async fn native_json_and_sse_traverse_resolution_and_relay() {.post().bearer_auth("gateway-json").header("x-langfuse-project-id", "attacker-project")- .body("{ \"model\":\"unchanged\", \"unknown\":true }")+ .body("{ \"model\":\"Production key:unchanged\", \"unknown\":true }").send();let sse = gateway.post().bearer_auth("gateway-sse")- .body(r#"{"model":"anything","stream":true}"#)+ .body(r#"{"model":"Production key:anything","stream":true}"#).send();let (json, sse) = tokio::join!(json, sse);let json = json.unwrap();@@ -492,8 +679,20 @@ async fn native_json_and_sse_traverse_resolution_and_relay() {}#[tokio::test]-async fn compact_relays_opaque_json_and_models_proxies_openai_list_without_query() {- let web = FakeServer::start(|_| async { resolution_response("provider-secret") }).await;+async fn compact_rewrites_model_and_models_come_from_the_control_plane() {+ let web = FakeServer::start(|request| async move {+ if request.uri().path() == "/api/internal/ai-gateway/v1/models" {+ assert_eq!(request.uri().query(), Some("api_format=openai.responses"));+ return Response::builder()+ .header("content-type", "application/json")+ .body(Body::from(+ r#"{"data":[{"id":"Production key:gpt-4.1","provider":"OPENAI","canonicalSlug":"gpt-4.1","displayName":"GPT-4.1","createdAt":"1970-01-01T00:00:10.000Z"}]}"#,+ ))+ .unwrap();+ }+ resolution_response("provider-secret")+ })+ .await;let provider = FakeServer::start(|request| async move {let method = request.method().clone();let uri = request.uri().to_string();@@ -502,7 +701,7 @@ async fn compact_relays_opaque_json_and_models_proxies_openai_list_without_query("POST", "/v1/responses/compact") => {assert_eq!(body,- r#"{"model":"gpt-4.1","encrypted_content":"opaque-ciphertext"}"#+ r#"{"model":"provider-model","encrypted_content":"opaque-ciphertext"}"#);Response::builder().header("content-type", "application/json")@@ -511,15 +710,6 @@ async fn compact_relays_opaque_json_and_models_proxies_openai_list_without_query)).unwrap()}- ("GET", "/v1/models") => {- assert!(body.is_empty());- Response::builder()- .header("content-type", "application/json")- .body(Body::from(- r#"{"object":"list","data":[{"id":"gpt-4.1","object":"model"}]}"#,- ))- .unwrap()- }other => panic!("unexpected provider request {other:?}"),}})@@ -528,7 +718,7 @@ async fn compact_relays_opaque_json_and_models_proxies_openai_list_without_querylet compact = gateway.compact().bearer_auth("gateway-key")- .body(r#"{"model":"gpt-4.1","encrypted_content":"opaque-ciphertext"}"#)+ .body(r#"{"model":"Production key:gpt-4.1","encrypted_content":"opaque-ciphertext"}"#).send().await.unwrap();@@ -549,10 +739,65 @@ async fn compact_relays_opaque_json_and_models_proxies_openai_list_without_queryassert_eq!(models.status(), StatusCode::OK);assert_eq!(models.text().await.unwrap(),- r#"{"object":"list","data":[{"id":"gpt-4.1","object":"model"}]}"#+ r#"{"object":"list","data":[{"id":"Production key:gpt-4.1","object":"model","created":10,"owned_by":"openai"}]}"#);assert_eq!(web.calls(), 2);- assert_eq!(provider.calls(), 2);+ assert_eq!(provider.calls(), 1);+}++#[tokio::test]+async fn model_routes_request_the_matching_control_plane_catalog() {+ let web = FakeServer::start(|request| async move {+ assert_eq!(request.uri().path(), "/api/internal/ai-gateway/v1/models");+ let api_format = request+ .uri()+ .query()+ .unwrap()+ .strip_prefix("api_format=")+ .unwrap();+ let body = match api_format {+ "openai.responses" => {+ r#"{"data":[{"id":"Production key:gpt-4.1","provider":"OPENAI","canonicalSlug":"gpt-4.1","displayName":"GPT-4.1","createdAt":"1970-01-01T00:00:10.000Z"}]}"#+ }+ "anthropic.messages" => {+ r#"{"data":[{"id":"Production key:claude-sonnet-4-5","provider":"ANTHROPIC","canonicalSlug":"claude-sonnet-4-5","displayName":"Claude Sonnet 4.5","createdAt":"2026-01-01T00:00:00.000Z"}]}"#+ }+ other => panic!("unexpected API format {other}"),+ };+ Response::builder()+ .header("content-type", "application/json")+ .body(Body::from(body))+ .unwrap()+ })+ .await;+ let provider = FakeServer::start(|request| async move {+ panic!("model catalog must not reach a provider: {}", request.uri())+ })+ .await;+ let gateway = Gateway::start(Some(inference(&web, &provider))).await;+ for (path, credential, expected) in [+ (+ "openai/v1/models",+ ("authorization", "Bearer gateway-openai"),+ r#"{"object":"list","data":[{"id":"Production key:gpt-4.1","object":"model","created":10,"owned_by":"openai"}]}"#,+ ),+ (+ "anthropic/v1/models",+ ("x-api-key", "gateway-anthropic"),+ r#"{"data":[{"type":"model","id":"Production key:claude-sonnet-4-5","display_name":"Claude Sonnet 4.5","created_at":"2026-01-01T00:00:00.000Z"}],"has_more":false,"first_id":"Production key:claude-sonnet-4-5","last_id":"Production key:claude-sonnet-4-5"}"#,+ ),+ ] {+ let response = reqwest::Client::new()+ .get(format!("{}/{path}", gateway.url))+ .header(credential.0, credential.1)+ .send()+ .await+ .unwrap();+ assert_eq!(response.status(), StatusCode::OK);+ assert_eq!(response.text().await.unwrap(), expected);+ }+ assert_eq!(web.calls(), 2);+ assert_eq!(provider.calls(), 0);}#[tokio::test]@@ -610,19 +855,76 @@ async fn malformed_credentials_skip_resolution_and_oversized_body_releases_capac.await.unwrap();assert_eq!(response.status(), StatusCode::PAYLOAD_TOO_LARGE);- assert_eq!(web.calls(), 1);+ assert_eq!(web.calls(), 0);assert_eq!(provider.calls(), 0);let next = gateway.post().bearer_auth("valid-key")- .body("{}")+ .body(r#"{"model":"Production key:test"}"#).send().await.unwrap();assert_eq!(next.status(), StatusCode::OK);+ assert_eq!(web.calls(), 1);assert_eq!(provider.calls(), 1);}+#[tokio::test]+async fn compressed_request_bodies_are_explicitly_rejected() {+ let web = FakeServer::start(|_| async { resolution_response("provider-secret") }).await;+ let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;+ let gateway = Gateway::start(Some(inference(&web, &provider))).await;+ let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default());+ encoder+ .write_all(br#"{"model":"connection:model","input":"hello"}"#)+ .unwrap();+ for (encoding, body) in [+ ("gzip", encoder.finish().unwrap()),+ (+ "zstd",+ b"encoded bytes are rejected before parsing".to_vec(),+ ),+ ] {+ let response = gateway+ .post()+ .bearer_auth("valid-key")+ .header(header::CONTENT_ENCODING, encoding)+ .header(header::CONTENT_TYPE, "application/json")+ .body(body)+ .send()+ .await+ .unwrap();+ assert_eq!(response.status(), StatusCode::UNSUPPORTED_MEDIA_TYPE);+ let body: serde_json::Value =+ serde_json::from_slice(&response.bytes().await.unwrap()).unwrap();+ assert_eq!(body["error"]["code"], "unsupported_content_encoding");+ }+ assert_eq!(web.calls(), 0);+ assert_eq!(provider.calls(), 0);+}++#[tokio::test]+async fn requests_without_a_model_are_rejected_before_resolution() {+ let web = FakeServer::start(|_| async { resolution_response("provider-secret") }).await;+ let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;+ let gateway = Gateway::start(Some(inference(&web, &provider))).await;++ let response = gateway+ .post()+ .bearer_auth("valid-key")+ .header(header::CONTENT_TYPE, "application/json")+ .body(r#"{"prompt":{"id":"pmpt_123"}}"#)+ .send()+ .await+ .unwrap();++ assert_eq!(response.status(), StatusCode::BAD_REQUEST);+ let body: serde_json::Value = serde_json::from_slice(&response.bytes().await.unwrap()).unwrap();+ assert_eq!(body["error"]["code"], "invalid_body");+ assert_eq!(web.calls(), 0);+ assert_eq!(provider.calls(), 0);+}+#[tokio::test]async fn resolution_failures_are_sanitized_and_never_call_provider() {let web = FakeServer::start(|request| async move {@@ -650,6 +952,7 @@ async fn resolution_failures_are_sanitized_and_never_call_provider() {let provider = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;let gateway = Gateway::start(Some(inference(&web, &provider))).await;for (key, status) in [+ ("400", 400),("401", 401),("403", 403),("404", 404),@@ -660,7 +963,7 @@ async fn resolution_failures_are_sanitized_and_never_call_provider() {let response = gateway.post().bearer_auth(key)- .body("{}")+ .body(r#"{"model":"Production key:test"}"#).send().await.unwrap();@@ -687,7 +990,7 @@ async fn slow_resolution_has_a_bounded_wait() {gateway.post().bearer_auth("slow-resolution")- .body("{}")+ .body(r#"{"model":"Production key:test"}"#).send(),).await@@ -825,18 +1128,7 @@ async fn chat_completions_streams_carry_the_request_and_generation_ids() {let telemetry =Telemetry::for_test(&ControlPlaneConfig::new(&sink.url, "service-key").unwrap());let (resolved, mut resolution_ids) = tokio::sync::mpsc::channel(1);- let web = FakeServer::start(move |request| {- let resolved = resolved.clone();- async move {- let id = request.headers()["langfuse-gateway-request-id"]- .to_str()- .unwrap()- .to_owned();- resolved.send(id).await.unwrap();- resolution_response_for(ApiFormat::OpenAiChatCompletions, "provider-chat")- }- })- .await;+ let web = chat_completion_resolution_server(resolved).await;let provider = FakeServer::start(|request| async move {assert_eq!(request.uri(), "/v1/chat/completions");Response::builder()@@ -908,6 +1200,23 @@ async fn chat_completions_streams_carry_the_request_and_generation_ids() {.await;}+async fn chat_completion_resolution_server(+ resolved: tokio::sync::mpsc::Sender<String>,+) -> FakeServer {+ FakeServer::start(move |request| {+ let resolved = resolved.clone();+ async move {+ let id = request.headers()["langfuse-gateway-request-id"]+ .to_str()+ .unwrap()+ .to_owned();+ resolved.send(id).await.unwrap();+ resolution_response_for(ApiFormat::OpenAiChatCompletions, "provider-chat")+ }+ })+ .await+}+#[tokio::test]async fn unconfigured_gateway_is_live_but_not_ready_and_inference_is_unavailable() {let gateway = Gateway::start(None).await;@@ -975,7 +1284,7 @@ async fn active_admission_is_held_through_stream_and_shutdown_drains_it() {let mut response = gateway.post().bearer_auth("key")- .body("{}")+ .body(r#"{"model":"Production key:test"}"#).send().await.unwrap();@@ -986,7 +1295,7 @@ async fn active_admission_is_held_through_stream_and_shutdown_drains_it() {gateway.post().bearer_auth("key")- .body("{}")+ .body(r#"{"model":"Production key:test"}"#).send().await.unwrap()@@ -1033,7 +1342,7 @@ async fn client_disconnect_releases_capacity_for_the_next_request() {let mut first = gateway.post().bearer_auth("key")- .body("{}")+ .body(r#"{"model":"Production key:test"}"#).send().await.unwrap();@@ -1044,7 +1353,7 @@ async fn client_disconnect_releases_capacity_for_the_next_request() {let response = gateway.post().bearer_auth("key")- .body("{}")+ .body(r#"{"model":"Production key:test"}"#).send().await.unwrap();@@ -1076,7 +1385,7 @@ async fn anthropic_messages_accept_either_credential_header_and_relay_native_sselet key = anthropic_gateway_key(&request).unwrap();assert_eq!(to_bytes(request.into_body(), 1024).await.unwrap(),- r#"{"apiFormat":"anthropic.messages"}"#+ r#"{"apiFormat":"anthropic.messages","model":"Production key:claude-sonnet-4-5"}"#);if key == "gateway-anthropic" {resolution_response_for(ApiFormat::AnthropicMessages, "provider-anthropic")@@ -1093,6 +1402,10 @@ async fn anthropic_messages_accept_either_credential_header_and_relay_native_sseassert_eq!(request.headers()["x-api-key"], "provider-anthropic");assert_eq!(request.headers()["anthropic-version"], "2023-06-01");assert!(!request.headers().contains_key(header::AUTHORIZATION));+ assert_eq!(+ to_bytes(request.into_body(), 4096).await.unwrap(),+ r#"{"model":"provider-model","stream":true,"messages":[]}"#+ );Response::builder().header("content-type", "text/event-stream").header("request-id", "req_upstream")@@ -1120,7 +1433,7 @@ async fn anthropic_messages_accept_either_credential_header_and_relay_native_sse.post(format!("{}/anthropic/v1/messages?beta=true", gateway.url)).header("anthropic-version", "2023-06-01").header("content-type", "application/json")- .body(r#"{"model":"claude-sonnet-4-5","stream":true,"messages":[]}"#);+ .body(r#"{"model":"Production key:claude-sonnet-4-5","stream":true,"messages":[]}"#);for (name, value) in credential_headers {request = request.header(name, value);}@@ -1198,7 +1511,7 @@ async fn anthropic_credential_conflicts_and_failures_use_the_native_error_envelofor (credential_headers, status, kind, message) in cases {let mut request = client.post(format!("{}/anthropic/v1/messages", gateway.url))- .body("{}");+ .body(r#"{"model":"Production key:test"}"#);for (name, value) in credential_headers {request = request.header(*name, *value);}@@ -1243,7 +1556,7 @@ async fn anthropic_capacity_rejections_use_the_native_envelope() {let response = reqwest::Client::new().post(format!("{}/anthropic/v1/messages", gateway.url)).header("x-api-key", "gateway-anthropic")- .body("{}")+ .body(r#"{"model":"Production key:test"}"#).send().await.unwrap();@@ -1283,11 +1596,26 @@ async fn openai_routes_ignore_x_api_key_and_keep_their_envelope() {#[tokio::test]async fn chat_completions_resolve_their_own_format_and_relay_native_json_and_errors() {let web = FakeServer::start(|request| async move {- assert_eq!(- to_bytes(request.into_body(), 1024).await.unwrap(),- r#"{"apiFormat":"openai.chat-completions"}"#- );- resolution_response_for(ApiFormat::OpenAiChatCompletions, "provider-chat")+ let request: serde_json::Value =+ serde_json::from_slice(&to_bytes(request.into_body(), 1024).await.unwrap()).unwrap();+ assert_eq!(request["apiFormat"], "openai.chat-completions");+ let provider_model = request["model"]+ .as_str()+ .and_then(|model| model.split_once(':'))+ .map(|(_, provider_model)| provider_model)+ .unwrap();+ let bytes = to_bytes(+ resolution_response_for(ApiFormat::OpenAiChatCompletions, "provider-chat").into_body(),+ 4096,+ )+ .await+ .unwrap();+ let mut response: serde_json::Value = serde_json::from_slice(&bytes).unwrap();+ response["connection"]["provider_model"] = provider_model.into();+ Response::builder()+ .header("content-type", "application/json")+ .body(Body::from(response.to_string()))+ .unwrap()}).await;let provider = FakeServer::start(|request| async move {@@ -1314,12 +1642,12 @@ async fn chat_completions_resolve_their_own_format_and_relay_native_json_and_errlet gateway = Gateway::start(Some(inference(&web, &provider))).await;for (body, status, expected) in [(- r#"{ "model":"gpt-4.1-mini", "messages":[] }"#,+ r#"{ "model":"Production key:gpt-4.1-mini", "messages":[] }"#,StatusCode::OK,r#"{"id":"chatcmpl-1","choices":[]}"#,),(- r#"{"model":"unknown-model","messages":[]}"#,+ r#"{"model":"Production key:unknown-model","messages":[]}"#,StatusCode::NOT_FOUND,r#"{"error":{"code":"model_not_found"}}"#,),@@ -1348,22 +1676,18 @@ async fn anthropic_routes_share_the_request_body_limit() {resolution_response_for(ApiFormat::AnthropicMessages, "provider-anthropic")}).await;- let provider = FakeServer::start(|request| async move {- let body = to_bytes(request.into_body(), MAX_REQUEST_BYTES)- .await- .unwrap();- assert_eq!(body.len(), MAX_REQUEST_BYTES);- Response::new(Body::empty())- })- .await;+ let provider = FakeServer::start(|_| async move { Response::new(Body::empty()) }).await;let gateway = Gateway::start(Some(inference(&web, &provider))).await;let client = reqwest::Client::new();for path in ["messages", "messages/count_tokens"] {let send = |size| {+ let prefix = r#"{"model":"Production key:test","padding":""#;+ let suffix = r#""}"#;+ let padding = "x".repeat(size - prefix.len() - suffix.len());client.post(format!("{}/anthropic/v1/{path}", gateway.url)).header("x-api-key", "gateway-anthropic")- .body(vec![b'x'; size])+ .body(format!("{prefix}{padding}{suffix}")).send()};assert_eq!(diff --git a/ai-gateway/src/inference.rs b/ai-gateway/src/inference.rsindex 4c13ca773f..23282f2dda 100644--- a/ai-gateway/src/inference.rs+++ b/ai-gateway/src/inference.rs@@ -1,9 +1,11 @@//! Resolve the caller's credential, then execute using only the trusted context.+use std::ops::Range;+use axum::{body::{Body, Bytes},http::{HeaderMap, Response},};-use tokio::sync::Semaphore;+use tokio::sync::{Semaphore, SemaphorePermit};use tracing::Instrument;use crate::{@@ -17,6 +19,8 @@ use crate::{pub struct InferenceService {control_plane: ControlPlaneClient,provider: ProviderTransport,+ request_body_capacity: Semaphore,+ model_catalog_capacity: Semaphore,resolution_capacity: Semaphore,telemetry: Option<crate::telemetry::Telemetry>,}@@ -26,6 +30,10 @@ pub(crate) enum RequestPreparationError {Provider(ProviderError),}+pub(crate) struct RequestBodyPermit<'a> {+ _permit: SemaphorePermit<'a>,+}+impl InferenceService {/// Initialize pooled clients for resolution and provider requests.///@@ -46,6 +54,8 @@ impl InferenceService {provider: ProviderTransport::new(max_active_requests).map_err(|_| ResolutionError::Configuration)?.with_telemetry(telemetry.clone()),+ request_body_capacity: Semaphore::new(max_active_requests),+ model_catalog_capacity: Semaphore::new(max_concurrent_resolutions),resolution_capacity: Semaphore::new(max_concurrent_resolutions),telemetry: Some(telemetry),})@@ -55,13 +65,49 @@ impl InferenceService {self.telemetry.clone()}- /// Authenticate with a separate bounded budget before reserving execution capacity.- /// The API format comes from the public route, so Web selects a compatible connection.- /// A successful resolution is recorded even if execution admission then fails.+ fn try_reserve_resolution(&self) -> Result<SemaphorePermit<'_>, ResolutionError> {+ self.resolution_capacity.try_acquire().map_err(|_| {+ crate::observability::rejected("resolution");+ ResolutionError::Unavailable+ })+ }++ pub(crate) fn try_reserve_request_body(+ &self,+ ) -> Result<RequestBodyPermit<'_>, RequestPreparationError> {+ let permit = self.request_body_capacity.try_acquire().map_err(|_| {+ crate::observability::rejected("request_body");+ RequestPreparationError::Resolution(ResolutionError::Unavailable)+ })?;+ Ok(RequestBodyPermit { _permit: permit })+ }++ pub(crate) async fn fetch_model_catalog(+ &self,+ gateway_key: &str,+ api_format: ApiFormat,+ correlation: &RequestCorrelation,+ ) -> Result<Vec<u8>, ResolutionError> {+ let _permit = self.model_catalog_capacity.try_acquire().map_err(|_| {+ crate::observability::rejected("model_catalog");+ ResolutionError::Unavailable+ })?;+ let _active = crate::observability::Active::new("model_catalog");+ let body = self+ .control_plane+ .models(gateway_key, api_format, correlation.id())+ .await?;+ crate::providers::model_list::provider_response(&body, api_format)+ }++ /// Authenticate, then reserve execution capacity. The API format comes from the public route,+ /// so Web selects a compatible connection. A successful resolution is recorded even if+ /// execution admission then fails.pub(crate) async fn resolve_and_admit(&self,gateway_key: &str,api_format: ApiFormat,+ model: &str,correlation: &mut RequestCorrelation,) -> Result<(RequestPermit, ResolvedRequestContext), RequestPreparationError> {let span = tracing::info_span!(@@ -70,7 +116,9 @@ impl InferenceService {gateway.outcome = tracing::field::Empty);async {- let prepared = self.prepare(gateway_key, api_format, correlation).await;+ let prepared = self+ .prepare(gateway_key, api_format, model, correlation)+ .await;tracing::Span::current().record("gateway.outcome",match &prepared {@@ -89,19 +137,20 @@ impl InferenceService {&self,gateway_key: &str,api_format: ApiFormat,+ model: &str,correlation: &mut RequestCorrelation,) -> Result<(RequestPermit, ResolvedRequestContext), RequestPreparationError> {+ let resolution_permit = self+ .try_reserve_resolution()+ .map_err(RequestPreparationError::Resolution)?;let context = {- let _permit = self.resolution_capacity.try_acquire().map_err(|_| {- crate::observability::rejected("resolution");- RequestPreparationError::Resolution(ResolutionError::Unavailable)- })?;let _active = crate::observability::Active::new("resolution");self.control_plane- .resolve(gateway_key, api_format, correlation.id())+ .resolve(gateway_key, api_format, model, correlation.id()).await.map_err(RequestPreparationError::Resolution)?};+ drop(resolution_permit);correlation.record_resolution(&context);let permit = self.provider@@ -120,12 +169,20 @@ impl InferenceService {context: ResolvedRequestContext,headers: &HeaderMap,body: Bytes,+ model_range: Range<usize>,route: Route,- query: Option<&str>,correlation: &mut RequestCorrelation,) -> Result<Response<Body>, ProviderError> {self.provider- .forward_route(permit, context, headers, body, route, query, correlation)+ .forward_route(+ permit,+ context,+ headers,+ body,+ model_range,+ route,+ correlation,+ ).await}@@ -138,6 +195,8 @@ impl InferenceService {Self {control_plane,provider,+ request_body_capacity: Semaphore::new(resolutions),+ model_catalog_capacity: Semaphore::new(resolutions),resolution_capacity: Semaphore::new(resolutions),telemetry: None,}diff --git a/ai-gateway/src/observability/http/tests.rs b/ai-gateway/src/observability/http/tests.rsindex 25fd158371..79ab5832ab 100644--- a/ai-gateway/src/observability/http/tests.rs+++ b/ai-gateway/src/observability/http/tests.rs@@ -205,7 +205,7 @@ async fn provider_http_errors_are_traced_without_changing_the_response() {provider.try_admit().unwrap(),context,&HeaderMap::new(),- Bytes::from_static(b"{}"),+ Bytes::from_static(br#"{"model":"requested"}"#),).await.unwrap();@@ -330,7 +330,7 @@ async fn generation_context_is_isolated_from_operational_spans_and_outbound_headlet observed = observed.lock().unwrap();assert_eq!(observed.len(), 3);for (name, headers, payload) in &*observed {- let child = spans.iter().find(|span| span.name == *name).unwrap();+ let child = outbound_span(&spans, name);assert_outbound_context(headers, (*name != "provider.headers").then_some(child));if *name == "ingestion" {assert_generation_context(payload);@@ -341,6 +341,17 @@ async fn generation_context_is_isolated_from_operational_spans_and_outbound_headtype SpanData = opentelemetry_sdk::trace::SpanData;+fn outbound_span<'a>(spans: &'a [SpanData], name: &str) -> &'a SpanData {+ let resolution = spans.iter().find(|span| span.name == "resolution").unwrap();+ spans+ .iter()+ .find(|span| {+ span.name == name+ && (name != "resolver" || span.parent_span_id == resolution.span_context.span_id())+ })+ .unwrap()+}+/// Every phase between the caller's headers and the last body byte has a span in/// the server trace, so a waterfall shows no unattributed wall time. Batched/// ingestion runs in its own trace, linked back to each request it carries.@@ -349,7 +360,6 @@ fn assert_phase_spans(spans: &[SpanData], server: &SpanData) {assert_eq!(spans.len(), 9);for (name, parent) in [("resolution", server),- ("resolver", named("resolution")),("request.body", server),("request.capture", server),("provider.headers", server),@@ -368,6 +378,15 @@ fn assert_phase_spans(spans: &[SpanData], server: &SpanData) {"{name}");}+ let resolver_parents = spans+ .iter()+ .filter(|span| span.name == "resolver")+ .map(|span| span.parent_span_id)+ .collect::<Vec<_>>();+ assert_eq!(+ resolver_parents,+ [named("resolution").span_context.span_id()]+ );let batch = named("telemetry.batch");assert_eq!(batch.parent_span_id, opentelemetry::trace::SpanId::INVALID);assert_ne!(@@ -430,7 +449,7 @@ fn assert_attribute(span: &SpanData, key: &str, expected: impl Into<opentelemetrassert_eq!(actual.value, expected.into(), "{} {key}", span.name);}-const GENERATION_REQUEST: &str = r#"{"model":"test-model"}"#;+const GENERATION_REQUEST: &str = r#"{"model":"Production key:test-model"}"#;const PROVIDER_RESPONSE: &str = r#"{"id":"resp_1","output":[]}"#;fn generation_request() -> HttpRequest<Body> {@@ -524,7 +543,7 @@ fn assert_request_correlation("provider_connection_name","Production key",),- ("gateway.upstream.model", "upstream_model", "test-model"),+ ("gateway.upstream.model", "upstream_model", "provider-model"),];for (attribute, _, expected) in resolution {assert_attribute(server, attribute, expected);diff --git a/ai-gateway/src/providers/chat_completions.rs b/ai-gateway/src/providers/chat_completions.rsindex f1ad94e254..7ef460468a 100644--- a/ai-gateway/src/providers/chat_completions.rs+++ b/ai-gateway/src/providers/chat_completions.rs@@ -2,25 +2,23 @@//! gateway opts in on the caller's behalf. The resulting usage chunk is relayed.use std::fmt;-use axum::{body::Bytes, http::HeaderMap};+use axum::body::Bytes;use serde::{Deserialize, Deserializer,de::{MapAccess, Visitor},};use serde_json::{Value, value::RawValue};-use crate::transport::identity_encoding;-/// The rewritten body for a streamed request that did not ask for usage, or/// `None` when the original bytes must be forwarded unchanged. Every other/// top-level member keeps its original bytes and position.-pub(super) fn request_stream_usage(headers: &HeaderMap, body: &[u8]) -> Option<Bytes> {- if !identity_encoding(headers) {- return None;- }+pub(super) fn request_stream_usage(body: &[u8], provider_model: &str) -> Option<Bytes> {let Members(mut members) = serde_json::from_slice(body).ok()?;// Duplicate members are resolved differently by different parsers.- if count(&members, "stream") != 1 || count(&members, "stream_options") > 1 {+ if count(&members, "model") != 1+ || count(&members, "stream") != 1+ || count(&members, "stream_options") > 1+ {return None;}if member(&members, "stream")? != "true" {@@ -40,6 +38,8 @@ pub(super) fn request_stream_usage(headers: &HeaderMap, body: &[u8]) -> Option<Blet mut options = options;options.insert("include_usage".to_owned(), Value::Bool(true));let options = RawValue::from_string(Value::Object(options).to_string()).ok()?;+ let model = RawValue::from_string(serde_json::to_string(provider_model).ok()?).ok()?;+ members.iter_mut().find(|(key, _)| key == "model")?.1 = model;match members.iter_mut().find(|(key, _)| key == "stream_options") {Some((_, raw)) => *raw = options,None => members.push(("stream_options".to_owned(), options)),@@ -95,10 +95,9 @@ impl<'de> Deserialize<'de> for Members {#[cfg(test)]mod tests {use super::*;- use axum::http::{HeaderValue, header};fn rewrite(body: &str) -> Option<String> {- request_stream_usage(&HeaderMap::new(), body.as_bytes())+ request_stream_usage(body.as_bytes(), "provider-model").map(|bytes| String::from_utf8(bytes.to_vec()).unwrap())}@@ -108,19 +107,21 @@ mod tests {rewrite(r#"{"model":"gpt-4.1", "messages":[{"role":"user","content":"1.0e2 \u00e9"}],"stream":true}"#).as_deref(),Some(- r#"{"model":"gpt-4.1","messages":[{"role":"user","content":"1.0e2 \u00e9"}],"stream":true,"stream_options":{"include_usage":true}}"#+ r#"{"model":"provider-model","messages":[{"role":"user","content":"1.0e2 \u00e9"}],"stream":true,"stream_options":{"include_usage":true}}"#));assert_eq!(rewrite(r#"{"stream_options":{"include_obfuscation":false,"include_usage":false},"stream":true,"model":"m"}"#).as_deref(),Some(- r#"{"stream_options":{"include_obfuscation":false,"include_usage":true},"stream":true,"model":"m"}"#+ r#"{"stream_options":{"include_obfuscation":false,"include_usage":true},"stream":true,"model":"provider-model"}"#));assert_eq!(- rewrite(r#"{"stream":true,"stream_options":null}"#).as_deref(),- Some(r#"{"stream":true,"stream_options":{"include_usage":true}}"#)+ rewrite(r#"{"model":"m","stream":true,"stream_options":null}"#).as_deref(),+ Some(+ r#"{"model":"provider-model","stream":true,"stream_options":{"include_usage":true}}"#+ ));}@@ -140,8 +141,5 @@ mod tests {] {assert_eq!(rewrite(body), None, "{body}");}- let mut encoded = HeaderMap::new();- encoded.insert(header::CONTENT_ENCODING, HeaderValue::from_static("gzip"));- assert!(request_stream_usage(&encoded, br#"{"stream":true}"#).is_none());}}diff --git a/ai-gateway/src/providers/mod.rs b/ai-gateway/src/providers/mod.rsindex 3c0202f94b..cda1a77e27 100644--- a/ai-gateway/src/providers/mod.rs+++ b/ai-gateway/src/providers/mod.rs@@ -1,10 +1,11 @@mod chat_completions;+pub(crate) mod model_list;-use std::{sync::Arc, time::Duration};+use std::{ops::Range, sync::Arc, time::Duration};use axum::{body::{Body, Bytes},- http::{HeaderMap, HeaderName, HeaderValue, Method, Response, header},+ http::{HeaderMap, HeaderName, HeaderValue, Response, header},};use reqwest::Client;use reqwest_middleware::ClientWithMiddleware;@@ -27,22 +28,17 @@ pub(crate) enum Route {OpenAiResponses,OpenAiResponsesCompact,OpenAiChatCompletions,- OpenAiModels,AnthropicMessages,AnthropicCountTokens,- AnthropicModels,}impl Route {pub(crate) fn provider(self) -> Provider {match self {- Self::OpenAiResponses- | Self::OpenAiResponsesCompact- | Self::OpenAiChatCompletions- | Self::OpenAiModels => Provider::OpenAi,- Self::AnthropicMessages | Self::AnthropicCountTokens | Self::AnthropicModels => {- Provider::Anthropic+ Self::OpenAiResponses | Self::OpenAiResponsesCompact | Self::OpenAiChatCompletions => {+ Provider::OpenAi}+ Self::AnthropicMessages | Self::AnthropicCountTokens => Provider::Anthropic,}}@@ -56,19 +52,11 @@ impl Route {}}- pub(crate) fn method(self) -> Method {- match self {- Self::OpenAiModels | Self::AnthropicModels => Method::GET,- _ => Method::POST,- }- }-fn path(self) -> &'static str {match self {Self::OpenAiResponses => "/responses",Self::OpenAiResponsesCompact => "/responses/compact",Self::OpenAiChatCompletions => "/chat/completions",- Self::OpenAiModels | Self::AnthropicModels => "/models",Self::AnthropicMessages => "/messages",Self::AnthropicCountTokens => "/messages/count_tokens",}@@ -83,13 +71,6 @@ impl Route {| Self::AnthropicMessages)}-- pub(crate) fn forwarded_query_parameters(self) -> &'static [&'static str] {- match self {- Self::AnthropicModels => &["limit", "after_id", "before_id"],- _ => &[],- }- }}pub(crate) struct ProviderLimits {@@ -181,6 +162,7 @@ impl ProviderTransport {/// # Errors/// See [`Self::forward_route`].+ #[cfg(test)]pub async fn forward(&self,permit: RequestPermit,@@ -188,13 +170,14 @@ impl ProviderTransport {headers: &HeaderMap,body: Bytes,) -> Result<Response<Body>, ProviderError> {+ let model_range = model_range(&body)?;self.forward_route(permit,context,headers,body,+ model_range,Route::OpenAiResponses,- None,&mut RequestCorrelation::from_headers(headers),).await@@ -210,13 +193,14 @@ impl ProviderTransport {context: ResolvedRequestContext,headers: &HeaderMap,body: Bytes,+ model_range: Range<usize>,route: Route,- query: Option<&str>,correlation: &mut RequestCorrelation,) -> Result<Response<Body>, ProviderError> {let api_format = route.api_format();let (credential_name, mut credential) = provider_credential(&context)?;credential.set_sensitive(true);+ let provider_model = context.connection().provider_model();let mut capture = if route.captures_generation() {let mut capture = ExecutionCapture::for_request(api_format, &context, headers, &body);if let Some(telemetry) = &self.telemetry {@@ -226,26 +210,32 @@ impl ProviderTransport {} else {ExecutionCapture::unobserved()};- // The capture keeps the caller's own request; only the upstream copy asks for usage.+ // Capture the caller's body shape, but keep upstream-only changes out of the input.let body = match route {Route::OpenAiChatCompletions => {- chat_completions::request_stream_usage(headers, &body).unwrap_or(body)+ match chat_completions::request_stream_usage(&body, provider_model) {+ Some(rewritten) => {+ drop(body);+ rewritten+ }+ None => rewrite_model(body, model_range, provider_model)?,+ }}- _ => body,+ _ => rewrite_model(body, model_range, provider_model)?,};- // The relay forwards the caller's `model` unchanged; only a captured route has parsed it.- let model = capture.requested_model().map(str::to_owned);- correlation.record_upstream(UpstreamTarget::for_connection(context.connection(), model));+ capture.set_provider_model(provider_model);+ correlation.record_upstream(UpstreamTarget::for_connection(+ context.connection(),+ Some(provider_model.to_owned()),+ ));let mut upstream = self.client- .request(route.method(), self.request_url(route, query))+ .post(self.request_url(route)).with_extension(DisableOtelPropagation).headers(transport::request_headers(headers, api_format)).header(header::ACCEPT_ENCODING, "identity").header(credential_name, credential);- if route.method() != Method::GET {- upstream = upstream.body(body);- }+ upstream = upstream.body(body);let response = tokio::time::timeout_at(permit.deadline@@ -286,7 +276,7 @@ impl ProviderTransport {Ok(downstream)}- fn request_url(&self, route: Route, query: Option<&str>) -> String {+ fn request_url(&self, route: Route) -> String {let base = self.origin_override.as_deref()@@ -295,10 +285,6 @@ impl ProviderTransport {let mut url = String::with_capacity(base.len() + route.path().len());url.push_str(base);url.push_str(route.path());- if let Some(query) = query.filter(|query| !query.is_empty()) {- url.push('?');- url.push_str(query);- }url}@@ -310,6 +296,50 @@ impl ProviderTransport {}}+#[expect(+ clippy::needless_pass_by_value,+ reason = "ownership releases the caller body when rewriting returns, before the upstream wait"+)]+fn rewrite_model(+ body: Bytes,+ model_range: Range<usize>,+ provider_model: &str,+) -> Result<Bytes, ProviderError> {+ if model_range.start > model_range.end || model_range.end > body.len() {+ return Err(ProviderError::Configuration);+ }+ let model = serde_json::to_vec(provider_model).map_err(|_| ProviderError::Configuration)?;+ let capacity = body.len() - model_range.len() + model.len();+ let mut rewritten = Vec::with_capacity(capacity);+ rewritten.extend_from_slice(&body[..model_range.start]);+ rewritten.extend_from_slice(&model);+ rewritten.extend_from_slice(&body[model_range.end..]);+ Ok(Bytes::from(rewritten))+}++#[cfg(test)]+pub(crate) fn model_range(body: &[u8]) -> Result<Range<usize>, ProviderError> {+ #[derive(serde::Deserialize)]+ struct ModelMember<'a> {+ #[serde(borrow)]+ model: &'a serde_json::value::RawValue,+ }++ let request = serde_json::from_slice::<ModelMember<'_>>(body)+ .map_err(|_| ProviderError::Configuration)?;+ let model = request.model.get();+ let start = model+ .as_ptr()+ .addr()+ .checked_sub(body.as_ptr().addr())+ .ok_or(ProviderError::Configuration)?;+ let end = start+ .checked_add(model.len())+ .filter(|end| *end <= body.len())+ .ok_or(ProviderError::Configuration)?;+ Ok(start..end)+}+fn provider_credential(context: &ResolvedRequestContext,) -> Result<(HeaderName, HeaderValue), ProviderError> {@@ -321,18 +351,5 @@ fn provider_credential(Ok((name, value))}-pub(crate) fn forwarded_query(route: Route, query: Option<&str>) -> Option<String> {- let allowed = route.forwarded_query_parameters();- let query = query?;- let kept: Vec<&str> = query- .split('&')- .filter(|pair| {- let key = pair.split('=').next().unwrap_or("");- allowed.contains(&key)- })- .collect();- (!kept.is_empty()).then(|| kept.join("&"))-}-#[cfg(test)]mod tests;diff --git a/ai-gateway/src/providers/model_list/anthropic.rs b/ai-gateway/src/providers/model_list/anthropic.rsnew file mode 100644index 0000000000..87573d85a9--- /dev/null+++ b/ai-gateway/src/providers/model_list/anthropic.rs@@ -0,0 +1,46 @@+use chrono::DateTime;+use serde::Serialize;++use super::CatalogEntry;+use crate::resolution::ResolutionError;++// API contract: https://platform.claude.com/docs/en/api/models/list+#[derive(Serialize)]+struct Response<'a> {+ data: Vec<Model<'a>>,+ has_more: bool,+ first_id: Option<&'a str>,+ last_id: Option<&'a str>,+}++#[derive(Serialize)]+struct Model<'a> {+ #[serde(rename = "type")]+ kind: &'static str,+ id: &'a str,+ display_name: &'a str,+ created_at: &'a str,+}++pub(super) fn response(entries: &[CatalogEntry]) -> Result<Vec<u8>, ResolutionError> {+ let data = entries+ .iter()+ .map(|entry| {+ DateTime::parse_from_rfc3339(&entry.created_at)+ .map_err(|_| ResolutionError::InvalidResponse)?;+ Ok(Model {+ kind: "model",+ id: &entry.id,+ display_name: &entry.display_name,+ created_at: &entry.created_at,+ })+ })+ .collect::<Result<Vec<_>, _>>()?;+ serde_json::to_vec(&Response {+ data,+ has_more: false,+ first_id: entries.first().map(|entry| entry.id.as_str()),+ last_id: entries.last().map(|entry| entry.id.as_str()),+ })+ .map_err(|_| ResolutionError::InvalidResponse)+}diff --git a/ai-gateway/src/providers/model_list/mod.rs b/ai-gateway/src/providers/model_list/mod.rsnew file mode 100644index 0000000000..a1c0128da2--- /dev/null+++ b/ai-gateway/src/providers/model_list/mod.rs@@ -0,0 +1,49 @@+mod anthropic;+mod openai;++use serde::Deserialize;++use crate::resolution::{ApiFormat, ResolutionError};++#[derive(Deserialize)]+struct Catalog {+ data: Vec<CatalogEntry>,+}++#[derive(Deserialize)]+#[serde(rename_all = "camelCase")]+struct CatalogEntry {+ id: String,+ provider: CatalogProvider,+ #[serde(rename = "canonicalSlug")]+ _canonical_slug: String,+ display_name: String,+ created_at: String,+}++#[derive(Clone, Copy, Deserialize, PartialEq, Eq)]+enum CatalogProvider {+ #[serde(rename = "OPENAI")]+ OpenAi,+ #[serde(rename = "ANTHROPIC")]+ Anthropic,+}++pub(crate) fn provider_response(+ body: &[u8],+ api_format: ApiFormat,+) -> Result<Vec<u8>, ResolutionError> {+ let catalog: Catalog =+ serde_json::from_slice(body).map_err(|_| ResolutionError::InvalidResponse)?;+ let provider = match api_format {+ ApiFormat::OpenAiResponses | ApiFormat::OpenAiChatCompletions => CatalogProvider::OpenAi,+ ApiFormat::AnthropicMessages => CatalogProvider::Anthropic,+ };+ if catalog.data.iter().any(|entry| entry.provider != provider) {+ return Err(ResolutionError::InvalidResponse);+ }+ match provider {+ CatalogProvider::OpenAi => openai::response(&catalog.data),+ CatalogProvider::Anthropic => anthropic::response(&catalog.data),+ }+}diff --git a/ai-gateway/src/providers/model_list/openai.rs b/ai-gateway/src/providers/model_list/openai.rsnew file mode 100644index 0000000000..63b02976cb--- /dev/null+++ b/ai-gateway/src/providers/model_list/openai.rs@@ -0,0 +1,45 @@+use chrono::DateTime;+use serde::Serialize;++use super::CatalogEntry;+use crate::resolution::ResolutionError;++// API contract: https://developers.openai.com/api/reference/resources/models/methods/list+#[derive(Serialize)]+struct Response<'a> {+ object: &'static str,+ data: Vec<Model<'a>>,+}++#[derive(Serialize)]+struct Model<'a> {+ id: &'a str,+ object: &'static str,+ created: i64,+ owned_by: &'static str,+}++pub(super) fn response(entries: &[CatalogEntry]) -> Result<Vec<u8>, ResolutionError> {+ let data = entries+ .iter()+ .map(|entry| {+ let created = DateTime::parse_from_rfc3339(&entry.created_at)+ .map_err(|_| ResolutionError::InvalidResponse)?+ .timestamp();+ if created < 0 {+ return Err(ResolutionError::InvalidResponse);+ }+ Ok(Model {+ id: &entry.id,+ object: "model",+ created,+ owned_by: "openai",+ })+ })+ .collect::<Result<Vec<_>, _>>()?;+ serde_json::to_vec(&Response {+ object: "list",+ data,+ })+ .map_err(|_| ResolutionError::InvalidResponse)+}diff --git a/ai-gateway/src/providers/tests.rs b/ai-gateway/src/providers/tests.rsindex 85038269a0..5699269b16 100644--- a/ai-gateway/src/providers/tests.rs+++ b/ai-gateway/src/providers/tests.rs@@ -36,9 +36,99 @@ const CALLER_CONTEXT_HEADERS: [(&str, &str); 9] = [("langfuse-metadata", "private:value"),];+#[test]+fn model_rewrite_changes_only_the_top_level_model_bytes() {+ let original = Bytes::from_static(+ br#"{ "z": {"second":2, "first":1}, "model" : "connection:model", "number": 1.0e2, "escaped":"\u00e9" }"#,+ );+ let expected = Bytes::from_static(+ br#"{ "z": {"second":2, "first":1}, "model" : "provider-model", "number": 1.0e2, "escaped":"\u00e9" }"#,+ );+ let model_range = model_range(&original).unwrap();+ assert_eq!(+ rewrite_model(original, model_range, "provider-model").unwrap(),+ expected+ );+}++#[test]+fn model_rewrite_fails_closed_for_an_invalid_model_range() {+ let missing = Bytes::from_static(br#"{"input":{"model":"nested-model"}}"#);+ assert!(matches!(+ rewrite_model(missing, 100..110, "provider-model"),+ Err(ProviderError::Configuration)+ ));+}+#[tokio::test]-async fn preserves_opaque_bytes_and_isolates_request_and_response_headers() {- const REQUEST: &[u8] = b"{ \"model\": \"opaque\", \"future_field\": [true, 3] }\n";+async fn invalid_model_rewrite_never_reaches_the_provider() {+ let upstream = FakeServer::start(|_| async { Response::new(Body::empty()) }).await;+ let relay = provider(&upstream, 1);+ let result = relay+ .forward_route(+ relay.try_admit().unwrap(),+ resolved_request_context("provider-secret").await,+ &HeaderMap::new(),+ Bytes::from_static(br#"{"model":"requested"}"#),+ 100..110,+ Route::OpenAiResponses,+ &mut crate::correlation::RequestCorrelation::default(),+ )+ .await;++ assert!(matches!(result, Err(ProviderError::Configuration)));+ assert_eq!(upstream.calls(), 0);+}++#[test]+fn model_rewrite_accepts_a_shorter_provider_model() {+ let original = Bytes::from_static(br#"{"model":"connection:model","input":"large"}"#);+ let model_range = model_range(&original).unwrap();+ assert_eq!(+ rewrite_model(original, model_range, "m").unwrap(),+ r#"{"model":"m","input":"large"}"#+ );+}++#[tokio::test]+async fn missing_provider_model_fails_resolution() {+ let web = FakeServer::start(|_| async {+ let bytes = to_bytes(+ crate::test_support::resolution_response("provider-secret").into_body(),+ 4096,+ )+ .await+ .unwrap();+ let mut body: serde_json::Value = serde_json::from_slice(&bytes).unwrap();+ body["connection"]+ .as_object_mut()+ .unwrap()+ .remove("provider_model");+ Response::new(Body::from(body.to_string()))+ })+ .await;+ let result = web+ .control_plane()+ .resolve(+ "gateway-secret",+ ApiFormat::OpenAiResponses,+ "Production key:gpt-4o",+ "test-request-id",+ )+ .await;++ assert!(matches!(+ result,+ Err(crate::resolution::ResolutionError::InvalidResponse)+ ));+}++#[tokio::test]+async fn rewrites_the_model_and_isolates_request_and_response_headers() {+ const REQUEST: &[u8] =+ br#"{ "model": "opaque", "future_field": {"model":"opaque","values":[true,3]} }"#;+ const UPSTREAM_REQUEST: &[u8] =+ br#"{ "model": "provider-model", "future_field": {"model":"opaque","values":[true,3]} }"#;const RESPONSE: &[u8] = b"\x1f\x8b\x08\x00opaque-encoded-response";let upstream = FakeServer::start(|request| async move {assert_eq!(request.uri(), "/v1/responses");@@ -49,6 +139,10 @@ async fn preserves_opaque_bytes_and_isolates_request_and_response_headers() {);assert_eq!(request.headers()[header::CONTENT_TYPE], "application/json");assert_eq!(request.headers()[header::ACCEPT_ENCODING], "identity");+ assert_eq!(+ request.headers()[header::CONTENT_LENGTH].to_str().unwrap(),+ UPSTREAM_REQUEST.len().to_string()+ );for name in ["cookie","langfuse-gateway-authorization",@@ -62,7 +156,10 @@ async fn preserves_opaque_bytes_and_isolates_request_and_response_headers() {{assert!(!request.headers().contains_key(name), "forwarded {name}");}- assert_eq!(to_bytes(request.into_body(), 1024).await.unwrap(), REQUEST);+ assert_eq!(+ to_bytes(request.into_body(), 1024).await.unwrap(),+ UPSTREAM_REQUEST+ );Response::builder().status(201).header("content-type", "application/json")@@ -149,7 +246,7 @@ async fn preserves_provider_failures_without_retries_or_redirects() {relay.try_admit().unwrap(),resolved_request_context("provider-secret").await,&HeaderMap::new(),- Bytes::new(),+ Bytes::from_static(br#"{"model":"requested"}"#),).await.unwrap();@@ -191,7 +288,7 @@ async fn streams_sse_incrementally_and_holds_admission_until_eof() {relay.try_admit().unwrap(),resolved_request_context("provider-secret").await,&HeaderMap::new(),- Bytes::new(),+ Bytes::from_static(br#"{"model":"requested"}"#),).await.unwrap();@@ -238,22 +335,22 @@ async fn simultaneous_requests_keep_provider_credentials_and_bodies_isolated() {relay.try_admit().unwrap(),alice,&headers,- Bytes::from_static(b"alice-body")+ Bytes::from_static(br#"{"model":"requested","owner":"alice"}"#)),relay.forward(relay.try_admit().unwrap(),bob,&headers,- Bytes::from_static(b"bob-body")+ Bytes::from_static(br#"{"model":"requested","owner":"bob"}"#)),);assert_eq!(to_bytes(alice.unwrap().into_body(), 1024).await.unwrap(),- "Bearer alice-secret:alice-body"+ r#"Bearer alice-secret:{"model":"provider-model","owner":"alice"}"#);assert_eq!(to_bytes(bob.unwrap().into_body(), 1024).await.unwrap(),- "Bearer bob-secret:bob-body"+ r#"Bearer bob-secret:{"model":"provider-model","owner":"bob"}"#);assert_eq!(upstream.calls(), 2);}@@ -291,7 +388,7 @@ async fn dropping_downstream_cancels_upstream_and_releases_admission() {relay.try_admit().unwrap(),resolved_request_context("provider-secret").await,&HeaderMap::new(),- Bytes::new(),+ Bytes::from_static(br#"{"model":"requested"}"#),).await.unwrap();@@ -324,7 +421,7 @@ async fn deadlines_bound_headers_and_stalled_bodies_without_exposing_transport_drelay.try_admit().unwrap(),resolved_request_context("provider-secret").await,&HeaderMap::new(),- Bytes::new(),+ Bytes::from_static(br#"{"model":"requested"}"#),).await;assert!(matches!(response, Err(ProviderError::Timeout)));@@ -345,7 +442,7 @@ async fn deadlines_bound_headers_and_stalled_bodies_without_exposing_transport_drelay.try_admit().unwrap(),context,&HeaderMap::new(),- Bytes::new(),+ Bytes::from_static(br#"{"model":"requested"}"#),).await.unwrap();@@ -543,6 +640,13 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rel.unwrap()["value"]["stringValue"],if cancelled { "WARNING" } else { "ERROR" });+ assert_eq!(+ attrs+ .iter()+ .find(|a| a["key"] == "langfuse.observation.model.name")+ .unwrap()["value"]["stringValue"],+ "provider-model"+ );assert_eq!(metadata["langfuse.gateway.response.status_code"], 200);assert!(metadata.get("relay_outcome").is_none());telemetry@@ -554,15 +658,15 @@ async fn cancelled_and_timed_out_executions_upload_after_provider_context_is_rel}#[tokio::test]-async fn compact_posts_compact_path_and_models_get_skips_ingestion() {+async fn compact_posts_compact_path_and_captures_ingestion() {use crate::{resolution::ControlPlaneConfig, telemetry::Telemetry};const COMPACT: &[u8] =br#"{"model":"gpt-4.1","input":[{"encrypted_content":"opaque-ciphertext"}]}"#;+ const UPSTREAM_COMPACT: &[u8] =+ br#"{"model":"provider-model","input":[{"encrypted_content":"opaque-ciphertext"}]}"#;const COMPACT_RESPONSE: &str =r#"{"id":"comp_1","output":[{"encrypted_content":"opaque-ciphertext"}]}"#;- const MODELS: &str = r#"{"object":"list","data":[{"id":"gpt-4.1","object":"model"}]}"#;-let (sent, mut received) = tokio::sync::mpsc::channel(2);let sink = FakeServer::start(move |request| {let sent = sent.clone();@@ -578,26 +682,14 @@ async fn compact_posts_compact_path_and_models_get_skips_ingestion() {let telemetry =Telemetry::for_test(&ControlPlaneConfig::new(&sink.url, "service-key").unwrap());let upstream = FakeServer::start(|request| async move {- let uri = request.uri().to_string();- let method = request.method().clone();+ assert_eq!(request.uri(), "/v1/responses/compact");+ assert_eq!(request.method(), "POST");let body = to_bytes(request.into_body(), 4096).await.unwrap();- match (method.as_str(), uri.as_str()) {- ("POST", "/v1/responses/compact") => {- assert_eq!(body, COMPACT);- Response::builder()- .header("content-type", "application/json")- .body(Body::from(COMPACT_RESPONSE))- .unwrap()- }- ("GET", "/v1/models") => {- assert!(body.is_empty());- Response::builder()- .header("content-type", "application/json")- .body(Body::from(MODELS))- .unwrap()- }- other => panic!("unexpected provider request {other:?}"),- }+ assert_eq!(body, UPSTREAM_COMPACT);+ Response::builder()+ .header("content-type", "application/json")+ .body(Body::from(COMPACT_RESPONSE))+ .unwrap()}).await;let relay = provider(&upstream, 1).with_telemetry(telemetry.clone());@@ -607,8 +699,8 @@ async fn compact_posts_compact_path_and_models_get_skips_ingestion() {resolved_request_context("provider-secret").await,&HeaderMap::new(),Bytes::from_static(COMPACT),+ model_range(COMPACT).unwrap(),Route::OpenAiResponsesCompact,- None,&mut crate::correlation::RequestCorrelation::default(),).await@@ -625,26 +717,12 @@ async fn compact_posts_compact_path_and_models_get_skips_ingestion() {payload["resourceSpans"][0]["scopeSpans"][0]["spans"][0]["name"],"openai.responses");- let models = relay- .forward_route(- relay.try_admit().unwrap(),- resolved_request_context("provider-secret").await,- &HeaderMap::new(),- Bytes::new(),- Route::OpenAiModels,- None,- &mut crate::correlation::RequestCorrelation::default(),- )- .await- .unwrap();- assert_eq!(to_bytes(models.into_body(), 4096).await.unwrap(), MODELS);- tokio::time::sleep(Duration::from_millis(50)).await;assert!(received.try_recv().is_err());telemetry.shutdown(Instant::now() + Duration::from_secs(1)).await;assert_eq!(sink.calls(), 1);- assert_eq!(upstream.calls(), 2);+ assert_eq!(upstream.calls(), 1);}fn assert_completed_upload(payload: &serde_json::Value, streaming: bool) {@@ -764,6 +842,7 @@ fn headers(pairs: &[(&'static str, &'static str)]) -> HeaderMap {#[tokio::test]async fn sends_the_x_api_key_credential_and_forwards_the_open_anthropic_header_family() {const REQUEST: &[u8] = br#"{"model":"claude-sonnet-4-5","system":[{"type":"text","text":"attribution","cache_control":{"type":"ephemeral"}}],"future_field":true}"#;+ const UPSTREAM_REQUEST: &[u8] = br#"{"model":"provider-model","system":[{"type":"text","text":"attribution","cache_control":{"type":"ephemeral"}}],"future_field":true}"#;let upstream = FakeServer::start(|request| async move {assert_eq!(request.uri(), "/v1/messages");assert_eq!(request.method(), "POST");@@ -779,7 +858,10 @@ async fn sends_the_x_api_key_credential_and_forwards_the_open_anthropic_header_ffor (name, _) in CONSUMED_HEADERS {assert!(!request.headers().contains_key(*name), "forwarded {name}");}- assert_eq!(to_bytes(request.into_body(), 1024).await.unwrap(), REQUEST);+ assert_eq!(+ to_bytes(request.into_body(), 1024).await.unwrap(),+ UPSTREAM_REQUEST+ );Response::builder().status(StatusCode::OK).header("content-type", "application/json")@@ -822,8 +904,8 @@ async fn sends_the_x_api_key_credential_and_forwards_the_open_anthropic_header_f.concat(),),Bytes::from_static(REQUEST),+ model_range(REQUEST).unwrap(),Route::AnthropicMessages,- None,&mut crate::correlation::RequestCorrelation::default(),).await@@ -852,96 +934,42 @@ async fn sends_the_x_api_key_credential_and_forwards_the_open_anthropic_header_f}#[tokio::test]-async fn count_tokens_relays_the_body_and_models_forwards_only_pagination() {+async fn count_tokens_relays_the_body() {let upstream = FakeServer::start(|request| async move {- let path = request.uri().path().to_owned();- let query = request.uri().query().map(str::to_owned);- match path.as_str() {- "/v1/messages/count_tokens" => {- assert_eq!(request.method(), "POST");- assert_eq!(query, None);- assert_eq!(- to_bytes(request.into_body(), 1024).await.unwrap(),- r#"{"model":"claude-sonnet-4-5","messages":[]}"#- );- Response::builder()- .header("content-type", "application/json")- .body(Body::from(r#"{"input_tokens":42}"#))- .unwrap()- }- "/v1/models" => {- assert_eq!(request.method(), "GET");- assert_eq!(query.as_deref(), Some("limit=1000&after_id=model_x"));- assert!(request.headers().get(header::CONTENT_LENGTH).is_none());- Response::builder()- .header("content-type", "application/json")- .body(Body::from(r#"{"data":[{"id":"claude-sonnet-4-5"}]}"#))- .unwrap()- }- other => panic!("unexpected upstream path {other}"),- }+ assert_eq!(request.uri(), "/v1/messages/count_tokens");+ assert_eq!(request.method(), "POST");+ assert_eq!(+ to_bytes(request.into_body(), 1024).await.unwrap(),+ r#"{"model":"provider-model","messages":[]}"#+ );+ Response::builder()+ .header("content-type", "application/json")+ .body(Body::from(r#"{"input_tokens":42}"#))+ .unwrap()}).await;let relay = transport(&upstream);- for (route, query, body, expected) in [- (+ let context =+ resolved_request_context_for(ApiFormat::AnthropicMessages, "provider-secret", "usage")+ .await;+ let response = relay+ .forward_route(+ relay.try_admit().unwrap(),+ context,+ &headers(&[("content-type", "application/json")]),+ Bytes::from_static(br#"{"model":"claude-sonnet-4-5","messages":[]}"#),+ model_range(br#"{"model":"claude-sonnet-4-5","messages":[]}"#).unwrap(),Route::AnthropicCountTokens,- Some("beta=true"),- r#"{"model":"claude-sonnet-4-5","messages":[]}"#,- r#"{"input_tokens":42}"#,- ),- (- Route::AnthropicModels,- Some("limit=1000&evil=1&after_id=model_x"),- "",- r#"{"data":[{"id":"claude-sonnet-4-5"}]}"#,- ),- ] {- let context =- resolved_request_context_for(ApiFormat::AnthropicMessages, "provider-secret", "usage")- .await;- let query = forwarded_query(route, query);- let response = relay- .forward_route(- relay.try_admit().unwrap(),- context,- &headers(&[("content-type", "application/json")]),- Bytes::from(body),- route,- query.as_deref(),- &mut crate::correlation::RequestCorrelation::default(),- )- .await- .unwrap();- assert_eq!(response.status(), StatusCode::OK);- assert_eq!(- to_bytes(response.into_body(), 1024).await.unwrap(),- expected- );- }- assert_eq!(upstream.calls(), 2);-}--#[test]-fn only_inference_routes_capture_generations_and_only_models_forwards_query() {- assert!(Route::AnthropicMessages.captures_generation());- assert!(!Route::AnthropicCountTokens.captures_generation());- assert!(!Route::AnthropicModels.captures_generation());- assert_eq!(- forwarded_query(Route::AnthropicMessages, Some("beta=true&limit=5")),- None- );- assert_eq!(- forwarded_query(- Route::AnthropicModels,- Some("before_id=a&limit=&evil&after_id=b%20c")- ),- Some("before_id=a&limit=&after_id=b%20c".to_owned())- );+ &mut crate::correlation::RequestCorrelation::default(),+ )+ .await+ .unwrap();+ assert_eq!(response.status(), StatusCode::OK);assert_eq!(- forwarded_query(Route::AnthropicModels, Some("evil=1")),- None+ to_bytes(response.into_body(), 1024).await.unwrap(),+ r#"{"input_tokens":42}"#);+ assert_eq!(upstream.calls(), 1);}#[test]@@ -989,13 +1017,7 @@ async fn chat_completions_streams_request_usage_and_relay_the_usage_chunk() {const REQUESTED: &str = r#"{"model":"gpt-4.1-mini","messages":[{"role":"user","content":"Say hello"}],"stream":true,"stream_options":{"include_usage":true}}"#;// Callers receive the usage chunk whether or not they asked for it.let relayed = [CHAT_CONTENT_CHUNK, CHAT_USAGE_CHUNK, CHAT_DONE].concat();- for (request, forwarded) in [- (- UNREQUESTED,- r#"{"model":"gpt-4.1-mini","messages":[{"role":"user","content":"Say hello"}],"stream":true,"stream_options":{"include_usage":true}}"#,- ),- (REQUESTED, REQUESTED),- ] {+ for request in [UNREQUESTED, REQUESTED] {let (sent, mut received) = tokio::sync::mpsc::channel(2);let sink = FakeServer::start(move |request| {let sent = sent.clone();@@ -1021,7 +1043,7 @@ async fn chat_completions_streams_request_usage_and_relay_the_usage_chunk() {}assert_eq!(to_bytes(request.into_body(), 4096).await.unwrap(),- forwarded+ r#"{"model":"provider-model","messages":[{"role":"user","content":"Say hello"}],"stream":true,"stream_options":{"include_usage":true}}"#);// Arrive split mid-event, as a provider stream may.let native = [CHAT_CONTENT_CHUNK, CHAT_USAGE_CHUNK, CHAT_DONE].concat();@@ -1054,8 +1076,8 @@ async fn chat_completions_streams_request_usage_and_relay_the_usage_chunk() {.concat(),),Bytes::from_static(request.as_bytes()),+ model_range(request.as_bytes()).unwrap(),Route::OpenAiChatCompletions,- None,&mut crate::correlation::RequestCorrelation::default(),).await@@ -1077,12 +1099,12 @@ async fn chat_completions_streams_request_usage_and_relay_the_usage_chunk() {}#[tokio::test]-async fn chat_completions_forward_non_streamed_bodies_unchanged() {+async fn chat_completions_rewrite_non_streamed_models() {const JSON_REQUEST: &str = r#"{"model":"gpt-4.1-mini", "messages":[]}"#;let upstream = FakeServer::start(|request| async move {assert_eq!(to_bytes(request.into_body(), 4096).await.unwrap(),- JSON_REQUEST+ r#"{"model":"provider-model", "messages":[]}"#);Response::builder().header("content-type", "application/json")@@ -1102,8 +1124,8 @@ async fn chat_completions_forward_non_streamed_bodies_unchanged() {.await,&headers(&[("content-type", "application/json")]),Bytes::from_static(JSON_REQUEST.as_bytes()),+ model_range(JSON_REQUEST.as_bytes()).unwrap(),Route::OpenAiChatCompletions,- None,&mut crate::correlation::RequestCorrelation::default(),).awaitdiff --git a/ai-gateway/src/resolution/contracts.rs b/ai-gateway/src/resolution/contracts.rsindex 5e3ec18975..da7424fbb6 100644--- a/ai-gateway/src/resolution/contracts.rs+++ b/ai-gateway/src/resolution/contracts.rs@@ -16,6 +16,16 @@ pub enum ApiFormat {AnthropicMessages,}+impl ApiFormat {+ pub fn as_str(self) -> &'static str {+ match self {+ Self::OpenAiResponses => "openai.responses",+ Self::OpenAiChatCompletions => "openai.chat-completions",+ Self::AnthropicMessages => "anthropic.messages",+ }+ }+}+#[derive(Debug, Clone, Copy, PartialEq, Eq, Deserialize)]#[serde(rename_all = "lowercase")]pub enum Provider {@@ -143,6 +153,7 @@ pub struct ProviderConnection {name: Option<String>,#[serde(deserialize_with = "string_enum")]provider: Provider,+ provider_model: String,#[serde(deserialize_with = "string_enum")]api_format: ApiFormat,base_url: String,@@ -160,6 +171,9 @@ impl ProviderConnection {pub fn provider(&self) -> Provider {self.provider}+ pub fn provider_model(&self) -> &str {+ &self.provider_model+ }pub fn api_format(&self) -> ApiFormat {self.api_format}@@ -314,6 +328,7 @@ pub(super) fn decode(].iter().any(|value| value.trim().is_empty())+ || connection.provider_model.trim().is_empty(){return Err(ResolutionError::InvalidResponse);}diff --git a/ai-gateway/src/resolution/mod.rs b/ai-gateway/src/resolution/mod.rsindex 98993a867d..fd58217ce5 100644--- a/ai-gateway/src/resolution/mod.rs+++ b/ai-gateway/src/resolution/mod.rs@@ -7,10 +7,10 @@ pub use contracts::{ProviderCredential, RequestAttribution, ResolvedRequestContext,};use reqwest::{- Client, Url,+ Client, Method, Url,header::{AUTHORIZATION, CONTENT_TYPE, HeaderValue},};-use reqwest_middleware::ClientWithMiddleware;+use reqwest_middleware::{ClientWithMiddleware, RequestBuilder};use std::{fmt,net::IpAddr,@@ -18,6 +18,7 @@ use std::{};const RESOLVE_PATH: &str = "/api/internal/ai-gateway/v1/resolve";+const MODELS_PATH: &str = "/api/internal/ai-gateway/v1/models";const REQUEST_ID_HEADER: &str = "langfuse-gateway-request-id";// Web sits behind load balancers that drop idle connections after 60s by default// (AWS ALB). Evicting pooled connections first keeps the resolver from sending on a@@ -25,7 +26,8 @@ const REQUEST_ID_HEADER: &str = "langfuse-gateway-request-id";const POOL_IDLE_TIMEOUT: Duration = Duration::from_secs(50);const TCP_KEEPALIVE: Duration = Duration::from_secs(30);-struct ResolutionLimits {+#[derive(Clone, Copy)]+struct ControlPlaneLimits {timeout: Duration,max_response_bytes: usize,}@@ -34,7 +36,8 @@ struct ResolutionLimits {pub struct ControlPlaneConfig {web_url: Url,service_key: String,- limits: ResolutionLimits,+ resolve_limits: ControlPlaneLimits,+ models_limits: ControlPlaneLimits,}impl ControlPlaneConfig {@@ -66,10 +69,15 @@ impl ControlPlaneConfig {Ok(Self {web_url: url,service_key: service_key.to_owned(),- limits: ResolutionLimits {+ resolve_limits: ControlPlaneLimits {timeout: Duration::from_secs(5),max_response_bytes: 256 * 1024,},+ // Model catalogs are unpaginated and can require cold provider discovery.+ models_limits: ControlPlaneLimits {+ timeout: Duration::from_secs(30),+ max_response_bytes: 10 * 1024 * 1024,+ },})}@@ -125,32 +133,68 @@ impl ControlPlaneClient {&self,gateway_key: &str,api_format: ApiFormat,+ model: &str,request_id: &str,) -> Result<ResolvedRequestContext, ResolutionError> {let timestamp = SystemTime::now().duration_since(UNIX_EPOCH).map_err(|_| ResolutionError::Unavailable)?;- self.resolve_at(gateway_key, api_format, timestamp, request_id)+ self.resolve_at(gateway_key, api_format, model, timestamp, request_id).await}+ /// Fetch the caller-visible model catalog from trusted Web.+ ///+ /// # Errors+ /// Returns the same sanitized errors as [`Self::resolve`].+ pub async fn models(+ &self,+ gateway_key: &str,+ api_format: ApiFormat,+ request_id: &str,+ ) -> Result<Vec<u8>, ResolutionError> {+ let timestamp = SystemTime::now()+ .duration_since(UNIX_EPOCH)+ .map_err(|_| ResolutionError::Unavailable)?;+ let mut endpoint = self.config.endpoint(MODELS_PATH);+ endpoint+ .query_pairs_mut()+ .append_pair("api_format", api_format.as_str());+ let request = self.authenticated_request(+ Method::GET,+ endpoint,+ gateway_key,+ timestamp.as_secs(),+ request_id,+ )?;+ self.execute(request, self.config.models_limits).await+ }+async fn resolve_at(&self,gateway_key: &str,api_format: ApiFormat,+ model: &str,timestamp: Duration,request_id: &str,) -> Result<ResolvedRequestContext, ResolutionError> {- if gateway_key.len() > 8192 || !valid_token(gateway_key) {- return Err(ResolutionError::InvalidCredential);- }let started = Instant::now();- let body = tokio::time::timeout(- self.config.limits.timeout,- self.fetch(gateway_key, api_format, timestamp.as_secs(), request_id),- )- .await- .map_err(|_| ResolutionError::Timeout)??;+ let body = serde_json::to_vec(&serde_json::json!({+ "apiFormat": api_format,+ "model": model,+ }))+ .map_err(|_| ResolutionError::Configuration)?;+ let request = self+ .authenticated_request(+ Method::POST,+ self.config.endpoint(RESOLVE_PATH),+ gateway_key,+ timestamp.as_secs(),+ request_id,+ )?+ .header(CONTENT_TYPE, "application/json")+ .body(body);+ let body = self.execute(request, self.config.resolve_limits).await?;contracts::decode(&body,api_format,@@ -158,13 +202,17 @@ impl ControlPlaneClient {)}- async fn fetch(+ fn authenticated_request(&self,+ method: Method,+ endpoint: Url,gateway_key: &str,- api_format: ApiFormat,timestamp: u64,request_id: &str,- ) -> Result<Vec<u8>, ResolutionError> {+ ) -> Result<RequestBuilder, ResolutionError> {+ if gateway_key.len() > 8192 || !valid_token(gateway_key) {+ return Err(ResolutionError::InvalidCredential);+ }let mut credential = HeaderValue::from_str(&format!("Bearer {gateway_key}")).map_err(|_| ResolutionError::InvalidCredential)?;credential.set_sensitive(true);@@ -175,47 +223,62 @@ impl ControlPlaneClient {)).map_err(|_| ResolutionError::Configuration)?;signature.set_sensitive(true);- let body = serde_json::to_vec(&serde_json::json!({ "apiFormat": api_format }))- .map_err(|_| ResolutionError::Configuration)?;- let mut response = self+ Ok(self.client- .post(self.config.endpoint(RESOLVE_PATH))+ .request(method, endpoint).header(AUTHORIZATION, credential).header("langfuse-gateway-authorization", signature)- .header(REQUEST_ID_HEADER, request_id)- .header(CONTENT_TYPE, "application/json")- .body(body)- .send()- .await- .map_err(|_| ResolutionError::Transport)?;- match response.status().as_u16() {- 200 => {}- 401 => return Err(ResolutionError::Authentication),- 403 => return Err(ResolutionError::Forbidden),- 404 => return Err(ResolutionError::NoRoute),- 429 | 500..=599 => return Err(ResolutionError::Unavailable),- _ => return Err(ResolutionError::InvalidResponse),- }- let limit = self.config.limits.max_response_bytes;- if response- .content_length()- .is_some_and(|size| size > limit as u64)- {+ .header(REQUEST_ID_HEADER, request_id))+ }++ async fn execute(+ &self,+ request: RequestBuilder,+ limits: ControlPlaneLimits,+ ) -> Result<Vec<u8>, ResolutionError> {+ tokio::time::timeout(limits.timeout, async {+ let response = request+ .send()+ .await+ .map_err(|_| ResolutionError::Transport)?;+ read_response(response, limits.max_response_bytes).await+ })+ .await+ .map_err(|_| ResolutionError::Timeout)?+ }+}++async fn read_response(+ mut response: reqwest::Response,+ limit: usize,+) -> Result<Vec<u8>, ResolutionError> {+ match response.status().as_u16() {+ 200 => {}+ 400 => return Err(ResolutionError::InvalidRequest),+ 401 => return Err(ResolutionError::Authentication),+ 403 => return Err(ResolutionError::Forbidden),+ 404 => return Err(ResolutionError::NoRoute),+ 429 | 500..=599 => return Err(ResolutionError::Unavailable),+ _ => return Err(ResolutionError::InvalidResponse),+ }+ if response+ .content_length()+ .is_some_and(|size| size > limit as u64)+ {+ return Err(ResolutionError::ResponseTooLarge);+ }+ let mut bytes = Vec::new();+ while let Some(chunk) = response+ .chunk()+ .await+ .map_err(|_| ResolutionError::Transport)?+ {+ if chunk.len() > limit - bytes.len() {return Err(ResolutionError::ResponseTooLarge);}- let mut bytes = Vec::new();- while let Some(chunk) = response- .chunk()- .await- .map_err(|_| ResolutionError::Transport)?- {- if chunk.len() > limit - bytes.len() {- return Err(ResolutionError::ResponseTooLarge);- }- bytes.extend_from_slice(&chunk);- }- Ok(bytes)+ bytes.extend_from_slice(&chunk);}+ Ok(bytes)}fn valid_token(value: &str) -> bool {@@ -227,6 +290,7 @@ fn valid_token(value: &str) -> bool {pub enum ResolutionError {Configuration,InvalidCredential,+ InvalidRequest,Authentication,Forbidden,NoRoute,@@ -242,6 +306,7 @@ impl fmt::Display for ResolutionError {f.write_str(match self {Self::Configuration => "invalid resolver configuration",Self::InvalidCredential => "invalid gateway credential",+ Self::InvalidRequest => "invalid gateway request",Self::Authentication => "gateway authentication failed",Self::Forbidden => "gateway access forbidden",Self::NoRoute => "no eligible connection",diff --git a/ai-gateway/src/resolution/tests.rs b/ai-gateway/src/resolution/tests.rsindex 009cc19513..4949754b0a 100644--- a/ai-gateway/src/resolution/tests.rs+++ b/ai-gateway/src/resolution/tests.rs@@ -58,16 +58,16 @@ impl FakeWeb {}fn control_plane(&self) -> ControlPlaneClient {- self.control_plane_with_limits(Duration::from_secs(2), 256 * 1024)+ self.control_plane_with_resolve_limits(Duration::from_secs(2), 256 * 1024)}- fn control_plane_with_limits(+ fn control_plane_with_resolve_limits(&self,timeout: Duration,max_response_bytes: usize,) -> ControlPlaneClient {let mut config = ControlPlaneConfig::new(&self.url, SERVICE_KEY).unwrap();- config.limits = ResolutionLimits {+ config.resolve_limits = ControlPlaneLimits {timeout,max_response_bytes,};@@ -97,6 +97,7 @@ fn success(project: &str, provider_token: &str) -> Value {"id": "connection-1","name": "Primary OpenAI","provider": "openai",+ "provider_model": "gpt-4.1","api_format": "openai.responses","base_url": "https://api.openai.com/v1","auth": {"type": "Bearer", "token": provider_token}@@ -114,7 +115,7 @@ fn success(project: &str, provider_token: &str) -> Value {}#[tokio::test]-async fn signs_the_exact_key_and_sends_only_the_api_format() {+async fn signs_the_exact_key_and_sends_api_format_and_model() {let web = FakeWeb::start(|request| async move {assert_eq!(request.method(), "POST");assert_eq!(request.uri(), "/api/internal/ai-gateway/v1/resolve");@@ -124,7 +125,7 @@ async fn signs_the_exact_key_and_sends_only_the_api_format() {assert_eq!(request.headers()["langfuse-gateway-authorization"],"HMAC timestamp=1800000000,signature=e104b6baa50d9b766b036e27876683aa1f4bff7f31bb6af66dce6b636a53b2bd");assert_eq!(to_bytes(request.into_body(), 1024).await.unwrap(),- r#"{"apiFormat":"openai.responses"}"#);+ r#"{"apiFormat":"openai.responses","model":"primary:gpt-4.1"}"#);response(success("project-1", "private-provider-token"))}).await;@@ -133,6 +134,7 @@ async fn signs_the_exact_key_and_sends_only_the_api_format() {.resolve_at(GATEWAY_KEY,ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",NOW_TIME,REQUEST_ID,)@@ -142,6 +144,7 @@ async fn signs_the_exact_key_and_sends_only_the_api_format() {assert_eq!(context.connection().name(), Some("Primary OpenAI"));assert_eq!(context.connection().provider(), Provider::OpenAi);assert_eq!(context.connection().provider().as_str(), "openai");+ assert_eq!(context.connection().provider_model(), "gpt-4.1");assert!(matches!(context.connection().credential(),ProviderCredential::Bearer("private-provider-token")@@ -171,9 +174,81 @@ async fn signs_the_exact_key_and_sends_only_the_api_format() {assert_eq!(web.calls(), 1);}+#[tokio::test]+async fn fetches_models_from_the_control_plane() {+ let web = FakeWeb::start(|request| async move {+ assert_eq!(request.method(), "GET");+ assert_eq!(+ request.uri(),+ "/api/internal/ai-gateway/v1/models?api_format=openai.responses"+ );+ assert_eq!(+ request.headers()[header::AUTHORIZATION],+ "Bearer gw_test_alice"+ );+ assert!(+ request+ .headers()+ .contains_key("langfuse-gateway-authorization")+ );+ assert_eq!(request.headers()["langfuse-gateway-request-id"], REQUEST_ID);+ assert!(+ to_bytes(request.into_body(), 1024)+ .await+ .unwrap()+ .is_empty()+ );+ response(json!({+ "data": [{+ "id": "Primary OpenAI:gpt-4.1",+ "provider": "OPENAI"+ }]+ }))+ })+ .await;++ let body = web+ .control_plane()+ .models(GATEWAY_KEY, ApiFormat::OpenAiResponses, REQUEST_ID)+ .await+ .unwrap();+ assert_eq!(+ serde_json::from_slice::<Value>(&body).unwrap()["data"][0]["id"],+ "Primary OpenAI:gpt-4.1"+ );+}++#[tokio::test]+async fn model_catalogs_have_a_larger_response_limit_than_resolution() {+ let body = "x".repeat(300 * 1024);+ let web = FakeWeb::start(move |_| {+ let body = body.clone();+ async move { Response::new(Body::from(body)) }+ })+ .await;+ let control_plane = web.control_plane();++ let resolution = control_plane+ .resolve_at(+ GATEWAY_KEY,+ ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",+ NOW_TIME,+ REQUEST_ID,+ )+ .await;+ assert!(matches!(resolution, Err(ResolutionError::ResponseTooLarge)));++ let models = control_plane+ .models(GATEWAY_KEY, ApiFormat::OpenAiResponses, REQUEST_ID)+ .await+ .unwrap();+ assert_eq!(models.len(), 300 * 1024);+}+#[tokio::test]async fn denied_and_failed_resolutions_are_classified_without_retries_or_body_leaks() {- for status in [401, 403, 404, 500, 503] {+ for status in [400, 401, 403, 404, 500, 503] {let web = FakeWeb::start(move |_| async move {Response::builder().status(status)@@ -186,12 +261,14 @@ async fn denied_and_failed_resolutions_are_classified_without_retries_or_body_le.resolve_at(GATEWAY_KEY,ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",NOW_TIME,REQUEST_ID,).await.unwrap_err();assert!(match status {+ 400 => matches!(error, ResolutionError::InvalidRequest),401 => matches!(error, ResolutionError::Authentication),403 => matches!(error, ResolutionError::Forbidden),404 => matches!(error, ResolutionError::NoRoute),@@ -223,6 +300,7 @@ async fn never_follows_redirects_with_credentials() {.resolve_at(GATEWAY_KEY,ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",NOW_TIME,REQUEST_ID)@@ -250,10 +328,11 @@ async fn rejects_oversized_declared_and_chunked_responses() {}).await;let result = web- .control_plane_with_limits(Duration::from_secs(2), 128)+ .control_plane_with_resolve_limits(Duration::from_secs(2), 128).resolve_at(GATEWAY_KEY,ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",NOW_TIME,REQUEST_ID,)@@ -282,10 +361,11 @@ async fn deadline_covers_response_headers_and_body() {.await;let result = tokio::time::timeout(Duration::from_millis(500),- web.control_plane_with_limits(Duration::from_millis(30), 256 * 1024)+ web.control_plane_with_resolve_limits(Duration::from_millis(30), 256 * 1024).resolve_at(GATEWAY_KEY,ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",NOW_TIME,REQUEST_ID,),@@ -303,7 +383,13 @@ async fn malformed_credentials_never_reach_web() {for key in ["", "has space", "has\r\nheader", "has\ttab"] {let result = web.control_plane()- .resolve_at(key, ApiFormat::OpenAiResponses, NOW_TIME, REQUEST_ID)+ .resolve_at(+ key,+ ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",+ NOW_TIME,+ REQUEST_ID,+ ).await;assert!(matches!(result, Err(ResolutionError::InvalidCredential)));}@@ -324,6 +410,7 @@ async fn rejects_a_grant_that_expires_while_resolving_across_a_second_boundary().resolve_at(GATEWAY_KEY,ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",NOW_TIME + Duration::from_millis(900),REQUEST_ID,)@@ -350,12 +437,14 @@ async fn concurrent_resolutions_keep_credentials_and_contexts_isolated() {control_plane.resolve_at("gw_test_alice",ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",NOW_TIME,REQUEST_ID),control_plane.resolve_at("gw_test_bob",ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",NOW_TIME,REQUEST_ID),@@ -380,6 +469,7 @@ fn anthropic_success(provider_key: &str) -> Value {body["connection"] = json!({"id": "connection-anthropic","provider": "anthropic",+ "provider_model": "claude-sonnet-4-5","api_format": "anthropic.messages","base_url": "https://api.anthropic.com/v1","auth": {"type": "x-api-key", "header": "x-api-key", "value": provider_key}@@ -393,7 +483,7 @@ async fn resolves_anthropic_messages_with_the_native_credential_scheme() {let web = FakeWeb::start(|request| async move {assert_eq!(to_bytes(request.into_body(), 1024).await.unwrap(),- r#"{"apiFormat":"anthropic.messages"}"#+ r#"{"apiFormat":"anthropic.messages","model":"primary:claude-sonnet-4-5"}"#);response(anthropic_success("private-anthropic-key"))})@@ -404,6 +494,7 @@ async fn resolves_anthropic_messages_with_the_native_credential_scheme() {.resolve_at(GATEWAY_KEY,ApiFormat::AnthropicMessages,+ "primary:claude-sonnet-4-5",NOW_TIME,REQUEST_ID,)@@ -443,7 +534,7 @@ async fn chat_completions_resolve_only_an_openai_connection_for_that_format() {async move {assert_eq!(to_bytes(request.into_body(), 1024).await.unwrap(),- r#"{"apiFormat":"openai.chat-completions"}"#+ r#"{"apiFormat":"openai.chat-completions","model":"primary:gpt-4.1"}"#);response(body)}@@ -454,6 +545,7 @@ async fn chat_completions_resolve_only_an_openai_connection_for_that_format() {.resolve_at(GATEWAY_KEY,ApiFormat::OpenAiChatCompletions,+ "primary:gpt-4.1",NOW_TIME,REQUEST_ID,)@@ -566,6 +658,7 @@ async fn rejects_incompatible_or_incomplete_execution_contexts() {("/connection/auth/token", json!("")),("/connection/auth/token", json!("bad\r\nheader")),("/connection/id", json!("")),+ ("/connection/provider_model", json!("")),("/attribution/project_id", json!("")),("/attribution/provider_connection_id", json!("")),(@@ -640,6 +733,7 @@ async fn rejects_ingestion_tokens_that_authorize_a_different_tenant() {.resolve_at(GATEWAY_KEY,ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",NOW_TIME,REQUEST_ID)@@ -660,7 +754,13 @@ async fn assert_invalid_context_for(body: Value, api_format: ApiFormat) {.await;let result = web.control_plane()- .resolve_at(GATEWAY_KEY, api_format, NOW_TIME, REQUEST_ID)+ .resolve_at(+ GATEWAY_KEY,+ api_format,+ "primary:provider-model",+ NOW_TIME,+ REQUEST_ID,+ ).await;assert!(matches!(result, Err(ResolutionError::InvalidResponse)));assert_eq!(web.calls(), 1);@@ -677,6 +777,7 @@ async fn rejects_malformed_json_without_exposing_response_contents() {.resolve_at(GATEWAY_KEY,ApiFormat::OpenAiResponses,+ "primary:gpt-4.1",NOW_TIME,REQUEST_ID,)diff --git a/ai-gateway/src/test_support.rs b/ai-gateway/src/test_support.rsindex 37b59075bb..6cefe779b7 100644--- a/ai-gateway/src/test_support.rs+++ b/ai-gateway/src/test_support.rs@@ -100,16 +100,19 @@ pub(crate) fn resolution_response_for(let connection = match api_format {ApiFormat::OpenAiResponses => json!({"id": "connection-1", "name": "Production key", "provider": "openai", "api_format": "openai.responses",+ "provider_model": "provider-model","base_url": "https://api.openai.com/v1","auth": {"type": "Bearer", "token": provider_secret}}),ApiFormat::OpenAiChatCompletions => json!({"id": "connection-1", "name": "Production key", "provider": "openai", "api_format": "openai.chat-completions",+ "provider_model": "provider-model","base_url": "https://api.openai.com/v1","auth": {"type": "Bearer", "token": provider_secret}}),ApiFormat::AnthropicMessages => json!({"id": "connection-1", "name": "Production key", "provider": "anthropic", "api_format": "anthropic.messages",+ "provider_model": "provider-model","base_url": "https://api.anthropic.com/v1","auth": {"type": "x-api-key", "header": "x-api-key", "value": provider_secret}}),@@ -170,7 +173,12 @@ pub(crate) async fn resolved_request_context_for(}).await;web.control_plane()- .resolve("gateway-secret", api_format, "test-request-id")+ .resolve(+ "gateway-secret",+ api_format,+ "Production key:provider-model",+ "test-request-id",+ ).await.unwrap()}diff --git a/packages/shared/prisma/migrations/20261012134000_add_unique_gateway_connection_names/migration.sql b/packages/shared/prisma/migrations/20261012134000_add_unique_gateway_connection_names/migration.sqlnew file mode 100644index 0000000000..9fbb371274--- /dev/null+++ b/packages/shared/prisma/migrations/20261012134000_add_unique_gateway_connection_names/migration.sql@@ -0,0 +1,2 @@+CREATE UNIQUE INDEX gateway_ai_connections_organization_id_name_key+ON gateway_ai_connections(organization_id, name);diff --git a/packages/shared/prisma/schema.prisma b/packages/shared/prisma/schema.prismaindex e2c431c3b2..2597b1c145 100644--- a/packages/shared/prisma/schema.prisma+++ b/packages/shared/prisma/schema.prisma@@ -361,6 +361,7 @@ model GatewayAiConnection {createdAt DateTime @default(now()) @map("created_at")updatedAt DateTime @default(now()) @updatedAt @map("updated_at")+ @@unique([organizationId, name])@@unique([organizationId, routingPriority])@@index([organizationId, status, routingPriority])@@map("gateway_ai_connections")diff --git a/web/src/__e2e__/ai-gateway.gatewaye2e.ts b/web/src/__e2e__/ai-gateway.gatewaye2e.tsindex 43950ef48c..df7af2c53e 100644--- a/web/src/__e2e__/ai-gateway.gatewaye2e.ts+++ b/web/src/__e2e__/ai-gateway.gatewaye2e.ts@@ -313,11 +313,17 @@ describe("AI gateway live end-to-end", () => {async ({ provider, apiFormat, baseUrl, authType }) => {await withGatewayApiKey({ provider, apiFormat },- async ({ secretKey, id: keyId, providerConnectionId }) => {+ async ({+ secretKey,+ id: keyId,+ providerConnectionId,+ providerConnectionName,+ }) => {const resolveBody = await gatewayControlPlaneRequest({path: RESOLVE_PATH,apiFormat,virtualSecretKey: secretKey,+ model: `${providerConnectionName}:ai-gateway-e2e`,});const resolved = GatewayResolveResponseSchema.parse(resolveBody);@@ -377,6 +383,7 @@ async function withGatewayApiKey(secretKey: string;id: string;providerConnectionId: string;+ providerConnectionName: string;}) => Promise<void>,) {const targetConnectionId = connectionIdsByProvider.get(input.provider);@@ -386,6 +393,10 @@ async function withGatewayApiKey();}await selectOnlyConnection(targetConnectionId);+ const targetConnection = await prisma.gatewayAiConnection.findUniqueOrThrow({+ where: { id: targetConnectionId },+ select: { name: true },+ });const gatewayKey = await admin.caller.aiGateway.createApiKey({orgId: ORGANIZATION_ID,@@ -410,6 +421,7 @@ async function withGatewayApiKey(secretKey: gatewayKey.secretKey,id: gatewayKey.id,providerConnectionId: targetConnectionId,+ providerConnectionName: targetConnection.name,});} finally {await admin.caller.aiGateway.revokeApiKey({@@ -547,6 +559,7 @@ async function gatewayControlPlaneRequest(input: {path: string;apiFormat: GatewayApiFormat;virtualSecretKey: string;+ model?: string;}): Promise<unknown> {const timestamp = Math.floor(Date.now() / 1000);const canonicalMessage = [@@ -571,7 +584,7 @@ async function gatewayControlPlaneRequest(input: {},body: isModelsRequest? undefined- : JSON.stringify({ apiFormat: input.apiFormat }),+ : JSON.stringify({ apiFormat: input.apiFormat, model: input.model }),});return readJsonResponse(response, 200, input.path);}diff --git a/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts b/web/src/__tests__/server/ai-gateway-control-plane.servertest.tsindex 050ea08197..f7359bcb6b 100644--- a/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts+++ b/web/src/__tests__/server/ai-gateway-control-plane.servertest.ts@@ -527,20 +527,20 @@ describe("AI gateway control plane", () => {).rejects.toThrow();});- it("routes by format and priority and issues a verifiable ingestion token", async () => {+ it("routes by connection slug and issues a verifiable ingestion token", async () => {const { caller, org, project } = await prepare();await caller.aiGateway.updateConfig({orgId: org.id,defaultIngestionProjectId: project.id,ingestionMode: "FULL",});- const openAiPrimary = await caller.aiGateway.createConnection({+ await caller.aiGateway.createConnection({orgId: org.id,name: "OpenAI first",provider: "OPENAI",credential: "sk-test-openai-primary",});- await caller.aiGateway.createConnection({+ const openAiSecondary = await caller.aiGateway.createConnection({orgId: org.id,name: "OpenAI second",provider: "OPENAI",@@ -563,16 +563,17 @@ describe("AI gateway control plane", () => {const result = await authenticateRequest({fastHashedSecretKey: createShaHash(gatewayKey.secretKey, env.SALT),apiFormat,+ model: "OpenAI second:openai/gpt-4o",});- expect(openAiPrimary.routingPriority).toBe(0);expect(result.connection).toEqual({- id: openAiPrimary.id,- name: "OpenAI first",+ id: openAiSecondary.id,+ name: "OpenAI second",provider: "openai",+ provider_model: "openai/gpt-4o",api_format: apiFormat,base_url: "https://api.openai.com/v1",- auth: { type: "Bearer", token: "sk-test-openai-primary" },+ auth: { type: "Bearer", token: "sk-test-openai-secondary" },});const ingestionToken = result.ingestion!.access_token;const ingestionClaims = jwtVerifier.verify({ token: ingestionToken });@@ -597,6 +598,132 @@ describe("AI gateway control plane", () => {expect(result.ingestion?.expires_at).toBe(ingestionClaims.exp);});+ it("routes unprefixed models by provider and rejects unknown models", async () => {+ configureGatewayJwtSigning();+ const { caller, org, project } = await prepare();+ await caller.aiGateway.updateConfig({+ orgId: org.id,+ defaultIngestionProjectId: project.id,+ ingestionMode: "FULL",+ });+ const openAiPrimary = await caller.aiGateway.createConnection({+ orgId: org.id,+ name: "OpenAI primary",+ provider: "OPENAI",+ credential: "sk-test-openai-primary",+ });+ const openAiSecondary = await caller.aiGateway.createConnection({+ orgId: org.id,+ name: "OpenAI secondary",+ provider: "OPENAI",+ credential: "sk-test-openai-secondary",+ });+ const anthropic = await caller.aiGateway.createConnection({+ orgId: org.id,+ name: "Anthropic",+ provider: "ANTHROPIC",+ credential: "sk-test-anthropic",+ });+ const gatewayKey = await caller.aiGateway.createApiKey({+ orgId: org.id,+ metadata: {},+ });+ const otherOrganization = await prepare();+ await otherOrganization.caller.aiGateway.createConnection({+ orgId: otherOrganization.org.id,+ name: "Unknown",+ provider: "OPENAI",+ credential: "sk-other-organization",+ });+ const params = {+ fastHashedSecretKey: createShaHash(gatewayKey.secretKey, env.SALT),+ apiFormat: "openai.responses" as const,+ };++ await expect(+ authenticateRequest({ ...params, model: "gpt-4o" }),+ ).resolves.toMatchObject({+ connection: {+ id: openAiPrimary.id,+ provider: "openai",+ provider_model: "gpt-4o",+ },+ });+ await expect(+ authenticateRequest({+ ...params,+ apiFormat: "anthropic.messages",+ model: "claude-haiku-4-5-20251001",+ }),+ ).resolves.toMatchObject({+ connection: {+ id: anthropic.id,+ provider: "anthropic",+ provider_model: "claude-haiku-4-5-20251001",+ },+ });+ await expect(+ authenticateRequest({+ ...params,+ model: "OpenAI secondary:gpt-4o",+ }),+ ).resolves.toMatchObject({+ connection: {+ id: openAiSecondary.id,+ provider: "openai",+ provider_model: "gpt-4o",+ },+ });+ await expect(+ authenticateRequest({ ...params, model: "random-model" }),+ ).rejects.toMatchObject({ status: 400 });+ await expect(+ authenticateRequest({ ...params, model: "Unknown:gpt-4o" }),+ ).rejects.toMatchObject({ status: 404 });+ });++ it("requires connection names to be unique within an organization", async () => {+ const { caller, org } = await prepare();+ await expect(+ caller.aiGateway.createConnection({+ orgId: org.id,+ name: "Production:Primary",+ provider: "OPENAI",+ credential: "sk-openai",+ }),+ ).rejects.toThrow("Connection names cannot contain a colon.");++ await caller.aiGateway.createConnection({+ orgId: org.id,+ name: "Production",+ provider: "OPENAI",+ credential: "sk-openai",+ });++ await expect(+ caller.aiGateway.createConnection({+ orgId: org.id,+ name: "Production",+ provider: "ANTHROPIC",+ credential: "sk-anthropic",+ }),+ ).rejects.toThrow("A provider credential with this name already exists.");++ const backup = await caller.aiGateway.createConnection({+ orgId: org.id,+ name: "Backup",+ provider: "ANTHROPIC",+ credential: "sk-anthropic",+ });+ await expect(+ caller.aiGateway.updateConnection({+ orgId: org.id,+ id: backup.id,+ name: "Production",+ }),+ ).rejects.toThrow("A provider credential with this name already exists.");+ });+it("changes ERROR only for credential auth failures and explicit recovery", async () => {const { caller, org, session } = await prepare();const connection = await caller.aiGateway.createConnection({@@ -787,6 +914,12 @@ describe("AI gateway control plane", () => {provider: "OPENAI",credential: "sk-test",});+ await caller.aiGateway.createConnection({+ orgId: org.id,+ name: "OpenAI backup",+ provider: "OPENAI",+ credential: "sk-test-backup",+ });const key = await caller.aiGateway.createApiKey({orgId: org.id,metadata: {},@@ -794,6 +927,7 @@ describe("AI gateway control plane", () => {const resolveParams = {fastHashedSecretKey: createShaHash(key.secretKey, env.SALT),apiFormat: "openai.responses" as const,+ model: "OpenAI:gpt-4o",};const lookup = vi.spyOn(prisma.gatewayApiKeyAssociation, "findFirst");@@ -802,20 +936,37 @@ describe("AI gateway control plane", () => {connection: { auth: { type: "Bearer", token: "sk-test" } },});}+ await expect(+ authenticateRequest({+ ...resolveParams,+ model: "OpenAI backup:gpt-4o",+ }),+ ).resolves.toMatchObject({+ connection: { auth: { type: "Bearer", token: "sk-test-backup" } },+ });// The second call must not reach Postgres: this endpoint runs on every LLM// request through the gateway.expect(lookup).toHaveBeenCalledOnce();- // Disabling a connection has to take effect now, not when the TTL expires.+ // Renaming a connection invalidates its old model slugs immediately.await caller.aiGateway.updateConnection({orgId: org.id,id: connection.id,- status: "DISABLED",+ name: "OpenAI renamed",});await expect(authenticateRequest(resolveParams)).rejects.toEqual(expect.objectContaining<Partial<GatewayResolveError>>({ status: 404 }),);expect(lookup).toHaveBeenCalledTimes(2);+ await expect(+ authenticateRequest({+ ...resolveParams,+ model: "OpenAI renamed:gpt-4o",+ }),+ ).resolves.toMatchObject({+ connection: { name: "OpenAI renamed" },+ });+ expect(lookup).toHaveBeenCalledTimes(2);lookup.mockRestore();});@@ -823,6 +974,7 @@ describe("AI gateway control plane", () => {const resolveParams = {fastHashedSecretKey: createShaHash(`sk-lf-${randomUUID()}`, env.SALT),apiFormat: "openai.responses" as const,+ model: "gpt-4o",};const lookup = vi.spyOn(prisma.gatewayApiKeyAssociation, "findFirst");@@ -855,7 +1007,7 @@ describe("AI gateway control plane", () => {Object.assign(env, { API_AUTH_MIGRATION: "enforce" });await expect(- authenticator.authenticateRequest(request),+ authenticator.authenticateModelsRequest(request),).resolves.toMatchObject({organizationId: org.id,});@@ -865,7 +1017,7 @@ describe("AI gateway control plane", () => {data: { systemRole: "INGEST", roleId: "system/INGEST" },});await expect(- authenticator.authenticateRequest(request),+ authenticator.authenticateModelsRequest(request),).rejects.toMatchObject({status: 403,message: "API key is not allowed to invoke the gateway",@@ -873,7 +1025,7 @@ describe("AI gateway control plane", () => {Object.assign(env, { API_AUTH_MIGRATION: "legacy" });await expect(- authenticator.authenticateRequest(request),+ authenticator.authenticateModelsRequest(request),).resolves.toMatchObject({organizationId: org.id,});@@ -930,20 +1082,27 @@ describe("AI gateway control plane", () => {authenticateRequest({fastHashedSecretKey: createShaHash(key.secretKey, env.SALT),apiFormat,+ model: "OpenAI:gpt-4o",}),).rejects.toEqual(expect.objectContaining<Partial<GatewayResolveError>>({ status: 403 }),);});- it("returns a canonical model catalog across enabled compatible connections", async () => {+ it("returns connection-prefixed models without consolidating connections", async () => {const { caller, org } = await prepare();await caller.aiGateway.createConnection({orgId: org.id,- name: "OpenAI",+ name: "OpenAI primary",provider: "OPENAI",credential: "sk-openai",});+ await caller.aiGateway.createConnection({+ orgId: org.id,+ name: "OpenAI backup",+ provider: "OPENAI",+ credential: "sk-openai-backup",+ });await caller.aiGateway.createConnection({orgId: org.id,name: "Anthropic",@@ -1001,7 +1160,7 @@ describe("AI gateway control plane", () => {const fastHashedSecretKey = createShaHash(key.secretKey, env.SALT);const openAiContext = await new GatewayApiKeyAuthenticator(prisma,- ).authenticateRequest({+ ).authenticateModelsRequest({fastHashedSecretKey,apiFormat: "openai.responses",});@@ -1013,14 +1172,28 @@ describe("AI gateway control plane", () => {expect(result).toEqual({data: [{- id: "a-model",+ id: "OpenAI backup:a-model",provider: "OPENAI",canonicalSlug: "a-model",displayName: "a-model",createdAt: "1970-01-01T00:00:10.000Z",},{- id: "shared-model",+ id: "OpenAI backup:shared-model",+ provider: "OPENAI",+ canonicalSlug: "shared-model",+ displayName: "shared-model",+ createdAt: "1970-01-01T00:00:15.000Z",+ },+ {+ id: "OpenAI primary:a-model",+ provider: "OPENAI",+ canonicalSlug: "a-model",+ displayName: "a-model",+ createdAt: "1970-01-01T00:00:10.000Z",+ },+ {+ id: "OpenAI primary:shared-model",provider: "OPENAI",canonicalSlug: "shared-model",displayName: "shared-model",@@ -1030,7 +1203,7 @@ describe("AI gateway control plane", () => {});const anthropicContext = await new GatewayApiKeyAuthenticator(prisma,- ).authenticateRequest({+ ).authenticateModelsRequest({fastHashedSecretKey,apiFormat: "anthropic.messages",});@@ -1041,14 +1214,14 @@ describe("AI gateway control plane", () => {expect(anthropicResult).toEqual({data: [{- id: "claude-a",+ id: "Anthropic:claude-a",provider: "ANTHROPIC",canonicalSlug: "claude-a",displayName: "Claude A",createdAt: "2026-01-01T00:00:00.000Z",},{- id: "claude-b",+ id: "Anthropic:claude-b",provider: "ANTHROPIC",canonicalSlug: "claude-b",displayName: "Claude B",@@ -1056,10 +1229,10 @@ describe("AI gateway control plane", () => {},],});- expect(fetcher).toHaveBeenCalledTimes(2);+ expect(fetcher).toHaveBeenCalledTimes(3);});- it("fails model discovery instead of returning a partial catalog", async () => {+ it("returns partial catalogs and fails only when every connection fails", async () => {const { caller, org } = await prepare();await caller.aiGateway.createConnection({orgId: org.id,@@ -1088,7 +1261,7 @@ describe("AI gateway control plane", () => {const context = await new GatewayApiKeyAuthenticator(prisma,- ).authenticateRequest({+ ).authenticateModelsRequest({fastHashedSecretKey: createShaHash(key.secretKey, env.SALT),apiFormat: "openai.chat-completions",});@@ -1097,6 +1270,26 @@ describe("AI gateway control plane", () => {context,apiFormat: "openai.chat-completions",}),+ ).resolves.toEqual({+ data: [+ {+ id: "OpenAI fallback:available-model",+ provider: "OPENAI",+ canonicalSlug: "available-model",+ displayName: "available-model",+ createdAt: "1970-01-01T00:00:10.000Z",+ },+ ],+ });++ const unavailableFetcher = vi+ .fn<typeof fetch>()+ .mockResolvedValue(new Response(null, { status: 500 }));+ await expect(+ new GatewayModelsService(prisma, unavailableFetcher, null).list({+ context,+ apiFormat: "openai.chat-completions",+ }),).rejects.toEqual(expect.objectContaining<Partial<GatewayResolveError>>({ status: 503 }),);diff --git a/web/src/__tests__/server/unit/ai-gateway-resolve-route.servertest.ts b/web/src/__tests__/server/unit/ai-gateway-resolve-route.servertest.tsindex d4783b8ba7..e5ce62da0d 100644--- a/web/src/__tests__/server/unit/ai-gateway-resolve-route.servertest.ts+++ b/web/src/__tests__/server/unit/ai-gateway-resolve-route.servertest.ts@@ -62,20 +62,49 @@ describe("POST /api/internal/ai-gateway/v1/resolve", () => {])("accepts api format %s and advances to authorization", async (format) => {// 401 rather than 400 proves the body passed the schema: the route only// reaches signature verification after parsing succeeds.- expect((await post({ apiFormat: format })).statusCode).toBe(401);- });-- it("accepts a model without using it for resolution", async () => {expect((await post({- apiFormat: "openai.responses",- model: "gpt-4o-mini",+ apiFormat: format,+ model: "production:model-with:separator",})).statusCode,).toBe(401);});+ it.each([+ { model: "gpt-4o-mini", apiFormat: "openai.responses" },+ { model: "claude-haiku-4-5-20251001", apiFormat: "anthropic.messages" },+ ])(+ "accepts inferred provider model $model and advances to authorization",+ async ({ model, apiFormat }) => {+ expect((await post({ apiFormat, model })).statusCode).toBe(401);+ },+ );++ it("rejects a missing model before authorization", async () => {+ expect((await post({ apiFormat: "openai.responses" })).statusCode).toBe(+ 400,+ );+ });++ it.each([+ {+ case: "unknown unprefixed model",+ body: { apiFormat: "openai.responses", model: "random-model" },+ },+ {+ case: "empty connection name",+ body: { apiFormat: "openai.responses", model: ":gpt-4o-mini" },+ },+ {+ case: "empty provider model",+ body: { apiFormat: "openai.responses", model: "production:" },+ },+ ])("rejects an invalid model slug: $case", async ({ body }) => {+ expect((await post(body)).statusCode).toBe(400);+ });+it("rejects a body that is not valid JSON", async () => {const { req, res } = createMocks<NextApiRequest, NextApiResponse>({method: "POST",diff --git a/web/src/__tests__/server/unit/gatewayModelSlug.servertest.ts b/web/src/__tests__/server/unit/gatewayModelSlug.servertest.tsnew file mode 100644index 0000000000..6973e94660--- /dev/null+++ b/web/src/__tests__/server/unit/gatewayModelSlug.servertest.ts@@ -0,0 +1,86 @@+import { describe, expect, it } from "vitest";++import { resolveGatewayModelRoute } from "@/src/features/ai-gateway/fns/gatewayModelSlug";++const connections = [+ { id: "openai-primary", name: "OpenAI primary", provider: "OPENAI" },+ { id: "anthropic", name: "Anthropic", provider: "ANTHROPIC" },+ { id: "openai-secondary", name: "OpenAI secondary", provider: "OPENAI" },+] as const;++describe("resolveGatewayModelRoute", () => {+ it.each([+ {+ model: "OpenAI secondary:gpt-4o",+ connectionId: "openai-secondary",+ providerModel: "gpt-4o",+ },+ {+ model: "OpenAI secondary:model:with:colon",+ connectionId: "openai-secondary",+ providerModel: "model:with:colon",+ },+ {+ model: "gpt-6-sol",+ connectionId: "openai-primary",+ providerModel: "gpt-6-sol",+ },+ {+ model: "ft:gpt-4o-mini-2024-07-18:acme::abc123",+ connectionId: "openai-primary",+ providerModel: "ft:gpt-4o-mini-2024-07-18:acme::abc123",+ },+ {+ model: "claude-sonnet-4-5",+ connectionId: "anthropic",+ providerModel: "claude-sonnet-4-5",+ },+ ])(+ "routes $model to $connectionId",+ ({ model, connectionId, providerModel }) => {+ const route = resolveGatewayModelRoute(model, connections);++ expect(route).not.toBeNull();+ expect(route?.connection?.id).toBe(connectionId);+ expect(route?.providerModel).toBe(providerModel);+ },+ );++ it.each(["gpt", "claude", "random-model", ":model", "OpenAI primary:"])(+ "rejects unsupported model %s",+ (model) => {+ expect(resolveGatewayModelRoute(model, connections)).toBeNull();+ },+ );++ it("returns a valid unresolved route for a missing connection", () => {+ expect(resolveGatewayModelRoute("Unknown:gpt-4o", connections)).toEqual({+ connection: null,+ providerModel: "gpt-4o",+ });+ expect(+ resolveGatewayModelRoute("claude-sonnet-4-5", connections.slice(0, 1)),+ ).toEqual({+ connection: null,+ providerModel: "claude-sonnet-4-5",+ });+ });++ it("prefers an explicit ft connection over the OpenAI fine-tune fallback", () => {+ const ftConnection = {+ id: "ft-connection",+ name: "ft",+ provider: "ANTHROPIC",+ } as const;++ expect(+ resolveGatewayModelRoute("ft:claude-sonnet-4-5", [+ ...connections,+ ftConnection,+ ]),+ ).toEqual({+ connection: ftConnection,+ providerModel: "claude-sonnet-4-5",+ });+ });+});diff --git a/web/src/__tests__/server/unit/gatewayResolveCache.servertest.ts b/web/src/__tests__/server/unit/gatewayResolveCache.servertest.tsindex c935496ebc..fd3f0b49b6 100644--- a/web/src/__tests__/server/unit/gatewayResolveCache.servertest.ts+++ b/web/src/__tests__/server/unit/gatewayResolveCache.servertest.ts@@ -19,7 +19,7 @@ const context: CachedResolveContext = {keyMetadata: {},ingestionProjectId: null,ingestionMode: null,- connection: null,+ connections: [],};describe("GatewayResolveCache", () => {diff --git a/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.clienttest.ts b/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.clienttest.tsindex 9500ec50b8..6e8f55d593 100644--- a/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.clienttest.ts+++ b/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.clienttest.ts@@ -17,17 +17,32 @@ describe("aggregateModels", () => {provider: "ANTHROPIC",} as Connection;const results = [- { connectionId: "second", success: true, models: ["a-model"] },- { connectionId: "first", success: true, models: ["z-model", "b-model"] },+ {+ connectionId: "second",+ success: true,+ models: ["a-model", "shared-model"],+ },+ {+ connectionId: "first",+ success: true,+ models: ["z-model", "b-model", "shared-model"],+ },] as Result[];expect(aggregateModels(results, [first]).map((model) => model.id)).toEqual([- "b-model",- "z-model",+ "First:b-model",+ "First:shared-model",+ "First:z-model",]);expect(aggregateModels(results, [first, second]).map((model) => model.id),- ).toEqual(["b-model", "z-model", "a-model"]);+ ).toEqual([+ "First:b-model",+ "First:shared-model",+ "First:z-model",+ "Second:a-model",+ "Second:shared-model",+ ]);});it("does not duplicate badges for connections repeated across pages", () => {diff --git a/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.ts b/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.tsindex 38433b41f1..fa819278e5 100644--- a/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.ts+++ b/web/src/features/ai-gateway/components/GatewayModelsPage/aggregateModels.ts@@ -1,4 +1,5 @@import type { GatewayProvider } from "@/src/features/ai-gateway/types/gatewayProvider";+import { formatGatewayModelSlug } from "@/src/features/ai-gateway/fns/gatewayModelSlug";import type { RouterOutputs } from "@/src/utils/api";type Connection = RouterOutputs["aiGateway"]["listConnections"]["data"][number];@@ -28,7 +29,7 @@ export function aggregateModels(const resultsByConnectionId = new Map(results.map((result) => [result.connectionId, result]),);- const models = new Map<string, ModelRow>();+ const models: ModelRow[] = [];const seenConnectionIds = new Set<string>();for (const connection of connections) {@@ -39,29 +40,18 @@ export function aggregateModels(for (const modelId of result.models.toSorted((left, right) =>left.localeCompare(right),)) {- const existing = models.get(modelId);const availableVia = {connectionId: connection.id,connectionName: connection.name,provider: connection.provider,};- if (existing) {- existing.availableVia.push(availableVia);- existing.apiFormats = [- ...new Set([- ...existing.apiFormats,- ...providerFormats[connection.provider],- ]),- ];- } else {- models.set(modelId, {- id: modelId,- availableVia: [availableVia],- apiFormats: [...providerFormats[connection.provider]],- });- }+ models.push({+ id: formatGatewayModelSlug(connection.name, modelId),+ availableVia: [availableVia],+ apiFormats: [...providerFormats[connection.provider]],+ });}}- return [...models.values()];+ return models;}diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/GatewayProvidersPage.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/GatewayProvidersPage.tsxindex 8184e95a64..8b404f5423 100644--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/GatewayProvidersPage.tsx+++ b/web/src/features/ai-gateway/components/GatewayProvidersPage/GatewayProvidersPage.tsx@@ -1,4 +1,4 @@-import { useRef, useState } from "react";+import { useState } from "react";import { Plus } from "lucide-react";import Header from "@/src/components/layouts/header";@@ -8,7 +8,7 @@ import { PaginationBar } from "@/src/components/design-system/PaginationBar/Pagiimport { Button } from "@/src/components/ui/button";import { Skeleton } from "@/src/components/ui/skeleton";import { ProviderDialogController } from "@/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderDialogController/ProviderDialogController";-import { ConnectedGatewayProvidersTable } from "@/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/ConnectedGatewayProvidersTable";+import { GatewayProvidersTable } from "@/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable";import { buildGatewayModelsUrl } from "@/src/features/ai-gateway/fns/gatewayUrls/buildGatewayModelsUrl";import type { GatewayConnection } from "@/src/features/ai-gateway/types/gatewayProvider";import { api, reportNonTrpcError } from "@/src/utils/api";@@ -30,10 +30,7 @@ export function GatewayProvidersPage({Record<string, number>>({});const [pageIndex, setPageIndex] = useState(0);- const [isReordering, setIsReordering] = useState(false);- const reorderLock = useRef(false);const utils = api.useUtils();- const reorder = api.aiGateway.reorderConnections.useMutation();const retry = api.aiGateway.retryConnection.useMutation();const remove = api.aiGateway.deleteConnection.useMutation();@@ -101,9 +98,6 @@ export function GatewayProvidersPage({const visiblePageIndex = Math.min(pageIndex, Math.max(0, pages.length - 1));const allConnections = pages.flatMap((page) => page.data);const connections = pages[visiblePageIndex]?.data ?? [];- const pageOffset = pages- .slice(0, visiblePageIndex)- .reduce((count, page) => count + page.data.length, 0);const modelCounts: Record<string, number | "loading"> = modelsQuery.isPending? Object.fromEntries(allConnections@@ -133,9 +127,8 @@ export function GatewayProvidersPage({}/><p className="text-muted-foreground text-sm">- Requests use the first compatible enabled credential in routing priority- order. Credentials are validated against the provider when they are- saved.+ Model names are prefixed with their connection name. Credentials are+ validated against the provider when they are saved.</p><ConfirmationDialogController<GatewayConnection>title="Delete provider credential"@@ -151,58 +144,12 @@ export function GatewayProvidersPage({<ProviderDialogController organizationId={organizationId}>{({ openEditDialog }) => (<div className="flex max-h-[60dvh] flex-col overflow-hidden">- <ConnectedGatewayProvidersTable+ <GatewayProvidersTableconnections={connections}- pageOffset={pageOffset}- previousConnectionId={allConnections[pageOffset - 1]?.id}- nextConnectionId={- allConnections[pageOffset + connections.length]?.id- }modelCounts={modelCounts}getModelsUrl={(connection) =>buildGatewayModelsUrl(organizationId, connection.id)}- canReorder={!connectionsQuery.hasNextPage && !isReordering}- onReorder={async (sourceId, targetId) => {- if (reorderLock.current) return false;- const sourceIndex = allConnections.findIndex(- (connection) => connection.id === sourceId,- );- const targetIndex = allConnections.findIndex(- (connection) => connection.id === targetId,- );- if (sourceIndex < 0 || targetIndex < 0) return false;- reorderLock.current = true;- setIsReordering(true);- const connectionIds = allConnections.map(- (connection) => connection.id,- );- const [movedId] = connectionIds.splice(sourceIndex, 1);- if (!movedId) return false;- connectionIds.splice(targetIndex, 0, movedId);- try {- await reorder.mutateAsync({- orgId: organizationId,- connectionIds,- });- await utils.aiGateway.listConnections.invalidate({- orgId: organizationId,- });- if (- Math.floor(sourceIndex / 50) !==- Math.floor(targetIndex / 50)- ) {- setPageIndex(Math.floor(targetIndex / 50));- }- return true;- } catch (error) {- reportNonTrpcError(error, "ai-gateway-providers");- return false;- } finally {- reorderLock.current = false;- setIsReordering(false);- }- }}onRetry={retryConnection}retryDisabled={retry.isPendingdiff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.stories.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.stories.tsxnew file mode 100644index 0000000000..1c19ea26f8--- /dev/null+++ b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.stories.tsx@@ -0,0 +1,31 @@+import { fn } from "storybook/test";++import preview from "@/.storybook/preview";+import { CredentialFields } from "./CredentialFields";++const meta = preview.meta({ component: CredentialFields });++const callbacks = {+ onNameChange: fn(),+ onCredentialChange: fn(),+};++export const Default = meta.story({+ args: {+ name: "Production",+ nameError: null,+ credential: "",+ credentialPlaceholder: "Enter secret key",+ ...callbacks,+ },+});++export const DuplicateNameError = meta.story({+ args: {+ name: "Production",+ nameError: "A provider credential with this name already exists.",+ credential: "",+ credentialPlaceholder: "Enter secret key",+ ...callbacks,+ },+});diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.tsxindex d93855616d..c2f336990a 100644--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.tsx+++ b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields.tsx@@ -1,14 +1,16 @@-import { Input } from "@/src/components/ui/input";+import { Input } from "@/src/components/design-system/Input/Input";import { Label } from "@/src/components/ui/label";export function CredentialFields({name,+ nameError,credential,credentialPlaceholder,onNameChange,onCredentialChange,}: {name: string;+ nameError: string | null;credential: string;credentialPlaceholder: string;onNameChange: (name: string) => void;@@ -18,25 +20,41 @@ export function CredentialFields({<><div><Label htmlFor="gateway-credential-name">Name</Label>- <Input- id="gateway-credential-name"- className="mt-1.5"- placeholder="Production"- value={name}- onChange={(event) => onNameChange(event.target.value)}- />+ <div className="mt-1.5 space-y-1">+ <Input+ id="gateway-credential-name"+ placeholder="Production"+ value={name}+ aria-invalid={Boolean(nameError)}+ aria-describedby={+ nameError ? "gateway-credential-name-error" : undefined+ }+ error={Boolean(nameError)}+ onChange={(event) => onNameChange(event.target.value)}+ />+ {nameError ? (+ <p+ id="gateway-credential-name-error"+ role="alert"+ className="text-destructive text-sm"+ >+ {nameError}+ </p>+ ) : null}+ </div></div><div className="ph-no-capture"><Label htmlFor="gateway-secret-key">Secret key</Label>- <Input- id="gateway-secret-key"- className="mt-1.5"- type="password"- autoComplete="new-password"- placeholder={credentialPlaceholder}- value={credential}- onChange={(event) => onCredentialChange(event.target.value)}- />+ <div className="mt-1.5">+ <Input+ id="gateway-secret-key"+ type="password"+ autoComplete="new-password"+ placeholder={credentialPlaceholder}+ value={credential}+ onChange={(event) => onCredentialChange(event.target.value)}+ />+ </div></div></>);diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/ConnectedGatewayProvidersTable.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/ConnectedGatewayProvidersTable.tsxdeleted file mode 100644index 6dfc1a5b4c..0000000000--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/ConnectedGatewayProvidersTable.tsx+++ /dev/null@@ -1,59 +0,0 @@-import { useState } from "react";--import { reorderProviderIds } from "@/src/features/ai-gateway/fns/providerReorder/reorderProviderIds";-import {- GatewayProvidersTable,- type GatewayProvidersTableProps,-} from "./GatewayProvidersTable";--type Props = Omit<GatewayProvidersTableProps, "onMove"> & {- onReorder: (sourceId: string, targetId: string) => Promise<boolean>;-};--export function ConnectedGatewayProvidersTable(props: Props) {- const { onReorder, ...tableProps } = props;- const serverOrderKey = JSON.stringify(- props.connections.map((connection) => connection.id),- );- const [optimisticOrder, setOptimisticOrder] = useState<{- base: string;- ids: string[];- } | null>(null);- const serverIds = props.connections.map((connection) => connection.id);- const sameMembers =- optimisticOrder &&- JSON.stringify([...optimisticOrder.ids].sort()) ===- JSON.stringify([...serverIds].sort());- const orderedIds =- optimisticOrder &&- sameMembers &&- (serverOrderKey === optimisticOrder.base ||- serverOrderKey === JSON.stringify(optimisticOrder.ids))- ? optimisticOrder.ids- : serverIds;- const connections = props.connections.toSorted(- (left, right) => orderedIds.indexOf(left.id) - orderedIds.indexOf(right.id),- );-- const onMove = async (sourceId: string, targetId: string) => {- const previousIds = orderedIds;- const nextIds = reorderProviderIds(previousIds, sourceId, targetId);- if (nextIds === previousIds) {- if (sourceId !== targetId) await onReorder(sourceId, targetId);- return;- }-- setOptimisticOrder({ base: serverOrderKey, ids: nextIds });- if (!(await onReorder(sourceId, targetId))) {- setOptimisticOrder(null);- }- };-- return (- <GatewayProvidersTable- {...tableProps}- connections={connections}- onMove={onMove}- />- );-}diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.clienttest.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.clienttest.tsxdeleted file mode 100644index 415404767a..0000000000--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.clienttest.tsx+++ /dev/null@@ -1,148 +0,0 @@-/* eslint-disable @repo/prefer-stories-over-client-tests */-import { fireEvent, render, screen, waitFor } from "@testing-library/react";-import { describe, expect, it, vi } from "vitest";--import type { GatewayConnection } from "@/src/features/ai-gateway/types/gatewayProvider";-import { ConnectedGatewayProvidersTable } from "./ConnectedGatewayProvidersTable";--const connections = ["Alpha", "Beta"].map((name, index) => ({- id: name.toLowerCase(),- name,- provider: "OPENAI" as const,- displaySecret: "sk-...test",- status: "ENABLED" as const,- organizationId: "org-1",- createdById: "user-1",- routingPriority: index,- createdAt: new Date("2026-09-08T12:00:00.000Z"),- updatedAt: new Date("2026-09-08T12:00:00.000Z"),-})) satisfies GatewayConnection[];--describe("gateway providers table", () => {- it("renders priority first and exposes a table actions menu", () => {- render(- <ConnectedGatewayProvidersTable- connections={connections}- pageOffset={0}- modelCounts={{}}- getModelsUrl={(connection) => `/models/${connection.id}`}- onRetry={() => {}}- onEdit={() => {}}- onDelete={() => {}}- canReorder- onReorder={async () => true}- />,- );-- expect(screen.getAllByRole("columnheader")[0]).toHaveTextContent(- "Priority",- );- expect(- screen.getAllByRole("button", { name: "Open actions menu" }),- ).toHaveLength(2);- });-- it("keeps a successful move through a stale refresh and shows absolute priorities", async () => {- const onReorder = vi.fn(async () => true);- const props = {- connections,- pageOffset: 50,- modelCounts: {},- getModelsUrl: (connection: GatewayConnection) =>- `/models/${connection.id}`,- onRetry: () => {},- onEdit: () => {},- onDelete: () => {},- canReorder: true,- onReorder,- };- const { rerender } = render(<ConnectedGatewayProvidersTable {...props} />);-- expect(screen.getAllByRole("row")[1]).toHaveTextContent("51");- fireEvent.click(- screen.getAllByRole("button", { name: "Move credential down" })[0]!,- );- await waitFor(() => expect(onReorder).toHaveBeenCalledOnce());- rerender(- <ConnectedGatewayProvidersTable- {...props}- connections={[...connections]}- />,- );- expect(screen.getAllByRole("row")[1]).toHaveTextContent("Beta");- expect(screen.getAllByRole("row")[1]).toHaveTextContent("51");-- rerender(- <ConnectedGatewayProvidersTable- {...props}- connections={[...connections].reverse()}- />,- );- expect(screen.getAllByRole("row")[1]).toHaveTextContent("Beta");- });-- it("allows a boundary credential to move to the adjacent page", async () => {- const onReorder = vi.fn(async () => true);- render(- <ConnectedGatewayProvidersTable- connections={connections}- pageOffset={50}- previousConnectionId="previous-page"- nextConnectionId="next-page"- modelCounts={{}}- getModelsUrl={(connection) => `/models/${connection.id}`}- onRetry={() => {}}- onEdit={() => {}}- onDelete={() => {}}- canReorder- onReorder={onReorder}- />,- );-- fireEvent.click(- screen.getAllByRole("button", { name: "Move credential up" })[0]!,- );- await waitFor(() =>- expect(onReorder).toHaveBeenCalledWith("alpha", "previous-page"),- );- fireEvent.click(- screen.getAllByRole("button", { name: "Move credential down" })[1]!,- );- await waitFor(() =>- expect(onReorder).toHaveBeenCalledWith("beta", "next-page"),- );- });-- it("adopts a new server order that differs from an optimistic move", async () => {- const gamma = { ...connections[0]!, id: "gamma", name: "Gamma" };- const onReorder = vi.fn(async () => true);- const props = {- pageOffset: 0,- modelCounts: {},- getModelsUrl: (connection: GatewayConnection) =>- `/models/${connection.id}`,- onRetry: () => {},- onEdit: () => {},- onDelete: () => {},- canReorder: true,- onReorder,- };- const { rerender } = render(- <ConnectedGatewayProvidersTable- {...props}- connections={[...connections, gamma]}- />,- );- fireEvent.click(- screen.getAllByRole("button", { name: "Move credential down" })[0]!,- );- await waitFor(() => expect(onReorder).toHaveBeenCalledOnce());- rerender(- <ConnectedGatewayProvidersTable- {...props}- connections={[gamma, ...connections]}- />,- );- expect(screen.getAllByRole("row")[1]).toHaveTextContent("Gamma");- });-});diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.stories.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.stories.tsxindex ba38edbecb..392cdf8e18 100644--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.stories.tsx+++ b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.stories.tsx@@ -1,15 +1,12 @@-import { useState, type ComponentProps } from "react";-import { Button } from "@/src/components/ui/button";-import { expect, fn, userEvent } from "storybook/test";+import type { ComponentProps } from "react";+import { fn } from "storybook/test";import preview from "@/.storybook/preview";import { GatewayProvidersTable } from "./GatewayProvidersTable";-import { ConnectedGatewayProvidersTable } from "./ConnectedGatewayProvidersTable";const meta = preview.meta({ component: GatewayProvidersTable });const onCredentialAction = fn();-const onReorder = fn(async () => true);const connections = [{@@ -63,18 +60,9 @@ const actions = {onRetry: (connection) => onCredentialAction(connection.id),onEdit: (connection) => onCredentialAction(connection.id),onDelete: (connection) => onCredentialAction(connection.id),- canReorder: true,- pageOffset: 0,- onMove: fn(),} satisfies Pick<ComponentProps<typeof GatewayProvidersTable>,- | "getModelsUrl"- | "onRetry"- | "onEdit"- | "onDelete"- | "canReorder"- | "onMove"- | "pageOffset"+ "getModelsUrl" | "onRetry" | "onEdit" | "onDelete">;export const OrderedCredentials = meta.story({@@ -103,79 +91,3 @@ export const Empty = meta.story({...actions,},});--export const MovesImmediately = meta.story({- name: "(Test) Moves Immediately",- args: {- connections,- modelCounts: {},- ...actions,- },- render: (args) => (- <ConnectedGatewayProvidersTable {...args} onReorder={onReorder} />- ),- play: async ({ canvas }) => {- onReorder.mockClear();- await userEvent.click(- canvas.getAllByRole("button", { name: "Move credential down" })[0]!,- );-- const rows = canvas.getAllByRole("row").slice(1);- await expect(rows[0]).toHaveTextContent("Fallback");- await expect(rows[1]).toHaveTextContent("Primary");- await expect(onReorder).toHaveBeenCalledWith(- "connection-openai",- "connection-anthropic",- );- },-});--export const ResetsWhenServerOrderChanges = meta.story({- name: "(Test) Resets When Server Order Changes",- args: {- connections: [- ...connections,- {- ...connections[0]!,- id: "connection-third",- name: "Third",- routingPriority: 2,- },- ],- modelCounts: {},- ...actions,- },- render: (args) => <ServerOrderExample {...args} />,- play: async ({ canvas }) => {- await userEvent.click(- canvas.getAllByRole("button", { name: "Move credential down" })[0]!,- );- await expect(canvas.getAllByRole("row")[1]).toHaveTextContent("Fallback");- await userEvent.click(- canvas.getByRole("button", { name: "Update server order" }),- );- await expect(canvas.getAllByRole("row")[1]).toHaveTextContent("Third");- await expect(canvas.getAllByRole("row")[2]).toHaveTextContent("Fallback");- await expect(canvas.getAllByRole("row")[3]).toHaveTextContent("Primary");- },-});--function ServerOrderExample(- args: ComponentProps<typeof GatewayProvidersTable>,-) {- const [serverConnections, setServerConnections] = useState(args.connections);- return (- <div>- <Button- onClick={() => setServerConnections([...args.connections].reverse())}- >- Update server order- </Button>- <ConnectedGatewayProvidersTable- {...args}- connections={serverConnections}- onReorder={async () => true}- />- </div>- );-}diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.tsxindex 453cf542e9..0b1ed89960 100644--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.tsx+++ b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/GatewayProvidersTable/GatewayProvidersTable.tsx@@ -1,13 +1,6 @@/* eslint-disable no-nested-ternary */import { useMemo } from "react";-import {- ArrowDown,- ArrowUp,- Pencil,- RefreshCw,- Route,- Trash2,-} from "lucide-react";+import { Pencil, RefreshCw, Route, Trash2 } from "lucide-react";import Link from "next/link";import { SiAnthropic, SiOpenai } from "react-icons/si";@@ -15,7 +8,6 @@ import { SettingsTable } from "@/src/components/SettingsTable/SettingsTable";import { createStatusTableColumn } from "@/src/components/design-system/table/columns/createStatusTableColumn";import { createTextTableColumn } from "@/src/components/design-system/table/columns/createTextTableColumn";import type { LangfuseColumnDef } from "@/src/components/table/types";-import { Button } from "@/src/components/ui/button";import { providerLabels } from "@/src/features/ai-gateway/constants/providerLabels";import type {GatewayConnection,@@ -24,9 +16,6 @@ import type {export type GatewayProvidersTableProps = {connections: GatewayConnection[];- pageOffset: number;- previousConnectionId?: string;- nextConnectionId?: string;modelCounts: Record<string, number | "loading">;getModelsUrl: (connection: GatewayConnection) => string;onRetry: (connection: GatewayConnection) => void;@@ -35,8 +24,6 @@ export type GatewayProvidersTableProps = {retryDisabled?: { reason: string };editDisabled?: { reason: string };deleteDisabled?: { reason: string };- canReorder: boolean;- onMove: (sourceId: string, targetId: string) => void;};const connectionStatus = {@@ -47,9 +34,6 @@ const connectionStatus = {export function GatewayProvidersTable({connections,- pageOffset,- previousConnectionId,- nextConnectionId,modelCounts,getModelsUrl,onRetry,@@ -58,67 +42,20 @@ export function GatewayProvidersTable({retryDisabled,editDisabled,deleteDisabled,- canReorder,- onMove,}: GatewayProvidersTableProps) {const columns = useMemo<LangfuseColumnDef<GatewayConnection>[]>(() => [- {- accessorKey: "routingPriority",- header: "Priority",- size: 110,- cell: ({ row }) => (- <div className="group flex items-center gap-1">- <span className="flex size-6 shrink-0 items-center justify-center font-mono">- {pageOffset + row.index + 1}- </span>- <div className="flex items-center opacity-40 transition-opacity group-hover:opacity-100 focus-within:opacity-100">- <Button- size="icon-xs"- variant="ghost"- disabled={- !canReorder || (row.index === 0 && !previousConnectionId)- }- aria-label="Move credential up"- onClick={() => {- const targetId =- connections[row.index - 1]?.id ?? previousConnectionId;- if (targetId) onMove(row.original.id, targetId);- }}- >- <ArrowUp className="icon-sm text-icon-foreground" />- </Button>- <Button- size="icon-xs"- variant="ghost"- disabled={- !canReorder ||- (row.index === connections.length - 1 && !nextConnectionId)- }- aria-label="Move credential down"- onClick={() => {- const targetId =- connections[row.index + 1]?.id ?? nextConnectionId;- if (targetId) onMove(row.original.id, targetId);- }}- >- <ArrowDown className="icon-sm text-icon-foreground" />- </Button>- </div>- </div>- ),- },+ createTextTableColumn<GatewayConnection>({+ accessorKey: "name",+ header: "Name",+ size: 180,+ }),{accessorKey: "provider",header: "Provider",size: 170,cell: ({ row }) => <ProviderName provider={row.original.provider} />,},- createTextTableColumn<GatewayConnection>({- accessorKey: "name",- header: "Name",- size: 180,- }),createTextTableColumn<GatewayConnection>({accessorKey: "displaySecret",header: "Credential",@@ -145,16 +82,7 @@ export function GatewayProvidersTable({),},],- [- canReorder,- connections,- pageOffset,- previousConnectionId,- nextConnectionId,- getModelsUrl,- modelCounts,- onMove,- ],+ [getModelsUrl, modelCounts],);return (diff --git a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderDialogController/ProviderDialogController.tsx b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderDialogController/ProviderDialogController.tsxindex 46522ceefa..b25946e57e 100644--- a/web/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderDialogController/ProviderDialogController.tsx+++ b/web/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderDialogController/ProviderDialogController.tsx@@ -1,5 +1,6 @@/* eslint-disable no-nested-ternary */import { useState, type ReactNode } from "react";+import { TriangleAlert } from "lucide-react";import { Alert } from "@/src/components/design-system/Alert/Alert";import { Button } from "@/src/components/ui/button";@@ -14,6 +15,7 @@ import { Input } from "@/src/components/ui/input";import { Label } from "@/src/components/ui/label";import { CredentialFields } from "@/src/features/ai-gateway/components/GatewayProvidersPage/components/CredentialFields";import { ProviderSelect } from "@/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderSelect";+import { DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR } from "@/src/features/ai-gateway/constants/gatewayConnection";import { providerLabels } from "@/src/features/ai-gateway/constants/providerLabels";import type {GatewayConnection,@@ -25,6 +27,9 @@ import {reportTrpcErrorWithoutToast,} from "@/src/utils/api";+const INVALID_CONNECTION_NAME_ERROR =+ "Connection names cannot contain a colon.";+export function ProviderDialogController({organizationId,connection,@@ -41,6 +46,7 @@ export function ProviderDialogController({connection?.provider ?? "OPENAI",);const [name, setName] = useState(connection?.name ?? "");+ const [nameError, setNameError] = useState<string | null>(null);const [credential, setCredential] = useState("");const utils = api.useUtils();const test = api.aiGateway.testConnection.useMutation({@@ -57,14 +63,28 @@ export function ProviderDialogController({});const isSaving = create.isPending || update.isPending;const isPending = test.isPending || isSaving;- const isError = test.isError || create.isError || update.isError;+ const saveError = create.error ?? update.error;+ const isNameMutationError =+ saveError?.message === DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR;+ const isError =+ test.isError ||+ ((create.isError || update.isError) && !isNameMutationError);+ const errorTitle = test.isError+ ? "Provider validation failed"+ : "Provider credential could not be saved";const reset = () => {setProvider("OPENAI");setName("");+ setNameError(null);setCredential("");};+ const updateName = (value: string) => {+ setName(value);+ setNameError(null);+ };+const testCredential = async (selectedConnection: GatewayConnection | null,) => {@@ -83,6 +103,12 @@ export function ProviderDialogController({selectedConnection: GatewayConnection | null,closeDialog: () => void,) => {+ if (name.includes(":")) {+ setNameError(INVALID_CONNECTION_NAME_ERROR);+ return;+ }+ setNameError(null);+try {if (selectedConnection) {await update.mutateAsync({@@ -107,6 +133,13 @@ export function ProviderDialogController({closeDialog();reset();} catch (error) {+ if (+ error instanceof Error &&+ error.message === DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR+ ) {+ setNameError(DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR);+ return;+ }reportNonTrpcError(error, "ai-gateway-providers");}};@@ -141,18 +174,28 @@ export function ProviderDialogController({)}<CredentialFieldsname={name}+ nameError={nameError}credential={credential}credentialPlaceholder={selectedConnection? "Leave blank to keep current key": "Enter secret key"}- onNameChange={setName}+ onNameChange={updateName}onCredentialChange={setCredential}/>+ {selectedConnection && name.trim() !== selectedConnection.name ? (+ <Alert variant="warning" icon={TriangleAlert}>+ <Alert.Title>Renaming changes model slugs</Alert.Title>+ <Alert.Description>+ Requests using model slugs with the old connection name will+ stop resolving.+ </Alert.Description>+ </Alert>+ ) : null}{isError ? (<Alert variant="destructive">- <Alert.Title>Provider validation failed</Alert.Title>+ <Alert.Title>{errorTitle}</Alert.Title><Alert.Description>{test.error?.message ??create.error?.message ??@@ -196,12 +239,14 @@ export function ProviderDialogController({openAddDialog: () => {setProvider(connection?.provider ?? "OPENAI");setName(connection?.name ?? "");+ setNameError(null);setCredential("");openDialog(connection ?? null);},openEditDialog: (selectedConnection) => {setProvider(selectedConnection.provider);setName(selectedConnection.name);+ setNameError(null);setCredential("");openDialog(selectedConnection);},diff --git a/web/src/features/ai-gateway/components/gatewayMutationErrors.clienttest.tsx b/web/src/features/ai-gateway/components/gatewayMutationErrors.clienttest.tsxdeleted file mode 100644index 684c64dbfb..0000000000--- a/web/src/features/ai-gateway/components/gatewayMutationErrors.clienttest.tsx+++ /dev/null@@ -1,379 +0,0 @@-import { TRPCClientError } from "@trpc/client";-import { fireEvent, render, screen, waitFor } from "@testing-library/react";--import { Button } from "@/src/components/ui/button";-import { TooltipProvider } from "@/src/components/ui/tooltip";-import type { GatewayConnection } from "@/src/features/ai-gateway/types/gatewayProvider";--const {- reportTrpcErrorWithoutToastMock,- reportNonTrpcErrorMock,- showErrorToastMock,- testConnectionMutateAsync,- createApiKeyMutateAsync,- mutationStates,-} = vi.hoisted(() => {- const mutationStates = {- testConnection: {- isError: false,- isPending: false,- error: null as Error | null,- },- createConnection: {- isError: false,- isPending: false,- error: null as Error | null,- },- updateConnection: {- isError: false,- isPending: false,- error: null as Error | null,- },- createApiKey: {- isError: false,- isPending: false,- error: null as Error | null,- },- };-- return {- reportTrpcErrorWithoutToastMock: vi.fn(),- reportNonTrpcErrorMock: vi.fn(),- showErrorToastMock: vi.fn(),- testConnectionMutateAsync: vi.fn(),- createApiKeyMutateAsync: vi.fn(),- mutationStates,- };-});--const providerValidationError = () =>- new TRPCClientError("Provider credential validation failed", {- result: {- error: {- message: "Provider credential validation failed",- code: -32600,- data: {- code: "BAD_REQUEST",- httpStatus: 400,- path: "aiGateway.createConnection",- },- },- },- });--const gatewayApiKeyError = () =>- new TRPCClientError("Gateway key limit reached", {- result: {- error: {- message: "Gateway key limit reached",- code: -32600,- data: {- code: "BAD_REQUEST",- httpStatus: 400,- path: "aiGateway.createApiKey",- },- },- },- });--vi.mock("@/src/utils/api", () => {- const React = require("react");-- const createStatefulMutation = <TArgs extends unknown[]>(- mutateImpl: (...args: TArgs) => Promise<unknown>,- stateKey: keyof typeof mutationStates,- ) => {- return (options?: { onError?: (error: unknown) => void }) => {- const [mutationState, setMutationState] = React.useState({- isError: false,- isPending: false,- error: null as Error | null,- });-- return {- ...mutationState,- mutateAsync: async (...args: TArgs) => {- setMutationState(- (current: {- isError: boolean;- isPending: boolean;- error: Error | null;- }) => ({ ...current, isPending: true }),- );- try {- return await mutateImpl(...args);- } catch (error) {- const normalizedError =- error instanceof Error ? error : new Error(String(error));- mutationStates[stateKey] = {- isError: true,- isPending: false,- error: normalizedError,- };- options?.onError?.(error);- setMutationState({- isError: true,- isPending: false,- error: normalizedError,- });- throw error;- }- },- };- };- };-- return {- api: {- useUtils: () => ({- aiGateway: {- listConnections: { invalidate: vi.fn() },- listApiKeys: { invalidate: vi.fn() },- },- }),- aiGateway: {- testConnection: {- useMutation: createStatefulMutation(- async (input: {- orgId: string;- provider: string;- credential: string;- }) => testConnectionMutateAsync(input),- "testConnection",- ),- },- createConnection: {- useMutation: createStatefulMutation(async () => {- throw providerValidationError();- }, "createConnection"),- },- updateConnection: {- useMutation: createStatefulMutation(async () => {- throw providerValidationError();- }, "updateConnection"),- },- createApiKey: {- useMutation: createStatefulMutation(- async (input: {- orgId: string;- note?: string;- metadata: Record<string, string>;- }) => {- createApiKeyMutateAsync(input);- throw gatewayApiKeyError();- },- "createApiKey",- ),- },- },- },- reportTrpcErrorWithoutToast: reportTrpcErrorWithoutToastMock,- reportNonTrpcError: reportNonTrpcErrorMock,- };-});--vi.mock("@/src/features/notifications/showErrorToast", () => ({- showErrorToast: showErrorToastMock,-}));--vi.mock("next/router", () => ({- useRouter: () => ({ query: {}, push: vi.fn() }),-}));--import { CreateGatewayApiKeyDialogController } from "@/src/features/ai-gateway/components/GatewayApiKeysPage/components/CreateGatewayApiKeyDialogController/CreateGatewayApiKeyDialogController";-import { ProviderDialogController } from "@/src/features/ai-gateway/components/GatewayProvidersPage/components/ProviderDialogController/ProviderDialogController";--const testConnection: GatewayConnection = {- id: "conn-1",- organizationId: "org-1",- name: "Production",- provider: "OPENAI",- status: "ENABLED",- displaySecret: "sk-...abcd",- createdById: "user-1",- routingPriority: 0,- createdAt: new Date("2026-09-04T12:00:00.000Z"),- updatedAt: new Date("2026-09-04T12:00:00.000Z"),-};--describe("gateway mutation local error handling", () => {- beforeAll(() => {- vi.stubGlobal(- "ResizeObserver",- class {- observe() {}- unobserve() {}- disconnect() {}- },- );- Element.prototype.scrollIntoView = vi.fn();- Element.prototype.hasPointerCapture = vi.fn();- });-- beforeEach(() => {- reportTrpcErrorWithoutToastMock.mockClear();- reportNonTrpcErrorMock.mockClear();- showErrorToastMock.mockClear();- testConnectionMutateAsync.mockClear();- createApiKeyMutateAsync.mockClear();- mutationStates.testConnection = {- isError: false,- isPending: false,- error: null,- };- mutationStates.createConnection = {- isError: false,- isPending: false,- error: null,- };- mutationStates.updateConnection = {- isError: false,- isPending: false,- error: null,- };- mutationStates.createApiKey = {- isError: false,- isPending: false,- error: null,- };- });-- it("tests a provider credential without saving", async () => {- render(- <ProviderDialogController organizationId="org-1">- {({ openAddDialog }) => (- <Button onClick={openAddDialog}>Add provider</Button>- )}- </ProviderDialogController>,- );-- fireEvent.click(screen.getByRole("button", { name: "Add provider" }));- fireEvent.change(screen.getByLabelText("Secret key"), {- target: { value: "sk-test-only" },- });- fireEvent.click(screen.getByRole("button", { name: "Test" }));-- await waitFor(() => {- expect(testConnectionMutateAsync).toHaveBeenCalledWith({- orgId: "org-1",- provider: "OPENAI",- credential: "sk-test-only",- });- });- expect(- screen.getByRole("heading", { name: "Add provider credential" }),- ).toBeInTheDocument();- });-- it("create provider dialog routes tRPC failures locally without a global toast", async () => {- render(- <ProviderDialogController organizationId="org-1">- {({ openAddDialog }) => (- <Button onClick={openAddDialog}>Add provider</Button>- )}- </ProviderDialogController>,- );-- fireEvent.click(screen.getByRole("button", { name: "Add provider" }));- fireEvent.change(screen.getByLabelText("Name"), {- target: { value: "Production" },- });- fireEvent.change(screen.getByLabelText("Secret key"), {- target: { value: "sk-invalid" },- });- fireEvent.click(screen.getByRole("button", { name: "Test and save" }));-- await waitFor(() => {- expect(reportTrpcErrorWithoutToastMock).toHaveBeenCalledWith(- expect.any(TRPCClientError),- "ai-gateway-providers",- );- });- expect(reportNonTrpcErrorMock).toHaveBeenCalledWith(- expect.any(TRPCClientError),- "ai-gateway-providers",- );- expect(showErrorToastMock).not.toHaveBeenCalled();- expect(- await screen.findByText("Provider credential validation failed"),- ).toBeInTheDocument();- });-- it("opens the selected credential through the dialog controller", () => {- render(- <ProviderDialogController organizationId="org-1">- {({ openEditDialog }) => (- <Button onClick={() => openEditDialog(testConnection)}>- Edit provider- </Button>- )}- </ProviderDialogController>,- );-- fireEvent.click(screen.getByRole("button", { name: "Edit provider" }));- expect(- screen.getByRole("heading", { name: "Edit provider credential" }),- ).toBeInTheDocument();- expect(screen.getByLabelText("Name")).toHaveValue(testConnection.name);- });-- it("edit provider dialog routes tRPC failures locally without a global toast", async () => {- render(- <ProviderDialogController- organizationId="org-1"- connection={testConnection}- >- {({ openAddDialog }) => (- <Button onClick={openAddDialog}>Edit provider</Button>- )}- </ProviderDialogController>,- );-- fireEvent.click(screen.getByRole("button", { name: "Edit provider" }));- fireEvent.change(screen.getByLabelText("Secret key"), {- target: { value: "sk-invalid" },- });- fireEvent.click(screen.getByRole("button", { name: "Test and save" }));-- await waitFor(() => {- expect(reportTrpcErrorWithoutToastMock).toHaveBeenCalledWith(- expect.any(TRPCClientError),- "ai-gateway-providers",- );- });- expect(showErrorToastMock).not.toHaveBeenCalled();- expect(- await screen.findByText("Provider credential validation failed"),- ).toBeInTheDocument();- });-- it("create gateway api key dialog routes tRPC failures locally and ignores blank metadata rows", async () => {- render(- <TooltipProvider>- <CreateGatewayApiKeyDialogController organizationId="org-1">- {({ openDialog }) => <Button onClick={openDialog}>Issue key</Button>}- </CreateGatewayApiKeyDialogController>- </TooltipProvider>,- );-- fireEvent.click(screen.getByRole("button", { name: "Issue key" }));- fireEvent.click(screen.getByRole("button", { name: "Create key" }));-- await waitFor(() => {- expect(reportTrpcErrorWithoutToastMock).toHaveBeenCalledWith(- expect.any(TRPCClientError),- "ai-gateway-api-keys",- );- });- expect(showErrorToastMock).not.toHaveBeenCalled();- expect(- await screen.findByText("Gateway key limit reached"),- ).toBeInTheDocument();- expect(createApiKeyMutateAsync).toHaveBeenCalledWith({- orgId: "org-1",- name: undefined,- metadata: {},- expiresAt: null,- });- });-});diff --git a/web/src/features/ai-gateway/constants/gatewayConnection.ts b/web/src/features/ai-gateway/constants/gatewayConnection.tsnew file mode 100644index 0000000000..5757e9b255--- /dev/null+++ b/web/src/features/ai-gateway/constants/gatewayConnection.ts@@ -0,0 +1,2 @@+export const DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR =+ "A provider credential with this name already exists.";diff --git a/web/src/features/ai-gateway/fns/gatewayModelSlug.ts b/web/src/features/ai-gateway/fns/gatewayModelSlug.tsnew file mode 100644index 0000000000..74ac65a482--- /dev/null+++ b/web/src/features/ai-gateway/fns/gatewayModelSlug.ts@@ -0,0 +1,75 @@+export function formatGatewayModelSlug(+ connectionName: string,+ providerModel: string,+) {+ return `${connectionName}:${providerModel}`;+}++export function resolveGatewayModelRoute<+ Connection extends GatewayRouteConnection,+>(+ model: string,+ connections: readonly Connection[],+): { connection: Connection | null; providerModel: string } | null {+ const modelSlug = parseGatewayModelSlug(model);+ const explicitConnection = modelSlug+ ? (connections.find(+ (connection) => connection.name === modelSlug.connectionName,+ ) ?? null)+ : null;+ if (modelSlug && explicitConnection) {+ return {+ connection: explicitConnection,+ providerModel: modelSlug.providerModel,+ };+ }++ if (model.startsWith("ft:")) {+ return {+ connection:+ connections.find((connection) => connection.provider === "OPENAI") ??+ null,+ providerModel: model,+ };+ }++ if (modelSlug) {+ return {+ connection: null,+ providerModel: modelSlug.providerModel,+ };+ }++ if (model.includes(":")) return null;++ let provider: "OPENAI" | "ANTHROPIC" | null = null;+ if (model.startsWith("gpt-")) provider = "OPENAI";+ if (model.startsWith("claude-")) provider = "ANTHROPIC";+ if (!provider) return null;++ return {+ connection:+ connections.find((connection) => connection.provider === provider) ??+ null,+ providerModel: model,+ };+}++function parseGatewayModelSlug(model: string) {+ const separatorIndex = model.indexOf(":");+ if (separatorIndex <= 0) {+ return null;+ }+ const providerModel = model.slice(separatorIndex + 1);+ if (!providerModel.trim()) return null;++ return {+ connectionName: model.slice(0, separatorIndex),+ providerModel,+ };+}++type GatewayRouteConnection = {+ name: string;+ provider: string;+};diff --git a/web/src/features/ai-gateway/fns/providerReorder/reorderProviderIds.ts b/web/src/features/ai-gateway/fns/providerReorder/reorderProviderIds.tsdeleted file mode 100644index 910d77946a..0000000000--- a/web/src/features/ai-gateway/fns/providerReorder/reorderProviderIds.ts+++ /dev/null@@ -1,17 +0,0 @@-export function reorderProviderIds(- ids: string[],- sourceId: string,- targetId: string,-) {- const sourceIndex = ids.indexOf(sourceId);- const targetIndex = ids.indexOf(targetId);- if (sourceIndex < 0 || targetIndex < 0 || sourceIndex === targetIndex) {- return ids;- }-- const nextIds = [...ids];- const [movedId] = nextIds.splice(sourceIndex, 1);- if (!movedId) return ids;- nextIds.splice(targetIndex, 0, movedId);- return nextIds;-}diff --git a/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts b/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.tsindex 48cbd0bfc7..a9d61bf8f7 100644--- a/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts+++ b/web/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator.ts@@ -13,6 +13,7 @@ import {gatewayProviders,providerSupportsApiFormat,} from "@/src/features/ai-gateway/server/provider/registry";+import { resolveGatewayModelRoute } from "@/src/features/ai-gateway/fns/gatewayModelSlug";import {type CachedResolveContext,GATEWAY_RESOLVE_KEY_NON_EXISTENT,@@ -20,7 +21,13 @@ import {} from "@/src/features/ai-gateway/server/resolve/gatewayResolveCache";import { GatewayResolveRepository } from "@/src/features/ai-gateway/server/resolve/gatewayResolveRepository";-export type GatewayApiKeyAuthContext = CachedResolveContext;+export type GatewayApiKeyAuthContext = Omit<+ CachedResolveContext,+ "connections"+> & {+ connection: CachedResolveContext["connections"][number] | null;+ providerModel: string | null;+};type GatewayApiKeyAuthenticatorConfig = {resolveCache?: GatewayResolveCache;@@ -41,13 +48,47 @@ export class GatewayApiKeyAuthenticator {async authenticateRequest(params: {fastHashedSecretKey: string;apiFormat: GatewayApiFormat;+ model: string;+ }): Promise<GatewayApiKeyAuthContext> {+ const context = await this.authenticateKey(params);+ const route = resolveGatewayModelRoute(params.model, context.connections);+ if (!route) {+ throw new GatewayControlPlaneError("Invalid gateway model", 400);+ }++ return {+ ...context,+ providerModel: route.providerModel,+ connection: route.connection,+ };+ }++ async authenticateModelsRequest(params: {+ fastHashedSecretKey: string;+ apiFormat: GatewayApiFormat;}): Promise<GatewayApiKeyAuthContext> {- const cached = await this.cache.get(params);+ const context = await this.authenticateKey(params);+ return {+ ...context,+ providerModel: null,+ connection: null,+ };+ }++ private async authenticateKey(params: {+ fastHashedSecretKey: string;+ apiFormat: GatewayApiFormat;+ }): Promise<CachedResolveContext> {+ const cacheParams = {+ fastHashedSecretKey: params.fastHashedSecretKey,+ apiFormat: params.apiFormat,+ };+ const cached = await this.cache.get(cacheParams);if (cached === GATEWAY_RESOLVE_KEY_NON_EXISTENT) {throw new GatewayControlPlaneError("Invalid gateway key", 401);}- const context = cached ?? (await this.loadAndCacheContext(params));+ const context = cached ?? (await this.loadAndCacheContext(cacheParams));// Re-check on cache hits so removing an organization from the allowlist// takes effect without waiting for the cache TTL.@@ -63,7 +104,7 @@ export class GatewayApiKeyAuthenticator {private async loadAndCacheContext(params: {fastHashedSecretKey: string;apiFormat: GatewayApiFormat;- }): Promise<GatewayApiKeyAuthContext> {+ }): Promise<CachedResolveContext> {const row = await this.repository.resolveContext({fastHashedSecretKey: params.fastHashedSecretKey,providers: supportedProviders(params.apiFormat),@@ -73,7 +114,8 @@ export class GatewayApiKeyAuthenticator {const organization = row?.apiKey.organization;if (!row || !organizationId || !organization) {await this.cache.set({- ...params,+ fastHashedSecretKey: params.fastHashedSecretKey,+ apiFormat: params.apiFormat,context: GATEWAY_RESOLVE_KEY_NON_EXISTENT,});throw new GatewayControlPlaneError("Invalid gateway key", 401);@@ -85,9 +127,7 @@ export class GatewayApiKeyAuthenticator {Boolean(project) &&!project?.deletedAt &&project?.orgId === organizationId;- const connection = organization.gatewayAiConnections[0];-- const context: GatewayApiKeyAuthContext = {+ const context: CachedResolveContext = {organizationId,apiKeyId: row.apiKeyId,keyMetadata: toKeyMetadata(row.metadata),@@ -95,14 +135,7 @@ export class GatewayApiKeyAuthenticator {? (gatewayConfig?.defaultIngestionProjectId ?? null): null,ingestionMode: gatewayConfig?.ingestionMode ?? null,- connection: connection- ? {- id: connection.id,- name: connection.name,- provider: connection.provider,- encryptedCredential: connection.encryptedCredential,- }- : null,+ connections: organization.gatewayAiConnections,};if (env.API_AUTH_MIGRATION === "enforce") {@@ -110,7 +143,8 @@ export class GatewayApiKeyAuthenticator {}await this.cache.set({- ...params,+ fastHashedSecretKey: params.fastHashedSecretKey,+ apiFormat: params.apiFormat,context,expiresAt: row.apiKey.expiresAt,});diff --git a/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.servertest.ts b/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.servertest.tsindex 69e6b83e6d..4727ab68ff 100644--- a/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.servertest.ts+++ b/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.servertest.ts@@ -108,7 +108,7 @@ describe("withGatewayResolveSignatureVerification", () => {async (secret) => {vi.useFakeTimers();vi.setSystemTime(now);- const body = '{ "apiFormat": "openai.responses" }\n';+ const body = '{ "apiFormat": "openai.responses", "model": "gpt-5" }\n';const req = request({authorization: " Bearer sk-gateway ",gatewayAuthorization: gatewayAuthorization({ secret }),@@ -124,10 +124,34 @@ describe("withGatewayResolveSignatureVerification", () => {res,fastHashedSecretKey: createShaHash("sk-gateway", "test-salt"),apiFormat: "openai.responses",+ model: "gpt-5",});},);+ it.each(["production: ", "production:\t"])(+ "rejects a whitespace-only provider model in %j",+ async (model) => {+ const handler = vi.fn();+ const res = response();++ await withGatewayResolveSignatureVerification(handler)(+ request({+ authorization: "Bearer sk-gateway",+ body: JSON.stringify({+ apiFormat: "openai.responses",+ model,+ }),+ }),+ res,+ );++ expect(res.status).toHaveBeenCalledWith(400);+ expect(res.json).toHaveBeenCalledWith({ error: "Invalid request" });+ expect(handler).not.toHaveBeenCalled();+ },+ );+it("rejects a missing Bearer token on the models endpoint", async () => {const handler = vi.fn();const res = response();@@ -179,7 +203,7 @@ describe("withGatewayResolveSignatureVerification", () => {secret: "current-service-secret",key: "sk-other-gateway",}),- body: '{"apiFormat":"openai.responses"}',+ body: '{"apiFormat":"openai.responses","model":"gpt-5"}',}),res,);@@ -191,7 +215,7 @@ describe("withGatewayResolveSignatureVerification", () => {it("rejects stale signatures and headers containing a key id", async () => {vi.useFakeTimers();vi.setSystemTime(now);- const body = '{"apiFormat":"openai.responses"}';+ const body = '{"apiFormat":"openai.responses","model":"gpt-5"}';const staleHeader = gatewayAuthorization({secret: "current-service-secret",timestamp: Math.floor(now.getTime() / 1000) - 301,diff --git a/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.ts b/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.tsindex 92fbaf9fd6..6637e47ffb 100644--- a/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.ts+++ b/web/src/features/ai-gateway/server/auth/gatewayRequestSignatureVerifier.ts@@ -8,6 +8,7 @@ import { env } from "@/src/env.mjs";import { verifyHmacSha256 } from "@/src/server/utils/hmac";import { createShaHash } from "@langfuse/shared/src/server/auth/apiKeys";+import { resolveGatewayModelRoute } from "@/src/features/ai-gateway/fns/gatewayModelSlug";import { GatewayApiFormatSchema, type GatewayApiFormat } from "../provider";import { GatewayControlPlaneError } from "@/src/features/ai-gateway/server/gatewayControlPlaneError";@@ -18,7 +19,9 @@ const SIGNATURE_MAX_AGE_SECONDS = 5 * 60;const resolveBodySchema = z.object({apiFormat: GatewayApiFormatSchema,- model: z.string().min(1).optional(),+ model: z+ .string()+ .refine((model) => resolveGatewayModelRoute(model, []) !== null),}).strict();const modelsQuerySchema = z@@ -38,6 +41,7 @@ type GatewayControlPlaneHandlerParams = {type GatewayResolveHandler = (params: GatewayControlPlaneHandlerParams & {apiFormat: GatewayApiFormat;+ model: string;},) => Promise<unknown>;@@ -104,7 +108,12 @@ export function withGatewayResolveSignatureVerification(handler: GatewayResolveHandler,) {return withGatewayControlPlaneAuth(- ({ body, ...params }) => handler({ ...params, apiFormat: body.apiFormat }),+ ({ body, ...params }) =>+ handler({+ ...params,+ apiFormat: body.apiFormat,+ model: body.model,+ }),resolveBodySchema,);}diff --git a/web/src/features/ai-gateway/server/provider/connection/gatewayProviderService.ts b/web/src/features/ai-gateway/server/provider/connection/gatewayProviderService.tsindex a84e4e8e0c..d16a077fc4 100644--- a/web/src/features/ai-gateway/server/provider/connection/gatewayProviderService.ts+++ b/web/src/features/ai-gateway/server/provider/connection/gatewayProviderService.ts@@ -1,7 +1,8 @@-import type {- GatewayConnectionStatus,- GatewayProvider,- PrismaClient,+import {+ Prisma,+ type GatewayConnectionStatus,+ type GatewayProvider,+ type PrismaClient,} from "@langfuse/shared/src/db";import { InvalidRequestError, LangfuseNotFoundError } from "@langfuse/shared";import { encrypt } from "@langfuse/shared/encryption";@@ -9,6 +10,7 @@ import { LLMAdapter, testModelCall } from "@langfuse/shared/src/server";import { getDisplaySecretKey } from "@langfuse/shared/src/server/auth/apiKeys";import { auditLog } from "@/src/features/audit-logs/server";+import { DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR } from "@/src/features/ai-gateway/constants/gatewayConnection";import type { OrgAuthedContext } from "@/src/server/api/trpc";import {type GatewayProviderName,@@ -18,11 +20,6 @@ import { invalidateGatewayResolveCacheForOrganization } from "@/src/features/ai-import { GatewayProviderRepository } from "./gatewayProviderRepository";import { GatewayModelCatalogService } from "../models/gatewayModelCatalogService";-type CredentialValidator = (params: {- provider: GatewayProviderName;- credential: string;-}) => Promise<void>;-export class GatewayProviderService {private readonly repository: GatewayProviderRepository;private readonly modelCatalogService: GatewayModelCatalogService;@@ -72,15 +69,20 @@ export class GatewayProviderService {provider: params.provider,credential: params.credential,});- const connection = await this.repository.createConnection({- organizationId: params.organizationId,- name: params.name,- provider: params.provider,- encryptedCredential: encrypt(params.credential),- displaySecret: getDisplaySecretKey(params.credential),- createdById: params.session.user.id,- status: "ENABLED",- });+ let connection;+ try {+ connection = await this.repository.createConnection({+ organizationId: params.organizationId,+ name: params.name,+ provider: params.provider,+ encryptedCredential: encrypt(params.credential),+ displaySecret: getDisplaySecretKey(params.credential),+ createdById: params.session.user.id,+ status: "ENABLED",+ });+ } catch (error) {+ rethrowConnectionMutationError(error);+ }await invalidateGatewayResolveCacheForOrganization(params.organizationId);await auditLog({@@ -124,18 +126,23 @@ export class GatewayProviderService {});}- const updated = await this.repository.updateConnection({- organizationId: params.organizationId,- id: params.id,- name: params.name,- status: params.credential ? "ENABLED" : params.status,- encryptedCredential: params.credential- ? encrypt(params.credential)- : undefined,- displaySecret: params.credential- ? getDisplaySecretKey(params.credential)- : undefined,- });+ let updated;+ try {+ updated = await this.repository.updateConnection({+ organizationId: params.organizationId,+ id: params.id,+ name: params.name,+ status: params.credential ? "ENABLED" : params.status,+ encryptedCredential: params.credential+ ? encrypt(params.credential)+ : undefined,+ displaySecret: params.credential+ ? getDisplaySecretKey(params.credential)+ : undefined,+ });+ } catch (error) {+ rethrowConnectionMutationError(error);+ }await this.modelCatalogService.clearModelCache(params.organizationId,params.id,@@ -222,6 +229,16 @@ export class GatewayProviderService {}}+function rethrowConnectionMutationError(error: unknown): never {+ if (+ error instanceof Prisma.PrismaClientKnownRequestError &&+ error.code === "P2002"+ ) {+ throw new InvalidRequestError(DUPLICATE_GATEWAY_CONNECTION_NAME_ERROR);+ }+ throw error;+}+async function validateGatewayCredential(params: {provider: GatewayProviderName;credential: string;@@ -257,3 +274,8 @@ async function validateGatewayCredential(params: {throw new InvalidRequestError("Provider credential validation failed");}}++type CredentialValidator = (params: {+ provider: GatewayProviderName;+ credential: string;+}) => Promise<void>;diff --git a/web/src/features/ai-gateway/server/provider/index.ts b/web/src/features/ai-gateway/server/provider/index.tsindex f3c53b8921..2c012a00cc 100644--- a/web/src/features/ai-gateway/server/provider/index.ts+++ b/web/src/features/ai-gateway/server/provider/index.ts@@ -10,4 +10,5 @@ export {type GatewayApiFormat,type GatewayMetadata,type GatewayProviderId,+ type GatewayResolveResponse,} from "./registry";diff --git a/web/src/features/ai-gateway/server/provider/models/gatewayModelsApiHandler.ts b/web/src/features/ai-gateway/server/provider/models/gatewayModelsApiHandler.tsindex 5a22db936a..5cb9d1553d 100644--- a/web/src/features/ai-gateway/server/provider/models/gatewayModelsApiHandler.ts+++ b/web/src/features/ai-gateway/server/provider/models/gatewayModelsApiHandler.ts@@ -23,7 +23,7 @@ export async function gatewayModelsApiHandler({try {const context = await new GatewayApiKeyAuthenticator(prisma,- ).authenticateRequest({ fastHashedSecretKey, apiFormat });+ ).authenticateModelsRequest({ fastHashedSecretKey, apiFormat });const result = await new GatewayModelsService(prisma).list({context,apiFormat,diff --git a/web/src/features/ai-gateway/server/provider/models/gatewayModelsRepository.ts b/web/src/features/ai-gateway/server/provider/models/gatewayModelsRepository.tsindex a914b7211e..aaa89c0ad4 100644--- a/web/src/features/ai-gateway/server/provider/models/gatewayModelsRepository.ts+++ b/web/src/features/ai-gateway/server/provider/models/gatewayModelsRepository.ts@@ -14,7 +14,7 @@ export class GatewayModelsRepository {status: "ENABLED",},orderBy: [{ routingPriority: "asc" }, { id: "asc" }],- select: { id: true },+ select: { id: true, name: true },});}}diff --git a/web/src/features/ai-gateway/server/provider/models/gatewayModelsService.ts b/web/src/features/ai-gateway/server/provider/models/gatewayModelsService.tsindex 4964157366..dc1855b14d 100644--- a/web/src/features/ai-gateway/server/provider/models/gatewayModelsService.ts+++ b/web/src/features/ai-gateway/server/provider/models/gatewayModelsService.ts@@ -1,8 +1,9 @@import type { GatewayProvider, PrismaClient } from "@langfuse/shared/src/db";-import { redis as defaultRedis } from "@langfuse/shared/src/server";+import { logger, redis as defaultRedis } from "@langfuse/shared/src/server";import type { Cluster, Redis } from "ioredis";import type { GatewayApiKeyAuthContext } from "@/src/features/ai-gateway/server/auth/gatewayApiKeyAuthenticator";+import { formatGatewayModelSlug } from "@/src/features/ai-gateway/fns/gatewayModelSlug";import { GatewayControlPlaneError } from "@/src/features/ai-gateway/server/gatewayControlPlaneError";import {type GatewayApiFormat,@@ -43,29 +44,45 @@ export class GatewayModelsService {) as GatewayProvider[],});const results = await Promise.all(- connections.map((connection) =>- this.modelCatalogService.getModelCatalog({+ connections.map(async (connection) => ({+ connection,+ catalog: await this.modelCatalogService.getModelCatalog({organizationId: params.context.organizationId,connectionId: connection.id,}),- ),+ })),);- if (results.some((result) => !result.success)) {+ const successfulResults = results.filter(+ (result) => result.catalog.success,+ );+ for (const { connection, catalog } of results) {+ if (!catalog.success) {+ logger.warn("Failed to discover models for AI Gateway connection", {+ organizationId: params.context.organizationId,+ connectionId: connection.id,+ connectionName: connection.name,+ error: catalog.error,+ });+ }+ }+ if (results.length > 0 && successfulResults.length === 0) {throw new GatewayControlPlaneError("Gateway model discovery is temporarily unavailable",503,);}- const modelsById = new Map<string, GatewayModelCatalogEntry>();- for (const result of results) {- if (!result.success) continue;- for (const model of result.models) {- if (!modelsById.has(model.id)) modelsById.set(model.id, model);- }- }return {- data: [...modelsById.values()].sort(compareModels),+ data: successfulResults+ .flatMap(({ connection, catalog }) =>+ catalog.success+ ? catalog.models.map((model) => ({+ ...model,+ id: formatGatewayModelSlug(connection.name, model.id),+ }))+ : [],+ )+ .sort(compareModels),};}}diff --git a/web/src/features/ai-gateway/server/provider/registry.ts b/web/src/features/ai-gateway/server/provider/registry.tsindex c9bb3b20df..8cebca0319 100644--- a/web/src/features/ai-gateway/server/provider/registry.ts+++ b/web/src/features/ai-gateway/server/provider/registry.ts@@ -27,6 +27,7 @@ export const GatewayResolveResponseSchema = zid: z.string(),name: z.string(),provider: z.enum(gatewayProviderIds),+ provider_model: z.string(),api_format: GatewayApiFormatSchema,base_url: z.url(),auth: z.union([@@ -60,6 +61,9 @@ export const GatewayResolveResponseSchema = z.strict(),}).strict();+export type GatewayResolveResponse = z.infer<+ typeof GatewayResolveResponseSchema+>;export const GatewayModelsResponseSchema = z.object({diff --git a/web/src/features/ai-gateway/server/resolve/gatewayResolveApiHandler.ts b/web/src/features/ai-gateway/server/resolve/gatewayResolveApiHandler.tsindex 7f7b08d7b1..374b6f6528 100644--- a/web/src/features/ai-gateway/server/resolve/gatewayResolveApiHandler.ts+++ b/web/src/features/ai-gateway/server/resolve/gatewayResolveApiHandler.ts@@ -13,16 +13,18 @@ export async function gatewayResolveApiHandler({res,fastHashedSecretKey,apiFormat,+ model,}: {req: NextApiRequest;res: NextApiResponse;fastHashedSecretKey: string;apiFormat: GatewayApiFormat;+ model: string;}) {try {const context = await new GatewayApiKeyAuthenticator(prisma,- ).authenticateRequest({ fastHashedSecretKey, apiFormat });+ ).authenticateRequest({ fastHashedSecretKey, apiFormat, model });const result = await new GatewayResolveService().resolve({context,apiFormat,diff --git a/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts b/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.tsindex f905c1be27..1d18b5dfe7 100644--- a/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts+++ b/web/src/features/ai-gateway/server/resolve/gatewayResolveCache.ts@@ -29,14 +29,14 @@ export const CachedResolveContext = z.object({keyMetadata: GatewayMetadataSchema,ingestionProjectId: z.string().nullable(),ingestionMode: z.enum(["USAGE", "FULL"]).nullable(),- connection: z- .object({+ connections: z.array(+ z.object({id: z.string(),name: z.string(),provider: z.enum(["OPENAI", "ANTHROPIC"]),encryptedCredential: z.string(),- })- .nullable(),+ }),+ ),});export type CachedResolveContext = z.infer<typeof CachedResolveContext>;@@ -172,11 +172,11 @@ export class GatewayResolveCache {if (!this.enabled) return;try {- await Promise.all(- gatewayApiFormats.map((apiFormat) =>+ await Promise.all([+ ...gatewayApiFormats.map((apiFormat) =>this.redis!.del(contextCacheKey(fastHashedSecretKey, apiFormat)),),- );+ ]);} catch (error) {logger.error("Error invalidating gateway resolve cache", error);}diff --git a/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts b/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.tsindex 22360c2897..31dcdeee8e 100644--- a/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts+++ b/web/src/features/ai-gateway/server/resolve/gatewayResolveRepository.ts@@ -42,7 +42,6 @@ export class GatewayResolveRepository {status: "ENABLED",},orderBy: { routingPriority: "asc" },- take: 1,select: {id: true,name: true,diff --git a/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts b/web/src/features/ai-gateway/server/resolve/resolveService.servertest.tsindex faf755d079..322b2644fd 100644--- a/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts+++ b/web/src/features/ai-gateway/server/resolve/resolveService.servertest.ts@@ -42,7 +42,10 @@ const serviceWith = (findFirst: () => Promise<unknown>) => {fastHashedSecretKey: string;apiFormat: GatewayApiFormat;}) => {- const context = await contextResolver.authenticateRequest(params);+ const context = await contextResolver.authenticateRequest({+ ...params,+ model: "Production OpenAI:gpt-4o",+ });return service.resolve({ context, apiFormat: params.apiFormat });},};@@ -98,6 +101,7 @@ describe("GatewayResolveService", () => {id: "connection-1",name: "Production OpenAI",provider: "openai",+ provider_model: "gpt-4o",api_format: "openai.responses",base_url: "https://api.openai.com/v1",auth: { type: "Bearer", token: "sk-test" },diff --git a/web/src/features/ai-gateway/server/resolve/resolveService.ts b/web/src/features/ai-gateway/server/resolve/resolveService.tsindex 6c774d0d3a..b59f0b784f 100644--- a/web/src/features/ai-gateway/server/resolve/resolveService.ts+++ b/web/src/features/ai-gateway/server/resolve/resolveService.ts@@ -14,6 +14,7 @@ import { GatewayControlPlaneError as GatewayResolveError } from "@/src/features/import {type GatewayApiFormat,type GatewayProviderId,+ type GatewayResolveResponse,getGatewayProviderDefinition,} from "@/src/features/ai-gateway/server/provider";@@ -66,7 +67,7 @@ export class GatewayResolveService {async resolve(params: {context: GatewayApiKeyAuthContext;apiFormat: GatewayApiFormat;- }) {+ }): Promise<GatewayResolveResponse> {return instrumentAsync({ name: "gateway-resolve" }, async (span) => {span.setAttribute("langfuse.organization.id",@@ -80,13 +81,12 @@ export class GatewayResolveService {403,);}- if (!context.connection) {+ if (!context.connection || !context.providerModel) {throw new GatewayResolveError("No enabled gateway connection supports this API format",404,);}-const provider = getGatewayProviderDefinition(context.connection.provider,);@@ -98,6 +98,7 @@ export class GatewayResolveService {name: context.connection.name,provider:context.connection.provider.toLowerCase() as GatewayProviderId,+ provider_model: context.providerModel,api_format: params.apiFormat,base_url: provider.baseUrl,auth:diff --git a/web/src/features/ai-gateway/server/router.ts b/web/src/features/ai-gateway/server/router.tsindex fd40068614..17633a0e30 100644--- a/web/src/features/ai-gateway/server/router.ts+++ b/web/src/features/ai-gateway/server/router.ts@@ -26,6 +26,14 @@ import {} from "./provider";const organizationInput = z.object({ orgId: z.string() });+const connectionNameSchema = z+ .string()+ .trim()+ .min(1)+ .max(200)+ .refine((name) => !name.includes(":"), {+ message: "Connection names cannot contain a colon.",+ });const paginatedOrganizationInput = organizationInput.extend({cursor: z.string().optional(),limit: z.number().int().min(1).max(100).default(50),@@ -100,7 +108,7 @@ export const aiGatewayRouter = createTRPCRouter({createConnection: protectedOrganizationProcedureWithoutTracing.input(organizationInput.extend({- name: z.string().trim().min(1).max(200),+ name: connectionNameSchema,provider: z.enum(GatewayProvider),credential: z.string().min(1),}),@@ -121,7 +129,7 @@ export const aiGatewayRouter = createTRPCRouter({.input(organizationInput.extend({id: z.string(),- name: z.string().trim().min(1).max(200).optional(),+ name: connectionNameSchema.optional(),credential: z.string().min(1).optional(),status: z.enum([