Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.
Python agentaianthropicclaude-codecompressioncontext-engineeringcontext-windowcursorfastapilangchainllmmcpopenaiprompt-engineeringproxypythonragtoken-optimizationtokenstypescript
fix(proxy): PR-D3 Bedrock observability + auth-mode integration
chopratejas90ef66213d85
diff --git a/Cargo.lock b/Cargo.lockindex 5dceb1ed8..510f05079 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -1826,6 +1826,7 @@ dependencies = ["hyper","hyper-util","pin-project-lite",+ "prometheus","proptest","reqwest","serde",@@ -2860,6 +2861,16 @@ version = "0.5.2"source = "registry+https://github.com/rust-lang/crates.io-index"checksum = "1a80800c0488c3a21695ea981a54918fbb37abf04f4d0720c453632255e2ff0e"+[[package]]+name = "parking_lot"+version = "0.12.5"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a"+dependencies = [+ "lock_api",+ "parking_lot_core",+]+[[package]]name = "parking_lot_core"version = "0.9.12"@@ -3027,6 +3038,20 @@ dependencies = ["syn",]+[[package]]+name = "prometheus"+version = "0.13.4"+source = "registry+https://github.com/rust-lang/crates.io-index"+checksum = "3d33c28a30771f7f96db69893f78b857f7450d7e0237e9c8fc6427a81bae7ed1"+dependencies = [+ "cfg-if",+ "fnv",+ "lazy_static",+ "memchr",+ "parking_lot",+ "thiserror 1.0.69",+]+[[package]]name = "proptest"version = "1.11.0"diff --git a/crates/headroom-proxy/Cargo.toml b/crates/headroom-proxy/Cargo.tomlindex c35039ecf..c462f4b74 100644--- a/crates/headroom-proxy/Cargo.toml+++ b/crates/headroom-proxy/Cargo.toml@@ -53,6 +53,12 @@ aws-smithy-runtime-api = { workspace = true }# transitively pulled in by `aws-smithy-*` (so this is not an# additional cold dep — promoting to a direct one for clarity).crc32fast = "1"+# Phase D PR-D3: Prometheus metrics for Bedrock observability. The+# `prometheus` crate's default-features pull in `protobuf`, which we+# don't need (we serve text-format scrapes only), so we disable+# defaults and re-enable nothing — pure registry + counter ++# histogram + text encoder is sufficient.+prometheus = { version = "0.13", default-features = false }[dev-dependencies]tower = { workspace = true, features = ["util"] }diff --git a/crates/headroom-proxy/src/bedrock/auth_mode_layer.rs b/crates/headroom-proxy/src/bedrock/auth_mode_layer.rsnew file mode 100644index 000000000..886724e20--- /dev/null+++ b/crates/headroom-proxy/src/bedrock/auth_mode_layer.rs@@ -0,0 +1,192 @@+//! Bedrock-route auth-mode middleware — Phase D PR-D3.+//!+//! # Why a dedicated middleware (vs inlining the classify call)?+//!+//! The Bedrock invoke + invoke-streaming handlers don't run through+//! `proxy::forward_http`'s catch-all (which is where Phase F PR-F1+//! classifies and stores the value in request extensions for+//! `/v1/messages`, `/v1/chat/completions`, `/v1/responses`). To+//! preserve the same downstream contract — Phase F PR-F2 and PR-F3+//! will read the [`AuthMode`] back out of `request.extensions()` and+//! gate compression policy on it — every Bedrock route applies this+//! middleware. The middleware:+//!+//! 1. Classifies the inbound headers via+//! [`headroom_core::auth_mode::classify`].+//! 2. **Asserts** the result is [`AuthMode::OAuth`] under the Bedrock+//! policy matrix. AWS SigV4 is an `Authorization` value that+//! isn't `Bearer ...` — F1's classifier already routes that to+//! OAuth (see `auth_mode.rs` decision rule 5). We additionally+//! catch the no-Authorization case (Bedrock SDK signs DOWNSTREAM+//! of our proxy on some setups, leaving the inbound request+//! unsigned), classify it, and if F1 returned `Payg` for it we+//! still force `OAuth` while emitting+//! `event = bedrock_auth_mode_unexpected` at WARN. Per the+//! realignment build constraint "no silent fallbacks", we+//! NEVER silently coerce — the divergence is loud.+//! 3. Stores the resolved [`AuthMode`] in `request.extensions()`+//! so downstream handlers can read it without re-classifying.+//! 4. Emits a structured info-level log with+//! `event = bedrock_auth_mode_classified` for ops correlation.+//!+//! # Performance+//!+//! `classify` is a pure function with one short owned `String` for+//! the lowercase UA copy; benched <10us. Inserting into request+//! extensions is `O(1)`. Total per-request overhead well under 1us+//! (excluding the classify call itself, which is shared with the+//! main proxy path).+//!+//! # Where the middleware is mounted+//!+//! See `crate::proxy::build_app` — the Bedrock router branch wraps+//! the three Bedrock POST routes+//! (`/model/:model_id/invoke`, `/converse`, and+//! `/invoke-with-response-stream`) with this layer using+//! `axum::middleware::from_fn`. The catch-all and the other+//! provider routes already classify in `forward_http`, so this+//! middleware does NOT apply to them.++use axum::body::Body;+use axum::extract::Request;+use axum::middleware::Next;+use axum::response::Response;++use headroom_core::auth_mode::{classify, AuthMode};++/// Inspect the inbound headers, classify the auth mode under+/// Bedrock policy (always [`AuthMode::OAuth`], with a loud WARN+/// when F1's classifier disagrees), and attach the resolved value+/// to `request.extensions()`.+///+/// Mounted as `axum::middleware::from_fn(classify_and_attach_auth_mode)`+/// so it composes with axum's standard router. The middleware is+/// infallible — it never short-circuits the request, never returns+/// an error response, and never panics. Worst case it logs and+/// proceeds.+pub async fn classify_and_attach_auth_mode(mut req: Request<Body>, next: Next) -> Response {+ let raw_classification = classify(req.headers());++ // Bedrock policy: always OAuth-equivalent. SigV4 IAM is an+ // OAuth-class signal under our policy matrix. F1 already+ // returns OAuth for non-Bearer Authorization (rule 5) and for+ // sk-ant-oat-* Bearer tokens (rule 2); the only paths that+ // wouldn't return OAuth are:+ //+ // - empty headers entirely (test setups, or AWS SDK that+ // signs after our hop) → F1 returns Payg.+ // - x-api-key set (Anthropic key on a Bedrock URL — wrong+ // surface, but possible in misconfigured setups) → F1+ // returns Payg.+ //+ // In both cases we coerce to OAuth (defence-in-depth: Bedrock+ // is OAuth-class regardless of the inbound surface) but log+ // loudly so operators see the misclassification. NO SILENT+ // FALLBACK — the divergence is the whole reason for the warn.+ let resolved = if raw_classification == AuthMode::OAuth {+ AuthMode::OAuth+ } else {+ tracing::warn!(+ event = "bedrock_auth_mode_unexpected",+ raw = raw_classification.as_str(),+ resolved = AuthMode::OAuth.as_str(),+ path = %req.uri().path(),+ "Bedrock route received headers that classified as non-OAuth; \+ coercing to OAuth per Bedrock policy and logging the divergence \+ so operators can investigate the source"+ );+ AuthMode::OAuth+ };++ tracing::info!(+ event = "bedrock_auth_mode_classified",+ mode = resolved.as_str(),+ raw = raw_classification.as_str(),+ path = %req.uri().path(),+ "bedrock route classified inbound auth mode"+ );++ req.extensions_mut().insert(resolved);+ next.run(req).await+}++#[cfg(test)]+mod tests {+ use super::*;+ use axum::body::Body;+ use axum::extract::Extension;+ use axum::http::{Request as HttpRequest, StatusCode};+ use axum::routing::post;+ use axum::Router;+ use http::HeaderValue;+ use tower::util::ServiceExt;++ /// Probe handler: returns the AuthMode it sees in extensions.+ async fn probe(Extension(auth_mode): Extension<AuthMode>) -> String {+ auth_mode.as_str().to_string()+ }++ fn router() -> Router {+ Router::new()+ .route("/probe", post(probe))+ .layer(axum::middleware::from_fn(classify_and_attach_auth_mode))+ }++ #[tokio::test]+ async fn empty_headers_classify_as_oauth_for_bedrock() {+ // No Authorization, no x-api-key, no UA — F1 returns Payg+ // by default. The middleware coerces to OAuth (with a+ // WARN logged at the call site) so downstream sees OAuth+ // and the policy matrix is consistent.+ let app = router();+ let req = HttpRequest::builder()+ .method("POST")+ .uri("/probe")+ .body(Body::empty())+ .unwrap();+ let resp = app.oneshot(req).await.unwrap();+ assert_eq!(resp.status(), StatusCode::OK);+ let body = axum::body::to_bytes(resp.into_body(), 64).await.unwrap();+ assert_eq!(&body[..], b"oauth");+ }++ #[tokio::test]+ async fn sigv4_authorization_classifies_as_oauth() {+ // Real Bedrock SDK does sign before reaching us in some+ // setups — `Authorization: AWS4-HMAC-SHA256 ...` is+ // routed to OAuth by F1's rule 5 directly. No coercion+ // needed; no WARN.+ let app = router();+ let mut req = HttpRequest::builder()+ .method("POST")+ .uri("/probe")+ .body(Body::empty())+ .unwrap();+ req.headers_mut().insert(+ "authorization",+ HeaderValue::from_static(+ "AWS4-HMAC-SHA256 Credential=AKIA.../20260101/us-east-1/bedrock/aws4_request",+ ),+ );+ let resp = app.oneshot(req).await.unwrap();+ let body = axum::body::to_bytes(resp.into_body(), 64).await.unwrap();+ assert_eq!(&body[..], b"oauth");+ }++ #[tokio::test]+ async fn x_api_key_inbound_is_coerced_to_oauth_loudly() {+ // x-api-key on the Bedrock surface is misconfigured but+ // possible. F1 returns Payg; we coerce to OAuth.+ let app = router();+ let mut req = HttpRequest::builder()+ .method("POST")+ .uri("/probe")+ .body(Body::empty())+ .unwrap();+ req.headers_mut()+ .insert("x-api-key", HeaderValue::from_static("sk-ant-api-fake"));+ let resp = app.oneshot(req).await.unwrap();+ let body = axum::body::to_bytes(resp.into_body(), 64).await.unwrap();+ assert_eq!(&body[..], b"oauth");+ }+}diff --git a/crates/headroom-proxy/src/bedrock/invoke.rs b/crates/headroom-proxy/src/bedrock/invoke.rsindex afa03b8f1..82d7c015a 100644--- a/crates/headroom-proxy/src/bedrock/invoke.rs+++ b/crates/headroom-proxy/src/bedrock/invoke.rs@@ -38,10 +38,10 @@//! unsigned.use std::net::SocketAddr;-use std::time::SystemTime;+use std::time::{Instant, SystemTime};use axum::body::Body;-use axum::extract::{ConnectInfo, Path, State};+use axum::extract::{ConnectInfo, Extension, Path, State};use axum::http::{HeaderMap, Method, StatusCode, Uri};use axum::response::{IntoResponse, Response};use bytes::Bytes;@@ -55,13 +55,49 @@ use crate::compression::{compress_anthropic_request, Outcome as AnthropicOutcome, PassthroughReason,};use crate::headers::filter_response_headers;+use crate::observability::{observe_bedrock_invoke_latency, record_bedrock_invoke};use crate::proxy::AppState;+// Phase F PR-F1 + PR-D3: the bedrock auth-mode layer+// (`classify_and_attach_auth_mode`) populates `request.extensions()`+// with `AuthMode` BEFORE this handler runs. We extract it via+// `Extension<AuthMode>` so the middleware-supplied value is the+// single source of truth — handler does NOT re-classify; that+// would risk drift from the middleware's resolution + WARN log.+use headroom_core::auth_mode::AuthMode;/// Anthropic vendor prefix as encoded in Bedrock model ids/// (`anthropic.claude-3-haiku-...`). Literal-match per project rule/// "no regexes for parsing the model ID".const ANTHROPIC_VENDOR_PREFIX: &str = "anthropic.";+/// RAII guard that observes the `bedrock_invoke_latency_seconds`+/// histogram on drop. Created at handler entry; observed when the+/// guard goes out of scope no matter how the handler exits. Owning+/// `String` rather than `&str` for the labels avoids capture-order+/// dramas with the borrow checker on early-return paths.+struct LatencyGuard {+ model: String,+ region: String,+ start: Instant,+}++impl LatencyGuard {+ fn start(model: &str, region: &str) -> Self {+ Self {+ model: model.to_string(),+ region: region.to_string(),+ start: Instant::now(),+ }+ }+}++impl Drop for LatencyGuard {+ fn drop(&mut self) {+ let elapsed = self.start.elapsed().as_secs_f64();+ observe_bedrock_invoke_latency(&self.model, &self.region, elapsed);+ }+}+/// AWS Bedrock Runtime DNS template. The `{}` placeholder is the/// region. Only used when `Config::bedrock_endpoint` is `None`.const BEDROCK_RUNTIME_HOST_TEMPLATE: &str = "bedrock-runtime.{region}.amazonaws.com";@@ -71,9 +107,11 @@ const BEDROCK_RUNTIME_HOST_TEMPLATE: &str = "bedrock-runtime.{region}.amazonaws./// Buffers the body so the live-zone compressor + SigV4 signer can/// inspect it. Both are required to be applied to the SAME byte slice/// — the signer hashes whatever the forwarder will actually send.+#[allow(clippy::too_many_arguments)] // axum extractors demand one argument per rolepub async fn handle_invoke(State(state): State<AppState>,ConnectInfo(client_addr): ConnectInfo<SocketAddr>,+ Extension(auth_mode): Extension<AuthMode>,Path(model_id): Path<String>,method: Method,uri: Uri,@@ -87,11 +125,32 @@ pub async fn handle_invoke(.map(|s| s.to_string()).unwrap_or_else(|| uuid::Uuid::new_v4().to_string());+ // PR-D3: latency stopwatch starts at handler entry (after+ // routing + middleware). The histogram observes wall-clock+ // time, so it captures upstream RTT + sign + compress as a+ // single number — operators can split contributions via the+ // `bedrock_*` structured-log timing fields if a slow path+ // shows up. Wrapped in a `LatencyGuard` so EVERY return path+ // (success, sign-failure, upstream-error, response-build error)+ // observes the histogram. RAII keeps the call site to one+ // line and rules out future regressions where someone adds a+ // new error path and forgets to instrument.+ let region = state.config.bedrock_region.clone();+ let _latency_guard = LatencyGuard::start(&model_id, ®ion);++ // PR-D3: count every invoke at handler entry (one per request,+ // before any error path can early-return). Pairs with the+ // structured log emitted below so operators can join the+ // counter with the trace by `request_id`.+ record_bedrock_invoke(&model_id, ®ion, auth_mode);+tracing::info!(event = "bedrock_invoke_received",request_id = %request_id,method = %method,model_id = %model_id,+ region = %region,+ auth_mode = auth_mode.as_str(),body_bytes = body.len(),"bedrock invoke route received request");diff --git a/crates/headroom-proxy/src/bedrock/invoke_streaming.rs b/crates/headroom-proxy/src/bedrock/invoke_streaming.rsindex daa9f683b..5da1667ec 100644--- a/crates/headroom-proxy/src/bedrock/invoke_streaming.rs+++ b/crates/headroom-proxy/src/bedrock/invoke_streaming.rs@@ -43,10 +43,10 @@use std::convert::Infallible;use std::net::SocketAddr;-use std::time::SystemTime;+use std::time::{Instant, SystemTime};use axum::body::Body;-use axum::extract::{ConnectInfo, Path, State};+use axum::extract::{ConnectInfo, Extension, Path, State};use axum::http::{HeaderMap, Method, StatusCode, Uri};use axum::response::{IntoResponse, Response};use bytes::Bytes;@@ -65,7 +65,14 @@ use crate::compression::{compress_anthropic_request, Outcome as AnthropicOutcome, PassthroughReason,};use crate::headers::filter_response_headers;+use crate::observability::{+ observe_bedrock_invoke_latency, record_bedrock_eventstream_message, record_bedrock_invoke,+};use crate::proxy::AppState;+// Phase F PR-F1 + PR-D3: pre-classified by `classify_and_attach_auth_mode`+// middleware on the bedrock router; we read it back via the+// `Extension<AuthMode>` extractor.+use headroom_core::auth_mode::AuthMode;/// Anthropic vendor prefix as encoded in Bedrock model ids.const ANTHROPIC_VENDOR_PREFIX: &str = "anthropic.";@@ -76,10 +83,41 @@ const BEDROCK_RUNTIME_HOST_TEMPLATE: &str = "bedrock-runtime.{region}.amazonaws./// Path action for the streaming route.const STREAMING_ACTION: &str = "invoke-with-response-stream";+/// RAII guard that observes the `bedrock_invoke_latency_seconds`+/// histogram on drop. Mirrors the [`crate::bedrock::invoke`] guard+/// — duplicated to avoid a cross-module type dependency for what+/// is fundamentally a 6-line struct. (When PR-D2 + PR-D3 settle,+/// the two handlers can share a `bedrock::common::LatencyGuard`+/// helper; that's a deferred refactor.)+struct LatencyGuard {+ model: String,+ region: String,+ start: Instant,+}++impl LatencyGuard {+ fn start(model: &str, region: &str) -> Self {+ Self {+ model: model.to_string(),+ region: region.to_string(),+ start: Instant::now(),+ }+ }+}++impl Drop for LatencyGuard {+ fn drop(&mut self) {+ let elapsed = self.start.elapsed().as_secs_f64();+ observe_bedrock_invoke_latency(&self.model, &self.region, elapsed);+ }+}+/// Axum POST handler for `/model/{model_id}/invoke-with-response-stream`.+#[allow(clippy::too_many_arguments)] // axum extractors demand one argument per rolepub async fn handle_invoke_streaming(State(state): State<AppState>,ConnectInfo(client_addr): ConnectInfo<SocketAddr>,+ Extension(auth_mode): Extension<AuthMode>,Path(model_id): Path<String>,method: Method,uri: Uri,@@ -93,11 +131,22 @@ pub async fn handle_invoke_streaming(.map(|s| s.to_string()).unwrap_or_else(|| uuid::Uuid::new_v4().to_string());+ // PR-D3: latency stopwatch + invoke counter at handler entry.+ // RAII guard observes the histogram regardless of which return+ // path the handler takes. Per-EventStream-message metrics are+ // recorded inside `translate_stream` once the upstream response+ // starts arriving.+ let region = state.config.bedrock_region.clone();+ let _latency_guard = LatencyGuard::start(&model_id, ®ion);+ record_bedrock_invoke(&model_id, ®ion, auth_mode);+tracing::info!(event = "bedrock_invoke_streaming_received",request_id = %request_id,method = %method,model_id = %model_id,+ region = %region,+ auth_mode = auth_mode.as_str(),body_bytes = body.len(),"bedrock invoke-with-response-stream route received request");@@ -344,6 +393,8 @@ pub async fn handle_invoke_streaming(upstream_stream,state.config.bedrock_validate_eventstream_crc,request_id.clone(),+ model_id.clone(),+ region.clone(),);let translated = tee_to_anthropic_state(translated, request_id.clone());let body_out = Body::from_stream(translated);@@ -366,6 +417,8 @@ fn translate_stream<S>(upstream: S,validate_crc: bool,request_id: String,+ model_id: String,+ region: String,) -> impl Stream<Item = Result<Bytes, std::io::Error>>whereS: Stream<Item = Result<Bytes, std::io::Error>> + Send + 'static,@@ -377,205 +430,247 @@ whereparser = parser.with_crc_validation(CrcValidation::No);}let upstream: ByteStream = Box::pin(upstream);- let init = (parser, upstream, false, request_id);- stream::unfold(init, |(mut parser, mut upstream, mut done, request_id)| {- Box::pin(async move {- if done {- return None;- }- // First, drain any complete messages already in the- // parser's buffer (bytes from the previous chunk).- loop {- match parser.next_message() {- Ok(Some(msg)) => match translate_message(&msg, OutputMode::Sse) {- Ok(TranslateOutcome::Emit(frame)) => {- tracing::debug!(- event = "bedrock_eventstream_message",- request_id = %request_id,- event_type = msg.event_type().unwrap_or(""),- payload_bytes = msg.payload.len(),- "translated bedrock eventstream message"- );- return Some((Ok(frame), (parser, upstream, false, request_id)));- }- Ok(TranslateOutcome::Skip { event_type }) => {- tracing::warn!(- event = "bedrock_eventstream_unknown_event_type",- request_id = %request_id,- event_type = %event_type,- "skipping unknown bedrock eventstream message"- );- // Loop and try the next message in the buffer.- continue;- }- Err(TranslateError::UpstreamException { payload_preview }) => {+ // Bundle the per-stream identifiers we thread through every+ // unfold step. `unfold` only allows a single state value, so+ // grouping these into a tuple of owned Strings keeps the+ // closure readable.+ let init = (parser, upstream, false, request_id, model_id, region);+ stream::unfold(+ init,+ |(mut parser, mut upstream, mut done, request_id, model_id, region)| {+ Box::pin(async move {+ if done {+ return None;+ }+ // First, drain any complete messages already in the+ // parser's buffer (bytes from the previous chunk).+ loop {+ match parser.next_message() {+ Ok(Some(msg)) => match translate_message(&msg, OutputMode::Sse) {+ Ok(TranslateOutcome::Emit(frame)) => {+ // PR-D3: per-message Prometheus metric.+ // The label `event_type` is bounded by+ // AWS's documented vocabulary (chunk,+ // metadata, exception variants); not+ // customer-controlled.+ let event_type = msg.event_type().unwrap_or("unknown").to_string();+ record_bedrock_eventstream_message(&model_id, ®ion, &event_type);+ tracing::debug!(+ event = "bedrock_eventstream_message",+ request_id = %request_id,+ event_type = %event_type,+ payload_bytes = msg.payload.len(),+ "translated bedrock eventstream message"+ );+ return Some((+ Ok(frame),+ (parser, upstream, false, request_id, model_id, region),+ ));+ }+ Ok(TranslateOutcome::Skip { event_type }) => {+ tracing::warn!(+ event = "bedrock_eventstream_unknown_event_type",+ request_id = %request_id,+ event_type = %event_type,+ "skipping unknown bedrock eventstream message"+ );+ // Loop and try the next message in the buffer.+ continue;+ }+ Err(TranslateError::UpstreamException { payload_preview }) => {+ tracing::warn!(+ event = "bedrock_eventstream_upstream_exception",+ request_id = %request_id,+ payload_preview = %payload_preview,+ "bedrock eventstream upstream exception"+ );+ // Emit the exception as an Anthropic-shape+ // SSE error frame so the client sees it.+ let json = serde_json::json!({+ "type": "error",+ "error": {+ "type": "bedrock_upstream_exception",+ "message": payload_preview,+ }+ })+ .to_string();+ let mut frame = Vec::with_capacity(json.len() + 32);+ frame.extend_from_slice(b"event: error\ndata: ");+ frame.extend_from_slice(json.as_bytes());+ frame.extend_from_slice(b"\n\n");+ return Some((+ Ok(Bytes::from(frame)),+ (parser, upstream, true, request_id, model_id, region),+ ));+ }+ Err(TranslateError::MissingEventType) => {+ tracing::warn!(+ event = "bedrock_eventstream_missing_event_type",+ request_id = %request_id,+ "bedrock eventstream message missing :event-type; emitting error frame"+ );+ let frame = error_sse_frame(+ "bedrock_eventstream_missing_event_type",+ "Bedrock message missing :event-type header",+ );+ return Some((+ Ok(frame),+ (parser, upstream, true, request_id, model_id, region),+ ));+ }+ },+ Ok(None) => break,+ Err(parse_err) => {+ let event_name = match &parse_err {+ ParseError::PreludeCrcMismatch { .. }+ | ParseError::MessageCrcMismatch { .. } => {+ "bedrock_eventstream_crc_mismatch"+ }+ _ => "bedrock_eventstream_parse_failed",+ };tracing::warn!(- event = "bedrock_eventstream_upstream_exception",+ event = event_name,request_id = %request_id,- payload_preview = %payload_preview,- "bedrock eventstream upstream exception"+ error = %parse_err,+ "bedrock eventstream parse failure; closing translated stream");- // Emit the exception as an Anthropic-shape- // SSE error frame so the client sees it.- let json = serde_json::json!({- "type": "error",- "error": {- "type": "bedrock_upstream_exception",- "message": payload_preview,- }- })- .to_string();- let mut frame = Vec::with_capacity(json.len() + 32);- frame.extend_from_slice(b"event: error\ndata: ");- frame.extend_from_slice(json.as_bytes());- frame.extend_from_slice(b"\n\n");+ let frame = error_sse_frame(event_name, &parse_err.to_string());return Some((- Ok(Bytes::from(frame)),- (parser, upstream, true, request_id),+ Ok(frame),+ (parser, upstream, true, request_id, model_id, region),));}- Err(TranslateError::MissingEventType) => {- tracing::warn!(- event = "bedrock_eventstream_missing_event_type",- request_id = %request_id,- "bedrock eventstream message missing :event-type; emitting error frame"- );- let frame = error_sse_frame(- "bedrock_eventstream_missing_event_type",- "Bedrock message missing :event-type header",- );- return Some((Ok(frame), (parser, upstream, true, request_id)));- }- },- Ok(None) => break,- Err(parse_err) => {- let event_name = match &parse_err {- ParseError::PreludeCrcMismatch { .. }- | ParseError::MessageCrcMismatch { .. } => {- "bedrock_eventstream_crc_mismatch"- }- _ => "bedrock_eventstream_parse_failed",- };- tracing::warn!(- event = event_name,- request_id = %request_id,- error = %parse_err,- "bedrock eventstream parse failure; closing translated stream"- );- let frame = error_sse_frame(event_name, &parse_err.to_string());- return Some((Ok(frame), (parser, upstream, true, request_id)));}}- }- // Buffer drained; pull the next chunk from upstream.- loop {- match upstream.next().await {- Some(Ok(chunk)) => {- parser.push(&chunk);- // Loop back through the parser to emit any- // newly-complete messages.- match parser.next_message() {- Ok(Some(msg)) => match translate_message(&msg, OutputMode::Sse) {- Ok(TranslateOutcome::Emit(frame)) => {- tracing::debug!(- event = "bedrock_eventstream_message",- request_id = %request_id,- event_type = msg.event_type().unwrap_or(""),- payload_bytes = msg.payload.len(),- "translated bedrock eventstream message"- );- return Some((- Ok(frame),- (parser, upstream, false, request_id),- ));- }- Ok(TranslateOutcome::Skip { event_type }) => {+ // Buffer drained; pull the next chunk from upstream.+ loop {+ match upstream.next().await {+ Some(Ok(chunk)) => {+ parser.push(&chunk);+ // Loop back through the parser to emit any+ // newly-complete messages.+ match parser.next_message() {+ Ok(Some(msg)) => match translate_message(&msg, OutputMode::Sse) {+ Ok(TranslateOutcome::Emit(frame)) => {+ // PR-D3: per-message Prometheus+ // metric (mirror of the parser-buffer+ // drain branch above).+ let event_type =+ msg.event_type().unwrap_or("unknown").to_string();+ record_bedrock_eventstream_message(+ &model_id,+ ®ion,+ &event_type,+ );+ tracing::debug!(+ event = "bedrock_eventstream_message",+ request_id = %request_id,+ event_type = %event_type,+ payload_bytes = msg.payload.len(),+ "translated bedrock eventstream message"+ );+ return Some((+ Ok(frame),+ (parser, upstream, false, request_id, model_id, region),+ ));+ }+ Ok(TranslateOutcome::Skip { event_type }) => {+ tracing::warn!(+ event = "bedrock_eventstream_unknown_event_type",+ request_id = %request_id,+ event_type = %event_type,+ "skipping unknown bedrock eventstream message"+ );+ // Continue draining the parser /+ // pulling more chunks.+ continue;+ }+ Err(TranslateError::UpstreamException { payload_preview }) => {+ let json = serde_json::json!({+ "type": "error",+ "error": {+ "type": "bedrock_upstream_exception",+ "message": payload_preview,+ }+ })+ .to_string();+ let mut frame = Vec::with_capacity(json.len() + 32);+ frame.extend_from_slice(b"event: error\ndata: ");+ frame.extend_from_slice(json.as_bytes());+ frame.extend_from_slice(b"\n\n");+ return Some((+ Ok(Bytes::from(frame)),+ (parser, upstream, true, request_id, model_id, region),+ ));+ }+ Err(TranslateError::MissingEventType) => {+ let frame = error_sse_frame(+ "bedrock_eventstream_missing_event_type",+ "Bedrock message missing :event-type header",+ );+ return Some((+ Ok(frame),+ (parser, upstream, true, request_id, model_id, region),+ ));+ }+ },+ Ok(None) => continue,+ Err(parse_err) => {+ let event_name = match &parse_err {+ ParseError::PreludeCrcMismatch { .. }+ | ParseError::MessageCrcMismatch { .. } => {+ "bedrock_eventstream_crc_mismatch"+ }+ _ => "bedrock_eventstream_parse_failed",+ };tracing::warn!(- event = "bedrock_eventstream_unknown_event_type",+ event = event_name,request_id = %request_id,- event_type = %event_type,- "skipping unknown bedrock eventstream message"+ error = %parse_err,+ "bedrock eventstream parse failure");- // Continue draining the parser /- // pulling more chunks.- continue;- }- Err(TranslateError::UpstreamException { payload_preview }) => {- let json = serde_json::json!({- "type": "error",- "error": {- "type": "bedrock_upstream_exception",- "message": payload_preview,- }- })- .to_string();- let mut frame = Vec::with_capacity(json.len() + 32);- frame.extend_from_slice(b"event: error\ndata: ");- frame.extend_from_slice(json.as_bytes());- frame.extend_from_slice(b"\n\n");+ let frame = error_sse_frame(event_name, &parse_err.to_string());return Some((- Ok(Bytes::from(frame)),- (parser, upstream, true, request_id),+ Ok(frame),+ (parser, upstream, true, request_id, model_id, region),));}- Err(TranslateError::MissingEventType) => {- let frame = error_sse_frame(- "bedrock_eventstream_missing_event_type",- "Bedrock message missing :event-type header",- );- return Some((Ok(frame), (parser, upstream, true, request_id)));- }- },- Ok(None) => continue,- Err(parse_err) => {- let event_name = match &parse_err {- ParseError::PreludeCrcMismatch { .. }- | ParseError::MessageCrcMismatch { .. } => {- "bedrock_eventstream_crc_mismatch"- }- _ => "bedrock_eventstream_parse_failed",- };- tracing::warn!(- event = event_name,- request_id = %request_id,- error = %parse_err,- "bedrock eventstream parse failure"- );- let frame = error_sse_frame(event_name, &parse_err.to_string());- return Some((Ok(frame), (parser, upstream, true, request_id)));}}- }- Some(Err(e)) => {- tracing::warn!(- event = "bedrock_eventstream_upstream_io_error",- request_id = %request_id,- error = %e,- "upstream io error mid-stream"- );- return Some((Err(e), (parser, upstream, true, request_id)));- }- None => {- // End of upstream stream. If buffered bytes- // remain that did not parse into a message,- // log loudly — we are NOT silently dropping- // them.- if parser.buffered_len() > 0 {+ Some(Err(e)) => {tracing::warn!(- event = "bedrock_eventstream_truncated",+ event = "bedrock_eventstream_upstream_io_error",request_id = %request_id,- buffered_bytes = parser.buffered_len(),- "upstream stream ended with un-parseable trailing bytes"+ error = %e,+ "upstream io error mid-stream");+ return Some((+ Err(e),+ (parser, upstream, true, request_id, model_id, region),+ ));+ }+ None => {+ // End of upstream stream. If buffered bytes+ // remain that did not parse into a message,+ // log loudly — we are NOT silently dropping+ // them.+ if parser.buffered_len() > 0 {+ tracing::warn!(+ event = "bedrock_eventstream_truncated",+ request_id = %request_id,+ buffered_bytes = parser.buffered_len(),+ "upstream stream ended with un-parseable trailing bytes"+ );+ }+ done = true;+ let _ = done;+ return None;}- done = true;- let _ = done;- return None;}}- }- })- })+ })+ },+ )}/// Tee the translated SSE stream into an `AnthropicStreamState` taskdiff --git a/crates/headroom-proxy/src/bedrock/mod.rs b/crates/headroom-proxy/src/bedrock/mod.rsindex e772a1b93..780ed4d18 100644--- a/crates/headroom-proxy/src/bedrock/mod.rs+++ b/crates/headroom-proxy/src/bedrock/mod.rs@@ -45,6 +45,7 @@//! parsing is in [`eventstream`]; the SSE translator is in//! [`eventstream_to_sse`].+pub mod auth_mode_layer;pub mod envelope;pub mod eventstream;pub mod eventstream_to_sse;@@ -52,6 +53,7 @@ pub mod invoke;pub mod invoke_streaming;pub mod sigv4;+pub use auth_mode_layer::classify_and_attach_auth_mode;pub use envelope::{BedrockEnvelope, EnvelopeError};pub use eventstream::{parse as parse_eventstream, CrcValidation, EventStreamMessage, EventStreamParser, HeaderValue,diff --git a/crates/headroom-proxy/src/lib.rs b/crates/headroom-proxy/src/lib.rsindex f6f152891..8b83eb099 100644--- a/crates/headroom-proxy/src/lib.rs+++ b/crates/headroom-proxy/src/lib.rs@@ -8,6 +8,7 @@ pub mod error;pub mod handlers;pub mod headers;pub mod health;+pub mod observability;pub mod proxy;pub mod responses_items;pub mod sse;diff --git a/crates/headroom-proxy/src/observability/mod.rs b/crates/headroom-proxy/src/observability/mod.rsnew file mode 100644index 000000000..d1c513a1d--- /dev/null+++ b/crates/headroom-proxy/src/observability/mod.rs@@ -0,0 +1,46 @@+//! Proxy observability surface — Phase D PR-D3.+//!+//! Centralises all Prometheus instrumentation in one place so that+//! metric names, label keys, and the global registry stay+//! co-located and discoverable. The Phase D acceptance criterion+//! (`Prometheus scrape includes Bedrock metrics`) demands a single+//! `/metrics` endpoint that serves the registry; that endpoint is+//! mounted by [`crate::proxy::build_app`] when the observability+//! module is in scope.+//!+//! # Module layout+//!+//! - [`prometheus`] — registry construction (lazy via `OnceLock`),+//! Bedrock-scoped counters / histograms, and the `/metrics`+//! text-format scrape handler. Per the realignment build+//! constraint "elegant + scalable" we keep one module per+//! concern; future Phase F / Phase H additions (auth-mode+//! counters, OpenAI request totals) live alongside the+//! Bedrock-prefixed ones below — never sprinkled across handlers.+//!+//! # Cardinality discipline+//!+//! Every label is bounded by infrastructure config, NOT by request+//! input. `model` comes from the axum path parameter (Bedrock vendor+//! prefix is enforced upstream of the metric increment); `region`+//! comes from `Config::bedrock_region`; `auth_mode` comes from the+//! `headroom_core::auth_mode::AuthMode` enum (3 variants total).+//! There is no path where a malicious client can drive label+//! cardinality unbounded — see `bedrock::invoke::handle_invoke` for+//! the call site.+//!+//! # Why not `metrics-rs`?+//!+//! `metrics-rs` is the more idiomatic Rust choice but it requires a+//! separate exporter binary. The Phase D scope is observability for+//! a single proxy binary; the simpler `prometheus` crate (with the+//! global default registry pinned in a `OnceLock`) keeps the+//! footprint small and the scrape endpoint trivial. Phase F may+//! revisit if multi-process aggregation lands.++pub mod prometheus;++pub use prometheus::{+ handle_metrics, observe_bedrock_invoke_latency, record_bedrock_eventstream_message,+ record_bedrock_invoke,+};diff --git a/crates/headroom-proxy/src/observability/prometheus.rs b/crates/headroom-proxy/src/observability/prometheus.rsnew file mode 100644index 000000000..bde34934f--- /dev/null+++ b/crates/headroom-proxy/src/observability/prometheus.rs@@ -0,0 +1,335 @@+//! Prometheus instrumentation for the Bedrock route — Phase D PR-D3.+//!+//! # Registered metrics+//!+//! - `bedrock_invoke_count_total{model, region, auth_mode}` — Counter.+//! One increment per `/model/{model}/invoke` (or `/converse` or+//! `/invoke-with-response-stream`) request that successfully+//! passed the path-parameter extractor and reached the handler+//! body. Failures BEFORE the handler runs (router 404s, axum+//! extractor errors) do not increment.+//! - `bedrock_invoke_latency_seconds{model, region}` — Histogram.+//! Observed at request completion (whether the upstream call+//! succeeded or returned 5xx). Buckets target typical Bedrock+//! latencies (50ms → 60s) so p50/p99 land in distinct buckets at+//! typical throughput. The `auth_mode` label is intentionally+//! absent: the cost of cross-multiplying it with `model` would+//! triple the per-model cardinality with little operator value+//! (auth-mode breakdown lives in the count metric instead).+//! - `bedrock_eventstream_message_count_total{model, region, event_type}`+//! — Counter. One increment per parsed binary EventStream message+//! in the streaming handler. `event_type` is the+//! `:event-type` header from the message (`chunk`, `metadata`,+//! `internalServerException`, etc.). The set is bounded by+//! AWS's documented event-type vocabulary, not customer input —+//! see `crates/headroom-proxy/src/bedrock/eventstream.rs` for+//! the parsed shape.+//!+//! # Wiring+//!+//! Every counter / histogram is created exactly once at first call+//! via `OnceLock`. Per-request work is `inc_with_label_values` /+//! `observe`, which is `O(1)` and lock-free in the common case+//! (the underlying `prometheus` crate uses a sharded RwLock per+//! metric vector). Total D3 overhead per request: a few hundred ns+//! plus one `Instant::elapsed()` for the latency histogram.+//!+//! # Logs paired with every increment+//!+//! Per the realignment build-constraint "comprehensive structured+//! logs", every metric increment in this module emits a+//! `tracing::debug!` with `event = "metric_recorded"` so operators+//! can correlate scrape values with log lines during incidents.+//! The cardinality of these debug logs is the same as the metric+//! itself (bounded by `model × region × auth_mode`), so leaving+//! them at `debug` level avoids per-request log volume in normal+//! operation while still being available under+//! `RUST_LOG=headroom_proxy::observability=debug`.++use std::sync::OnceLock;++use axum::body::Body;+use axum::http::{header, StatusCode};+use axum::response::Response;+use prometheus::{+ Encoder, HistogramOpts, HistogramVec, IntCounterVec, Opts, Registry, TextEncoder,+};++use headroom_core::auth_mode::AuthMode;++/// Latency-histogram buckets in seconds. Chosen to discriminate+/// across typical Bedrock latencies: cold-start (~1-2s), warm+/// streaming-start (~100-500ms), small completions (~50-200ms),+/// long completions (5-60s). Mirrors the bucket layout the AWS+/// CloudWatch sample dashboards use.+const LATENCY_BUCKETS_SECONDS: &[f64] = &[0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0];++/// Lazy singleton registry. Borrowed by `handle_metrics` for+/// scrape rendering and by every metric registration helper.+fn registry() -> &'static Registry {+ static REGISTRY: OnceLock<Registry> = OnceLock::new();+ REGISTRY.get_or_init(Registry::new)+}++/// `bedrock_invoke_count_total{model, region, auth_mode}` —+/// initialised on first call.+fn invoke_counter() -> &'static IntCounterVec {+ static COUNTER: OnceLock<IntCounterVec> = OnceLock::new();+ COUNTER.get_or_init(|| {+ let opts = Opts::new(+ "bedrock_invoke_count_total",+ "Total Bedrock invoke requests handled by the Rust proxy, \+ broken down by model id, AWS region, and inbound auth mode.",+ );+ let counter = IntCounterVec::new(opts, &["model", "region", "auth_mode"])+ .expect("bedrock_invoke_count_total descriptor is well-formed");+ registry()+ .register(Box::new(counter.clone()))+ .expect("bedrock_invoke_count_total registers exactly once");+ counter+ })+}++/// `bedrock_invoke_latency_seconds{model, region}` — initialised+/// on first call.+fn invoke_latency() -> &'static HistogramVec {+ static HIST: OnceLock<HistogramVec> = OnceLock::new();+ HIST.get_or_init(|| {+ let opts = HistogramOpts::new(+ "bedrock_invoke_latency_seconds",+ "Latency in seconds of Bedrock invoke requests as observed at the \+ Rust proxy entry boundary. Includes upstream call time plus any \+ pre-/post-compression and SigV4 signing.",+ )+ .buckets(LATENCY_BUCKETS_SECONDS.to_vec());+ let hist = HistogramVec::new(opts, &["model", "region"])+ .expect("bedrock_invoke_latency_seconds descriptor is well-formed");+ registry()+ .register(Box::new(hist.clone()))+ .expect("bedrock_invoke_latency_seconds registers exactly once");+ hist+ })+}++/// `bedrock_eventstream_message_count_total{model, region, event_type}`+/// — initialised on first call.+fn eventstream_counter() -> &'static IntCounterVec {+ static COUNTER: OnceLock<IntCounterVec> = OnceLock::new();+ COUNTER.get_or_init(|| {+ let opts = Opts::new(+ "bedrock_eventstream_message_count_total",+ "Total Bedrock binary EventStream messages parsed by the Rust proxy, \+ broken down by model id, AWS region, and the message's :event-type header \+ (chunk, metadata, modelStreamErrorException, etc.).",+ );+ let counter = IntCounterVec::new(opts, &["model", "region", "event_type"])+ .expect("bedrock_eventstream_message_count_total descriptor is well-formed");+ registry()+ .register(Box::new(counter.clone()))+ .expect("bedrock_eventstream_message_count_total registers exactly once");+ counter+ })+}++/// Record a single Bedrock invoke (non-streaming or streaming).+///+/// Pure increment + a paired `tracing::debug!` so operators can+/// correlate this metric with the request's structured log line via+/// the `request_id` field (callers thread that through).+pub fn record_bedrock_invoke(model: &str, region: &str, auth_mode: AuthMode) {+ invoke_counter()+ .with_label_values(&[model, region, auth_mode.as_str()])+ .inc();+ tracing::debug!(+ event = "metric_recorded",+ metric = "bedrock_invoke_count_total",+ model = %model,+ region = %region,+ auth_mode = auth_mode.as_str(),+ "incremented bedrock_invoke_count_total"+ );+}++/// Observe latency at the END of an invoke. The duration must be+/// computed by the caller via `Instant::elapsed()` — passing the+/// duration in (rather than the start time) keeps this helper+/// free of `Instant` types so unit tests can assert on synthetic+/// values.+pub fn observe_bedrock_invoke_latency(model: &str, region: &str, seconds: f64) {+ invoke_latency()+ .with_label_values(&[model, region])+ .observe(seconds);+ tracing::debug!(+ event = "metric_recorded",+ metric = "bedrock_invoke_latency_seconds",+ model = %model,+ region = %region,+ seconds = seconds,+ "observed bedrock_invoke_latency_seconds"+ );+}++/// Record a single parsed EventStream message in the streaming+/// path. The `event_type` argument is the `:event-type` header of+/// the message (or `unknown` when the message header was missing,+/// which itself is loud-logged at the call site).+pub fn record_bedrock_eventstream_message(model: &str, region: &str, event_type: &str) {+ eventstream_counter()+ .with_label_values(&[model, region, event_type])+ .inc();+ tracing::debug!(+ event = "metric_recorded",+ metric = "bedrock_eventstream_message_count_total",+ model = %model,+ region = %region,+ event_type = %event_type,+ "incremented bedrock_eventstream_message_count_total"+ );+}++/// Axum handler for `GET /metrics`. Renders the registry in the+/// Prometheus text format. Per Phase D acceptance: the scrape MUST+/// include the three Bedrock metrics above as soon as they have+/// been touched at least once; un-touched counter vectors expose+/// their HELP/TYPE lines but no labelled rows (Prometheus's+/// documented behaviour, not a regression).+pub async fn handle_metrics() -> Response {+ // Force lazy registration so the HELP/TYPE descriptor lines+ // appear in the scrape even before any request has hit the+ // Bedrock route. Operators who curl /metrics on a fresh boot+ // see the metric names already advertised — surprises are+ // worse than the cost of three function calls.+ let _ = invoke_counter();+ let _ = invoke_latency();+ let _ = eventstream_counter();++ let metric_families = registry().gather();+ let mut buffer = Vec::with_capacity(2048);+ let encoder = TextEncoder::new();+ if let Err(e) = encoder.encode(&metric_families, &mut buffer) {+ tracing::error!(+ event = "metrics_encode_failed",+ error = %e,+ "failed to encode Prometheus metrics scrape"+ );+ return Response::builder()+ .status(StatusCode::INTERNAL_SERVER_ERROR)+ .body(Body::from(format!("metrics encode error: {e}")))+ .expect("static error response");+ }+ Response::builder()+ .status(StatusCode::OK)+ .header(header::CONTENT_TYPE, encoder.format_type())+ .body(Body::from(buffer))+ .unwrap_or_else(|e| {+ Response::builder()+ .status(StatusCode::INTERNAL_SERVER_ERROR)+ .body(Body::from(format!("metrics response build error: {e}")))+ .expect("static error response")+ })+}++#[cfg(test)]+mod tests {+ use super::*;++ /// Helper: render the registry to a String for assertions.+ fn scrape() -> String {+ let mf = registry().gather();+ let encoder = TextEncoder::new();+ let mut buf = Vec::new();+ encoder.encode(&mf, &mut buf).expect("encode");+ String::from_utf8(buf).expect("utf8")+ }++ #[test]+ fn invoke_counter_advertises_metric_in_scrape() {+ // The `prometheus` crate only emits HELP/TYPE for vectors+ // that have AT LEAST ONE row (`gather()` skips empty+ // vectors), so we must fire one increment with a unique+ // label set this test owns to assert the metric family+ // appears in the scrape.+ invoke_counter()+ .with_label_values(&[+ "anthropic.unit-test-advertise-v1:0",+ "us-test-advertise-1",+ "oauth",+ ])+ .inc();+ let body = scrape();+ assert!(+ body.contains("bedrock_invoke_count_total"),+ "scrape missing bedrock_invoke_count_total: {body}"+ );+ assert!(+ body.contains("# TYPE bedrock_invoke_count_total counter"),+ "scrape missing TYPE line: {body}"+ );+ }++ #[test]+ fn invoke_increment_appears_with_labels() {+ record_bedrock_invoke(+ "anthropic.claude-3-haiku-20240307-v1:0",+ "us-east-1",+ AuthMode::OAuth,+ );+ let body = scrape();+ // The label-set rendering uses lexical ordering of label+ // names — auth_mode, model, region — so we assert on the+ // values without locking in the ordering of the columns.+ assert!(+ body.contains("auth_mode=\"oauth\""),+ "scrape missing auth_mode label: {body}"+ );+ assert!(+ body.contains("model=\"anthropic.claude-3-haiku-20240307-v1:0\""),+ "scrape missing model label: {body}"+ );+ assert!(+ body.contains("region=\"us-east-1\""),+ "scrape missing region label: {body}"+ );+ }++ #[test]+ fn latency_histogram_records_observation() {+ observe_bedrock_invoke_latency("anthropic.claude-3-haiku-20240307-v1:0", "us-east-1", 0.42);+ let body = scrape();+ assert!(+ body.contains("bedrock_invoke_latency_seconds_bucket"),+ "histogram bucket lines missing: {body}"+ );+ assert!(+ body.contains("bedrock_invoke_latency_seconds_sum"),+ "histogram sum line missing: {body}"+ );+ assert!(+ body.contains("bedrock_invoke_latency_seconds_count"),+ "histogram count line missing: {body}"+ );+ }++ #[test]+ fn eventstream_counter_records_event_type_label() {+ record_bedrock_eventstream_message(+ "anthropic.claude-3-haiku-20240307-v1:0",+ "us-east-1",+ "chunk",+ );+ record_bedrock_eventstream_message(+ "anthropic.claude-3-haiku-20240307-v1:0",+ "us-east-1",+ "metadata",+ );+ let body = scrape();+ assert!(+ body.contains("event_type=\"chunk\""),+ "scrape missing event_type=chunk: {body}"+ );+ assert!(+ body.contains("event_type=\"metadata\""),+ "scrape missing event_type=metadata: {body}"+ );+ }+}diff --git a/crates/headroom-proxy/src/proxy.rs b/crates/headroom-proxy/src/proxy.rsindex dd794669a..dddf46964 100644--- a/crates/headroom-proxy/src/proxy.rs+++ b/crates/headroom-proxy/src/proxy.rs@@ -88,6 +88,14 @@ pub fn build_app(state: AppState) -> Router {let mut router = Router::new().route("/healthz", get(healthz)).route("/healthz/upstream", get(healthz_upstream))+ // PR-D3: Prometheus scrape endpoint. Renders the global+ // registry in text format. The handler is stateless — no+ // `AppState` needed — and idempotent across concurrent+ // scrapes (`prometheus`'s registry uses internal locking).+ // Mounted unconditionally because it has no dependencies on+ // any feature flag; an operator who doesn't want it scraped+ // simply firewalls the path.+ .route("/metrics", get(crate::observability::handle_metrics))// PR-C2: explicit POST route for /v1/chat/completions. The// handler buffers the body and re-injects it into// `forward_http`, which runs the OpenAI live-zone gate@@ -116,7 +124,16 @@ pub fn build_app(state: AppState) -> Router {// identical for `anthropic.claude-*` model IDs (Bedrock just// accepts both legacy `invoke` and modern `converse` paths).if state.config.enable_bedrock_native {- router = router+ // PR-D3: Bedrock-scoped auth-mode middleware. Build a+ // sub-router with ONLY the Bedrock routes, attach the+ // auth-mode layer (so it fires before the handler runs and+ // is scoped to these routes alone — `/v1/messages`,+ // `/healthz`, etc. do NOT run through this middleware), and+ // merge it into the parent router. The merge composes+ // routes without changing their layer stacks; the parent's+ // `with_state` (applied at the end) hands `AppState` to the+ // Bedrock handlers identically.+ let bedrock_router: Router<AppState> = Router::new().route("/model/:model_id/invoke",post(crate::bedrock::invoke::handle_invoke),@@ -133,7 +150,11 @@ pub fn build_app(state: AppState) -> Router {.route("/model/:model_id/invoke-with-response-stream",post(crate::bedrock::invoke_streaming::handle_invoke_streaming),- );+ )+ .route_layer(axum::middleware::from_fn(+ crate::bedrock::classify_and_attach_auth_mode,+ ));+ router = router.merge(bedrock_router);if !state.config.bedrock_validate_eventstream_crc {tracing::warn!(event = "bedrock_eventstream_crc_validation_disabled",diff --git a/crates/headroom-proxy/tests/integration_bedrock_authmode.rs b/crates/headroom-proxy/tests/integration_bedrock_authmode.rsnew file mode 100644index 000000000..81fe30085--- /dev/null+++ b/crates/headroom-proxy/tests/integration_bedrock_authmode.rs@@ -0,0 +1,241 @@+//! Integration tests for the Bedrock auth-mode middleware+//! (Phase D PR-D3).+//!+//! Coverage:+//!+//! 1. `bedrock_classified_as_oauth` — POST a Bedrock invoke request+//! with no Authorization header (the most common SDK pattern when+//! AWS credentials live downstream of the proxy). Assert the+//! middleware coerces the result to `AuthMode::OAuth` per the+//! Bedrock policy matrix and that the value lands in+//! `request.extensions()` where downstream Phase F handlers can+//! pick it up.+//! 2. `oauth_policy_passthrough_prefer` — fire a request with an+//! Anthropic body containing NO `cache_control` markers; assert+//! the upstream-bound body is byte-equal to the inbound body.+//! The OAuth policy matrix forbids auto-injecting `cache_control`+//! or `prompt_cache_key`; D3 wires the marker, F2 enforces the+//! policy. Until F2 lands, the proof is the byte-equality (no+//! mutation observed at the upstream boundary).++mod common;++use aws_credential_types::Credentials;+use axum::body::Body;+use axum::extract::{Extension, State};+use axum::http::StatusCode;+use axum::routing::post;+use axum::Router;+use bytes::Bytes;+use common::start_proxy_with_state;+use headroom_core::auth_mode::AuthMode;+use headroom_proxy::AppState;+use serde_json::{json, Value};+use sha2::{Digest, Sha256};+use std::net::SocketAddr;+use std::sync::{Arc, Mutex};+use tokio::sync::oneshot;+use url::Url;+use wiremock::matchers::{method, path};+use wiremock::{Mock, MockServer, ResponseTemplate};++const TEST_MODEL: &str = "anthropic.claude-3-haiku-20240307-v1:0";++fn test_credentials() -> Credentials {+ Credentials::new(+ "AKIAEXAMPLEAKIDFORTEST",+ "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",+ None,+ None,+ "test",+ )+}++#[derive(Default, Clone, Debug)]+struct CapturedRequest {+ body: Option<Vec<u8>>,+}++type Capture = Arc<Mutex<CapturedRequest>>;++async fn mount_capture_invoke(upstream: &MockServer, response_body: &str) -> Capture {+ let captured: Capture = Arc::new(Mutex::new(CapturedRequest::default()));+ let captured_clone = captured.clone();+ let response_body = response_body.to_string();+ Mock::given(method("POST"))+ .and(path(format!("/model/{TEST_MODEL}/invoke")))+ .respond_with(move |req: &wiremock::Request| {+ let mut c = captured_clone.lock().unwrap();+ c.body = Some(req.body.clone());+ ResponseTemplate::new(200).set_body_string(response_body.clone())+ })+ .mount(upstream)+ .await;+ captured+}++async fn bedrock_proxy(+ upstream: &MockServer,+ customize: impl FnOnce(&mut headroom_proxy::Config),+) -> common::ProxyHandle {+ let endpoint: Url = upstream.uri().parse().unwrap();+ start_proxy_with_state(+ &upstream.uri(),+ |c| {+ c.bedrock_endpoint = Some(endpoint);+ customize(c);+ },+ |s| s.with_bedrock_credentials(test_credentials()),+ )+ .await+}++/// Test 1: With no Authorization header, the bedrock auth-mode+/// middleware classifies as OAuth (Bedrock policy matrix). We boot+/// a separate axum app that mounts the same middleware in front of+/// a probe handler; the probe reads the AuthMode out of+/// `request.extensions()` and echoes it back. This is the canonical+/// "extension was set" assertion the spec asks for.+#[tokio::test]+async fn bedrock_classified_as_oauth() {+ use headroom_proxy::bedrock::classify_and_attach_auth_mode;++ async fn probe(Extension(auth_mode): Extension<AuthMode>) -> String {+ auth_mode.as_str().to_string()+ }+ let app = Router::new()+ .route("/model/:model_id/invoke", post(probe))+ .route_layer(axum::middleware::from_fn(classify_and_attach_auth_mode));++ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();+ let addr = listener.local_addr().unwrap();+ let (tx, rx) = oneshot::channel::<()>();+ let task = tokio::spawn(async move {+ let _ = axum::serve(+ listener,+ app.into_make_service_with_connect_info::<SocketAddr>(),+ )+ .with_graceful_shutdown(async move {+ let _ = rx.await;+ })+ .await;+ });++ // Bedrock SDK style: no Authorization header in the inbound+ // request to our proxy (the SDK signs at the egress side, or+ // the customer is using IAM-instance-credential downstream of+ // our hop). NO x-api-key. NO x-goog-api-key. F1 returns Payg by+ // default; the bedrock middleware must coerce to OAuth.+ let resp = reqwest::Client::new()+ .post(format!(+ "http://{addr}/model/{TEST_MODEL}/invoke",+ addr = addr,+ TEST_MODEL = TEST_MODEL,+ ))+ .header("content-type", "application/json")+ .body(r#"{"anthropic_version":"bedrock-2023-05-31","max_tokens":8,"messages":[]}"#)+ .send()+ .await+ .unwrap();+ assert_eq!(resp.status(), 200);+ let body_text = resp.text().await.unwrap();+ assert_eq!(+ body_text, "oauth",+ "bedrock route must classify as OAuth; saw {body_text}"+ );+ let _ = tx.send(());+ let _ = task.await;+}++/// Test 2: confirm the upstream-bound body is byte-equal to the+/// inbound body. The OAuth policy forbids auto-injecting+/// `cache_control`; D3's contribution is to MARK the request as+/// OAuth so PR-F2 can gate the cache-control walker. For now the+/// invariant is "no mutation visible at the upstream boundary"+/// when compression mode is `off`.+#[tokio::test]+async fn oauth_policy_passthrough_prefer() {+ let upstream = MockServer::start().await;+ let captured = mount_capture_invoke(&upstream, r#"{"id":"msg_x","content":[]}"#).await;+ let proxy = bedrock_proxy(&upstream, |c| {+ c.compression = true;+ c.compression_mode = headroom_proxy::config::CompressionMode::Off;+ })+ .await;++ let payload = json!({+ "anthropic_version": "bedrock-2023-05-31",+ "max_tokens": 64,+ "messages": [+ {"role": "user", "content": "hi"}+ ]+ });+ let body = serde_json::to_vec(&payload).unwrap();+ let resp = reqwest::Client::new()+ .post(format!("{}/model/{TEST_MODEL}/invoke", proxy.url()))+ .header("content-type", "application/json")+ .body(body.clone())+ .send()+ .await+ .unwrap();+ assert_eq!(resp.status(), 200);++ let got = captured.lock().unwrap().clone();+ let received = got.body.expect("upstream got body");+ // Byte-equality (sha256 hashes match).+ let inbound_hash = sha256_hex(&body);+ let received_hash = sha256_hex(&received);+ assert_eq!(+ inbound_hash, received_hash,+ "upstream body must be byte-equal to inbound body under OAuth policy: \+ inbound={inbound_hash}, received={received_hash}"+ );+ // Belt-and-braces: parse the upstream body and assert NO+ // cache_control marker was added to any message.+ let parsed: Value = serde_json::from_slice(&received).unwrap();+ let messages = parsed["messages"].as_array().expect("messages array");+ for (i, msg) in messages.iter().enumerate() {+ // `cache_control` may live on either the message itself or+ // on individual content blocks. Assert neither path got+ // synthesised by us.+ assert!(+ msg.get("cache_control").is_none(),+ "messages[{i}] gained a cache_control marker; OAuth policy forbids auto-injection"+ );+ if let Some(content) = msg.get("content").and_then(|v| v.as_array()) {+ for (j, block) in content.iter().enumerate() {+ assert!(+ block.get("cache_control").is_none(),+ "messages[{i}].content[{j}] gained a cache_control marker"+ );+ }+ }+ }+ // And NO prompt_cache_key at the top level.+ assert!(+ parsed.get("prompt_cache_key").is_none(),+ "top-level prompt_cache_key must NOT be auto-injected under OAuth"+ );+ proxy.shutdown().await;+}++/// Helper: SHA-256 hex of bytes. Mirrors `integration_bedrock_invoke.rs`.+fn sha256_hex(bytes: &[u8]) -> String {+ let mut hasher = Sha256::new();+ hasher.update(bytes);+ hasher+ .finalize()+ .iter()+ .fold(String::with_capacity(64), |mut acc, b| {+ use std::fmt::Write as _;+ let _ = write!(acc, "{b:02x}");+ acc+ })+}++/// Pin the unused-import lint silencers — these symbols are+/// referenced by the assertions but the linter is paranoid about+/// `axum::body::Body` and `AppState` only being used in a single+/// type-position.+#[allow(dead_code)]+fn _pin(_: Body, _: State<AppState>, _: Bytes, _: StatusCode) {}diff --git a/crates/headroom-proxy/tests/integration_bedrock_metrics.rs b/crates/headroom-proxy/tests/integration_bedrock_metrics.rsnew file mode 100644index 000000000..0bdb79180--- /dev/null+++ b/crates/headroom-proxy/tests/integration_bedrock_metrics.rs@@ -0,0 +1,435 @@+//! Integration tests for the Phase D PR-D3 Prometheus instrumentation.+//!+//! Coverage:+//!+//! 1. `metrics_increment_per_invoke` — fire 3 invoke calls; assert+//! `bedrock_invoke_count_total` registers 3 increments tagged+//! with the right `model` + `region` + `auth_mode=oauth`.+//! 2. `metrics_observe_latency` — fire one invoke; assert+//! `bedrock_invoke_latency_seconds` observed exactly one sample.+//! 3. `eventstream_metrics_per_message_type` — drive D2's streaming+//! path with a captured Bedrock binary stream that yields N+//! `chunk` messages and assert the counter registers+//! `event_type=chunk` with N. (The Anthropic-on-Bedrock vocabulary+//! in D2's translator only accepts `:event-type=chunk`; metadata+//! frames are not produced by Bedrock for the Anthropic shape.+//! We assert the chunk path; a future PR-H2 may add metadata+//! frame support and extend this test.)+//! 4. `metrics_endpoint_serves_scrape` — GET `/metrics` and assert+//! the three Bedrock metric families appear in the text-format+//! output.+//!+//! All tests use wiremock as the upstream — no live AWS dependency.++mod common;++use aws_credential_types::Credentials;+use bytes::{Bytes, BytesMut};+use common::start_proxy_with_state;+use headroom_proxy::bedrock::MessageBuilder;+use serde_json::json;+use url::Url;+use wiremock::matchers::{method, path};+use wiremock::{Mock, MockServer, ResponseTemplate};++// Each test in this file owns a UNIQUE (model, region) tuple so the+// global Prometheus registry — shared across all parallel tests in+// the same binary — gives each test isolated label rows. Without+// this isolation, parallel-running tests would cross-contaminate the+// counters they read back. Bumping a counter never tears down its+// row, so absolute counts are not assertable after-the-fact;+// per-tuple isolation gives each test a fresh row to assert deltas+// against.+const TEST_MODEL_INVOKE_COUNT: &str = "anthropic.claude-3-haiku-test-invoke-count-v1:0";+const TEST_MODEL_LATENCY: &str = "anthropic.claude-3-haiku-test-latency-v1:0";+const TEST_MODEL_EVENTSTREAM: &str = "anthropic.claude-3-haiku-test-eventstream-v1:0";+const TEST_MODEL_SCRAPE: &str = "anthropic.claude-3-haiku-test-scrape-v1:0";+const TEST_REGION_INVOKE_COUNT: &str = "us-test-invoke-count-1";+const TEST_REGION_LATENCY: &str = "us-test-latency-1";+const TEST_REGION_EVENTSTREAM: &str = "us-test-eventstream-1";+const TEST_REGION_SCRAPE: &str = "us-test-scrape-1";++fn test_credentials() -> Credentials {+ Credentials::new(+ "AKIAEXAMPLEAKIDFORTEST",+ "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",+ None,+ None,+ "test",+ )+}++async fn bedrock_proxy_with_region(+ upstream: &MockServer,+ region: &str,+ customize: impl FnOnce(&mut headroom_proxy::Config),+) -> common::ProxyHandle {+ let endpoint: Url = upstream.uri().parse().unwrap();+ let region = region.to_string();+ start_proxy_with_state(+ &upstream.uri(),+ |c| {+ c.bedrock_endpoint = Some(endpoint);+ c.bedrock_region = region;+ customize(c);+ },+ |s| s.with_bedrock_credentials(test_credentials()),+ )+ .await+}++async fn mount_simple_invoke_for(upstream: &MockServer, model: &str) {+ Mock::given(method("POST"))+ .and(path(format!("/model/{model}/invoke")))+ .respond_with(ResponseTemplate::new(200).set_body_string(r#"{"id":"msg_x","content":[]}"#))+ .mount(upstream)+ .await;+}++/// Fetch the proxy's `/metrics` text-format scrape.+async fn scrape_metrics(proxy_url: &str) -> String {+ let resp = reqwest::Client::new()+ .get(format!("{proxy_url}/metrics"))+ .send()+ .await+ .expect("metrics scrape");+ assert_eq!(resp.status(), 200, "metrics endpoint must return 200");+ let ct = resp+ .headers()+ .get("content-type")+ .and_then(|v| v.to_str().ok())+ .unwrap_or("")+ .to_string();+ assert!(+ ct.starts_with("text/plain"),+ "metrics content-type must be text/plain (Prometheus text format); got {ct}"+ );+ resp.text().await.unwrap()+}++/// Count the number of Prometheus text lines that contain the+/// metric name + every label key/value pair in `label_pairs`. The+/// label-set rendering uses lexical ordering of label names so we+/// MUST NOT compare exact substrings — instead, every pair must+/// appear in the same line, in any order.+fn count_lines_with_labels(+ scrape: &str,+ metric: &str,+ label_pairs: &[(&str, &str)],+) -> Option<u64> {+ for line in scrape.lines() {+ if !line.starts_with(metric) {+ continue;+ }+ if !label_pairs+ .iter()+ .all(|(k, v)| line.contains(&format!("{k}=\"{v}\"")))+ {+ continue;+ }+ // Counter / gauge: " <value>" tail. We split on the last+ // whitespace, parse as u64.+ if let Some(value_str) = line.rsplit_once(' ').map(|(_, v)| v.trim()) {+ if let Ok(value) = value_str.parse::<u64>() {+ return Some(value);+ }+ if let Ok(f) = value_str.parse::<f64>() {+ return Some(f as u64);+ }+ }+ }+ None+}++/// Test 1: `bedrock_invoke_count_total` increments per request+/// with the right model / region / auth_mode labels.+#[tokio::test]+async fn metrics_increment_per_invoke() {+ let upstream = MockServer::start().await;+ mount_simple_invoke_for(&upstream, TEST_MODEL_INVOKE_COUNT).await;+ let proxy = bedrock_proxy_with_region(&upstream, TEST_REGION_INVOKE_COUNT, |c| {+ c.compression_mode = headroom_proxy::config::CompressionMode::Off;+ })+ .await;++ // Per-tuple-isolated counter — start at 0 (no other test+ // touches this label set), so absolute count == invocations.+ let payload = json!({+ "anthropic_version": "bedrock-2023-05-31",+ "max_tokens": 8,+ "messages": [{"role":"user","content":"hi"}]+ });+ let body = serde_json::to_vec(&payload).unwrap();+ for _ in 0..3 {+ let resp = reqwest::Client::new()+ .post(format!(+ "{}/model/{TEST_MODEL_INVOKE_COUNT}/invoke",+ proxy.url()+ ))+ .header("content-type", "application/json")+ .body(body.clone())+ .send()+ .await+ .unwrap();+ assert_eq!(resp.status(), 200);+ }++ let after = scrape_metrics(&proxy.url()).await;+ let after_count = count_lines_with_labels(+ &after,+ "bedrock_invoke_count_total",+ &[+ ("model", TEST_MODEL_INVOKE_COUNT),+ ("region", TEST_REGION_INVOKE_COUNT),+ ("auth_mode", "oauth"),+ ],+ )+ .expect("counter row must appear after first request");+ assert_eq!(+ after_count, 3,+ "expected exactly 3 increments on isolated labels; got {after_count}"+ );++ proxy.shutdown().await;+}++/// Test 2: `bedrock_invoke_latency_seconds` records exactly one+/// sample for one request.+#[tokio::test]+async fn metrics_observe_latency() {+ let upstream = MockServer::start().await;+ mount_simple_invoke_for(&upstream, TEST_MODEL_LATENCY).await;+ let proxy = bedrock_proxy_with_region(&upstream, TEST_REGION_LATENCY, |c| {+ c.compression_mode = headroom_proxy::config::CompressionMode::Off;+ })+ .await;++ let payload = json!({+ "anthropic_version": "bedrock-2023-05-31",+ "max_tokens": 8,+ "messages": [{"role":"user","content":"hi"}]+ });+ let body = serde_json::to_vec(&payload).unwrap();+ let resp = reqwest::Client::new()+ .post(format!("{}/model/{TEST_MODEL_LATENCY}/invoke", proxy.url()))+ .header("content-type", "application/json")+ .body(body)+ .send()+ .await+ .unwrap();+ assert_eq!(resp.status(), 200);++ let after = scrape_metrics(&proxy.url()).await;+ let after_count = count_lines_with_labels(+ &after,+ "bedrock_invoke_latency_seconds_count",+ &[+ ("model", TEST_MODEL_LATENCY),+ ("region", TEST_REGION_LATENCY),+ ],+ )+ .expect("histogram count row must appear after first request");+ assert_eq!(+ after_count, 1,+ "expected exactly 1 latency observation on isolated labels; got {after_count}"+ );++ // Sum line for the same labels must appear and be > 0.+ let sum_line = after+ .lines()+ .find(|l| {+ l.starts_with("bedrock_invoke_latency_seconds_sum")+ && l.contains(&format!("model=\"{TEST_MODEL_LATENCY}\""))+ })+ .expect("histogram sum line must appear for our labels");+ let sum_value: f64 = sum_line+ .rsplit_once(' ')+ .map(|(_, v)| v.trim())+ .and_then(|s| s.parse().ok())+ .unwrap_or(0.0);+ assert!(+ sum_value > 0.0,+ "histogram sum must reflect a real observation > 0s; saw {sum_value}"+ );++ proxy.shutdown().await;+}++/// Synthesise N chunk EventStream messages.+fn synthesize_chunks(n: usize) -> Bytes {+ let mut buf = BytesMut::new();+ for i in 0..n {+ let payload = serde_json::to_string(&json!({+ "type": "content_block_delta",+ "index": 0,+ "delta": {"type": "text_delta", "text": format!("t{i}")}+ }))+ .unwrap();+ let bytes = MessageBuilder::new()+ .header_string(":event-type", "chunk")+ .header_string(":content-type", "application/json")+ .header_string(":message-type", "event")+ .payload(Bytes::from(payload))+ .build();+ buf.extend_from_slice(&bytes);+ }+ buf.freeze()+}++/// Test 3: per-EventStream-message metrics increment with the+/// correct `event_type` label.+#[tokio::test]+async fn eventstream_metrics_per_message_type() {+ let upstream = MockServer::start().await;+ let chunks = synthesize_chunks(5);+ Mock::given(method("POST"))+ .and(path(format!(+ "/model/{TEST_MODEL_EVENTSTREAM}/invoke-with-response-stream"+ )))+ .respond_with(+ ResponseTemplate::new(200)+ .insert_header("content-type", "application/vnd.amazon.eventstream")+ .set_body_bytes(chunks.to_vec()),+ )+ .mount(&upstream)+ .await;++ let proxy = bedrock_proxy_with_region(&upstream, TEST_REGION_EVENTSTREAM, |c| {+ c.compression_mode = headroom_proxy::config::CompressionMode::Off;+ })+ .await;++ // Default Accept → SSE translation, which is the path that+ // parses messages and increments the counter (passthrough mode+ // forwards bytes verbatim and therefore can't categorize event+ // types — the spec defers that to a future H2 PR).+ let payload = json!({+ "anthropic_version": "bedrock-2023-05-31",+ "max_tokens": 8,+ "messages": [{"role":"user","content":"hi"}]+ });+ let body = serde_json::to_vec(&payload).unwrap();+ let resp = reqwest::Client::new()+ .post(format!(+ "{}/model/{TEST_MODEL_EVENTSTREAM}/invoke-with-response-stream",+ proxy.url()+ ))+ .header("content-type", "application/json")+ .body(body)+ .send()+ .await+ .unwrap();+ assert_eq!(resp.status(), 200);+ // Drain the response body so the translator runs to completion.+ let _ = resp.bytes().await.unwrap();++ let after = scrape_metrics(&proxy.url()).await;+ let after_count = count_lines_with_labels(+ &after,+ "bedrock_eventstream_message_count_total",+ &[+ ("model", TEST_MODEL_EVENTSTREAM),+ ("region", TEST_REGION_EVENTSTREAM),+ ("event_type", "chunk"),+ ],+ )+ .expect("eventstream chunk counter row must appear after first stream");+ assert_eq!(+ after_count, 5,+ "expected 5 chunk increments on isolated labels; got {after_count}"+ );++ proxy.shutdown().await;+}++/// Test 4: `/metrics` endpoint serves a valid Prometheus text-format+/// scrape that includes the three Bedrock metric families. Every+/// metric family must be touched at least once for the+/// `prometheus` crate to render its HELP/TYPE lines (`gather()`+/// skips empty vectors), so this test explicitly drives both the+/// invoke and the streaming routes — each populates a different+/// family, and the latency histogram comes for free with the+/// invoke route.+#[tokio::test]+async fn metrics_endpoint_serves_scrape() {+ let upstream = MockServer::start().await;+ mount_simple_invoke_for(&upstream, TEST_MODEL_SCRAPE).await;+ // Mount the streaming endpoint too, so the third metric family+ // (eventstream message counter) gets at least one increment+ // and its HELP/TYPE lines render in the scrape.+ let chunks = synthesize_chunks(1);+ Mock::given(method("POST"))+ .and(path(format!(+ "/model/{TEST_MODEL_SCRAPE}/invoke-with-response-stream"+ )))+ .respond_with(+ ResponseTemplate::new(200)+ .insert_header("content-type", "application/vnd.amazon.eventstream")+ .set_body_bytes(chunks.to_vec()),+ )+ .mount(&upstream)+ .await;+ let proxy = bedrock_proxy_with_region(&upstream, TEST_REGION_SCRAPE, |c| {+ c.compression_mode = headroom_proxy::config::CompressionMode::Off;+ })+ .await;++ // Fire one invoke (populates invoke_count + invoke_latency)+ // and one streaming invoke (populates eventstream_count).+ let payload = json!({+ "anthropic_version": "bedrock-2023-05-31",+ "max_tokens": 8,+ "messages": [{"role":"user","content":"hi"}]+ });+ let body = serde_json::to_vec(&payload).unwrap();+ let resp = reqwest::Client::new()+ .post(format!("{}/model/{TEST_MODEL_SCRAPE}/invoke", proxy.url()))+ .header("content-type", "application/json")+ .body(body.clone())+ .send()+ .await+ .unwrap();+ assert_eq!(resp.status(), 200);+ let stream_resp = reqwest::Client::new()+ .post(format!(+ "{}/model/{TEST_MODEL_SCRAPE}/invoke-with-response-stream",+ proxy.url()+ ))+ .header("content-type", "application/json")+ .body(body)+ .send()+ .await+ .unwrap();+ assert_eq!(stream_resp.status(), 200);+ let _ = stream_resp.bytes().await.unwrap();++ let scrape = scrape_metrics(&proxy.url()).await;+ // HELP + TYPE lines are advertised even before any increment;+ // after the increment the labelled rows also appear.+ assert!(+ scrape.contains("# HELP bedrock_invoke_count_total"),+ "scrape missing bedrock_invoke_count_total HELP: {scrape}"+ );+ assert!(+ scrape.contains("# TYPE bedrock_invoke_count_total counter"),+ "scrape missing bedrock_invoke_count_total TYPE: {scrape}"+ );+ assert!(+ scrape.contains("# HELP bedrock_invoke_latency_seconds"),+ "scrape missing bedrock_invoke_latency_seconds HELP"+ );+ assert!(+ scrape.contains("# TYPE bedrock_invoke_latency_seconds histogram"),+ "scrape missing bedrock_invoke_latency_seconds TYPE"+ );+ assert!(+ scrape.contains("# HELP bedrock_eventstream_message_count_total"),+ "scrape missing bedrock_eventstream_message_count_total HELP"+ );+ assert!(+ scrape.contains("# TYPE bedrock_eventstream_message_count_total counter"),+ "scrape missing bedrock_eventstream_message_count_total TYPE"+ );++ proxy.shutdown().await;+}diff --git a/docs/bedrock.md b/docs/bedrock.mdnew file mode 100644index 000000000..31d502c44--- /dev/null+++ b/docs/bedrock.md@@ -0,0 +1,182 @@+# AWS Bedrock — Operator Guide++Headroom's Rust proxy ships a native AWS Bedrock InvokeModel surface. After Phase D (PRs D1–D3), Anthropic-on-Bedrock requests are signed, compressed, and observed by the proxy directly — no LiteLLM Python shim on the request path.++This document covers how to deploy the Bedrock-native surface, how compression policy is applied, and how to read the Prometheus metrics the proxy exports.++## What's in scope++| Capability | Status |+|---|---|+| `POST /model/{model}/invoke` | PR-D1 — native Rust handler |+| `POST /model/{model}/converse` | PR-D1 — same handler (Bedrock accepts both paths for the Anthropic envelope) |+| `POST /model/{model}/invoke-with-response-stream` | PR-D2 — binary EventStream parsed and translated to SSE |+| AWS SigV4 signing (post-compression) | PR-D1 |+| `AuthMode::OAuth` classification | PR-D3 — Bedrock IAM is OAuth-equivalent under the policy matrix |+| Per-model + per-region Prometheus metrics | PR-D3 — exposed at `GET /metrics` |+| OAuth compression policy gates (no auto cache_control, lossless-only) | Phase F PR-F2/F3 (gates the marker D3 wires) |++## AWS credential configuration++The proxy uses the [aws-config default credential chain](https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html), resolved once at startup.++The chain searches in this order, stopping at the first source that yields valid credentials:++1. **Environment variables** — `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, optional `AWS_SESSION_TOKEN`. Useful for ECS task roles that inject creds via env or for `aws sts assume-role` shells.+2. **Shared credentials file** — `~/.aws/credentials`. Profile selected by `--aws-profile` (or `AWS_PROFILE`). Falls back to `[default]`.+3. **IAM instance profile / IMDS** — when running on EC2.+4. **ECS task role / EKS pod identity** — when running on the AWS-managed compute platforms.+5. **AWS SSO** — `~/.aws/sso/cache/...` when `aws sso login` has been run.++If the chain does NOT resolve any credentials at startup, the proxy logs `event=bedrock_credentials_unavailable` at WARN and continues to start. Bedrock invoke routes will then return `500` with `event=bedrock_credentials_missing` per request — there is **no silent fallback to unsigned requests**, by design.++### Required IAM permissions++The proxy needs:++- `bedrock:InvokeModel` for non-streaming+- `bedrock:InvokeModelWithResponseStream` for streaming++Scope these to the specific model ARNs you intend to use. Example IAM policy snippet:++```json+{+ "Version": "2012-10-17",+ "Statement": [+ {+ "Effect": "Allow",+ "Action": [+ "bedrock:InvokeModel",+ "bedrock:InvokeModelWithResponseStream"+ ],+ "Resource": [+ "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-haiku-20240307-v1:0",+ "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0"+ ]+ }+ ]+}+```++## Region configuration++```sh+headroom-proxy \+ --upstream http://unused-when-bedrock-only \+ --bedrock-region us-east-1+```++| Flag | Env var | Default | Notes |+|---|---|---|---|+| `--bedrock-region` | `HEADROOM_PROXY_BEDROCK_REGION` (or `AWS_REGION`) | `us-east-1` | Drives both the SigV4 region and the derived endpoint hostname. |+| `--bedrock-endpoint` | `HEADROOM_PROXY_BEDROCK_ENDPOINT` | derived from region | Override for FIPS endpoints (`bedrock-runtime-fips.{region}.amazonaws.com`), VPC endpoints, or local mock servers. |+| `--aws-profile` | `HEADROOM_PROXY_AWS_PROFILE` | unset | Selects the named profile from the shared credentials file. |+| `--enable-bedrock-native` | `HEADROOM_PROXY_ENABLE_BEDROCK_NATIVE` | `true` | Set to `false` to mount no Bedrock routes at all (Bedrock requests will then fall through to the catch-all and fail without SigV4). |++## Supported model IDs++The proxy classifies model IDs by **literal vendor prefix** — no regexes. Any model ID starting with `anthropic.` is treated as Anthropic-shape: the live-zone compression dispatcher runs over the body, the envelope is re-emitted with `anthropic_version` preserved as the first key, and the request is signed with SigV4.++Examples that hit the Anthropic compression path:++- `anthropic.claude-3-haiku-20240307-v1:0`+- `anthropic.claude-3-5-sonnet-20241022-v2:0`+- `anthropic.claude-3-opus-20240229-v1:0`++Other Bedrock vendors (`amazon.titan-...`, `meta.llama3-...`, `cohere.command-...`, `ai21.j2-...`, `stability.stable-diffusion-...`) are signed and forwarded **without compression** — the proxy does not yet understand their body shapes and would risk corrupting them. These model IDs log `event=bedrock_compression_skipped, reason=non_anthropic_vendor` per request. Full Anthropic envelopes only.++The contract: **any new model ID that AWS adds under the `anthropic.` prefix automatically picks up the full compression + signing pipeline.** No code change in the proxy is needed for new versions of Claude on Bedrock.++## Compression behaviour++Bedrock requests are subject to the **same** live-zone compression rules as direct Anthropic (`/v1/messages`):++- Only the live-zone messages (latest user turn, latest tool/output blocks) are eligible for compression.+- The cache hot zone (older messages, system prompt, tools list) is byte-faithful passthrough.+- The dispatcher only mutates body bytes when at least one block compressed. The byte-equality invariant for unchanged blocks is enforced at `debug_assert!` granularity.++### OAuth policy (PR-D3 → PR-F2/F3)++The Bedrock auth-mode middleware classifies every Bedrock request as `AuthMode::OAuth`. Even when the inbound request has no Authorization header (the common case where the AWS SDK signs after our proxy), the middleware **coerces** to OAuth and emits `event=bedrock_auth_mode_unexpected` at WARN if F1's classifier disagreed — so the divergence is loud, not silent.++Under the OAuth policy matrix (see `docs/auth-modes.md`):++- **No auto-`cache_control` injection.** OAuth subscriptions pin the cache scope to `(account, model, session)`; auto-injecting markers can void cache hits.+- **No auto-`prompt_cache_key`.** Same reasoning.+- **Lossless-only compressors.** Lossy compressors (text rewriting, summarisation) are gated off for OAuth.++PR-D3 lands the classification + the resulting `AuthMode` in `request.extensions()`. PR-F2 and PR-F3 wire the actual policy gates that read it. Until those PRs land, the Bedrock route uses the existing dispatcher (which is a no-op in `compression_mode=off`); the OAuth contract above is the documented forward direction.++### Cache safety++The bytes signed by SigV4 are exactly the bytes Bedrock receives — the signer hashes the post-compression body. There is no "sign before compress" shortcut that would produce a signature mismatched to the wire payload. Compression mutates the body once, then the signer runs once, then the bytes are forwarded once.++## Prometheus metrics++The proxy exposes a `GET /metrics` endpoint that serves the standard Prometheus text-format scrape. Three Bedrock-specific metric families are exported:++| Metric | Type | Labels | Source |+|---|---|---|---|+| `bedrock_invoke_count_total` | Counter | `model`, `region`, `auth_mode` | One increment per `/model/.../invoke` (and `/converse` and `/invoke-with-response-stream`) request. |+| `bedrock_invoke_latency_seconds` | Histogram | `model`, `region` | Observed at request completion (success or failure). |+| `bedrock_eventstream_message_count_total` | Counter | `model`, `region`, `event_type` | One increment per parsed binary EventStream message in the streaming path. `event_type` is the `:event-type` header (`chunk`, `metadata`, `internalServerException`, etc.). |++All labels are bounded by infrastructure config (`region` from `--bedrock-region`, `auth_mode` from the 3-variant enum) or by the path parameter (`model`, supplied by the axum extractor — never by user-controlled body bytes). Cardinality is bounded by deployment fan-out, not by traffic volume.++### Sample PromQL queries++**p99 latency by model:**+```promql+histogram_quantile(+ 0.99,+ sum by (model, le) (rate(bedrock_invoke_latency_seconds_bucket[5m]))+)+```++**Request rate by region (RPS):**+```promql+sum by (region) (rate(bedrock_invoke_count_total[1m]))+```++**EventStream message rate by event type (debugging the streaming path):**+```promql+sum by (event_type) (rate(bedrock_eventstream_message_count_total[1m]))+```++**Error breakdown by HTTP status (cross-references the structured logs `event=bedrock_upstream_error`):**+```promql+sum by (model) (rate(bedrock_invoke_count_total{auth_mode="oauth"}[5m]))+ / sum by (model) (rate(bedrock_invoke_latency_seconds_count[5m]))+```++(The denominator is the total observed latency samples — useful for sanity-checking that every counted invoke also got a histogram observation. They should be equal.)++### Structured-log correlation++Every metric increment in the Bedrock path is paired with a `tracing::debug!` log line carrying:++- `event = "metric_recorded"`+- `metric = "bedrock_invoke_count_total" | "bedrock_invoke_latency_seconds" | "bedrock_eventstream_message_count_total"`+- the same labels as the metric++Enable with `RUST_LOG=headroom_proxy::observability=debug` for incident correlation. In normal operation keep this at the default `info` level — debug volume per request is bounded by the same cardinality the metric uses.++## Live cloud validation++The PR-D1, D2, D3 implementations are exercised end-to-end against a wiremock upstream (`crates/headroom-proxy/tests/integration_bedrock_*.rs`). The wiremock-based tests are the canonical correctness gate.++A real Bedrock smoke test (`aws bedrock-runtime invoke-model ...` through the proxy) requires `bedrock:InvokeModel` permissions in the developer's AWS account. Set the proxy upstream to the proxy URL (`http://localhost:8787`) via the AWS SDK's `AWS_ENDPOINT_URL_BEDROCK_RUNTIME` env var:++```sh+AWS_ENDPOINT_URL_BEDROCK_RUNTIME=http://localhost:8787 \+ aws bedrock-runtime invoke-model \+ --model-id anthropic.claude-3-haiku-20240307-v1:0 \+ --body '{"anthropic_version":"bedrock-2023-05-31","max_tokens":32,"messages":[{"role":"user","content":"hi"}]}' \+ /tmp/out.json+```++If the SDK signs the request before sending to the proxy, the proxy will see a SigV4 `Authorization` header and classify as OAuth via the standard rule. If the SDK is configured to sign downstream of the proxy (some IAM-instance-profile setups), the proxy still classifies as OAuth via the middleware's coerce-and-log fallback.++## Rollback++Set `--enable-bedrock-native=false` to unmount all Bedrock routes; the catch-all proxy then forwards Bedrock requests unchanged to `--upstream`. This is an emergency rollback only — without SigV4 re-signing, the catch-all path will fail closed unless the upstream is itself a Bedrock-aware proxy (e.g., the Python LiteLLM converter on a different port).