:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(capture-apm-metrics): decouple remote-write expansion cap from body limit (#120229)
Frank Hamand16b0e072430b
diff --git a/rust/capture-apm-metrics/README.md b/rust/capture-apm-metrics/README.mdindex 307a8f5aa2a9..55c2298a768d 100644--- a/rust/capture-apm-metrics/README.md+++ b/rust/capture-apm-metrics/README.md@@ -10,6 +10,7 @@ This binary is the metrics half of [`capture-logs`](../capture-logs/README.md).| Variable | Default | Description ||----------|---------|-------------|+| REMOTE_WRITE_MAX_EXPANDED_BYTES | 167772160 | Largest estimated row set (160 MiB) one remote-write request may expand to. A larger request gets a 400. Separate from `MAX_REQUEST_BODY_SIZE_BYTES` || REDIS_URL | (none) | Redis that shares the set of recently labelled metric series between pods. Unset means local cache only || METRICS_SERIES_LABEL_GATE_ENABLED | false | Strip label maps from repeat metric rows (see below). When false the gate only reports metrics || METRICS_SERIES_LABEL_INTERVAL_SECS | 1800 | Longest gap between two labelled rows of one metric series. Each series uses a window up to a quarter shorter, derived from its key |diff --git a/rust/capture-apm-metrics/src/config.rs b/rust/capture-apm-metrics/src/config.rsindex d7a03f7794b4..07e8343cd9eb 100644--- a/rust/capture-apm-metrics/src/config.rs+++ b/rust/capture-apm-metrics/src/config.rs@@ -11,6 +11,12 @@ pub struct Config {#[envconfig(from = "REDIS_URL")]pub redis_url: Option<String>,+ /// Largest estimated row set one remote-write request may expand to. It is+ /// separate from MAX_REQUEST_BODY_SIZE_BYTES, so the body limit can grow for+ /// large vmagent blocks without raising the memory one request can hold.+ #[envconfig(from = "REMOTE_WRITE_MAX_EXPANDED_BYTES", default = "167772160")] // 160 MiB+ pub remote_write_max_expanded_bytes: u64,+/// When false the gate still tracks series and reports metrics but leaves/// every row labelled, so the switch can flip once the metrics2 read path/// takes labels from the series rollup.diff --git a/rust/capture-apm-metrics/src/main.rs b/rust/capture-apm-metrics/src/main.rsindex eaa060caee91..41607a6de32e 100644--- a/rust/capture-apm-metrics/src/main.rs+++ b/rust/capture-apm-metrics/src/main.rs@@ -246,6 +246,7 @@ async fn run(kafka_sink,authorizer,config.base.max_request_body_size_bytes,+ config.remote_write_max_expanded_bytes,series_label_gate,);let http_bind = format!("{}:{}", config.base.host, config.base.port);diff --git a/rust/capture-apm-metrics/src/prometheus.rs b/rust/capture-apm-metrics/src/prometheus.rsindex 412af1638da9..4360cb0289a9 100644--- a/rust/capture-apm-metrics/src/prometheus.rs+++ b/rust/capture-apm-metrics/src/prometheus.rs@@ -21,14 +21,6 @@ use tracing::{debug, error, instrument};use crate::histogram_assembly::fold_classic_histograms;use crate::service::MetricsService;-/// How much larger than the (already decompression-capped) request body the-/// expanded row set may be. Row building clones the series label map per-/// sample and the whole batch becomes a single Kafka message, so a 2 wire-byte-/// sample expands to hundreds of bytes — legitimate senders (Prometheus-/// defaults to 2k samples per send, vmagent to 10k rows per block) stay far-/// below this; only expansion-bomb payloads exceed it.-const MAX_EXPANSION_FACTOR: u64 = 16;-#[derive(Deserialize)]pub struct RemoteWriteQueryParams {token: Option<String>,@@ -128,10 +120,14 @@ pub async fn export_prometheus_remote_write_http(}};+ // Row building clones the series label map per sample and the whole batch+ // becomes a single Kafka message, so a 2 wire-byte sample expands to+ // hundreds of bytes. The cap bounds the memory of one request; only+ // expansion-bomb payloads and oversized blocks exceed it.// 400, not 5xx: a batch that expands past the limit will never fit, so the// sender must drop it rather than retry it forever.let expanded = estimate_expanded_bytes(&write_request);- let max_expanded = MAX_EXPANSION_FACTOR * service.max_request_body_size_bytes as u64;+ let max_expanded = service.remote_write_max_expanded_bytes;if expanded > max_expanded {error!("Rejecting remote-write request expanding to ~{expanded} bytes (limit {max_expanded})"diff --git a/rust/capture-apm-metrics/src/service.rs b/rust/capture-apm-metrics/src/service.rsindex bdf53afebfee..750facf9ec06 100644--- a/rust/capture-apm-metrics/src/service.rs+++ b/rust/capture-apm-metrics/src/service.rs@@ -32,6 +32,7 @@ pub struct MetricsService {pub(crate) sink: KafkaSink,pub(crate) authorizer: Authorizer,pub(crate) max_request_body_size_bytes: usize,+ pub(crate) remote_write_max_expanded_bytes: u64,pub(crate) series_label_gate: Arc<SeriesLabelGate>,}@@ -45,12 +46,14 @@ impl MetricsService {kafka_sink: KafkaSink,authorizer: Authorizer,max_request_body_size_bytes: usize,+ remote_write_max_expanded_bytes: u64,series_label_gate: Arc<SeriesLabelGate>,) -> Self {Self {sink: kafka_sink,authorizer,max_request_body_size_bytes,+ remote_write_max_expanded_bytes,series_label_gate,}}