Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(capture-apm-metrics): decouple remote-write expansion cap from body limit (#120229)

Frank Hamand16b0e072430b
diff --git a/rust/capture-apm-metrics/README.md b/rust/capture-apm-metrics/README.md
index 307a8f5aa2a9..55c2298a768d 100644
--- a/rust/capture-apm-metrics/README.md
+++ b/rust/capture-apm-metrics/README.md
@@ -10,6 +10,7 @@ This binary is the metrics half of [`capture-logs`](../capture-logs/README.md).
| Variable | Default | Description |
|----------|---------|-------------|
+| REMOTE_WRITE_MAX_EXPANDED_BYTES | 167772160 | Largest estimated row set (160 MiB) one remote-write request may expand to. A larger request gets a 400. Separate from `MAX_REQUEST_BODY_SIZE_BYTES` |
| REDIS_URL | (none) | Redis that shares the set of recently labelled metric series between pods. Unset means local cache only |
| METRICS_SERIES_LABEL_GATE_ENABLED | false | Strip label maps from repeat metric rows (see below). When false the gate only reports metrics |
| METRICS_SERIES_LABEL_INTERVAL_SECS | 1800 | Longest gap between two labelled rows of one metric series. Each series uses a window up to a quarter shorter, derived from its key |
diff --git a/rust/capture-apm-metrics/src/config.rs b/rust/capture-apm-metrics/src/config.rs
index d7a03f7794b4..07e8343cd9eb 100644
--- a/rust/capture-apm-metrics/src/config.rs
+++ b/rust/capture-apm-metrics/src/config.rs
@@ -11,6 +11,12 @@ pub struct Config {
#[envconfig(from = "REDIS_URL")]
pub redis_url: Option<String>,
+ /// Largest estimated row set one remote-write request may expand to. It is
+ /// separate from MAX_REQUEST_BODY_SIZE_BYTES, so the body limit can grow for
+ /// large vmagent blocks without raising the memory one request can hold.
+ #[envconfig(from = "REMOTE_WRITE_MAX_EXPANDED_BYTES", default = "167772160")] // 160 MiB
+ pub remote_write_max_expanded_bytes: u64,
+
/// When false the gate still tracks series and reports metrics but leaves
/// every row labelled, so the switch can flip once the metrics2 read path
/// takes labels from the series rollup.
diff --git a/rust/capture-apm-metrics/src/main.rs b/rust/capture-apm-metrics/src/main.rs
index eaa060caee91..41607a6de32e 100644
--- a/rust/capture-apm-metrics/src/main.rs
+++ b/rust/capture-apm-metrics/src/main.rs
@@ -246,6 +246,7 @@ async fn run(
kafka_sink,
authorizer,
config.base.max_request_body_size_bytes,
+ config.remote_write_max_expanded_bytes,
series_label_gate,
);
let http_bind = format!("{}:{}", config.base.host, config.base.port);
diff --git a/rust/capture-apm-metrics/src/prometheus.rs b/rust/capture-apm-metrics/src/prometheus.rs
index 412af1638da9..4360cb0289a9 100644
--- a/rust/capture-apm-metrics/src/prometheus.rs
+++ b/rust/capture-apm-metrics/src/prometheus.rs
@@ -21,14 +21,6 @@ use tracing::{debug, error, instrument};
use crate::histogram_assembly::fold_classic_histograms;
use crate::service::MetricsService;
-/// How much larger than the (already decompression-capped) request body the
-/// expanded row set may be. Row building clones the series label map per
-/// sample and the whole batch becomes a single Kafka message, so a 2 wire-byte
-/// sample expands to hundreds of bytes — legitimate senders (Prometheus
-/// defaults to 2k samples per send, vmagent to 10k rows per block) stay far
-/// below this; only expansion-bomb payloads exceed it.
-const MAX_EXPANSION_FACTOR: u64 = 16;
-
#[derive(Deserialize)]
pub struct RemoteWriteQueryParams {
token: Option<String>,
@@ -128,10 +120,14 @@ pub async fn export_prometheus_remote_write_http(
}
};
+ // Row building clones the series label map per sample and the whole batch
+ // becomes a single Kafka message, so a 2 wire-byte sample expands to
+ // hundreds of bytes. The cap bounds the memory of one request; only
+ // expansion-bomb payloads and oversized blocks exceed it.
// 400, not 5xx: a batch that expands past the limit will never fit, so the
// sender must drop it rather than retry it forever.
let expanded = estimate_expanded_bytes(&write_request);
- let max_expanded = MAX_EXPANSION_FACTOR * service.max_request_body_size_bytes as u64;
+ let max_expanded = service.remote_write_max_expanded_bytes;
if expanded > max_expanded {
error!(
"Rejecting remote-write request expanding to ~{expanded} bytes (limit {max_expanded})"
diff --git a/rust/capture-apm-metrics/src/service.rs b/rust/capture-apm-metrics/src/service.rs
index bdf53afebfee..750facf9ec06 100644
--- a/rust/capture-apm-metrics/src/service.rs
+++ b/rust/capture-apm-metrics/src/service.rs
@@ -32,6 +32,7 @@ pub struct MetricsService {
pub(crate) sink: KafkaSink,
pub(crate) authorizer: Authorizer,
pub(crate) max_request_body_size_bytes: usize,
+ pub(crate) remote_write_max_expanded_bytes: u64,
pub(crate) series_label_gate: Arc<SeriesLabelGate>,
}
@@ -45,12 +46,14 @@ impl MetricsService {
kafka_sink: KafkaSink,
authorizer: Authorizer,
max_request_body_size_bytes: usize,
+ remote_write_max_expanded_bytes: u64,
series_label_gate: Arc<SeriesLabelGate>,
) -> Self {
Self {
sink: kafka_sink,
authorizer,
max_request_body_size_bytes,
+ remote_write_max_expanded_bytes,
series_label_gate,
}
}