Skip to content

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

fix(trace-batching): emit readiness gauges as totals only (#18723)

Lysander Kiesel455924ffc004
diff --git a/worker/AGENTS.md b/worker/AGENTS.md
index 87a5b6d8ea..bf56193bd9 100644
--- a/worker/AGENTS.md
+++ b/worker/AGENTS.md
@@ -73,8 +73,10 @@
still pools 1,000 candidates. Drain the old layout before upgrade
or shard-count/retention changes. The trace-batch consumer checks existing
summaries against the job enqueue time so retries skip completed facets while
- later redispatches append updated summaries. Map gauges include shard and
- total series; never sum duplicate reporters or add totals to shard series.
+ later redispatches append updated summaries. Map and dispatcher gauges are
+ summed across shards before emission; do not add per-shard series, which
+ multiply custom metrics on every reporting worker. Never sum duplicate
+ reporters.
`src/features/traceBatching/traceBatchTranscript.ts` measures per-trace assembly
phases, thread count, current-turn/history token estimates and their sum. Each
admitted trace records full assembled-transcript JSON lengths, renders the
diff --git a/worker/src/__tests__/traceBatching.test.ts b/worker/src/__tests__/traceBatching.test.ts
index a229aa01d1..e6a21fc269 100644
--- a/worker/src/__tests__/traceBatching.test.ts
+++ b/worker/src/__tests__/traceBatching.test.ts
@@ -982,7 +982,7 @@ describe("trace micro-batch scheduling with Redis", () => {
"langfuse.trace_batch.active_reads",
0,
);
- expect(vi.mocked(recordGauge).mock.calls).toHaveLength(15);
+ expect(vi.mocked(recordGauge).mock.calls).toHaveLength(9);
expect(
vi.mocked(recordGauge).mock.calls.every(([, value]) => value === 0),
).toBe(true);
diff --git a/worker/src/features/traceBatching/README.md b/worker/src/features/traceBatching/README.md
index c3f276daa3..37f8feb7ac 100644
--- a/worker/src/features/traceBatching/README.md
+++ b/worker/src/features/traceBatching/README.md
@@ -539,15 +539,16 @@ The following metrics use the `langfuse.trace_batch` prefix. All additions are
inside the existing cloud experiment paths; normal ingestion with tracking
disabled performs no new aggregation or Redis calls.
-| Metric | Kind / tags | Meaning |
-| ---------------------------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `event_updates`, `serialized_event_bytes` | Counters; `stage:eligible\|recorded` | Accepted, valid-start-time updates before sampling, and in Redis-acknowledged chunks. Repeated updates count again. Selected volume is `sampling_decisions`. |
-| `active_reads` | Per-process gauge | Streams currently being consumed; failures decrement the count too. |
-| `failed_read_observation_count`, `failed_read_input_bytes` | Distributions | Partial logical rows and input bytes consumed before a stream failure. Separate from successful throughput. Output and metadata bytes stay on the completion log. |
-| `queue_depth` | Gauge; `type:delayed` | Delayed BullMQ jobs. Waiting, active, and failed depth stay on `langfuse.queue.trace_batch.depth`. Request rate, wait time, and processing time use the generic `langfuse.queue.trace_batch` series, the same as every other queue. Processing time is recorded when the processor returns. Discard reasons stay on `discarded_jobs`. |
-| `queue_waiting_head_age_ms` | Gauge | Maximum creation age of the next FIFO jobs in waiting/paused lists. Zero when empty. |
-| `redis_key_bytes` | Gauge; `key:due\|state\|first`, `shard:N\|total` | Estimated memory of each readiness key, including Redis overhead. |
-| `redis_key_entries` | Gauge; `key:due\|state\|first`, `shard:N\|total` | `ZCARD` / `HLEN` at the memory snapshot. |
+| Metric | Kind / tags | Meaning |
+| ---------------------------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `event_updates`, `serialized_event_bytes` | Counters; `stage:eligible\|recorded` | Accepted, valid-start-time updates before sampling, and in Redis-acknowledged chunks. Repeated updates count again. Selected volume is `sampling_decisions`. |
+| `active_reads` | Per-process gauge | Streams currently being consumed; failures decrement the count too. |
+| `failed_read_observation_count`, `failed_read_input_bytes` | Distributions | Partial logical rows and input bytes consumed before a stream failure. Separate from successful throughput. Output and metadata bytes stay on the completion log. |
+| `queue_depth` | Gauge; `type:delayed` | Delayed BullMQ jobs. Waiting, active, and failed depth stay on `langfuse.queue.trace_batch.depth`. Request rate, wait time, and processing time use the generic `langfuse.queue.trace_batch` series, the same as every other queue. Processing time is recorded when the processor returns. Discard reasons stay on `discarded_jobs`. |
+| `queue_waiting_head_age_ms` | Gauge | Maximum creation age of the next FIFO jobs in waiting/paused lists. Zero when empty. |
+| `redis_key_bytes` | Gauge; `key:due\|state\|first`, `shard:total` | Estimated memory of each readiness key, including Redis overhead. |
+| `redis_key_entries` | Gauge; `key:due\|state\|first`, `shard:total` | `ZCARD` / `HLEN` at the memory snapshot. |
+| `pending_traces`, `ready_traces`, `oldest_due_age_ms` | Gauges, untagged | Dispatcher snapshot summed (age: maximum) across shards, emitted by the current dispatcher lease holder. |
The eligible and recorded counters reuse the already-computed serialized size and
per-trace aggregates. Recorded volume is emitted after each acknowledged Lua
@@ -571,10 +572,13 @@ SAMPLES 5`, preserving the shared key prefix and Redis Cluster slot routing.
Missing keys report zero; there is no keyspace scan or exact full-hash traversal.
Memory sampling estimates storage, not process RSS or event payload size.
+Map gauges are summed across readiness shards before they are emitted. Each
+worker emits them, so a shard tag would multiply custom metrics by the shard
+count on every reporting worker. The dispatcher snapshot gauges are untagged for
+the same reason. Inspect a single shard's keys in Redis when diagnosing skew.
+
Queue/map snapshots are global and can be emitted by several workers: use
-max/latest across reporters, never sum duplicates. For map totals, first
-deduplicate reporters per `key,shard`, then add shard estimates, or use the
-`shard:total` series directly. Never add total and shard series together. First-seen
+max/latest across reporters, never sum duplicates. First-seen
counts include physically retained expired entries until their bucket expires.
In contrast,
`active_reads` is local to a worker; distinct live worker series may be summed.
diff --git a/worker/src/features/traceBatching/TraceBatchMetricsRunner.ts b/worker/src/features/traceBatching/TraceBatchMetricsRunner.ts
index b6a1406524..3ec7bdb888 100644
--- a/worker/src/features/traceBatching/TraceBatchMetricsRunner.ts
+++ b/worker/src/features/traceBatching/TraceBatchMetricsRunner.ts
@@ -135,13 +135,14 @@ export class TraceBatchMetricsRunner extends PeriodicRunner {
state,
...first.keys,
)) as number[];
- this.recordMapMemory(values, String(shard));
values.forEach((value, index) => (totals[index] += Number(value)));
}
- this.recordMapMemory(totals, "total");
+ // Per-shard series would multiply custom metrics by the shard count on
+ // every reporting worker; only the summed totals are emitted.
+ this.recordMapMemory(totals);
}
- private recordMapMemory(values: number[], shard: string): void {
+ private recordMapMemory(values: number[]): void {
const [
dueBytes,
dueEntries,
@@ -159,12 +160,12 @@ export class TraceBatchMetricsRunner extends PeriodicRunner {
]) {
recordGauge(`${METRIC_PREFIX}.redis_key_bytes`, bytes, {
key,
- shard,
+ shard: "total",
unit: "bytes",
});
recordGauge(`${METRIC_PREFIX}.redis_key_entries`, entries, {
key,
- shard,
+ shard: "total",
unit: "records",
});
}
diff --git a/worker/src/features/traceBatching/traceBatching.ts b/worker/src/features/traceBatching/traceBatching.ts
index 85e57fe8b9..9491b80cdf 100644
--- a/worker/src/features/traceBatching/traceBatching.ts
+++ b/worker/src/features/traceBatching/traceBatching.ts
@@ -1110,17 +1110,6 @@ export class TraceBatchDispatcher extends PeriodicExclusiveRunner {
recordIncrement("langfuse.trace_batch.expired_traces", expired, {
source: "dispatcher",
});
- recordGauge("langfuse.trace_batch.pending_traces", pending, {
- shard: String(shard),
- });
- recordGauge("langfuse.trace_batch.ready_traces", ready, {
- shard: String(shard),
- });
- recordGauge(
- "langfuse.trace_batch.oldest_due_age_ms",
- Math.max(0, now - oldest),
- { shard: String(shard) },
- );
if (expired < CLEANUP_SIZE) {
totals.pending += pending;
totals.ready += ready;
@@ -1154,6 +1143,8 @@ export class TraceBatchDispatcher extends PeriodicExclusiveRunner {
});
}
}
+ // The lease moves between workers after every run, so per-shard series
+ // would multiply custom metrics by the shard count on each dispatcher.
recordGauge("langfuse.trace_batch.pending_traces", totals.pending);
recordGauge("langfuse.trace_batch.ready_traces", totals.ready);
recordGauge("langfuse.trace_batch.oldest_due_age_ms", totals.oldestAge);