🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
TypeScript analyticsautogenevaluationlangchainlarge-language-modelsllama-indexllmllm-evaluationllm-observabilityllmopsmonitoringobservabilityopen-sourceopenaiplaygroundprompt-engineeringprompt-managementself-hostedycombinator
fix(trace-batching): emit readiness gauges as totals only (#18723)
Lysander Kiesel455924ffc004
diff --git a/worker/AGENTS.md b/worker/AGENTS.mdindex 87a5b6d8ea..bf56193bd9 100644--- a/worker/AGENTS.md+++ b/worker/AGENTS.md@@ -73,8 +73,10 @@still pools 1,000 candidates. Drain the old layout before upgradeor shard-count/retention changes. The trace-batch consumer checks existingsummaries against the job enqueue time so retries skip completed facets while- later redispatches append updated summaries. Map gauges include shard and- total series; never sum duplicate reporters or add totals to shard series.+ later redispatches append updated summaries. Map and dispatcher gauges are+ summed across shards before emission; do not add per-shard series, which+ multiply custom metrics on every reporting worker. Never sum duplicate+ reporters.`src/features/traceBatching/traceBatchTranscript.ts` measures per-trace assemblyphases, thread count, current-turn/history token estimates and their sum. Eachadmitted trace records full assembled-transcript JSON lengths, renders thediff --git a/worker/src/__tests__/traceBatching.test.ts b/worker/src/__tests__/traceBatching.test.tsindex a229aa01d1..e6a21fc269 100644--- a/worker/src/__tests__/traceBatching.test.ts+++ b/worker/src/__tests__/traceBatching.test.ts@@ -982,7 +982,7 @@ describe("trace micro-batch scheduling with Redis", () => {"langfuse.trace_batch.active_reads",0,);- expect(vi.mocked(recordGauge).mock.calls).toHaveLength(15);+ expect(vi.mocked(recordGauge).mock.calls).toHaveLength(9);expect(vi.mocked(recordGauge).mock.calls.every(([, value]) => value === 0),).toBe(true);diff --git a/worker/src/features/traceBatching/README.md b/worker/src/features/traceBatching/README.mdindex c3f276daa3..37f8feb7ac 100644--- a/worker/src/features/traceBatching/README.md+++ b/worker/src/features/traceBatching/README.md@@ -539,15 +539,16 @@ The following metrics use the `langfuse.trace_batch` prefix. All additions areinside the existing cloud experiment paths; normal ingestion with trackingdisabled performs no new aggregation or Redis calls.-| Metric | Kind / tags | Meaning |-| ---------------------------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |-| `event_updates`, `serialized_event_bytes` | Counters; `stage:eligible\|recorded` | Accepted, valid-start-time updates before sampling, and in Redis-acknowledged chunks. Repeated updates count again. Selected volume is `sampling_decisions`. |-| `active_reads` | Per-process gauge | Streams currently being consumed; failures decrement the count too. |-| `failed_read_observation_count`, `failed_read_input_bytes` | Distributions | Partial logical rows and input bytes consumed before a stream failure. Separate from successful throughput. Output and metadata bytes stay on the completion log. |-| `queue_depth` | Gauge; `type:delayed` | Delayed BullMQ jobs. Waiting, active, and failed depth stay on `langfuse.queue.trace_batch.depth`. Request rate, wait time, and processing time use the generic `langfuse.queue.trace_batch` series, the same as every other queue. Processing time is recorded when the processor returns. Discard reasons stay on `discarded_jobs`. |-| `queue_waiting_head_age_ms` | Gauge | Maximum creation age of the next FIFO jobs in waiting/paused lists. Zero when empty. |-| `redis_key_bytes` | Gauge; `key:due\|state\|first`, `shard:N\|total` | Estimated memory of each readiness key, including Redis overhead. |-| `redis_key_entries` | Gauge; `key:due\|state\|first`, `shard:N\|total` | `ZCARD` / `HLEN` at the memory snapshot. |+| Metric | Kind / tags | Meaning |+| ---------------------------------------------------------- | --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `event_updates`, `serialized_event_bytes` | Counters; `stage:eligible\|recorded` | Accepted, valid-start-time updates before sampling, and in Redis-acknowledged chunks. Repeated updates count again. Selected volume is `sampling_decisions`. |+| `active_reads` | Per-process gauge | Streams currently being consumed; failures decrement the count too. |+| `failed_read_observation_count`, `failed_read_input_bytes` | Distributions | Partial logical rows and input bytes consumed before a stream failure. Separate from successful throughput. Output and metadata bytes stay on the completion log. |+| `queue_depth` | Gauge; `type:delayed` | Delayed BullMQ jobs. Waiting, active, and failed depth stay on `langfuse.queue.trace_batch.depth`. Request rate, wait time, and processing time use the generic `langfuse.queue.trace_batch` series, the same as every other queue. Processing time is recorded when the processor returns. Discard reasons stay on `discarded_jobs`. |+| `queue_waiting_head_age_ms` | Gauge | Maximum creation age of the next FIFO jobs in waiting/paused lists. Zero when empty. |+| `redis_key_bytes` | Gauge; `key:due\|state\|first`, `shard:total` | Estimated memory of each readiness key, including Redis overhead. |+| `redis_key_entries` | Gauge; `key:due\|state\|first`, `shard:total` | `ZCARD` / `HLEN` at the memory snapshot. |+| `pending_traces`, `ready_traces`, `oldest_due_age_ms` | Gauges, untagged | Dispatcher snapshot summed (age: maximum) across shards, emitted by the current dispatcher lease holder. |The eligible and recorded counters reuse the already-computed serialized size andper-trace aggregates. Recorded volume is emitted after each acknowledged Lua@@ -571,10 +572,13 @@ SAMPLES 5`, preserving the shared key prefix and Redis Cluster slot routing.Missing keys report zero; there is no keyspace scan or exact full-hash traversal.Memory sampling estimates storage, not process RSS or event payload size.+Map gauges are summed across readiness shards before they are emitted. Each+worker emits them, so a shard tag would multiply custom metrics by the shard+count on every reporting worker. The dispatcher snapshot gauges are untagged for+the same reason. Inspect a single shard's keys in Redis when diagnosing skew.+Queue/map snapshots are global and can be emitted by several workers: use-max/latest across reporters, never sum duplicates. For map totals, first-deduplicate reporters per `key,shard`, then add shard estimates, or use the-`shard:total` series directly. Never add total and shard series together. First-seen+max/latest across reporters, never sum duplicates. First-seencounts include physically retained expired entries until their bucket expires.In contrast,`active_reads` is local to a worker; distinct live worker series may be summed.diff --git a/worker/src/features/traceBatching/TraceBatchMetricsRunner.ts b/worker/src/features/traceBatching/TraceBatchMetricsRunner.tsindex b6a1406524..3ec7bdb888 100644--- a/worker/src/features/traceBatching/TraceBatchMetricsRunner.ts+++ b/worker/src/features/traceBatching/TraceBatchMetricsRunner.ts@@ -135,13 +135,14 @@ export class TraceBatchMetricsRunner extends PeriodicRunner {state,...first.keys,)) as number[];- this.recordMapMemory(values, String(shard));values.forEach((value, index) => (totals[index] += Number(value)));}- this.recordMapMemory(totals, "total");+ // Per-shard series would multiply custom metrics by the shard count on+ // every reporting worker; only the summed totals are emitted.+ this.recordMapMemory(totals);}- private recordMapMemory(values: number[], shard: string): void {+ private recordMapMemory(values: number[]): void {const [dueBytes,dueEntries,@@ -159,12 +160,12 @@ export class TraceBatchMetricsRunner extends PeriodicRunner {]) {recordGauge(`${METRIC_PREFIX}.redis_key_bytes`, bytes, {key,- shard,+ shard: "total",unit: "bytes",});recordGauge(`${METRIC_PREFIX}.redis_key_entries`, entries, {key,- shard,+ shard: "total",unit: "records",});}diff --git a/worker/src/features/traceBatching/traceBatching.ts b/worker/src/features/traceBatching/traceBatching.tsindex 85e57fe8b9..9491b80cdf 100644--- a/worker/src/features/traceBatching/traceBatching.ts+++ b/worker/src/features/traceBatching/traceBatching.ts@@ -1110,17 +1110,6 @@ export class TraceBatchDispatcher extends PeriodicExclusiveRunner {recordIncrement("langfuse.trace_batch.expired_traces", expired, {source: "dispatcher",});- recordGauge("langfuse.trace_batch.pending_traces", pending, {- shard: String(shard),- });- recordGauge("langfuse.trace_batch.ready_traces", ready, {- shard: String(shard),- });- recordGauge(- "langfuse.trace_batch.oldest_due_age_ms",- Math.max(0, now - oldest),- { shard: String(shard) },- );if (expired < CLEANUP_SIZE) {totals.pending += pending;totals.ready += ready;@@ -1154,6 +1143,8 @@ export class TraceBatchDispatcher extends PeriodicExclusiveRunner {});}}+ // The lease moves between workers after every run, so per-shard series+ // would multiply custom metrics by the shard count on each dispatcher.recordGauge("langfuse.trace_batch.pending_traces", totals.pending);recordGauge("langfuse.trace_batch.ready_traces", totals.ready);recordGauge("langfuse.trace_batch.oldest_due_age_ms", totals.oldestAge);