🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.
Internal cloud trace-batch experiment
This experiment measures full-event reads and transcript assembly for idle
traces, then summarizes allowlisted projects from that assembled transcript.
It requires NEXT_PUBLIC_LANGFUSE_CLOUD_REGION and explicit opt-in. The enablement
flags and transcript-metrics switch below default to false; an ordinary release
needs no infrastructure changes. The four enablement flags stay commented in env
templates. The local dev template sets the idle time below.
Readiness requirements
- Every admitted event refreshes the trace's idle deadline. Tracking uses the project's configured idle interval, or the deployment default when unset.
- Keep pending scheduling state separate from retained historical bounds. Enqueue acknowledgement clears pending state; history survives to support later arrivals.
- A late arrival within history retention opens a new pending window and reads the whole known trace, using the earliest and latest retained event starts plus the existing two-minute buffer. Backdated arrivals must preserve the previous tail.
- The consumer checks existing summaries against the job enqueue time. Retries skip facets completed by this job; later redispatches append updated summaries.
- Bounds, counters, revision and due time are merged atomically. Readiness is revalidated before enqueue, and acknowledgement must preserve any newer revision.
- Pending state must survive its idle interval plus recovery grace. Historical values have an independent logical TTL and expire without field scans.
- Use Redis 7-compatible operations on standalone Redis, Sentinel and Cluster. Every multi-key Lua call stays in one slot; cleanup and Lua batches are bounded. One renewable dispatcher lease coordinates scheduling across all shards.
- Redis loss and events outside retained history can leave gaps. There is no reconciler, and retained timestamps cannot recover data that was never observed.
Controls
| Variable | Default | Role |
|---|---|---|
LANGFUSE_TRACE_BATCH_INGESTION_ENABLED |
false |
Track accepted direct-v4 event writes in Redis. |
LANGFUSE_TRACE_BATCH_DISPATCHER_ENABLED |
false |
Schedule idle traces into BullMQ jobs. |
QUEUE_CONSUMER_TRACE_BATCH_QUEUE_IS_ENABLED |
false |
Register the batch worker. |
LANGFUSE_TRACE_BATCH_READ_ENABLED |
false |
Allow the worker to query ClickHouse; otherwise discard jobs. |
LANGFUSE_TRACE_BATCH_TRANSCRIPT_METRICS_ENABLED |
false |
Emit transcript size distributions. Span attributes and failure counters stay on either way. |
LANGFUSE_TRACE_BATCH_SAMPLING_RATE |
0 |
Fraction of other projects admitted by ingestion, from 0 to 1. Unset or 0 is off. Topics-active projects are always admitted. |
LANGFUSE_TRACE_BATCH_STRATEGY |
project |
Choose project-order packing or opt-in locality grouping. |
LANGFUSE_TRACE_BATCH_MAX_SIZE |
60 |
Maximum traces per job, up to 10,000. |
LANGFUSE_TRACE_BATCH_MAX_BATCH_EVENT_UPDATES |
1500 |
Estimated event updates per multi-trace job; 0 disables. |
LANGFUSE_TRACE_BATCH_MAX_BATCH_SERIALIZED_BYTES |
67108864 |
Estimated serialized event bytes per multi-trace job (64 MiB); 0 disables. |
LANGFUSE_TRACE_BATCH_MAX_TRACE_EVENT_UPDATES |
5000 |
Exclude a trace whose retained read-window estimate exceeds this many event updates; 0 disables. |
LANGFUSE_TRACE_BATCH_MAX_TRACE_SERIALIZED_BYTES |
52428800 |
Exclude a trace whose retained read-window estimate exceeds these serialized bytes (50 MiB); 0 disables. |
LANGFUSE_TRACE_BATCH_MAX_THREADS |
2 |
ClickHouse threads per query (positive integer). |
LANGFUSE_TRACE_BATCH_REQUEST_TIMEOUT_MS |
40000 |
ClickHouse client socket idle timeout in milliseconds, including pauses for summary backpressure. Independent of the 30-second server query execution limit. |
LANGFUSE_TRACE_BATCH_MAX_PENDING_SUMMARY_BYTES |
104857600 |
Queued input/output/metadata payload threshold (100 MiB). Includes the active summary; excludes the current trace, tool fields, transport buffers and JavaScript object overhead. |
LANGFUSE_TRACE_BATCH_MAX_BLOCK_SIZE |
unset | Optional positive max_block_size hint; unset inherits the server profile. |
LANGFUSE_TRACE_BATCH_EXPERIMENT_ID |
unset | Optional 1–64 character attempt/query label: letters, digits, ., _, -; first character alphanumeric. |
LANGFUSE_TRACE_BATCH_CONCURRENCY |
2 |
Concurrent batch jobs per worker process. |
LANGFUSE_TOPICS_TRACE_CONCURRENCY |
20 |
Topics traces summarized and embedded at once per batch job (positive integer). Tokenization stays one trace at a time. |
LANGFUSE_TRACE_BATCH_IDLE_MS |
600000 |
Default idle time before a trace is ready. Unset and DEV uses 120000 (2 minutes); every other region uses 10 minutes. A project's tracing_readiness_settings.trace_idle_ms (60 seconds to 60 minutes) overrides it, capped at the pending TTL minus the dispatch interval. The project value is cached in Redis for 5 minutes; saving replaces it, and a failed read uses the default without caching. |
LANGFUSE_TRACE_BATCH_PENDING_TTL_MS |
7200000 |
Recovery grace after a trace becomes due (2 hours). Pending keys expire no earlier than the latest required due-plus-grace deadline. |
LANGFUSE_TRACE_BATCH_FIRST_SEEN_TTL_MS |
86400000 |
Historical event-start bounds and size estimates retained after activity or dispatch (24 hours; at most 48 hours). Internal and cloud-gated. |
LANGFUSE_TRACE_BATCH_SHARD_COUNT |
auto |
16 on Redis Cluster; 1 on standalone/Sentinel. Override with 1–256. Fixed across Redis resizing; drain before changing. |
LANGFUSE_TRACE_BATCH_DISPATCH_INTERVAL_MS |
30000 |
Dispatcher cadence (30 seconds). |
Producer, dispatcher and consumer can run independently. Sampling reuses the evaluator's deterministic trace-ID sampler; every observation of the same trace gets the same admission decision. Dispatch and consumption do not resample. Cloud guards cover ingestion, dispatcher execution, worker registration and processing, and the shared queue accessor, including admin queue inspection.
Redis state and expiry
{trace-batch:<shard>}:due: sorted set, with JSON[projectId, traceId]members and Redis server time plus idle delay as the score.{trace-batch:<shard>}:state: hash with the same members and JSON{minStart, maxStart, revision, historyBucket, eventUpdateCount, serializedEventBytes}values. Bounds describe observed event start times; revision protects newer ingestion from an older dispatch acknowledgment.{trace-batch}:dispatcher: renewable dispatcher lease.{trace-batch:<shard>}:first:<bucket>: hourly hashes retaining the minimum and maximum event starts independently of pending state. Fields are the first 16 binary bytes of SHA-256 of the JSON project/trace pair; values are MessagePack[minStart, maxStart, expiresAt, eventUpdateCount, serializedEventBytes]. A bounds-only tuple has unknown estimates. This accepts the negligible collision probability of a 128-bit identity. Key shape is independent of the processing unit.
Sharding uses the shared SHA-256 shard helper on the complete project/trace pair, so a large project spreads across shards. Every Lua call uses only keys with one shard's hash tag. The single dispatcher lease covers all shards; batch packing combines ready traces from every shard. Queue slots are unchanged.
When pending state is absent or has no bucket pointer, tracking looks up the bounded set of live time buckets (26 keys for the default TTL), never scans their fields, and atomically merges the historical bounds. Valid pending state contains both bounds, cumulative size estimates and its current history bucket, allowing direct movement to the new bucket on subsequent activity and guarded acknowledgement. The Redis server clock owns both retention and readiness. Values with an elapsed deadline are ignored; whole-key expiry at bucket end plus retention removes storage without scans. Physical removal can lag logical expiry by up to one hour (or the TTL for a shorter test override). Activity and guarded acknowledgement refresh retention, so the post-dispatch window lasts the configured TTL independently of idle delay. One live copy per identity is retained under a consistent TTL.
No event payloads are stored here. Atomic Lua updates accumulate estimates and refresh bounds, revision and readiness:
due = last admitted activity + effective project idle
pending expiry = due + recovery grace
Tracking uses the project's configured idle duration when available. The default is 600 seconds outside DEV and 120 seconds in DEV, with the existing environment override.
Both pending keys receive a whole-key PEXPIREAT covering the required expiry,
preserving any existing later deadline with PEXPIRETIME (Redis 7+). During
activity, cleanup removes at most 100 members per call with
due <= now - recovery grace, using the existing due index. This supports
different project idle durations without another per-member expiry index.
Cleanup runs during ingestion and dispatch; whole-key expiry also clears abandoned
state if all processes stop. Acknowledgement normally removes pending entries
as soon as enqueueing succeeds, independently of their safety deadline.
Sharding and atomicity
The shard is SHA256(JSON.stringify([projectId, traceId]))[first 32 bits] % N.
It is stable for a fixed shard count and hashes the trace identity rather than
the project alone. Redis Cluster then maps the shared {trace-batch:N} tag to
one of its 16,384 slots; the number of logical readiness shards is independent
of the number of Redis primaries. Standalone Redis executes the same layout.
Use a REDIS_KEY_PREFIX without its own hash tag: Redis uses the first tag in
the full key, so a tagged prefix would collapse every shard onto one slot.
flowchart LR
identity[Project + trace identity] --> hash[SHA-256 modulo shard count]
hash --> slot[One Redis Cluster slot per shard tag]
slot --> due[Due sorted set]
slot --> state[Pending state hash]
slot --> first[First-seen bucket hashes]
| Operation | Atomic boundary |
|---|---|
| Track activity | One Lua call reads retained starts and pending state, merges min/max and counters, replaces the revision, sets readiness, moves first-seen history and sets expiry. All keys share a slot. |
| Snapshot/expiry | One Lua call removes a bounded expired cohort and returns readiness counts and the cutoff. |
| Hydrate | One Lua call rechecks due scores and reads corresponding state/revisions. Freshly reactivated traces are not dispatched from an obsolete snapshot. |
| Acknowledge | One Lua call compares the revision, refreshes first-seen retention from dispatch time and removes due/state only when it still matches. |
| Enqueue | A separate BullMQ operation precedes acknowledgement. Stable job IDs reduce duplicates if acknowledgement fails. Concurrent arrivals survive the guarded acknowledgement. |
No application-side read-then-write decides a trace's bounds, revision or due
score. TIME before tracking selects the bounded bucket keys; Lua reads Redis
time again and rejects a stale bucket selection before mutating anything.
Scripts serialize within a Redis primary; different primaries can execute
different shards concurrently. Tracking, hydration and acknowledgement calls
contain at most 50 traces; cleanup handles at most 100. Ingestion sends at most
four tracking scripts per pipeline, all in one shard, using one Redis clock lookup
per ingestion call. A group requiring only one script uses a direct call.
Successful pipeline commands remain committed when a sibling
fails; only commands rejected with NOSCRIPT are retried using EVAL. A pipeline
is not one atomic operation; each Lua call is atomic independently.
Dispatch still pools
up to 1,000 hydrated candidates for locality selection, so script call size does
not constrain queue batch packing. Scripts use cached EVALSHA, falling back
to EVAL after a missing script cache on the selected primary. Dispatch holds one renewable lease across
all shards, and checks/extends it while making progress. The lease excludes other
dispatchers; it does not lock ingestion. Only each synchronous Redis command or
Lua call blocks its primary, not the dispatcher's entire read/pack/enqueue run.
Lua atomicity prevents command interleaving, but is not an ACID transaction across Redis, BullMQ and ClickHouse and does not roll back writes on a runtime script error. Duplicate runs and the accepted Redis failover gap remain part of this pipeline's delivery contract. There is no reconciler.
Dispatch and reads
One dispatcher holds the renewable lease and snapshots each shard in turn. It selects the complete due cohort at each shard's cutoff; there is no per-run trace limit. Each due-ID range read and the combined in-memory candidate list grow with the due cohort, even though subsequent Lua calls and selection pools are bounded. It sorts ready IDs by project and due time across shards, then hydrates and revalidates state in single-slot calls of up to 50. Locality selection combines those calls into pools of up to 1,000. Jobs contain up to the configured trace cap and weight budgets; project leftovers can share a job. The dispatcher enqueues before acknowledging matching revisions, so concurrent intake stays pending and failed enqueueing can be retried. Successful enqueues are acknowledged by shard after each hydration window, including when a later enqueue fails, so Redis calls stay bounded by 50 traces rather than job size. Stable job IDs limit duplicates.
Acknowledgement removes due/state only. Late arrivals within first-seen retention reopen readiness with the merged historical and incoming minimum and maximum starts; the existing two-minute query padding reads the trace again. Tracking does not classify events as late by comparing timestamps. Every admitted event updates readiness; retained history supplies the earlier bound even when the incoming start equals it. History without pending state can also result from pending expiry, so it does not prove a previous dispatch. Concurrent arrivals replace the revision and survive an acknowledgement of an older snapshot.
The consumer reads the supplied trace window and checks existing summaries for each facet. A summary processed since this job was enqueued is skipped on retry; a later redispatch generates and appends an updated row. There are no Redis completion markers or staged results. BullMQ still uses Redis for queue delivery.
ReplacingMergeTree(processed_at) retains the greatest processing timestamp
when the Topics sort key matches. That key includes the start minute: an event
revealing a genuinely earlier start in a different minute can leave the previous
summary under its original key. processed_at is assigned when summary
processing starts, so it orders processing attempts rather than dispatch windows.
An older, narrower dispatch processed later can replace a summary that includes
newer events. This is an accepted limitation; dispatch-order versioning must be
revisited before requiring the newest dispatch window to win.
Retaining the historical maximum preserves the previous tail even when a late event's start precedes that tail by more than the query buffer. Once historical bounds expire or are lost, the prior window cannot be reconstructed from this map.
Trace-window size estimates
Accepted direct-v4 writes contribute their final serialized UTF-8 event size after
field overflow handling, reusing the writer's existing event_bytes calculation.
The accounting field itself is excluded. Acceptance means admission to the writer,
not confirmation of a ClickHouse flush. eventUpdateCount counts accepted updates,
including retries and repeated observation versions, rather than unique observations.
Pending state and history carry cumulative estimates for the retained read window. Tracking adds newly accepted updates to valid pending totals. When reopening a trace, it initializes totals from retained history before adding the new updates. Acknowledgement preserves estimates with the bounds, including for oversized exclusions. A late window therefore cannot make a known oversized trace appear small or understate its batch weight by counting only its new events.
Missing estimates remain unknown through updates and redispatch until their retained window expires. They are not replaced with zero or a count of only the new events. These are admission estimates, not hard limits on actual ClickHouse rows or query memory; the two-minute query buffer can include additional rows. Tracking-volume metrics still count only updates accepted in the current ingestion batch; dispatcher size distributions use the cumulative estimates.
The dispatcher emits estimated_event_update_count and
estimated_serialized_event_bytes distributions under langfuse.trace_batch, tagged
with scope:trace|batch and strategy. Batch totals require known estimates for
every member; estimates_unavailable counts unknown traces and affected batches.
Estimates stay out of queue payloads and job IDs. Dispatch retries can emit
another sample.
Serialized event bytes differ from the reader's input_bytes, output_bytes
and metadata_bytes metrics: they include the serialized event fields and JSON
encoding, and do not represent RAM, network transfer or ClickHouse scan bytes.
The job return still sums those three as ioMetadataBytes; that sum is not a
custom metric.
Weight budgets and oversized traces
Both strategies close a job before its summed estimates would exceed
LANGFUSE_TRACE_BATCH_MAX_BATCH_EVENT_UPDATES or
LANGFUSE_TRACE_BATCH_MAX_BATCH_SERIALIZED_BYTES, in addition to the trace cap.
Locality treats an over-budget slice like one past its time envelope, and
partials coalesce only within the budget. A single trace always forms a job, so
a trace heavier than the budget runs alone. Locality packs traces above half of
either budget in a separate lane, so one heavy trace does not split the light
traces around it into extra jobs. Unknown estimates weigh nothing.
Budget-limited jobs report fill:partial; the scope:batch estimate
distributions show how close jobs run to the budget.
After hydration, a trace whose retained read-window estimates exceed
LANGFUSE_TRACE_BATCH_MAX_TRACE_EVENT_UPDATES or
LANGFUSE_TRACE_BATCH_MAX_TRACE_SERIALIZED_BYTES is never queued or read. The
dispatcher acknowledges it like a dispatched trace, increments
langfuse.trace_batch.excluded_traces tagged with
reason:event_updates|serialized_bytes and strategy, and logs
Trace batch excluded oversized trace at warn level with projectId, traceId,
reason and both estimates. Group that log by @projectId to find repeat
offenders. Activity after the acknowledgement starts a new pending window that
is evaluated again using its retained cumulative estimates, so a trace that keeps
growing may be logged more than once.
Traces with unknown estimates are never excluded.
The worker streams full events_full payloads and records batch observation,
trace and logical payload-byte metrics. The input_bytes, output_bytes
and metadata_bytes distributions under langfuse.trace_batch each report one
total per completed batch read. Distinct project count stays on the span and
completion log.
Sizes count UTF-8 bytes, including metadata keys and values, excluding JSON
transport framing/escaping and compression. They do not measure ClickHouse bytes
scanned or per-trace ingestion size; retries can record another batch sample.
Exact (projectId, traceId)
pairs protect tenant boundaries; project, trace hash and a shared outer time
window prune reads. Each selected trace also gets its own interval, so unrelated
batch companions cannot widen the payload returned for that trace.
TRACE_QUERY_BUFFER_MS pads each recorded interval by two minutes on both sides.
This fixed margin neither waits for arrivals nor guarantees trace completeness;
repeated intervals for the same pair form a union without duplicating rows.
Queries use response compression and chunked multipart parameters for up to
10,000 UUID-sized trace/project IDs. The fixed 30-second timeout fails the job
rather than accepting partial results. Beyond the dispatcher's weight budgets
and trace exclusion, this does not cap observation count, query memory or
dispatcher snapshot memory.
Optional input storage
LANGFUSE_TRACE_BATCH_TRANSCRIPT_STORAGE_ENABLED=false controls writes only.
When enabled, storeInputs.ts captures the assembled transcript for traces that
pass the internal-environment exclusion, Topics settings and rule sampling gate.
The latest event's environment determines eligibility for both capture and Topics.
It runs inside the
Topics concurrency limit alongside in-memory summarization. Both operations must
settle before releasing the slot, including when summarization fails. Each S3
request is bounded by LANGFUSE_TRACE_BATCH_TRANSCRIPT_STORAGE_REQUEST_TIMEOUT_MS
(default 10 seconds) so a stalled request cannot hold a slot indefinitely; the
SDK attempt budget stays at its default. It does not capture every trace admitted
by experimental batching.
The shared server/traceBatch module uses the event-upload bucket and credentials:
<prefix>trace-batch/trace/<hash>/<projectId>/<safeTraceId>/<revision>.json
hash is the first three hex characters of SHA-256 over
JSON.stringify([projectId, traceId]). Identifiers use the shared blob-key
escaping helpers. The revision comes from the queued readiness entry; a new
revision uses another key, while an ordinary PUT of the same revision can replace
its object. A reader needs all three identifiers; no latest-revision lookup or
durable job reference is stored.
Objects contain a versioned envelope:
{
schemaVersion: 1,
transcript: StoredTranscriptV1 | null,
traceContext: {
sessionId: string | null,
traceTimestamp: string, // ISO timestamp
environment: string,
traceName: string | null,
},
}
The queue creates inputStorage with createTraceBatchInputStorage() and passes
it into storeInputs. The factory returns the TraceBatchInputStorage interface
with put and get; all instances share one lazily initialized provider client.
Feature gating, instrumentation and tolerated failures remain in storeInputs.
inputStorage.put(reference, { transcript, traceContext }) requires all context
fields, with null for an absent session ID or name. Context fields mirror the
event-domain field shapes without importing the domain schema; the writer's typed
assignment keeps them compatible. Context uses the existing Topics selection: earliest observation start,
newest nonempty session ID, newest-event environment and first observation name.
The transcript alone cannot reconstruct this context. The original job enqueue
time remains execution state rather than trace context.
The trace-batch module freezes the complete stored V1 envelope in
traceBatch/versions/v1.ts, including trace context and copies of the transcript
and normalized message schemas without importing live schemas. Its serialization
module owns the typed adapters between the live transcript and V1. Message dates are
serialized as ISO strings. inputStorage.get(reference) returns
{ transcript, traceContext }, validates nested content and restores message
dates; traceContext.traceTimestamp remains an ISO string. Missing context,
invalid JSON and download errors propagate; an explicitly stored null transcript
returns transcript: null. A schemaVersion the reader does not know throws
UnsupportedTraceBatchInputVersionError, distinct from invalid data. Schema
version and readiness revision are independent.
Mixed versions happen: rolling deploys, and self-hosters updating web and worker
at different times. A reader that meets an unknown schemaVersion must fail with
the retryable UnsupportedTraceBatchInputVersionError, never discard the job or
treat the object as missing. Consumers read inside retried jobs whose backoff
spans a deploy, so the retry lands on an updated reader. Deploy readers for a new
version before enabling its writer; if a writer ships first, the retries cover the
gap. Compatible optional fields may be added to V1 when absent values are safe for
old readers. A breaking change adds versions/v2.ts, its adapters and a read
dispatch on schemaVersion; existing objects are never rewritten.
Serialize the full envelope once. Skip the entire upload if json.length exceeds
1,000,000; do not truncate. This measures UTF-16 code units, not UTF-8 bytes, and
does not bound assembly or serialization memory. Size skips emit an info log with identifiers,
actual length and limit. StorageService owns upload retries and logs: SlowDown at
warning level, other failures including request timeouts at error level. Exhausted
failures mark the trace-batch-store-inputs span as failed. Neither outcome stops
Topics. The span records outcome and JSON length without transcript content;
there are no additional outcome counters.
Deploy with writes disabled. Before enabling, configure and verify an enabled
bucket lifecycle rule for the full <prefix>trace-batch/trace/ path, matching
span-eval retention, including noncurrent versions where applicable. A rule for
evals/ alone does not cover this path. Enable on a small worker rollout and
check storage spans, size-skip logs, S3 failures and Topics latency. Roll back by
disabling writes. Existing Topics processing continues to use in-memory inputs;
future consumers that require files must schedule only after a successful PUT.
Like temporary span-eval inputs, these objects have no ingestion deletion references. Trace/project deletion and per-project retention do not immediately remove them; bucket expiry is the accepted cleanup mechanism, and the deletion paths are intentionally not extended. The lifecycle rule above remains the precondition for enabling writes.
Storage portability
Keep producers and consumers behind the TraceBatchInputStorage interface.
createTraceBatchInputStorage() wires the production implementation; consumers
receive the storage object so tests or another backend can supply the same contract.
TraceBatchInputData is the application contract; the S3 key layout, client,
serialization and physical retention belong to the storage implementation.
The logical identity (projectId, traceId, revision) must remain stable across
storage backends. A ClickHouse implementation can return the same transcript
and trace context without exposing its table layout to callers.
When adding file-dependent consumers, persist a backend-aware input reference on the execution record and carry it in the queue job. For S3 this reference must retain the exact object key. Only the storage module should interpret the backend and locator. During a migration, keep readers for both backends until queued references have drained or expired; changing the writer backend must not redirect reads of existing references.
Replacing S3 also requires preserving behavior: report stored only after the
input is durable and available to the consumer's read path; retain the size
limit and define retry/same-revision replacement semantics explicitly. ClickHouse
needs batched or safely acknowledged async inserts rather than one synchronous
insert per trace, and its retention/deletion policy must be configured separately.
These are requirements for a future backend, not a second implementation in this
optional-capture stage.
Per-trace transcripts
The reader orders only by project ID and trace ID. The worker keeps one trace's observations, assembles it when the pair changes, and flushes the last trace only after a successful end of stream. A stream error discards the current partial trace; earlier completed traces have already emitted samples. Each trace uses the shared depth-first ordering and transcript assembler. Transcripts are ephemeral unless optional input capture is enabled. Neither logs nor job results contain their content.
Completion means all rows returned for that pair's query window, not that no late events can arrive. Event versions follow the current ClickHouse merge state; the shared ordering function keeps one observation per ID. Retries can repeat per-trace samples.
These values are always attributes on the trace-batch-transcript span. With
LANGFUSE_TRACE_BATCH_TRANSCRIPT_METRICS_ENABLED=true they are also
langfuse.trace_batch.* distributions. That switch defaults to false.
Failure and unavailable estimates still increment the counters named in the
paragraphs below, with or without the switch.
| Metric | Sample |
|---|---|
transcript_assembly_duration_ms |
One trace's ordering and assembly time, excluding I/O conversion, stream waits and tokenization; has_transcript:true|false. |
transcript_json_characters |
UTF-16 length of JSON.stringify(assembleTranscript(...)), including thread/message wrappers and provenance; zero for a null transcript. Same admitted traces as the Topics length metrics. |
transcript_json_tokens |
o200k estimate of that complete JSON string, tagged tokenizer:o200k_base; zero for a null transcript. Attempted on the same admitted traces as topics_transcript_tokens; unavailable estimates are omitted. |
generic_transcript_characters |
UTF-16 length of the previous generic plain-text layout, rendered with the same Topics block caps and inclusions. Tool definitions or errors may still render when assembly returns null. |
generic_transcript_tokens |
o200k estimate of that generic text, tagged tokenizer:o200k_base. |
transcript_message_tokens |
Sum of current-turn and history message-projection token estimates; zero when empty, omitted if either estimate is unavailable. Distinct from full-transcript JSON. |
transcript_current_turn_tokens |
Token estimate of current-turn messages across all threads, using role and parts only; zero when empty. |
transcript_history_tokens |
Token estimate of history messages across all threads, using the same role/parts representation; zero when empty. |
transcript_content_characters |
Sum of JSON-serialized message-part lengths across history and current turn, in UTF-16 code units; excludes message wrappers and provenance. |
transcript_tool_response_characters |
The subset of content characters belonging to tool-role messages or unmatched tool-result parts; zero when absent. |
transcript_thread_count |
Number of assembled conversation threads, or zero for a null transcript. |
transcript_observation_count |
Ordered, deduplicated observations passed to assembly and the Topics renderer. The observation_count span attribute is the raw row count before deduplication. |
transcript_history_message_count, transcript_history_part_count |
Replayed messages and their normalized parts across all assembled threads. |
transcript_current_turn_message_count, transcript_current_turn_part_count |
This-trace messages and their normalized parts across all assembled threads. |
transcript_current_turn_tool_call_count, transcript_current_turn_tool_result_count |
Tool-call parts and tool-result parts in this trace; a tool-role part also counts as a result. |
transcript_assembly_phase_duration_ms |
Two non-overlapping samples per trace tagged phase:normalization or phase:matching. Normalization includes initial message-key construction and partitioning; matching covers remaining assembly work, including tool matching, deduplication, any rebuilt keys and finalization. Observation ordering is outside these phases but remains in total assembly time. |
topics_transcript_characters |
UTF-16 length of the complete rendered Topics text, including labels, section headings and line breaks; admitted traces only. |
topics_transcript_tokens |
o200k estimate of that same text, tagged tokenizer:o200k_base; admitted traces only. |
transcript_comparison_render_duration_ms |
Time to serialize assembled JSON and render generic and Topics text, excluding tokenization. |
transcript_comparison_tokenization_duration_ms |
Wall time for the three sequential JSON, generic and Topics token estimates, excluding the existing current-turn/history estimates. |
topics_transcript_block_characters |
Present block content length tagged block:user|assistant|system|reasoning|tool_calls|tool_results|tool_definitions|errors|run_io|observations and stage:raw|clipped; absent blocks emit no sample. |
topics_transcript_blocks_cut |
Number of content blocks cut by their per-block character caps. |
topics_transcript_history_characters, topics_transcript_current_turn_characters, topics_transcript_history_share |
Rendered content-line characters from replayed input and this run, plus replayed input divided by their sum (0 when both are empty). This-run content includes fallback trace-level I/O, observation markers, and inline errors. |
Topics measurement uses the existing assembled transcript and the inclusive
Topics preset in packages/shared/src/server/transcript/topics-renderer-config.ts. It
does not call a model or store the text. The preset includes all block types and
history, caps fallback trace-level input and output at 10,000 characters each,
and caps each system message at 600 characters. These are separate per-block
limits, not a combined trace budget. The preset has no total token budget. It
omits no middle lines. The rendered text has
<run_facts>, optional <tools> and <earlier_conversation source="replayed input">,
<this_run>, and <end_of_run> sections in that order. Observation markers show
the Langfuse operation type and name in walk order; matched tool results already
identify their operation. Trace input appears as the request when no current-run
user message exists. If it differs from the rendered input messages, it appears
as trace context alongside the user request, capped at the user block limit
(2,000 characters in this preset). A trace output matching the last assistant
message or tool
result labels that message as final output; distinct application output appears
once as [final output]. Thread segments follow observation
order; the replayed prefix does not claim to come from a previous trace. Tool
calls and results share a number, and observation errors appear after the last
message emitted at or before their position in the observation walk. The facts
line counts rendered user entries and tool calls across threads; repeated
content in separate threads is not necessarily a distinct user turn. raw block
characters are counted after media redaction and whitespace collapse but before
a block cap; clipped counts the resulting content including any omission marker.
Block values exclude
role labels, section headings, run facts and line breaks; media placeholders
can add characters to the complete text without adding to a block metric. The
history share uses content lines including role labels, tool exchanges, fallback
root I/O, observation markers and inline errors, excluding section headings and
tool definitions. Keep
numerator and denominator on the same basis when calculating shares. Ingestion
applies LANGFUSE_TRACE_BATCH_SAMPLING_RATE by trace ID before the dispatcher
and worker; every admitted trace receives Topics measurements. Projects where
isTopicsProcessingActive holds (in the Topics allowlist
LANGFUSE_TOPICS_ENABLED_PROJECT_IDS, Topics enabled with all three models, and
an active rule with an unblocked facet) bypass sampling so every trace is
summarized; a rate of 0 limits the flow to those projects and skips all
per-trace tracking work (no per-trace state or metrics) for the rest. Projects
outside the allowlist make no Redis or Postgres calls; allowlisted ones read the
cached gate answer (5 minutes), and a miss queries Postgres with a 2-second
deadline. If that lookup fails or times out, ingestion falls back to the
allowlist, caches that answer for 30 seconds and increments topics_gate_failures.
Transcript token counts use the existing local worker-thread pool and bundled
tiktoken WASM, without a network or model API call. The gpt-4o configuration
selects o200k_base, also used by GPT-5 mini and nano
(OpenAI mapping).
The span records tokenizer:o200k_base. These are serialized-payload estimates, not provider
billing counts or a model's full request framing.
Current turn and history use identical { messages: [{ role, parts }] } JSON
framing, without observation provenance. Only nonempty partitions are tokenized,
at most twice per trace. Their sum is transcript_message_tokens; it is not
directly comparable to the full assembled-transcript JSON or rendered text. The
transcript_json_*, generic_transcript_*, and topics_transcript_* lengths
are span attributes on the same admitted trace cohort. The generic
renderer preserves the previous plain-text layout under the same Topics
per-block preset; it has no trace-root I/O or run-state sections. Empty
transcripts have zero JSON length and tokens. Thread counts are numeric
span attributes.
To diagnose large Topics texts, compare topics_transcript_tokens with
transcript_observation_count, the history/current-turn message and part counts,
current-turn tool-call/result counts, topics_transcript_history_share, and
raw/clipped characters by block type. Structure counts describe the assembled
source, before the renderer's per-block character caps; they are not a count of
visible lines. A last-N history policy can use each thread's
conversationHistory, but a per-thread N would still grow with the number of
threads. A future total history budget should account for all threads; tool
loops should retain call/result pairs and errors when shortened.
For rough tool-response size share, divide transcript_tool_response_characters
by transcript_content_characters (when nonzero). Both sum serialized parts on
the same basis, including part JSON syntax but excluding message wrappers and
provenance. This is a character share, not a token share. For an overall traffic
share, divide the sums rather than averaging per-trace percentages. These metrics
are emitted before tokenization, including when it fails; no tool tokenizer runs.
Unknown estimates are omitted and counted in token_estimation_unavailable.
Rejected estimates increment token_estimation_failed and stop the remaining
estimates for that trace; successful earlier samples are retained. Unknown
estimates do not stop later estimates. The summed estimate is omitted unless
both partitions are known. Neither failure retries the batch.
Current-turn and history message projections are tokenized first. Each admitted
trace then adds one full-JSON, one generic plain-text, and one Topics plain-text
tokenization call.
Missing or failed JSON estimates increment transcript_json_token_estimation_unavailable
or transcript_json_failed without stopping later measurements. Generic text
rendering or token failures increment generic_transcript_failed, while an
unavailable token estimate increments
generic_transcript_token_estimation_unavailable. These do not stop Topics.
Topics rendering failures increment topics_transcript_failed and leave the
batch successful. An
unavailable Topics token estimate increments
topics_transcript_token_estimation_unavailable; a rejected estimate increments
topics_transcript_failed. Character metrics remain available in either case.
The reader queues completed traces behind one serial transcript/Topics chain
while continuing to consume ClickHouse rows. Projections are tokenized
sequentially, keeping at most one pending tokenizer request per batch. The chain
yields to the event loop before each trace's assembly. When queued logical
payload bytes exceed LANGFUSE_TRACE_BATCH_MAX_PENDING_SUMMARY_BYTES, reading
pauses until completed summaries reduce the queue to that threshold or below;
it does not wait for the entire chain. Each pause increments
langfuse.trace_batch.summary_backpressure. Time from enqueue to summary start
is recorded as langfuse.trace_batch.summary_queue_wait_ms.
The dispatcher byte budget estimates only pending updates, while reads can
include older observations, so it does not guarantee avoiding backpressure.
LANGFUSE_TRACE_BATCH_REQUEST_TIMEOUT_MS allows 40 seconds of socket inactivity
by default. This is a starting value, not a per-trace processing deadline:
slow tokenization or provider calls can still exceed it. Tune the timeout with
observed pauses and provider latency; the query's server execution limit remains
30 seconds. Success and stream failure both drain accepted summaries; a stream
failure never assembles its partial final trace. Assembly itself remains
synchronous. Wall-clock duration
of that work, including tokenization, is the span attribute duration_ms and
the experiment completion log. The generic queue series
langfuse.queue.trace_batch.time_distribution with type:processing records
the same duration when the processor returns.
Memory includes the current trace, queued observations, the active transcript
and stream buffers, multiplied by active batch jobs. The payload threshold is
checked at trace boundaries and is not a hard job-memory limit.
The active trace also retains its
assembled JSON, generic text, and Topics text for sequential tokenizer calls.
A single trace remains unbounded. Each message belongs to one JSON partition;
there is no additional tool-response tokenization pass.
The default two-thread tokenizer pool is shared with ingestion; overlap hides
waiting time but does not remove CPU use or contention. Its existing 30-second
timeout rejects the promise without cancelling queued/running encoding, so the
pending-promise bound is not a cancellation guarantee. Watch tokenization failures
alongside runtime.node.mem.rss, runtime.node.mem.heap_used, worker
CPU and queue depth before raising batch concurrency.
ClickHouse must sort the filtered result, so compare query memory and latency against the unordered baseline before increasing load.
Transcript sizes are always span attributes. The same measurements are also
dogstatsd distributions when LANGFUSE_TRACE_BATCH_TRANSCRIPT_METRICS_ENABLED
is true. That switch defaults to false. Use the trace explorer query below
either way.
Inspect individual transcripts in Datadog
Each completed trace emits a trace-batch-transcript child span under the batch
processing span. Its attributes include langfuse.project.id, langfuse.trace.id,
and langfuse.trace.url (a peek link using the configured product base URL).
Under langfuse.trace_batch, the span records transcript_message_tokens, transcript_characters,
transcript_json_characters, transcript_json_tokens,
generic_transcript_characters, generic_transcript_tokens,
topics_transcript_characters, topics_transcript_tokens,
transcript_comparison_render_duration_ms,
transcript_comparison_tokenization_duration_ms,
transcript_assembly_duration_ms, observation_count (rows before observation
deduplication), has_transcript, tokenizer, and experiment_id.
It also records the token breakdown and content/tool-response character metrics,
transcript_thread_count,
and each phase as transcript_assembly_<phase>_duration_ms.
It also records the remaining Topics metrics above. Block character attributes use
topics_transcript_block_<block>_<stage>_characters. Topics character metrics
are absent if rendering failed; the token metric can also be absent if its
estimate failed.
No transcript content is attached. Project and trace IDs stay on the span.
transcript_characters is the legacy span-only name for the same value as
transcript_json_characters. The full JSON includes syntax and provenance, so
use transcript_content_characters as the tool-response denominator. JSON
characters are recorded before tokenization and remain available if an estimate
fails. The JSON string is retained only until its sequential token count settles;
it is not included in the assembly-duration measurement.
The span stays open until token estimation settles, so its duration includes
tokenization and pool waits. Use the assembly-duration attribute for assembly
performance. Missing estimates have token_estimation:unavailable|failed; their
counts are omitted while successful earlier estimates remain. Null transcripts
have zero tokens. The child span does not become
active while the next trace streams, and failed streams do not emit a span for
their partial final trace.
In Datadog APM Trace Explorer, search for:
env:prod-eu service:worker-cpu resource_name:trace-batch-transcript @langfuse.trace_batch.transcript_message_tokens:>=100000
Add token count as a numeric measure to sort largest first, display the project
and trace IDs, and open langfuse.trace.url. The peek view shows the current
source observations, not a saved copy of the measured transcript. Late arrivals
and retries can produce different or repeated samples for the same trace.
These spans follow existing APM ingestion sampling and retention. Configure a custom retention filter at 100% for the outlier query to keep matching ingested spans searchable. This cannot recover spans dropped before ingestion. Transcript sizes live on these spans and follow APM sampling; they are not separate custom metrics.
Capacity measurements
The following metrics use the langfuse.trace_batch prefix. All additions are
inside the existing cloud experiment paths; normal ingestion with tracking
disabled performs no new aggregation or Redis calls.
| Metric | Kind / tags | Meaning |
|---|---|---|
event_updates, serialized_event_bytes |
Counters; stage:eligible|recorded |
Accepted, valid-start-time updates before sampling, and in Redis-acknowledged chunks. Repeated updates count again. Selected volume is sampling_decisions. |
active_reads |
Per-process gauge | Streams currently being consumed; failures decrement the count too. |
failed_read_observation_count, failed_read_input_bytes |
Distributions | Partial logical rows and input bytes consumed before a stream failure. Separate from successful throughput. Output and metadata bytes stay on the completion log. |
queue_depth |
Gauge; type:delayed |
Delayed BullMQ jobs. Waiting, active, and failed depth stay on langfuse.queue.trace_batch.depth. Request rate, wait time, and processing time use the generic langfuse.queue.trace_batch series, the same as every other queue. Processing time is recorded when the processor returns. Discard reasons stay on discarded_jobs. |
queue_waiting_head_age_ms |
Gauge | Maximum creation age of the next FIFO jobs in waiting/paused lists. Zero when empty. |
redis_key_bytes |
Gauge; key:due|state|first, shard:total |
Estimated memory of each readiness key, including Redis overhead. |
redis_key_entries |
Gauge; key:due|state|first, shard:total |
ZCARD / HLEN at the memory snapshot. |
pending_traces, ready_traces, oldest_due_age_ms |
Gauges, untagged | Dispatcher snapshot summed (age: maximum) across shards, emitted by the current dispatcher lease holder. |
The eligible and recorded counters reuse the already-computed serialized size and per-trace aggregates. Recorded volume is emitted after each acknowledged Lua chunk: earlier chunks remain counted if a later one fails. A timeout can mean Redis applied an update without acknowledgement, so the stage difference includes uncertain outcomes; it does not prove data loss. Writer acceptance still precedes the ClickHouse flush. No additional serialization is performed.
TraceBatchMetricsRunner starts when either the dispatcher or consumer is
enabled in a cloud worker, independently of the generic queue-metrics flag.
It polls the queue every 30 seconds and map memory every 60 seconds per process,
plus collection time. No completed jobs or progressing dispatcher are required.
Queue-age collection reads only the next waiting/paused job IDs and their
timestamp fields; it never fetches or parses batch payloads.
One failed collection does not suppress the others; failed samples are not
reported as zero. Check langfuse.periodic_runner.completed and
last_healthy_timestamp_seconds with runner:trace_batch_metrics for freshness.
Memory collection uses bounded per-shard Lua calls with MEMORY USAGE ... SAMPLES 5, preserving the shared key prefix and Redis Cluster slot routing.
Missing keys report zero; there is no keyspace scan or exact full-hash traversal.
Memory sampling estimates storage, not process RSS or event payload size.
Map gauges are summed across readiness shards before they are emitted. Each worker emits them, so a shard tag would multiply custom metrics by the shard count on every reporting worker. The dispatcher snapshot gauges are untagged for the same reason. Inspect a single shard's keys in Redis when diagnosing skew.
Queue/map snapshots are global and can be emitted by several workers: use
max/latest across reporters, never sum duplicates. First-seen
counts include physically retained expired entries until their bucket expires.
In contrast,
active_reads is local to a worker; distinct live worker series may be summed.
It is emitted on stream start/finish and on each metrics-runner cycle, including
zero when idle, so unchanged long-running reads remain visible.
Instantaneous gauges can miss short-lived peaks. A retried job may re-enter
behind newer jobs, so waiting-head age is a bounded-cost backlog indicator, not
an exact oldest-created-job measurement across the entire queue.
With an experiment ID, start/completion logs include jobId, attempt,
queryId and experimentId, with the completion outcome. The per-read UUID is
forwarded to ClickHouse as query_id and remains the root correlation ID for
the stream. Discards and validation failures have no query ID. Identifiers are
not metric tags and logs contain no event payloads. Successful batch metrics and
job return values retain their existing meanings.
Completion logs also correlate the requested batchTraceCount, distinct
batchProjectCount, eventTimeSpanMs (outer event-time span, without query
buffering), and maxTraceSpanMs with the outcome and durationMs. These are
batch-locality proxies, not measured ClickHouse scan costs. observationCount,
foundTraceCount, foundProjectCount and separate inputBytes, outputBytes,
metadataBytes describe rows consumed by that attempt. partial: true marks
failed reads, including failures before any row arrives; zero then does not mean
the requested batch was empty. Successful counts cover the completed stream,
not unique ingested events. Discarded/unparsed jobs omit stream counters.
Use these existing logs to compare batch shape and volume across outcomes by
query ID; no payload contents, project-ID lists or per-batch metric tags are added.
With an experiment ID, the BullMQ processing span also carries attributes under
langfuse.trace_batch.*: experiment_id, job_id, attempt (one-based),
query_id, batch_trace_count, batch_project_count, event_time_span_ms,
and max_trace_span_ms. Shape and query ID are attached before reading, so they
remain available if the stream fails. Completion adds outcome, duration_ms,
observation_count, found_trace_count, found_project_count, input_bytes,
output_bytes, metadata_bytes, and partial, with the same semantics as the
logs. Discards and validation failures omit query/stream fields. These reuse
existing aggregates without additional queries or serialization. Payloads and
project/trace-ID lists are not attached. Dispatcher size estimates remain aggregate
metrics, not per-job span attributes; consumed bytes are not estimated total size.
Span availability follows the existing tracing sampling and retention settings.
For the initial scaling curve, use deterministic trace sampling at 10%, 20%, 50%, then 100% across the same project population. Annotate fixed UTC windows; pause tracking and drain old readiness/queued work between settings. Warm up at least 20 minutes and compare a settled hour at each stage, holding batch/query settings fixed. Plot actual admitted updates/bytes and returned rows/bytes against CPU, memory and backlog rather than assuming configured percentages equal load. These additions do not generate amplified reads or change batching, query limits, late-arrival behavior, queue payloads or retention.
Pair the metrics with per-worker RSS/CPU/event-loop and network telemetry, Redis primary telemetry, and experiment-filtered ClickHouse query logs. Include failed attempts in CPU cost, count successful root results once, and verify distributed CPU-counter accounting before summing query parts. Query memory peaks are not actual concurrent replica RAM. Validate live metric delivery and units before raising production load.
Compare reader settings
Establish a fresh control measurement after deploying this reader: both arms must use the same per-trace windows and compression. Hold sampling, batch cap, concurrency, threads, idle delay and traffic cohort constant; then change only block size (and the identifying label):
| Variable | Control | Treatment |
|---|---|---|
LANGFUSE_TRACE_BATCH_MAX_THREADS |
2 |
2 |
LANGFUSE_TRACE_BATCH_MAX_BLOCK_SIZE |
unset | 512 |
LANGFUSE_TRACE_BATCH_EXPERIMENT_ID |
trace-block-control |
trace-block-512 |
Run these settings only in cloud workers with the existing four enablement
flags explicitly set to true. Changing a query setting does not enable any
part of the pipeline. max_block_size is a row-count hint, not a byte or memory
limit; large inputs may still dominate memory. Keep the initial cap at 120 when
comparing existing experiments; raising the supported ceiling is not a rollout
recommendation.
With an experiment ID, each read attempt logs its label, build ID, configured
threads/block size, batch size, concurrency, sampling and timing controls before
querying, including failed attempts. The same ID enters ClickHouse
log_comment alongside existing surface/route/project attribution. Leave the ID
unset to suppress these per-attempt logs. Strategy attribution uses the
dispatcher metrics described below. Compare
query memory/latency and worker memory as well as input/output/metadata byte
metrics. Redis storage metrics do not account for selector or query memory.
Opt-in locality grouping
LANGFUSE_TRACE_BATCH_STRATEGY=locality changes grouping only after dispatch is
explicitly enabled in a cloud region. The default project strategy preserves
project ordering and carries unfinished jobs across hydration chunks.
Locality sorts each hydrated window by project, minimum start minute, maximum
start minute and xxHash32(traceId). It finds the fewest feasible jobs under the
trace cap, the weight budgets and an event-time envelope of at most one hour, or
125% of the first trace's observed span if larger. Within that job count, it minimizes project
boundaries, then minute × trace count, then gaps between trace hashes. These are
read-locality proxies, not measured ClickHouse scan costs.
Locality combines each window of at most 1,000 hydrated candidates with retained
partials. Compatible same-project partials may merge when their buffered time
intervals overlap and the combined batch respects the cap and envelope limits.
At most maxBatchSize - 1 traces are retained across all partials; oldest-due
partials dispatch first when that budget is exceeded. The last window flushes
all remaining partials, so carry never outlives the current dispatch run.
The full ready-ID snapshot remains unchanged; only selector input is bounded. Selector cost is O(n × k × cap), where k is the fewest feasible jobs. Measure duration before increasing scale, especially with distant event times that force many jobs.
Compare dispatched_batches (tags strategy, fill),
event_time_envelope_ms, candidate_buffer_size and
selector_duration_ms under langfuse.trace_batch. Per-trace event span stays
on the completion log as maxTraceSpanMs. The envelope metric includes
the reader's two-minute margin on each side; the selector's envelope limit does
not. Payload and Redis state are unchanged.
Monitor dispatcher memory
The baseline intentionally snapshots the complete ready cohort. Batch size limits jobs, not snapshot memory. Retain this behavior for the initial experiment and watch memory alongside backlog growth before deciding whether to add paging.
The worker enables Datadog Node runtime metrics. In Metrics Explorer, filter to the experiment environment and worker service, and inspect the maximum per worker/container rather than a fleet average:
runtime.node.mem.rss: total process memory; compare with the container limit.runtime.node.mem.heap_used: JavaScript heap in use.langfuse.trace_batch.snapshot_size,pending_traces,ready_tracesandoldest_due_age_ms: snapshot and backlog growth.langfuse.trace_batch.expired_traces: taggedsource:ingestionorsource:dispatcher. Sum within one source; the two writers are different events.langfuse.periodic_runner.duration_msandcompleted, filtered byrunner:trace_batch_dispatcher: run duration and outcome.
Runtime memory is shared with other work in the process, so correlation with
dispatch runs does not isolate the dispatcher's allocations. Periodic sampling
can miss short peaks, and snapshot_size is emitted only after parsing/sorting
finishes. Check worker restarts/OOM events as well. Verify live metric delivery
and available instance tags in Datadog before relying on these signals.
Stop and drain
Upgrading from the unsharded layout requires a drain; mixed writers and
dispatchers must not run against different layouts. Disable producers, let the
old dispatcher empty {trace-batch}:due and {trace-batch}:state, and drain the
queue with the old consumers. Stop old dispatchers before deploying all new
producers/dispatchers. Resume tracking only once all use the new layout. There
is no retained first-seen history for traces dispatched before this upgrade.
The same procedure applies before changing shard count or retention. Do not
delete pending keys to simulate a drain. Old queue payloads remain parseable.
Draining pending work does not migrate first-seen history. A shard-count change
must also wait for the old first-seen buckets to expire (TTL plus up to one
bucket), or explicitly accept losing that retained history. Offline migration
of first-seen entries is not implemented here.
Disable ingestion and dispatch to stop new work. Leave the consumer enabled and
set the read flag to false to complete and remove jobs without querying or
parsing their payloads. Jobs at least two hours old are also discarded on pickup.
Reads already in progress finish under their query timeout.
On process shutdown, the dispatcher stops scheduling and waits up to five seconds for its active dispatch. If Redis stalls, it logs a warning and lets the remaining shutdown steps proceed. This timeout does not cancel an already issued Redis command.
BullMQ waiting jobs have no autonomous TTL: disabling all consumers leaves them queued until consumption resumes. Draining requires an active, unpaused worker; delayed jobs become eligible later. Normal completed jobs retain IDs for up to one hour / 10,000 jobs, and failures retain up to 1,000 jobs under BullMQ's lazy retention rules. Drain and expired jobs are removed immediately on success.
For an already enabled cloud experiment, explicitly set the new read flag to
true when deploying this version if reads should continue; leaving it unset
drains instead. Upgrade all producers before relying on native expiry: older
producers do not install or refresh that TTL.