:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Parity with the Node ingestion framework
This crate is a port of nodejs/src/ingestion/framework.
The two implementations are kept in step: a change to one lands with the same change to the other, and the mirrored tests below change together.
File map
Node (nodejs/src/ingestion/framework/) |
Rust (rust/ingestion-framework/src/) |
|---|---|
results.ts |
results.rs |
pipeline.interface.ts |
context.rs (types), pipeline.rs (Pipeline trait) |
chunk-pipeline.interface.ts |
chunk_pipeline.rs (ChunkPipeline trait), context.rs (ChunkResults) |
steps.ts |
steps.rs |
start-pipeline.ts, step-pipeline.ts |
pipeline.rs |
branching-pipeline.ts |
branching.rs |
timeout-barrier-pipeline.ts |
timeout_barrier.rs |
batch-budget.ts |
budget.rs |
base-chunk-pipeline.ts |
chunk_pipeline.rs (BaseChunkPipeline, apply_chunk_step_to_results) |
buffering-chunk-pipeline.ts |
chunk_pipeline.rs (BufferingChunkPipeline) |
sequential-chunk-pipeline.ts |
chunk_pipeline.rs (SequentialChunkPipeline) |
gathering-chunk-pipeline.ts |
chunk_pipeline.rs (GatheringChunkPipeline) |
chunk-pipeline-unwrapper.ts |
chunk_pipeline.rs (ChunkPipelineUnwrapper) |
concurrent-chunk-pipeline.ts |
concurrent.rs |
concurrently-grouping-chunk-pipeline.ts |
grouping.rs |
filter-map-chunk-pipeline.ts |
filter_map.rs |
fan-out-fan-in-chunk-pipeline.ts |
fan_out_fan_in.rs |
interleaving-chunk-pipeline.ts |
interleaving.rs |
resettable-signal.ts |
sync.rs (ResettableSignal); sync.rs (FifoLimiter) replaces p-limit |
result-handling-pipeline.ts |
result_handling.rs |
result-handling-helpers.ts |
result_handling.rs (produce_message_to_dlq, redirect_message_to_output, headers) |
side-effect-handling-pipeline.ts |
side_effect_handling.rs |
ingestion-warning-handling-chunk-pipeline.ts |
warning_handling.rs |
per-element-step.ts |
per_element.rs (per_element, also exported from builders.rs) |
retry.ts (+ retryIfRetriable) |
retry.rs |
batching-pipeline.ts |
batching.rs |
metrics.ts |
metrics.rs (same metric names and labels) |
helpers.ts |
helpers.rs |
builders/pipeline-builders.ts |
builders.rs (new_pipeline, PipelineBuilder, BranchingPipelineBuilder) |
builders/chunk-pipeline-builders.ts |
builders.rs (new_chunk_pipeline, ChunkPipelineBuilder, group/fan-out/result builders) |
builders/helpers.ts |
builders.rs (new_batching_pipeline) |
~/common/outputs (IngestionOutputs) |
outputs.rs (IngestionOutputs trait, KafkaMessage, Headers, MemoryOutputs) |
~/common/utils/promise-scheduler |
side_effects.rs (PromiseScheduler, SideEffectScheduler) |
~/ingestion/common/ingestion-warnings.ts (serializeIngestionWarning, emitIngestionWarning) |
warning_handling.rs (serialize_ingestion_warning, OutputsWarningEmitter), js_json.rs (JSON.stringify output) |
~/ingestion/common/ingestion-warning-types.ts (INGESTION_WARNING_TYPES) |
warning_handling.rs (warning_classification), read from rust/common/ingestion_warnings/warning_types.generated.json, which the Node registry exports |
span-attributes.ts, tophog/, extensions/ |
not ported (see below) |
Mirrored tests
Each living-docs chapter docs/NN-*.test.ts has a Rust twin tests/docs_NN_*.rs, with one #[tokio::test] per Node it(...).
The Rust test name is the snake_case form of the Node test name.
Jest fake timers map to #[tokio::test(start_paused = true)]: tokio advances the paused clock by itself when every task waits, and tokio::time::advance replaces a synchronous jest.advanceTimersByTime.
per-element-step.test.ts has the twin tests/per_element_step.rs.
tests/unit_semantics.rs pins behavior the chapters leave implicit: poison-then-drain for the concurrent and grouping stages, feed-while-parked, FIFO admission under max_concurrency, filter-map passthrough order, DLQ header layout, DLQ rethrow, warnings on timeouts, fan-out DLQ aggregation, and concurrent batch feeds on a multi-thread runtime.
tests/review_regressions.rs pins divergences that a review found and that are now fixed: the ingestion warning bytes, the fan-out dlq_reason, produce order on a multi-thread runtime, the redirect produce before next() returns, a dropped batching next(), scheduler cleanup, tries: 0, top-level retriability, a fan-in panic, max_concurrency: Some(0), the cost of a group start, overlapping and cancelled scheduler waits, the parents a fan-in panic leaves ready, a group processor that panics before it returns its future, a dropped limiter acquire(), and retry options that give an out-of-range sleep.
consumeAll(pipeline, ms) and jest.advanceTimersByTimeAsync(ms) before an await map to consume_all_within and within_ms in tests/common, which fail the test when the work needs more virtual time than Node gives it.
Skipped cases
| Node test | Why |
|---|---|
docs/16-tophog-metrics.test.ts (all four cases) |
TopHog (framework/tophog/) is a separate metrics aggregation subsystem, not part of the pipeline composition this crate ports. Port it as its own module if a Rust pipeline needs it. |
Everything else is mirrored. Where a Node assertion relies on a JS runtime detail, the Rust test checks the same observable behavior another way:
13-conventions"pipelines are created via factory functions" usesArc::ptr_eqfornot.toBe.07-result-handling"REDIRECT with awaitAck=true" checks thatnext()does not finish within 50 ms while the ack is held, instead ofsetImmediate.08-side-effects"steps can add side effects" checks that the side effect ran before the step result is awaited, becauseSideEffect::spawnruns the future up to its first pending point, as a Node promise executor runs.
Known differences
These are deliberate. Each one follows from a language difference. Items 1 to 15 do not change what a pipeline emits; items 16 to 19 change only timing or a failure path.
- Step names are explicit. Node reads
fn.name. Rust closures have no name, sostep("name", f)andchunk_step("name", f)take one. Use the Node function name. - A step error is
Err, not a throw.PipelineError::retriable/non_retriablestand in forerror.isRetriable. An error with neither flag behaves like a Node error with noisRetriable: retried, then propagated.is_retriablereads the outermost error only, as Node reads the property of the thrown object:PipelineError::non_retriable(..).context(..)has no flag, so it retries. ASharedErrorreads through, because it is the same error object shared by more than one caller. - Side effects start on creation. A Node promise runs its executor (or an async function its body up to the first
await) when it is created; a Rust future runs only when polled.SideEffect::spawnpolls the future once on the caller's thread and moves the unfinished rest to a tokio task, so DLQ, redirect and warning produces are issued in item order insidenext(), as in Node. A panic in that first poll becomes a failed side effect, as a throw before the firstawaitrejects a promise. ASideEffectcannot be cloned, so a step's side effects and warnings move from the result into the context (Node keeps a copy on both). ResultWithContext.resultis anOutcome. Side effects and warnings live only on the context once a step finishes.- The user context is a field. Node spreads
Cinto the context object; Rust stores it inPipelineContext.datawithDeref.message_id(Node'sBatchingContext) is a base context field, so batching does not change the context type.traceContextis not ported. - Structural context access is a trait.
message_aware/handle_resultsrequireHasMessage;team_aware/handle_ingestion_warningsrequireHasTeamId. filter_mapoutput context. Node types it as the unionCSubOutput | CIntermediate. Rust requiresCSubOutput: From<CIntermediate>and converts non-OK passthroughs with it.- Batch context is not spread into values. Node's sub-pipeline values are
TInput & CBatch. Rust's areWithBatch<T, CB>(value plusArc<CB>, withDerefto the value). The caller's feed context returns asBatchResult.feed_contextrather than merged intobatch_context. - Retries clone the input.
pipe_with_retryneedsT: Clone + Sync, because each attempt takes the value by value. Every attempt but the last possible one gets a clone, sotries: 1clones nothing; the first of several attempts cannot take the original, because a retry needs it again. Node passes the same reference. - Compile-time checks that become run-time. Node checks branch exhaustiveness, redirect output names and warning types in the type system. Rust uses string names, so an unknown branch is the run-time DLQ that Node also has, an unknown output fails in the
IngestionOutputsimplementation, and an unregistered warning type fails its side effect, as Node's registry lookup throws at run time. - Ingestion warnings. The Node
IngestionWarningLimiteris an app concern:IngestionWarningEmitteris the seam, andOutputsWarningEmitteremits with no limiter.serialize_ingestion_warningwrites the Node bytes withjs_json(key order,JSON.stringifynumbers and escapes, array-index keys first), with no dependence on serde_json'spreserve_orderfeature.categoryandseveritycome from the registry JSON that the Node registry exports.PipelineWarning.detailsis anIndexMap, so its keys keep insertion order; build ordered details withwith_detail, because ajson!object sorts its keys unless the build enablespreserve_order. A nested object inside a detail value is aserde_json::Mapand is written in that map's order, which is sorted in a build withoutpreserve_order; Node keeps insertion order there too. Keep details flat where the bytes must match. - Fan-out DLQ error. Node builds
AggregateError(errors, reason); Rust buildserrors::AggregateErrorwith the same message and every sub error. Thedlq_reasonheader is identical. - Concurrency. Concurrent items and groups run as tokio tasks, so on a multi-thread runtime they run in parallel; Node interleaves them on one thread. Ordering guarantees (FIFO emission, per-key order, FIFO admission under a cap) are the same. When a group completes, the queues still waiting all belong to running groups, so their relative order no longer matters and the remove is O(1). The interleaving stage keeps the pending sub-pipeline pull as a stored boxed future so it survives across
next()calls, which is how Node keeps its memoized promise. That future is polled only while anext()is awaited. Item and group work runs in its own tasks and never pauses, but sub-pipeline stages that the pull itself drives (for example asequentiallyinsidefilter_map) advance only while the caller waits onnext(). A Node promise advances in the background. This changes timing, not results or order; every consumer loops onnext()anyway. A spawned task pernext()was measured first and cost a cross-thread wake per emitted chunk. - Clock.
BatchBudgetreadstokio::time::Instant, so it follows a paused test clock as Node's followsDate.now()under fake timers. - Not ported:
span-attributes.tsand OTel span parenting (Rust usestracing),tophog/, andextensions/. - Cancel safety. Node has no cancellation.
BatchingPipeline::next()is cancel-safe: it keeps its sub-pipeline pull, the results it has not routed, and a running afterBatch hook under its pump lock, and the next call resumes them, so a dropped call (for example from an HTTP handler whose client went away) loses no batch and runs no hook twice. AChunkPipeline::next()is not cancel-safe: the interleaving stages keep their pull, but a stage such aspipe_chunk, result handling or side-effect handling holds the chunk in the dropped future. Do not race a chunk pipeline'snext()against a timeout. A dropped batchingnext()also leaves the hook paused until the next call polls it, where Node's would run on. PromiseSchedulerkeeps the first failure. Node removes a promise when it settles, so a rejection that settles beforewaitForAll()is not seen there. The Rust scheduler keeps it, as theFailureLatchingPromiseSchedulerof the Node metrics consumer does, because a consumer that misses it stores the offsets of a batch whose side effect failed. A settled side effect leaves the set at the nextschedule()that finds the set grown to twice its size after the last cleanup, atpending_count(), or when await_for_all()sees it settle. As in Node, a side effect stays in the set until it settles, so an overlappingwait_for_all()also waits for it, and a cancelled one leaves it, and its failure, for the next call.wait_for_all()waits for every side effect pending at the call and then returns the first failure; Node'sPromise.allrejects at the first failure without waiting for the rest.- User callbacks that panic. A panic in a grouping function, fan-out function or fan-in function becomes an error from
next(), as a Node throw rejectsnext()and poisons the stage. No lock is held while user code runs.max_concurrency: Some(0)panics when the stage is built, aspLimit(0)throws. per_elementrejects late. Node'sPromise.allrejects at the first failed step, and the other steps run on. Rust waits for the other steps and then returns the first error to occur.
Benchmarks
benches/framework.rs (criterion) and nodejs/src/ingestion/framework/benchmarks/framework-bench.ts run the same six workloads at sizes 1, 100 and 10 000, with trivial steps, so the numbers are framework overhead per element. The Rust benchmark installs a Prometheus recorder, so both sides pay for their metrics.
cd rust && cargo bench -p ingestion-framework --bench framework
cd nodejs && DATABASE_URL=postgres://localhost/bench pnpm exec tsx src/ingestion/framework/benchmarks/framework-bench.ts
Measured overhead
One 4-vCPU cloud container, the two runs one after the other. Mean ns per element; lower is better. Rust numbers are on a current-thread runtime, which matches the single Node event loop.
| Workload | Size | Node (ns/elem) | Rust (ns/elem) | Node ÷ Rust |
|---|---|---|---|---|
step_pipeline_5_steps |
1 | 30 383 | 2 379 | 12.8× |
step_pipeline_5_steps |
100 | 17 827 | 2 349 | 7.6× |
step_pipeline_5_steps |
10 000 | 19 693 | 2 307 | 8.5× |
pipe_chunk_5_steps |
1 | 25 940 | 3 483 | 7.4× |
pipe_chunk_5_steps |
100 | 722 | 431 | 1.7× |
pipe_chunk_5_steps |
10 000 | 740 | 541 | 1.4× |
concurrently_max50 |
1 | 24 758 | 3 613 | 6.9× |
concurrently_max50 |
100 | 8 488 | 1 873 | 4.5× |
concurrently_max50 |
10 000 | 10 007 | 2 187 | 4.6× |
per_element_max50 |
1 | 20 834 | 2 399 | 8.7× |
per_element_max50 |
100 | 1 912 | 1 118 | 1.7× |
per_element_max50 |
10 000 | 4 108 | 2 248 | 1.8× |
concurrently_per_group_16 |
1 | 14 492 | 5 165 | 2.8× |
concurrently_per_group_16 |
100 | 7 470 | 1 302 | 5.7× |
concurrently_per_group_16 |
10 000 | 8 586 | 1 091 | 7.9× |
batching_full |
1 | 53 331 | 8 726 | 6.1× |
batching_full |
100 | 7 091 | 1 454 | 4.9× |
batching_full |
10 000 | 6 868 | 1 400 | 4.9× |
On a 4-worker multi-thread runtime, the 10 000-element workloads run at 1 885 (concurrently_max50), 947 (concurrently_per_group_16) and 1 798 (batching_full) ns per element. Single-element batches cost about 50 µs there, which is worker wake-up latency, so a current-thread runtime suits a consumer that feeds tiny batches.
The Node per-step cost is dominated by the per-step instrumentation (instrumentFn, the prom-client step histogram); the Rust port records the same histogram through the metrics crate.