:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(ingestion-consumer): settle and commit per group completion (#115486)
Paweł Ledwoń0edba1845c74
diff --git a/rust/Cargo.lock b/rust/Cargo.lockindex b6553a8d3121..5879b1dd25fd 100644--- a/rust/Cargo.lock+++ b/rust/Cargo.lock@@ -3035,6 +3035,7 @@ version = "0.1.0"dependencies = ["metrics","rdkafka",+ "tracing",][[package]]diff --git a/rust/common/kafka-consumer/Cargo.toml b/rust/common/kafka-consumer/Cargo.tomlindex c5b3c3f92e98..0541bda6d3f7 100644--- a/rust/common/kafka-consumer/Cargo.toml+++ b/rust/common/kafka-consumer/Cargo.toml@@ -10,3 +10,4 @@ workspace = true[dependencies]metrics = { workspace = true }rdkafka = { workspace = true }+tracing = { workspace = true }diff --git a/rust/common/kafka-consumer/src/topic_offset_ledger.rs b/rust/common/kafka-consumer/src/topic_offset_ledger.rsindex 2bf76769bfe5..0e33aaeeb3eb 100644--- a/rust/common/kafka-consumer/src/topic_offset_ledger.rs+++ b/rust/common/kafka-consumer/src/topic_offset_ledger.rs@@ -3,6 +3,7 @@ use std::sync::atomic::{AtomicU64, Ordering};use std::sync::{Arc, Mutex};use metrics::{counter, gauge};+use tracing::warn;use crate::charge::Charge;use crate::partition_offset_ledger::{Held, LedgerError, PartitionOffsetLedger, TakenFrontier};@@ -207,7 +208,7 @@ impl TopicOffsetLedger {(taken, ledger.held())};publish_held(topic_partition, held);- count_gap_offsets(topic_partition, taken.gap_offset_count);+ report_gap_offsets(topic_partition, &taken);Some(taken)}@@ -275,13 +276,21 @@ fn publish_held(topic_partition: &TopicPartition, held: Held) {.set(held.charge.bytes as f64);}-fn count_gap_offsets(topic_partition: &TopicPartition, gap_offset_count: usize) {- if gap_offset_count == 0 {+fn report_gap_offsets(topic_partition: &TopicPartition, taken: &TakenFrontier) {+ if taken.gap_offset_count == 0 {return;}let (topic, partition) = labels(topic_partition);counter!(GAP_OFFSETS, "topic" => topic, "partition" => partition)- .increment(gap_offset_count as u64);+ .increment(taken.gap_offset_count as u64);+ warn!(+ topic = %topic_partition.topic,+ partition = topic_partition.partition,+ window_base = taken.first.0,+ frontier = taken.offset.0,+ gap_offsets = taken.gap_offset_count,+ "Commit frontier walked over offsets Kafka never delivered"+ );}/// Count one charge or settlement the ledger rejected. A stale slice isdiff --git a/rust/ingestion-consumer/README.md b/rust/ingestion-consumer/README.mdindex 3f0f8963cda7..3de9c7c24179 100644--- a/rust/ingestion-consumer/README.md+++ b/rust/ingestion-consumer/README.md@@ -1,24 +1,24 @@# ingestion-consumerRust Kafka consumer that routes analytics events to Node.js ingestion workers over ordered gRPC streams, with at most one request per key in flight.-It reads batches from Kafka, groups messages by Kafka message key, packs each key's messages into requests (preserving per-key ordering; unkeyed messages can go to any worker), sends each request on its worker's `WorkerIngest` stream, and commits offsets only after every message in the batch is accepted.+It reads batches from Kafka, groups messages by Kafka message key, packs each key's messages into requests (preserving per-key ordering; unkeyed messages can go to any worker), sends each request on its worker's `WorkerIngest` stream, and commits each partition's frontier: one past the prefix every worker accepted.Worker health combines active `/_ready` probes with passive send outcomes; workers that leave the pool drain gracefully: in-flight work finishes, and new work for their keys waits and goes to survivors in order.## Ordering sentinels-`order_sentinel.rs` turns the consumer's two core guarantees into always-on, alertable metrics (see the module docs for semantics).+`order_sentinel.rs` and the offset ledger turn the consumer's core guarantees into always-on, alertable metrics (see the module docs for semantics).Each guarantee has a violation counter that must stay flat and a denominator that must grow:| Guarantee | Violations (must stay 0) | Denominator / supporting || --- | --- | --- |-| Commits are contiguous, monotonic, non-empty per partition, and cover only completed offsets | `ingestion_consumer_commit_violations_total{kind=gap\|out_of_order\|overlap\|empty\|uncredited}` | `ingestion_consumer_commits_checked_total`, `ingestion_consumer_committed_offset{topic,partition}` |-| Async commits actually succeed | `ingestion_consumer_commit_confirmation_lag{topic,partition}` persistently > 0 | `ingestion_consumer_broker_committed_offset{topic,partition}`, `ingestion_consumer_last_successful_commit_timestamp_seconds`, `ingestion_consumer_commit_monitor_errors_total` |+| Commits are contiguous and monotonic per partition, each the ledger frontier one past the accepted prefix, and cover only completed offsets | `kafka_consumer_ledger_errors_total{stage,kind}` (a redelivered or out-of-window offset), `kafka_consumer_ledger_gap_offsets_total{topic,partition}` (offsets a frontier walked over without delivery), `ingestion_consumer_commit_violations_total{kind=empty\|uncredited}` | `ingestion_consumer_committed_offset{topic,partition}`, `kafka_consumer_ledger_uncommitted_offsets{topic,partition}` |+| Async commits land on the broker | `kminion_kafka_consumer_group_topic_partition_lag{group_id}` growing while `ingestion_consumer_messages_processed_total` holds its rate | `ingestion_consumer_committed_offset{topic,partition}` (the offset last handed to the client), `ingestion_consumer_offset_commits_total` || Per-key sends are in offset order, never re-sent after ACK (keyed messages only) | `ingestion_consumer_key_order_violations_total{kind=intra_group_disorder\|resend_after_ack}` | `ingestion_consumer_key_replays_total` (legal at-least-once retries), `ingestion_consumer_key_sentinel_keys`, `ingestion_consumer_key_sentinel_unkeyed_total` (skipped null-key messages) || Messages enter the worker pipeline in per-key offset order (end to end) | `ingestion_api_out_of_order_messages_total` (worker-side) | `ingestion_api_replayed_messages_total`, `ingestion_api_order_sentinel_keys` |Commit success can't be observed via rdkafka's `commit_callback` — librdkafka drops the result of manual async commits (no conf-level `offset_commit_cb` is ever registered by rust-rdkafka, and only sync commits attach a reply queue).-Instead a background commit monitor fetches the group's broker-committed offsets every 30s (one OffsetFetch) and compares them to what was attempted: `commit_confirmation_lag` is transiently positive while async commits are in flight, and persistently positive when commits are submitted but not landing (e.g. a stuck coordinator).-Alerts should gate on `ingestion_consumer_offset_commits_total > 0` — an idle consumer that never committed has nothing to confirm.+The broker's view comes from kminion, which exports every group's lag per partition; commits that stop landing show as lag that grows while the consumer's processed rate holds, or as lag that stays above zero on a partition with nothing new to read.+A commit that never lands costs a replay from the last landed commit on the next restart, not a loss.Both sides default to enabled and have kill switches: `CONSUMER_ORDER_SENTINEL_ENABLED` here, `INGESTION_API_FEED_ORDER_SENTINEL_ENABLED` (plus `INGESTION_API_FEED_ORDER_SENTINEL_MAX_KEYS`) on the worker.The rebalance metrics from the consumer context stay on regardless, and the `consumer_id`/`replay` request fields are always stamped so either side can be toggled independently.@@ -31,17 +31,23 @@ Null-key messages (e.g. overflow rerouting) are excluded from both checks: the p## Offset ledgerThe consumer holds the per-partition offset ledger from `common/kafka-consumer`, which owns the commit.-Every delivered message is charged to its partition's ledger during collection, and a committed batch completes its offsets there; the commit is then each partition's frontier, one past its longest completed prefix.-A partition that settles without a frontier is not committed and stays on its last commit.-A batch with no frontier on any partition is not committed at all; `ingestion_consumer_commits_skipped_total{reason}` counts those, where `rejected` means the ledger dropped every slice (expected around a rebalance) and `no_frontier` means an earlier batch is still incomplete at the front of every window the batch settled.-Each frontier the consumer takes goes to the commit sentinel (`commit_sentinel.rs`), which checks it and passes it to the commit pacer (`commit_pacer.rs`); after settling a poll the consumer asks the pacer what is due and commits it in one call.-The pacer is immediate for now, so every poll commits its frontiers as it completes; an interval pacer that coalesces per partition comes with per-partition commits (see the driver-model plan, cycle 8).-The commit pacer holds no I/O: the consumer commits what it is given, and the commit monitor (`commit_monitor.rs`) reports the broker's committed offsets to the sentinel.+Every delivered message is charged to its partition's ledger during collection, and accepted work completes its offsets there; the commit is then each partition's frontier, one past its longest completed prefix.+A partition that settles without a frontier is not committed and stays on its last commit: an earlier group on that partition is still out.+Each frontier the consumer takes goes to the commit pacer (`commit_pacer.rs`), and the consumer commits what the pacer hands back.+The pacer holds no I/O: the consumer commits what it is given.A partition that leaves the assignment drops its pending frontier with its ledger, so a later commit cannot commit under another owner.++`CONSUMER_COMPLETION_GRANULARITY` selects the unit that settles against the ledger; each unit is one implementation of `Settlement` in `src/settlement/`, and the consumer loop is the same for both.+`poll` (the default) settles a whole poll once every one of its groups is accepted, oldest poll first, and commits its frontiers at once.+`group` settles each group's offsets as its completion arrives, in any order, so a stalled key holds back only its own partition's commit.+At `group` granularity the pacer keeps the latest frontier per partition and hands them out at most once per `CONSUMER_COMMIT_INTERVAL_MS` (default 500 ms).+The consumer asks the pacer on each completion wake, at the top of each loop turn, and on the completion wait's one-second heartbeat, and commits everything pending on exit.+Admission is still per poll at both granularities: `CONSUMER_MAX_BACKGROUND_TASKS` caps in-flight polls, and a slot frees when its whole poll is covered, oldest first, so a stalled key still holds its poll's slot.+At `poll` granularity, `ingestion_consumer_commits_skipped_total{reason}` counts polls that settle with no frontier: `rejected` means the ledger dropped every slice (expected around a rebalance) and `no_frontier` means an earlier batch is still incomplete at the front of every window the batch settled.The ledger emits its own metrics, so any consumer built on the crate reports the same series.`kafka_consumer_ledger_uncommitted_offsets{topic,partition}` gauges each partition's window depth; `kafka_consumer_ledger_uncommitted_events` and `kafka_consumer_ledger_uncommitted_bytes` gauge the charge those offsets carry, where bytes is the payload plus key plus headers of each message.`kafka_consumer_ledger_stale_slices_total{stage}` counts charges and settlements dropped because their partition was reassigned while they were in flight; a few around a rebalance are expected.-`kafka_consumer_ledger_errors_total{stage,kind}` counts contract violations in the ledger's accounting; it must stay 0. A violation resets that partition's ledger and the consumer keeps running; the consumer logs the rejected slice with the batch and ledger generations and the window depth before the reset.+`kafka_consumer_ledger_errors_total{stage,kind}` counts contract violations in the ledger's accounting; it must stay 0. A violation stops the consumer, and the restart replays from the last commit: the ledger resets the partition's window, so offsets in flight below its next delivery would never complete and a later commit would move past them. The consumer logs the rejected slice with the batch and ledger generations and the window depth before it stops.## Batcher@@ -84,7 +90,7 @@ Set `DEBUG_API_ENABLED=true` **and** `DEBUG_API_SECRET` to mount a real-time debEvery request must present the secret as `X-Debug-Api-Secret`; enabling without a secret fails closed (nothing is mounted).The secret is dedicated to this control-plane→consumer hop — deliberately not `INTERNAL_API_SECRET` (see `.agents/security.md`).The ingestion control plane UI consumes these endpoints to render the consumer's live state.-`debug_recorder.rs` keeps a bounded in-memory buffer of structured lifecycle events, recorded at the same points that emit metrics, and it never influences routing: poll dispatch, commit and failure; every request the batcher sends (`RequestSent`) and how it settled (`RequestResolved`, with the unfinished count of a partial ack, or `RequestFailed` with its cause); worker health and membership.+`debug_recorder.rs` keeps a bounded in-memory buffer of structured lifecycle events, recorded at the same points that emit metrics, and it never influences routing: poll dispatch, completion, commit submission and failure; every request the batcher sends (`RequestSent`) and how it settled (`RequestResolved`, with the unfinished count of a partial ack, or `RequestFailed` with its cause); worker health and membership.A poll and a request both carry their assignment epoch and their offsets per topic-partition as runs of consecutive offsets, which is how the control plane UI matches a request to exactly the polls it carries, because a request can merge messages from several polls.Routing keys never enter the payload, because they hold distinct ids.@@ -108,9 +114,9 @@ Sub-batches are split at `INGESTION_TRANSPORT_MAX_BODY_BYTES` (default 10 MiB) i### 2. Commit-error observability — done-Addressed by the commit monitor plus `SentinelContext` (`order_sentinel.rs`): broker-committed offsets are polled and compared against attempted commits (see "Ordering sentinels" above), and rebalance callbacks log assignment changes and reset the ordering sentinels.-The originally envisioned offset-commit callback turned out to be unreachable for manual async commits (librdkafka drops their results unless a conf-level `offset_commit_cb` is registered, which rust-rdkafka never does) — hence the polling design.-Remaining: alert rules on the new metrics.+Addressed by `RebalanceContext` (`rebalance.rs`) and kminion: the consumer reports the offset it last handed to the client, kminion reports the group's lag per partition (see "Ordering sentinels" above), and rebalance callbacks log assignment changes and reset the ordering sentinels.+The originally envisioned offset-commit callback turned out to be unreachable for manual async commits (librdkafka drops their results unless a conf-level `offset_commit_cb` is registered, which rust-rdkafka never does).+Remaining: alert rules on the ledger counters and on kminion lag.### 3. DLQ for poison messagesdiff --git a/rust/ingestion-consumer/docs/driver-model-plan.md b/rust/ingestion-consumer/docs/driver-model-plan.mdindex 8f6742d142a3..d4b7f40faed0 100644--- a/rust/ingestion-consumer/docs/driver-model-plan.md+++ b/rust/ingestion-consumer/docs/driver-model-plan.md@@ -18,7 +18,7 @@ Master already contains two parts of the design:An assignment epoch also exists.A rebalance increments it, and every sub-batch carries it.-These components stay as they are: the stream runners, the P2C (power-of-two-choices) router, the aperture, the worker registry, the commit sentinel, the commit monitor, and the key-order sentinel.+These components stay as they are: the stream runners, the P2C (power-of-two-choices) router, the aperture, the worker registry, and the key-order sentinel.## Initial PR stacks@@ -56,7 +56,7 @@ Rules:- A cycle's cleanup lands only after its switchover is stable on all lanes.- Swap the smallest piece that removes the most complexity. Components that already match the target design are not rebuilt.- A change must not alter the meaning of an existing metric. Metrics of deleted machinery are deleted with it.-- The commit sentinel and the key-order sentinel stay enabled through every cycle.+- The key-order sentinel stays enabled through every cycle.Cycle 1 lands first: one fast change that removes complexity, preserves behavior, and depends on nothing else.After that, the cycles run in order, with two exceptions.@@ -153,7 +153,7 @@ Exit criterion: `ingestion_consumer_ledger_mismatch_total` stays zero across dep- The frontier is in Kafka's committed-offset representation: the next offset to read, not the last offset processed. The current commit path already commits in this representation (the poll's max offset plus one), so the frontier is commit-ready as returned — no call site adds or subtracts one.- The read/consume split matters: the comparison read is idempotent and cannot consume state by accident. Consumption happens at commit points in every mode, so the ring holds only uncommitted offsets.- Completions can arrive in any order. The frontier moves only over completed slots.-- Kafka can deliver offsets with gaps: transaction markers occupy offsets that consumers never receive, and compaction keeps offsets while removing records. Our ingestion topics have neither today, and the commit sentinel treats a gap as a real skip (see the caveat on `CommitSentinel`).+- Kafka can deliver offsets with gaps: transaction markers occupy offsets that consumers never receive, and compaction keeps offsets while removing records. Our ingestion topics have neither today, so a gap the frontier walks over (`kafka_consumer_ledger_gap_offsets_total`) is a real skip.- The ledger still defines contiguity over delivered offsets, not offset arithmetic. A legitimate gap can then never block the frontier.- The sentinel's gap alert and the ledger's gap handling cannot both stand. Once the frontier owns commits (change 4), a commit that walks over a legitimate gap is correct, and the sentinel would report it as a skip. The ledger therefore counts each gap it walked over (`kafka_consumer_ledger_gaps_total`), and the sentinel's gap alert retires in change 5. Until then the alert stays, because the ingestion topics have no legitimate gaps today and a gap is still a real skip.- Add property tests: out-of-order completion, monotonic frontier, `frontier` idempotence, the ring drains after `take_frontier`, offset gaps, ring capacity.@@ -568,8 +568,8 @@ This is a deliberate structural outcome, not routine cleanup: it is the largestOutcome: commits advance per partition as groups complete. Needs cycles 2 and 3.-**Verify:** the commit sentinel checks every commit, and `kafka_consumer_ledger_uncommitted_offsets` shows per-partition frontier lag.-Exit criterion at the canary: zero sentinel violations, and a stalled partition no longer moves the commit rate of other partitions.+**Verify:** `kafka_consumer_ledger_errors_total` and `kafka_consumer_ledger_gap_offsets_total` stay flat, and `kafka_consumer_ledger_uncommitted_offsets` shows per-partition frontier lag.+Exit criterion at the canary: zero ledger errors and gap walks, and a stalled partition no longer moves the commit rate of other partitions.### 21. Switch completion to group granularity (switchover)@@ -584,13 +584,14 @@ Exit criterion at the canary: zero sentinel violations, and a stalled partition- Keep `max_in_flight_batches` as the bound on outstanding work. A poll's slot frees when all its groups are accepted.- Behavior change: commit timing decouples across partitions and polls. Replay exposure stays inside the admission cap.- Rollback is the switch back to `poll`.-- Commit pacing is the seam for this cycle. The consumer already hands every taken frontier to a commit pacer (`commit_pacer.rs`) and commits what the pacer hands back. Today's pacer is `ImmediateCommitPacer`: everything handed over is due on the next take, so commits stay per poll. Per-partition commits at `group` granularity would commit on every group completion, which is too many commits, so this cycle adds an `IntervalCommitPacer` that keeps the latest frontier per partition and hands them out at most once per `CONSUMER_COMMIT_INTERVAL_MS`. The loop then asks the pacer on a tick, not only after a poll settles, and the tick must not sit in a `select!` next to `collect_batch`, because a dropped collection loses messages before they reach the ledger; the interval bounds the commit delay only if the collection loop itself asks the pacer, otherwise the bound is the interval plus the batch timeout. The commit sentinel must record a partition's attempted offset when the pacer hands it out, not when the frontier is handed over, or `ingestion_consumer_commit_confirmation_lag` reports pending frontiers as commits that never landed. On revoke the pacer forgets the partition; the drain in change 27 commits what the pacer still holds at shutdown. Select between the two pacers with a `CommitPacer` trait or an enum once the second one exists, not before. The sentinel's contiguity check compares the span a poll delivered against the previous commit, which catches a skip in what Kafka delivered; at `group` granularity a take no longer maps to one poll's slice, so that check retires here, after an alert on `kafka_consumer_ledger_gaps_total` stands in for it. Checking the take's own window base against the previous take is not a replacement: the ledger sets the base to the frontier it hands out, so that comparison holds by construction.+- Commit pacing is the seam for this cycle. The consumer already hands every taken frontier to a commit pacer (`commit_pacer.rs`) and commits what the pacer hands back. Today's pacer is `ImmediateCommitPacer`: everything handed over is due on the next take, so commits stay per poll. Per-partition commits at `group` granularity would commit on every group completion, which is too many commits, so this cycle gives the pacer an interval: it keeps the latest frontier per partition and hands them out at most once per `CONSUMER_COMMIT_INTERVAL_MS`, and a zero interval is today's immediate pacer. The loop then asks the pacer on a tick, not only after a poll settles, and the tick must not sit in a `select!` next to `collect_batch`, because a dropped collection loses messages before they reach the ledger; the interval bounds the commit delay only if the collection loop itself asks the pacer, otherwise the bound is the interval plus the batch timeout. `ingestion_consumer_committed_offset` is set when a commit is submitted to the client, so a frontier the pacer still holds is not reported as committed. On revoke the pacer forgets the partition; the drain in change 27 commits what the pacer still holds at shutdown. One `CommitPacer` with an interval covers both granularities; `poll` runs it at zero. The commit sentinel and the commit monitor retire here. The sentinel's contiguity check compared the span a poll delivered against the previous commit; at `group` granularity a take no longer maps to one poll's slice, and the ledger already rejects a redelivered offset at charge (`kafka_consumer_ledger_errors_total`) and counts the offsets a frontier walks over (`kafka_consumer_ledger_gap_offsets_total`). The monitor's broker comparison duplicated kminion's per-partition consumer group lag. Checking the take's own window base against the previous take is not a replacement: the ledger sets the base to the frontier it hands out, so that comparison holds by construction.**Interfaces:**- Modify `Config`: add the completion granularity and `CONSUMER_COMMIT_INTERVAL_MS`.-- Modify `IngestionConsumer::process`: at `group` granularity, select over `GroupCompletion` events. The per-poll path stays for rollback.-- Add `IntervalCommitPacer` next to `ImmediateCommitPacer`, and the seam that selects one.+- Add `src/settlement/`: a `Settlement` trait (`settle_run`, `settle_poll`, `take_due`, `take_all`, `forget_partitions`) with one implementation per granularity, each owning its commit pacer; the consumer loop calls it unconditionally and has no granularity branches. The poll implementation stays for rollback.+- `CommitPacer::every(interval)` paces the group implementation; the poll implementation runs it at zero. `ImmediateCommitPacer` goes.+- Delete `commit_sentinel.rs` and `commit_monitor.rs`, with `ingestion_consumer_commit_violations_total{kind=gap|out_of_order|overlap}`, `ingestion_consumer_commits_checked_total`, `ingestion_consumer_broker_committed_offset`, `ingestion_consumer_commit_confirmation_lag`, `ingestion_consumer_last_successful_commit_timestamp_seconds` and `ingestion_consumer_commit_monitor_errors_total`. The rdkafka context moves to `rebalance.rs` as `RebalanceContext`, an event source with a revoke hook and an assign hook; the consumer's hooks forget a partition's ledger window and pending frontier together, on either path.### 22. Delete the per-poll completion path (cleanup)@@ -602,49 +603,52 @@ Exit criterion at the canary: zero sentinel violations, and a stalled partition- Remove the ordered `InFlightBatch` queue, the per-poll aggregation from change 7, and the completion granularity from `Config`.-## Cycle 9: replay exposure is bounded by B+## Cycle 9: replay exposure is bounded per partition-Outcome: uncommitted work never exceeds the budget, in events and bytes. The in-flight admission cap is gone.+Outcome: no partition's uncommitted work exceeds its budget, in events and bytes. The in-flight admission cap is gone. Needs cycles 2 and 8.**Verify:** change 23's gauges run on all lanes with the budget unset.-Exit criterion: `ingestion_consumer_budget_outstanding_events` tracks the admission cap's occupancy, and the pause gauge stays flat at zero.+Exit criterion: `ingestion_consumer_budget_outstanding_events` summed per pod tracks the admission cap's occupancy, and `ingestion_consumer_budget_paused_partitions` stays flat at zero.++The budget is per partition, and so is the pause. One budget for the whole assignment would let a stalled partition fill it and pause every other partition with it, which undoes cycle 8. Per-partition commits are a hard prerequisite for the same reason: at `poll` granularity a stalled partition holds the commits, and so the refunds, of every partition in its poll, so every budget fills together.### 23. Add the budget accounting (implement)-**Task:** Track uncommitted work in events and bytes. Ship with no budget set: the admission cap still governs alone.+**Task:** Track each partition's uncommitted work in events and bytes. Ship with no budget set: the admission cap still governs alone.**Goal:** The budget exists and reports before it constrains.-- Charge each message on poll. The ledger slots already carry the costs (change 2).-- Refund the charge when the commit that covers the message is issued.-- Gauge the outstanding charge per partition and in total.-- New config: the budget in events and in bytes, and the low watermark as a fraction of the budget (default 0.8). Unset means no constraint.+- Charge each message to its partition on poll. The ledger slots already carry the costs (change 2).+- Refund the charge when the commit that covers the message is issued. A revoked partition refunds everything it holds.+- Gauge the outstanding charge per partition.+- New config: the budget per partition in events and in bytes, and the low watermark as a fraction of the budget (default 0.8). Unset means no constraint.+- The budget is a fixed amount per partition, not a share of a pod-wide amount. A rebalance changes how many partitions a pod holds, not how much each one may expose, so the value does not move at rebalances.**Interfaces:**-- Modify `Config`: add the budget, default unset.+- Modify `Config`: add the per-partition budget, default unset.- Modify `IngestionConsumer`: track charge and refund through `OffsetLedger`. No polling change while the budget is unset.**Metrics:**-- Add `ingestion_consumer_budget_outstanding_events` and `ingestion_consumer_budget_outstanding_bytes` (gauges).-- Add `ingestion_consumer_budget_paused` (gauge, 0 or 1) and `ingestion_consumer_budget_pauses_total` (counter). Both stay flat until change 24.+- Add `ingestion_consumer_budget_outstanding_events{topic,partition}` and `ingestion_consumer_budget_outstanding_bytes{topic,partition}` (gauges).+- Add `ingestion_consumer_budget_paused_partitions` (gauge) and `ingestion_consumer_budget_pauses_total{topic,partition}` (counter). Both stay flat until change 24.### 24. Enable the budget (switchover)-**Task:** Set the budget per lane. Pause the poll at the limit and resume on refund.+**Task:** Set the budget per lane. Pause a partition at its limit and resume it on refund.-**Goal:** Replay exposure is bounded directly.+**Goal:** Replay exposure is bounded directly, per partition. A stalled partition stops only its own fetch.- A charts values PR sets the budget lane by lane. The admission cap and the budget both bound work during the transition.-- Derive the first values from the current config: events near the batch size times the cap, bytes from the byte bound. The effective bound then does not change at the switchover.-- Pause the assignment when the outstanding charge reaches the budget on either axis. Resume only when it is back at or under the low watermark on both axes. A pause purges librdkafka's fetch queue and a resume refetches it, so a gate that reopens on the first refund refetches on every crossing.-- Do not stop polling while paused. A paused consumer still serves rebalance callbacks and keeps `max.poll.interval.ms` alive; an unpolled one does neither. When a partition is assigned while the gate is closed, pause it too: an assignment resets librdkafka's pause flags.-- Watch the pause gauge and `ingestion_lag_ms`. Rollback is unsetting the values.+- Derive the first values from the current config: the cap times the batch size, divided by the partitions a pod holds, for events; the byte bound likewise for bytes. The per-pod bound then does not change at the switchover.+- Pause a partition when its outstanding charge reaches the budget on either axis. Resume it only when its charge is back at or under the low watermark on both axes. A pause purges that partition's fetched messages from librdkafka's queue and a resume refetches them, so a gate that reopens on the first refund refetches on every crossing.+- Keep polling while partitions are paused. A paused partition still serves rebalance callbacks and keeps `max.poll.interval.ms` alive; an unpolled consumer does neither. An assignment resets librdkafka's pause flags, so a partition that arrives starts unpaused, with a fresh ledger generation and zero charge.+- Watch the paused-partitions gauge and `ingestion_lag_ms`. Rollback is unsetting the values.**Interfaces:**-- None in this repository. Config values only.+- Modify `IngestionConsumer`: pause and resume one partition at a time through the Kafka client.### 25. Remove the in-flight admission cap (cleanup)@@ -653,6 +657,7 @@ Exit criterion: `ingestion_consumer_budget_outstanding_events` tracks the admiss**Goal:** The budget is the only admission bound.- Only admission control goes. The collection bounds — batch size and batch timeout — stay: they shape one poll, not the amount of admitted work.+- The in-flight poll count goes with the cap. After change 22 the polls exist only to be counted.**Interfaces:**@@ -692,7 +697,7 @@ Exit criterion: `ingestion_consumer_drain_duration_seconds` stays inside the dea- While a revoke waits for the hand-back, librdkafka pauses every assigned partition, not only the revoked ones. The drain window is a pod-wide fetch pause, bounded by the request timeout. Set the rebalance timeout (`max.poll.interval.ms`) above the request timeout, and keep polling through the wait so the callbacks and the liveness clock keep running.- A lost assignment skips the final commit. When `assignment_lost()` is true at the revoke (the session or `max.poll.interval.ms` expired) or the callback reports an error, the broker has fenced the generation and the commit would fail. Drop the partition's ledger, refund its charge, hand it back at once, and still send the marker so the batcher clears its queues. Completions for it die by epoch.- Closing the consumer revokes the whole assignment through the same callback, and rdkafka's `Drop` polls until the close completes. A deferred hand-back there never comes and the drop spins forever. A `closing` flag makes the callback answer inline; set it before any path that drops the consumer, the clean shutdown and the fatal-error exit alike.-- Shutdown is a drain of every assigned partition: a revoke marker per partition through the same path, the final commit per drained marker, then one synchronous commit of every partition's last attempted offset, then close. A drain cut short by the graceful-shutdown deadline replays its un-acked tail, bounded by B.+- Shutdown is a drain of every assigned partition: a revoke marker per partition through the same path, the final commit per drained marker, then one synchronous commit of every partition's last attempted offset, then close. A drain cut short by the graceful-shutdown deadline replays its un-acked tail, bounded by each partition's budget.- Empty assign events arrive after a hand-back and at the end of a rebalance round. They are no-ops.- Check completions against the assignment epoch. The epoch already exists on master and rides on `GroupCompletion` since change 7.- Add a stall deadline per partition, reset only when its frontier moves. Outside a rebalance, an expired deadline fails the process. A partition being drained stands down: its wait is bounded by the request timeout. Replay is at most B.@@ -701,7 +706,7 @@ Exit criterion: `ingestion_consumer_drain_duration_seconds` stays inside the dea- Modify `KeyTableScheduler`: drop queued work per partition on the revoke marker.- Modify `IngestionConsumer` and `OffsetLedger`: the final commit, the refund of abandoned charge, and the stall deadline.-- Modify `SentinelContext`: override `rebalance` so a revoke reports without unassigning, with the `closing` flag and the lost-assignment path above.+- Modify `RebalanceContext`: override `rebalance` so a revoke reports without unassigning, with the `closing` flag and the lost-assignment path above.- Modify `Config`: add the drain switch, default off.**Metrics:**@@ -760,7 +765,7 @@ flowchart LRC6["6: adaptive concurrency"]C7["7: single-owner batcher"]C8["8: decoupled commits"]- C9["9: bounded replay"]+ C9["9: bounded replay per partition"]C10["10: clean handoff"]PR(["#115388 worker side"])@@ -778,7 +783,7 @@ flowchart LRC4 -.-> C6C5 -.-> C6C6 -.-> C7- C8 -.-> C9+ C8 --> C9C9 -.-> C10```@@ -786,7 +791,7 @@ What the graph permits:- Cycles 1 and 2 start in parallel. Neither depends on the other.- After cycle 3, four tracks can run side by side: the request track (4, then 5, then 6), the collapse (7), decoupled commits (8, once cycle 2 is done), and the drain (10).-- Cycle 9 hard-depends only on cycle 2. The dashed edge from cycle 8 exists because group-level completion makes the refunds fine-grained before the admission cap is removed.+- Cycle 9 hard-depends on cycles 2 and 8. The budget and the pause are per partition, and only per-partition commits refund one partition without waiting for the others.- The dashed edges into cycle 6 are the tune-once argument: packing makes request sizes uniform, and the time budget caps RTT, so the governor tunes against the final distribution.- The dashed edge into cycle 7 saves a move: the loop takes the scheduler, packer, and governor in one transformation instead of re-homing components later.- The dashed edge into cycle 10 covers one detail: the drain refunds budget charge, which exists after cycle 9. Without cycle 9 the drain refunds nothing and still works.@@ -795,3 +800,4 @@ What the graph permits:- The keyless lane: today an unkeyed message gets a synthetic per-message key and its own group. A separate keyless lane per partition can remove that overhead.- Partition affinity as a packing hint (open question 6). Measure frontier coupling first.+- Recover from a ledger violation without a restart. For now a violation stops the consumer, and the restart replays from the last commit. A seek back to the partition's last commit would keep the pod running, but it needs the revoke path for one partition (purge its queued messages, strip its in-flight slices, drop its pending frontier) and the offset of its last submitted commit, which nothing tracks yet. Worth it only if violations show up in practice.diff --git a/rust/ingestion-consumer/src/commit_monitor.rs b/rust/ingestion-consumer/src/commit_monitor.rsdeleted file mode 100644index 535a5afe6dee..000000000000--- a/rust/ingestion-consumer/src/commit_monitor.rs+++ /dev/null@@ -1,95 +0,0 @@-//! Confirms that async commits land. librdkafka never reports the result of-//! a manual async commit (see the note on-//! [`crate::order_sentinel::SentinelContext`]), so a task fetches the group's-//! broker-committed offsets on an interval and reports them to the commit-//! sentinel.--use std::sync::Arc;-use std::time::Duration;--use lifecycle::Handle;-use metrics::counter;-use rdkafka::consumer::{Consumer, StreamConsumer};-use tokio::task::JoinHandle;-use tracing::warn;--use crate::commit_sentinel::CommitSentinel;-use crate::order_sentinel::SentinelContext;--/// How often the monitor fetches the group's broker-committed offsets.-const COMMIT_MONITOR_INTERVAL: Duration = Duration::from_secs(30);--/// Aborts the wrapped task when dropped, covering every `process()` exit path.-pub struct AbortOnDrop(JoinHandle<()>);--impl Drop for AbortOnDrop {- fn drop(&mut self) {- self.0.abort();- }-}--/// Run the monitor until shutdown. Aborted when the guard drops, so a-/// consumer torn down mid-test doesn't keep the rdkafka client alive.-pub fn spawn_commit_monitor(- consumer: Arc<StreamConsumer<SentinelContext>>,- sentinel: Arc<CommitSentinel>,- handle: Handle,-) -> AbortOnDrop {- AbortOnDrop(tokio::spawn(run_commit_monitor(consumer, sentinel, handle)))-}--/// Periodically fetch the broker's committed offsets for the current-/// assignment (an OffsetFetch round trip) and feed them to the commit-/// sentinel, which compares them against attempted commits and stamps the-/// last-successful-commit gauge on progress.-async fn run_commit_monitor(- consumer: Arc<StreamConsumer<SentinelContext>>,- sentinel: Arc<CommitSentinel>,- handle: Handle,-) {- loop {- tokio::select! {- _ = handle.shutdown_recv() => return,- _ = tokio::time::sleep(COMMIT_MONITOR_INTERVAL) => {}- }-- let fetch_consumer = Arc::clone(&consumer);- // assignment() and committed_offsets() block on librdkafka.- let fetched = tokio::task::spawn_blocking(move || {- let assignment = fetch_consumer.assignment()?;- if assignment.count() == 0 {- return Ok(None);- }- fetch_consumer- .committed_offsets(assignment, Duration::from_secs(5))- .map(Some)- })- .await;-- match fetched {- Ok(Ok(Some(committed))) => {- let observed: Vec<(String, i32, i64)> = committed- .elements()- .iter()- .filter_map(|e| match e.offset() {- rdkafka::Offset::Offset(offset) => {- Some((e.topic().to_string(), e.partition(), offset))- }- // Invalid = no offset stored for the partition yet.- _ => None,- })- .collect();- sentinel.observe_broker_committed(observed);- }- Ok(Ok(None)) => {} // no assignment yet (e.g. before first rebalance)- Ok(Err(err)) => {- counter!("ingestion_consumer_commit_monitor_errors_total").increment(1);- warn!(error = %err, "Commit monitor failed to fetch committed offsets");- }- Err(err) => {- counter!("ingestion_consumer_commit_monitor_errors_total").increment(1);- warn!(error = %err, "Commit monitor task join error");- }- }- }-}diff --git a/rust/ingestion-consumer/src/commit_pacer.rs b/rust/ingestion-consumer/src/commit_pacer.rsindex 2eff31ba8a64..44beefe3c6b8 100644--- a/rust/ingestion-consumer/src/commit_pacer.rs+++ b/rust/ingestion-consumer/src/commit_pacer.rs@@ -1,28 +1,37 @@-//! Commit pacing for the ingestion consumer. The consumer hands over each-//! partition's frontier as it takes it from the ledger, then asks the pacer-//! what is due and commits that in one call. The pacer holds no I/O.+//! Commit pacing for the ingestion consumer. A settlement hands over each+//! partition's frontier as it takes it from the ledger; the consumer asks the+//! pacer what is due and commits that in one call. The pacer holds no I/O.//!-//! This pacer is immediate: every frontier handed over is due on the next-//! take, so the consumer commits once per poll. Coalescing frontiers on an-//! interval is a later pacer; see the driver-model plan, cycle 8.+//! The pacer keeps the latest frontier per partition and hands them out at+//! most once per interval. A zero interval makes every take due.use std::collections::HashMap;use std::sync::Mutex;+use std::time::{Duration, Instant};use common_kafka_consumer::{Offset, TakenFrontier, TopicPartition};-/// Hands out every frontier handed to it on the next take. The commit-/// sentinel wraps it to check what passes through.+pub struct CommitPacer {+ interval: Duration,+ pending: Mutex<Pending>,+}+#[derive(Default)]-pub struct ImmediateCommitPacer {- /// The next-to-read offset each partition is ready to commit. Frontiers- /// only move forward, so the latest one covers every earlier one.- pending: Mutex<HashMap<TopicPartition, Offset>>,+struct Pending {+ frontiers: HashMap<TopicPartition, Offset>,+ last_take: Option<Instant>,}-impl ImmediateCommitPacer {- pub fn new() -> Self {- Self::default()+impl CommitPacer {+ pub fn immediate() -> Self {+ Self::every(Duration::ZERO)+ }++ pub fn every(interval: Duration) -> Self {+ Self {+ interval,+ pending: Mutex::new(Pending::default()),+ }}/// Replace the partition's pending offset; frontiers only move forward.@@ -30,6 +39,7 @@ impl ImmediateCommitPacer {self.pending.lock().unwrap()+ .frontiers.insert(topic_partition.clone(), taken.offset);}@@ -40,18 +50,31 @@ impl ImmediateCommitPacer {pub fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {let mut pending = self.pending.lock().unwrap();for topic_partition in topic_partitions {- pending.remove(topic_partition);+ pending.frontiers.remove(topic_partition);}}- /// The offsets due for commit: everything pending, or `None` with- /// nothing pending.- pub fn take_due(&self) -> Option<HashMap<TopicPartition, Offset>> {+ pub fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, Offset>> {let mut pending = self.pending.lock().unwrap();- if pending.is_empty() {+ if pending.frontiers.is_empty() {return None;}- Some(std::mem::take(&mut pending))+ if pending+ .last_take+ .is_some_and(|last| now.duration_since(last) < self.interval)+ {+ return None;+ }+ pending.last_take = Some(now);+ Some(std::mem::take(&mut pending.frontiers))+ }++ pub fn take_all(&self) -> Option<HashMap<TopicPartition, Offset>> {+ let mut pending = self.pending.lock().unwrap();+ if pending.frontiers.is_empty() {+ return None;+ }+ Some(std::mem::take(&mut pending.frontiers))}}@@ -83,22 +106,25 @@ mod tests {#[test]fn a_pending_frontier_is_due_on_the_next_take() {- let pacer = ImmediateCommitPacer::new();+ let pacer = CommitPacer::immediate();pacer.advance_frontier(&tp(0), taken(0, 10));- let offsets = pacer.take_due().expect("due");+ let offsets = pacer.take_due(Instant::now()).expect("due");assert_eq!(offsets, HashMap::from([(tp(0), Offset(10))]));- assert!(pacer.take_due().is_none(), "a take leaves nothing behind");+ assert!(+ pacer.take_due(Instant::now()).is_none(),+ "a take leaves nothing behind"+ );}#[test]fn the_latest_frontier_per_partition_is_what_goes_out() {- let pacer = ImmediateCommitPacer::new();+ let pacer = CommitPacer::immediate();pacer.advance_frontier(&tp(0), taken(0, 10));pacer.advance_frontier(&tp(1), taken(0, 20));pacer.advance_frontier(&tp(0), taken(10, 12));- let offsets = pacer.take_due().expect("due");+ let offsets = pacer.take_due(Instant::now()).expect("due");assert_eq!(offsets,HashMap::from([(tp(0), Offset(12)), (tp(1), Offset(20))])@@ -107,15 +133,56 @@ mod tests {#[test]fn a_forgotten_partition_is_not_committed() {- let pacer = ImmediateCommitPacer::new();+ let pacer = CommitPacer::immediate();pacer.advance_frontier(&tp(0), taken(0, 10));pacer.advance_frontier(&tp(1), taken(0, 20));pacer.forget_partitions(&[tp(0)]);assert_eq!(- pacer.take_due().expect("due"),+ pacer.take_due(Instant::now()).expect("due"),HashMap::from([(tp(1), Offset(20))]));}++ #[test]+ fn frontiers_handed_over_within_the_interval_wait_for_it() {+ let pacer = CommitPacer::every(Duration::from_millis(500));+ let start = Instant::now();++ pacer.advance_frontier(&tp(0), taken(0, 10));+ assert!(+ pacer.take_due(start).is_some(),+ "the first frontier goes out at once"+ );++ pacer.advance_frontier(&tp(0), taken(10, 12));+ pacer.advance_frontier(&tp(1), taken(0, 20));+ assert!(+ pacer.take_due(start + Duration::from_millis(100)).is_none(),+ "inside the interval nothing goes out"+ );+ assert_eq!(+ pacer+ .take_due(start + Duration::from_millis(500))+ .expect("due"),+ HashMap::from([(tp(0), Offset(12)), (tp(1), Offset(20))]),+ "one take carries everything that arrived during the interval"+ );+ }++ #[test]+ fn take_all_ignores_the_interval() {+ let pacer = CommitPacer::every(Duration::from_secs(3600));+ let start = Instant::now();+ pacer.advance_frontier(&tp(0), taken(0, 10));+ pacer.take_due(start).expect("due");++ pacer.advance_frontier(&tp(0), taken(10, 12));+ assert!(pacer.take_due(start + Duration::from_secs(1)).is_none());+ assert_eq!(+ pacer.take_all().expect("pending"),+ HashMap::from([(tp(0), Offset(12))])+ );+ }}diff --git a/rust/ingestion-consumer/src/commit_sentinel.rs b/rust/ingestion-consumer/src/commit_sentinel.rsdeleted file mode 100644index 437d84e587b6..000000000000--- a/rust/ingestion-consumer/src/commit_sentinel.rs+++ /dev/null@@ -1,454 +0,0 @@-//! The commit sentinel: an always-on observer that turns the consumer's-//! commit guarantee into alertable metrics.-//!-//! **Commit order**: for every topic-partition, offsets must be committed-//! contiguously and monotonically — each span's first offset must equal the-//! previously committed offset (no skips), and the committed offset must-//! never move backwards (no out-of-order commits). Violations increment-//! `ingestion_consumer_commit_violations_total{kind}` and log the offending-//! offsets. `ingestion_consumer_commits_checked_total` is the denominator: the-//! guarantee holds while it grows and the violation counter stays flat.-//!-//! **Commit confirmation**: librdkafka never reports the result of a manual-//! async commit (see the note on [`crate::order_sentinel::SentinelContext`]),-//! so the commit monitor fetches the group's broker-committed offsets and-//! feeds [`CommitSentinel::observe_broker_committed`], which emits-//! `ingestion_consumer_broker_committed_offset` and-//! `ingestion_consumer_commit_confirmation_lag` gauges and stamps-//! `ingestion_consumer_last_successful_commit_timestamp_seconds` on progress.-//!-//! Rebalances reset the baselines, where Kafka legitimately re-deals-//! partitions and the invariant must re-baseline instead of firing false-//! positives.-//!-//! The sentinel wraps the [`ImmediateCommitPacer`] the consumer hands its-//! frontiers to: every frontier passes through it on the way in, and every-//! take passes through it on the way out. It is a pure observer: it never-//! changes what is committed.--use std::collections::HashMap;-use std::sync::atomic::{AtomicBool, Ordering};-use std::sync::{Arc, Mutex};-use std::time::{SystemTime, UNIX_EPOCH};--use common_kafka_consumer::{Offset, TakenFrontier, TopicPartition};-use metrics::{counter, gauge};-use tracing::warn;--use crate::commit_pacer::ImmediateCommitPacer;-use crate::order_sentinel::OffsetSpan;--/// How a commit violated the contiguous-monotonic invariant.-#[derive(Clone, Copy, Debug, PartialEq, Eq)]-pub enum CommitViolationKind {- /// The batch starts past the previously committed offset — the offsets in- /// between were never part of a committed batch (skipped messages).- Gap,- /// The whole batch lies at or behind the committed offset — the commit- /// moves the partition backwards.- OutOfOrder,- /// The batch partially re-covers already-committed offsets.- Overlap,-}--impl CommitViolationKind {- fn as_str(&self) -> &'static str {- match self {- CommitViolationKind::Gap => "gap",- CommitViolationKind::OutOfOrder => "out_of_order",- CommitViolationKind::Overlap => "overlap",- }- }-}--/// One detected commit-order violation, returned for tests and logged.-#[derive(Clone, Debug, PartialEq, Eq)]-pub struct CommitViolation {- pub kind: CommitViolationKind,- pub topic: String,- pub partition: i32,- /// The partition's committed offset (Kafka "next to read") before this batch.- pub prev_committed: i64,- pub span: OffsetSpan,-}--/// Per-partition commit tracking: what this process asked Kafka to commit-/// (attempted) and what the broker has confirmed as the group's committed-/// offset (observed by the commit monitor via OffsetFetch).-#[derive(Default, Clone, Copy)]-struct PartitionCommits {- /// The offset value last submitted for commit (Kafka "next to read").- attempted: Option<i64>,- /// The broker-confirmed committed offset from the last monitor poll.- confirmed: Option<i64>,-}--/// Tracks the last committed offset per topic-partition and checks each new-/// commit for contiguity and monotonicity. The first commit after a partition-/// is (re)assigned establishes a baseline and is never a violation — earlier-/// offsets may have been committed by another consumer in the group.-///-/// Caveat: legitimate offset gaps exist on topics with transactional producers-/// (control records consume offsets). The ingestion topics are produced by-/// capture without transactions, so a gap here is a real skip.-pub struct CommitSentinel {- partitions: Mutex<HashMap<(String, i32), PartitionCommits>>,- /// Kill switch (`CONSUMER_ORDER_SENTINEL_ENABLED`). When off, checks- /// no-op and no state accumulates.- enabled: AtomicBool,- /// The pacer being observed.- inner: ImmediateCommitPacer,-}--impl CommitSentinel {- pub fn new(inner: ImmediateCommitPacer) -> Self {- Self {- partitions: Mutex::new(HashMap::new()),- enabled: AtomicBool::new(true),- inner,- }- }-- pub fn set_enabled(&self, enabled: bool) {- self.enabled.store(enabled, Ordering::Relaxed);- }-- /// Check the span the poll delivered for the partition against the- /// previous commit as the frontier is handed over, so a violation is- /// attributed to the poll that caused it, then pass the frontier on.- /// Returns the violations for tests.- pub fn advance_frontier(- &self,- topic_partition: &TopicPartition,- delivered: OffsetSpan,- taken: TakenFrontier,- ) -> Vec<CommitViolation> {- let violations = self.check_commit([(topic_partition, &delivered)]);- self.inner.advance_frontier(topic_partition, taken);- violations- }-- /// Partitions leaving the assignment: drop their baselines, and whatever- /// the pacer holds for them.- pub fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {- self.forget_baselines(topic_partitions);- self.inner.forget_partitions(topic_partitions);- }-- pub fn take_due(&self) -> Option<HashMap<TopicPartition, Offset>> {- self.inner.take_due()- }-- /// Check a batch's offset spans against the previous commit per partition,- /// then advance the tracked committed offset to `span.last + 1`. Emits- /// metrics and logs; returns the violations for tests.- pub fn check_commit<'a>(- &self,- spans: impl IntoIterator<Item = (&'a TopicPartition, &'a OffsetSpan)>,- ) -> Vec<CommitViolation> {- if !self.enabled.load(Ordering::Relaxed) {- return Vec::new();- }- let mut partitions = self.partitions.lock().unwrap();- let mut violations = Vec::new();-- for (topic_partition, span) in spans {- counter!("ingestion_consumer_commits_checked_total").increment(1);-- let topic = &topic_partition.topic;- let partition = &topic_partition.partition;- let state = partitions.entry((topic.clone(), *partition)).or_default();- if let Some(prev) = state.attempted {- let kind = if span.first == prev {- None- } else if span.first > prev {- Some(CommitViolationKind::Gap)- } else if span.last < prev {- Some(CommitViolationKind::OutOfOrder)- } else {- Some(CommitViolationKind::Overlap)- };-- if let Some(kind) = kind {- counter!(- "ingestion_consumer_commit_violations_total",- "kind" => kind.as_str(),- )- .increment(1);- warn!(- kind = kind.as_str(),- topic = %topic,- partition = *partition,- prev_committed = prev,- batch_first = span.first,- batch_last = span.last,- "Commit order violation"- );- violations.push(CommitViolation {- kind,- topic: topic.clone(),- partition: *partition,- prev_committed: prev,- span: *span,- });- }- }-- state.attempted = Some(span.last + 1);- gauge!(- "ingestion_consumer_committed_offset",- "topic" => topic.clone(),- "partition" => partition.to_string(),- )- .set((span.last + 1) as f64);- }-- violations- }-- /// Feed broker-confirmed committed offsets (from an OffsetFetch of the- /// group's assigned partitions) and compare against what this process- /// attempted. Emits per-partition gauges:- ///- /// - `ingestion_consumer_broker_committed_offset` — the group's committed- /// offset as the broker reports it;- /// - `ingestion_consumer_commit_confirmation_lag` — attempted minus- /// confirmed. Transiently positive while async commits are in flight;- /// persistently positive means commits are being submitted but not- /// landing (e.g. a stuck coordinator).- ///- /// Returns true when commits verifiably progressed since the last- /// observation — the broker offset advanced, or everything attempted is- /// confirmed — so the caller can stamp the last-successful-commit gauge.- pub fn observe_broker_committed(- &self,- observed: impl IntoIterator<Item = (String, i32, i64)>,- ) -> bool {- if !self.enabled.load(Ordering::Relaxed) {- return false;- }- let mut partitions = self.partitions.lock().unwrap();- let mut advanced = false;-- for (topic, partition, committed) in observed {- gauge!(- "ingestion_consumer_broker_committed_offset",- "topic" => topic.clone(),- "partition" => partition.to_string(),- )- .set(committed as f64);-- let state = partitions.entry((topic.clone(), partition)).or_default();- if let Some(attempted) = state.attempted {- gauge!(- "ingestion_consumer_commit_confirmation_lag",- "topic" => topic.clone(),- "partition" => partition.to_string(),- )- .set((attempted - committed).max(0) as f64);- }- // Only an increase over a *previous* observation counts as- // progress — the first poll baselines (the broker may be reporting- // a prior incarnation's commits, which say nothing about ours).- if state.confirmed.is_some_and(|prev| committed > prev) {- advanced = true;- }- state.confirmed = Some(committed);- }-- let all_confirmed = {- let attempted_any = partitions.values().any(|s| s.attempted.is_some());- attempted_any- && partitions.values().all(|s| match s.attempted {- Some(attempted) => s.confirmed.is_some_and(|c| c >= attempted),- None => true,- })- };-- let progressed = advanced || all_confirmed;- if progressed {- gauge!("ingestion_consumer_last_successful_commit_timestamp_seconds").set(- SystemTime::now()- .duration_since(UNIX_EPOCH)- .unwrap_or_default()- .as_secs_f64(),- );- }- progressed- }-- /// Drop the baselines for departing partitions so the next commit after a- /// re-assignment baselines instead of reporting a false gap/overlap- /// (another group member may have committed in between).- fn forget_baselines(&self, topic_partitions: &[TopicPartition]) {- let mut partitions = self.partitions.lock().unwrap();- for topic_partition in topic_partitions {- let (topic, partition) = (&topic_partition.topic, topic_partition.partition);- let forgotten = partitions.remove(&(topic.clone(), partition));- if forgotten.is_some_and(|state| state.attempted.is_some()) {- let topic: Arc<str> = Arc::from(topic.as_str());- let partition: Arc<str> = Arc::from(partition.to_string());- gauge!(- "ingestion_consumer_commit_confirmation_lag",- "topic" => topic,- "partition" => partition,- )- .set(0.0);- }- }- }-}--#[cfg(test)]-mod tests {- use crate::commit_pacer::test_support::{taken, tp};-- use super::*;-- fn sentinel() -> CommitSentinel {- CommitSentinel::new(ImmediateCommitPacer::new())- }-- #[test]- fn a_delivered_span_is_checked_as_the_frontier_is_handed_over() {- let sentinel = sentinel();- let span = |first, last| OffsetSpan { first, last };- // The first poll baselines. The second delivered 10..=14 and the- // frontier reached 15, so it starts where the last commit left off.- assert!(sentinel- .advance_frontier(&tp(0), span(0, 9), taken(0, 10))- .is_empty());- assert!(sentinel- .advance_frontier(&tp(0), span(10, 14), taken(10, 15))- .is_empty());-- // A poll that delivered past the last commit skipped offsets, even- // though the ledger walked the gap and its take chains.- let violations = sentinel.advance_frontier(&tp(0), span(17, 19), taken(15, 20));- assert_eq!(violations.len(), 1);- assert_eq!(violations[0].kind, CommitViolationKind::Gap);- assert_eq!(violations[0].prev_committed, 15);- }-- fn spans(entries: &[(&str, i32, i64, i64)]) -> HashMap<TopicPartition, OffsetSpan> {- entries- .iter()- .map(|(topic, partition, first, last)| {- (- TopicPartition::new(*topic, *partition),- OffsetSpan {- first: *first,- last: *last,- },- )- })- .collect()- }-- // ---- CommitSentinel ------ #[test]- fn contiguous_commits_pass() {- let sentinel = sentinel();- assert!(- sentinel.check_commit(&spans(&[("t", 0, 0, 99)])).is_empty(),- "first commit baselines"- );- assert!(- sentinel- .check_commit(&spans(&[("t", 0, 100, 149)]))- .is_empty(),- "next batch starts exactly at the committed offset"- );- }-- #[test]- fn commit_gap_is_detected() {- let sentinel = sentinel();- sentinel.check_commit(&spans(&[("t", 0, 0, 99)]));- // Offsets 100..=104 were never committed — the batch skipped them.- let violations = sentinel.check_commit(&spans(&[("t", 0, 105, 150)]));- assert_eq!(violations.len(), 1);- assert_eq!(violations[0].kind, CommitViolationKind::Gap);- assert_eq!(violations[0].prev_committed, 100);- }-- #[test]- fn commit_regression_is_out_of_order() {- let sentinel = sentinel();- sentinel.check_commit(&spans(&[("t", 0, 0, 99)]));- let violations = sentinel.check_commit(&spans(&[("t", 0, 10, 50)]));- assert_eq!(violations.len(), 1);- assert_eq!(violations[0].kind, CommitViolationKind::OutOfOrder);- }-- #[test]- fn partial_recommit_is_overlap() {- let sentinel = sentinel();- sentinel.check_commit(&spans(&[("t", 0, 0, 99)]));- let violations = sentinel.check_commit(&spans(&[("t", 0, 90, 150)]));- assert_eq!(violations.len(), 1);- assert_eq!(violations[0].kind, CommitViolationKind::Overlap);- }-- #[test]- fn partitions_are_tracked_independently() {- let sentinel = sentinel();- sentinel.check_commit(&spans(&[("t", 0, 0, 99), ("t", 1, 0, 9)]));- // Partition 0 continues cleanly; partition 1 skips 10..=19.- let violations = sentinel.check_commit(&spans(&[("t", 0, 100, 120), ("t", 1, 20, 30)]));- assert_eq!(violations.len(), 1);- assert_eq!(violations[0].partition, 1);- assert_eq!(violations[0].kind, CommitViolationKind::Gap);- }-- #[test]- fn broker_observation_baselines_then_tracks_progress() {- let sentinel = sentinel();- sentinel.check_commit(&spans(&[("t", 0, 0, 99)])); // attempted next = 100- // First poll baselines: a stale broker offset (previous incarnation)- // is not evidence that OUR commits landed.- assert!(!sentinel.observe_broker_committed([("t".to_string(), 0, 40)]));- // Broker offset advancing across polls = commits are landing.- assert!(sentinel.observe_broker_committed([("t".to_string(), 0, 80)]));- // No advance and still behind the attempted offset = no progress.- assert!(!sentinel.observe_broker_committed([("t".to_string(), 0, 80)]));- // Catching up to everything attempted also counts as progress.- assert!(sentinel.observe_broker_committed([("t".to_string(), 0, 100)]));- // Fully confirmed and idle: repeated identical polls stay "progressed"- // via the all-confirmed arm, keeping the liveness gauge fresh.- assert!(sentinel.observe_broker_committed([("t".to_string(), 0, 100)]));- }-- #[test]- fn broker_observation_requires_every_attempted_partition_confirmed() {- let sentinel = sentinel();- sentinel.check_commit(&spans(&[("t", 0, 0, 99), ("t", 1, 0, 9)]));- sentinel.observe_broker_committed([("t".to_string(), 0, 100), ("t".to_string(), 1, 5)]);- // Partition 0 fully confirmed but partition 1 stuck below attempted and- // not advancing: not progress.- assert!(!sentinel- .observe_broker_committed([("t".to_string(), 0, 100), ("t".to_string(), 1, 5)]));- }-- #[test]- fn forgotten_partition_rebaselines_without_violation() {- let sentinel = sentinel();- sentinel.check_commit(&spans(&[("t", 0, 0, 99)]));- sentinel.forget_partitions(&[TopicPartition::new("t", 0)]);- // After revoke + re-assign another consumer may have committed past us;- // a non-contiguous first commit must baseline, not fire.- assert!(sentinel- .check_commit(&spans(&[("t", 0, 500, 599)]))- .is_empty());- }-- #[test]- fn a_disabled_sentinel_checks_nothing() {- let commit = sentinel();- commit.set_enabled(false);- commit.check_commit(&spans(&[("t", 0, 0, 99)]));- // A blatant regression passes: the kill switch disarms the check.- assert!(commit.check_commit(&spans(&[("t", 0, 10, 50)])).is_empty());- }-}diff --git a/rust/ingestion-consumer/src/config.rs b/rust/ingestion-consumer/src/config.rsindex c39b88a5b2db..6c3dd2b39b67 100644--- a/rust/ingestion-consumer/src/config.rs+++ b/rust/ingestion-consumer/src/config.rs@@ -1,4 +1,5 @@use std::num::{NonZeroU32, NonZeroUsize};+use std::str::FromStr;use std::time::{Duration, Instant};use common_continuous_profiling::ContinuousProfilingConfig;@@ -15,6 +16,27 @@ use crate::discovery::DiscoveryMode;use crate::routing::{Router, RoutingStrategy};use common_kafka_consumer::config::ConsumerConfigBuilder;+#[derive(Clone, Copy, Debug, PartialEq, Eq, Default)]+pub enum CompletionGranularity {+ #[default]+ Poll,+ Group,+}++impl FromStr for CompletionGranularity {+ type Err = String;++ fn from_str(value: &str) -> Result<Self, Self::Err> {+ match value.trim().to_lowercase().as_str() {+ "poll" => Ok(Self::Poll),+ "group" => Ok(Self::Group),+ other => Err(format!(+ "unknown consumer completion granularity '{other}' (expected 'poll' or 'group')"+ )),+ }+ }+}+/// Configuration for the ingestion consumer.////// Kafka env vars match the Node.js ingestion consumer so this can be a@@ -224,6 +246,15 @@ pub struct Config {#[envconfig(from = "CONSUMER_MAX_BACKGROUND_TASKS", default = "1")]pub consumer_max_background_tasks: usize,+ /// `poll` or `group`; see [`CompletionGranularity`].+ #[envconfig(from = "CONSUMER_COMPLETION_GRANULARITY", default = "poll")]+ pub consumer_completion_granularity: CompletionGranularity,++ /// Minimum spacing between offset commits at `group` granularity+ /// (milliseconds). Not read at `poll` granularity.+ #[envconfig(from = "CONSUMER_COMMIT_INTERVAL_MS", default = "500")]+ pub consumer_commit_interval_ms: u64,+// ---- Debug API ----/// Serve the real-time debug API (`/debug/load`, `/debug/state`,/// `/debug/events` SSE) on the health server, recording structured events@@ -242,12 +273,8 @@ pub struct Config {pub debug_api_secret: String,// ---- Ordering sentinels ----- /// Kill switch for the ordering sentinels (per-partition commit- /// contiguity/monotonicity checks and per-key send-order checks). They are- /// pure observers with per-batch lock/state overhead bounded by in-flight- /// work; disable only if that overhead is ever implicated. The commit-result- /// and rebalance metrics from the consumer context stay on regardless. The- /// worker-side feed-order sentinel has its own flag+ /// Kill switch for the per-key send-order sentinel. The worker-side+ /// feed-order sentinel has its own flag/// (INGESTION_API_FEED_ORDER_SENTINEL_ENABLED on the Node.js side).#[envconfig(from = "CONSUMER_ORDER_SENTINEL_ENABLED", default = "true")]pub consumer_order_sentinel_enabled: bool,@@ -670,4 +697,17 @@ mod tests {fn an_invalid_request_budget_is_rejected(#[case] vars: &[(&str, &str)]) {assert!(config(vars).request_budget().is_err());}++ #[test]+ fn completion_granularity_parses_known_values() {+ assert_eq!(+ CompletionGranularity::from_str(" POLL "),+ Ok(CompletionGranularity::Poll)+ );+ assert_eq!(+ CompletionGranularity::from_str("group"),+ Ok(CompletionGranularity::Group)+ );+ assert!(CompletionGranularity::from_str("batch").is_err());+ }}diff --git a/rust/ingestion-consumer/src/consumer.rs b/rust/ingestion-consumer/src/consumer.rsindex de35a9f2e029..0dc0a0525ffe 100644--- a/rust/ingestion-consumer/src/consumer.rs+++ b/rust/ingestion-consumer/src/consumer.rs@@ -3,7 +3,7 @@ use std::sync::{Arc, Mutex};use std::time::{Duration, Instant};use common_kafka_consumer::{- Charge, GroupCompletion, Offset, Partition, Rejection, TopicOffsetLedger, TopicPartition,+ AssignmentEpoch, Charge, GroupCompletion, Offset, Partition, TopicOffsetLedger, TopicPartition,};use futures::StreamExt;use lifecycle::Handle;@@ -15,17 +15,16 @@ use tokio::sync::mpsc;use tracing::{error, info, warn};use crate::batcher::{make_batch_id, Batcher, BatcherObserver, BatcherOutputs, Revoker};-use crate::commit_monitor::spawn_commit_monitor;-use crate::commit_pacer::ImmediateCommitPacer;-use crate::commit_sentinel::{CommitSentinel, CommitViolation};-use crate::config::Config;+use crate::config::{CompletionGranularity, Config};use crate::debug_recorder::{- record_if, DebugEventKind, DebugRecorder, PartitionOffset, PartitionOffsets,+ record_if, CommitOffset, DebugEventKind, DebugRecorder, PartitionOffset, PartitionOffsets,};use crate::discovery::DiscoveryMode;use crate::grpc_transport::GrpcTransport;-use crate::ledger_rejection::{warn_rejection, RejectedSlice};-use crate::order_sentinel::{OffsetSpan, RevokeHook, SentinelContext};+use crate::ledger_rejection::{report_rejection, RejectedSlice};+use crate::order_sentinel::OffsetSpan;+use crate::rebalance::{PartitionHook, RebalanceContext};+use crate::settlement::{self, ChargedSlice, Settlement};use crate::types::{Accumulator, SerializedKafkaMessage};/// Batch-wide statistics gathered while collecting, used to emit parity@@ -99,8 +98,21 @@ impl PartitionDeliveries {}}- fn credit(&mut self, offset: i64) {- self.credited.insert(offset);+ /// Credit a completion's run and return the offsets to settle: those+ /// charged under this generation that no earlier completion credited.+ fn credit_run(&mut self, run: &[Offset]) -> Vec<Offset> {+ let Self {+ credited, charges, ..+ } = self;+ run.iter()+ .copied()+ .filter(|offset| {+ credited.insert(offset.0)+ && charges+ .binary_search_by_key(&offset.0, |(charged, _)| charged.0)+ .is_ok()+ })+ .collect()}fn uncredited_charge(&self) -> Option<Offset> {@@ -161,7 +173,7 @@ struct CollectedBatch {/// One submitted poll, awaiting its group completions. The consumer/// correlates completions to it by assignment epoch, partition, and offset;-/// the poll commits only once completions cover every message.+/// the poll retires only once completions cover every message.struct InFlightPoll {/// Consumer-side id for logs and debug events only; the batcher's/// internal batch id never crosses the boundary.@@ -172,7 +184,7 @@ struct InFlightPoll {message_count: u32,/// Messages covered by completions so far, accepted or not.covered: u32,- /// Worker-accepted messages so far. The poll commits only when this+ /// Worker-accepted messages so far. The poll retires only when this/// reaches `message_count`.accepted: u32,dispatched_at: Instant,@@ -194,6 +206,18 @@ impl InFlightPoll {.map(|offset| (topic_partition, offset))})}++ fn charged_slices(&self) -> Vec<ChargedSlice<'_>> {+ self.partitions+ .iter()+ .map(|(topic_partition, deliveries)| ChargedSlice {+ topic_partition,+ generation: deliveries.generation,+ span: deliveries.span,+ charges: &deliveries.charges,+ })+ .collect()+ }}/// A partition revocation stamped with the ledger generation started by the@@ -207,27 +231,76 @@ struct RevokedPartition {/// The consumer's revocation hook: purge the batcher, and record the revoked/// partitions, because the purge means their polls never complete.+fn forget_partitions(+ ledger: &TopicOffsetLedger,+ settlement: &dyn Settlement,+ partitions: &[TopicPartition],+) {+ ledger.forget_partitions(+ partitions+ .iter()+ .map(|tp| (tp.topic.as_str(), tp.partition)),+ );+ settlement.forget_partitions(partitions);+}+fn revoke_hook(revoker: Revoker,ledger: Arc<TopicOffsetLedger>,+ settlement: Arc<dyn Settlement>,revoked_partitions: Arc<Mutex<Vec<RevokedPartition>>>,-) -> RevokeHook {+) -> PartitionHook {Box::new(move |partitions| {- revoker.purge_revoked(partitions);+ forget_partitions(&ledger, &*settlement, partitions);+ let pairs: Vec<(String, i32)> = partitions+ .iter()+ .map(|tp| (tp.topic.clone(), tp.partition))+ .collect();+ revoker.purge_revoked(&pairs);revoked_partitions.lock().unwrap()- .extend(partitions.iter().map(|(topic, partition)| {- let topic_partition = TopicPartition::new(topic, *partition);- let generation = ledger.generation(&topic_partition);- RevokedPartition {- topic_partition,- generation,- }+ .extend(partitions.iter().map(|topic_partition| RevokedPartition {+ topic_partition: topic_partition.clone(),+ generation: ledger.generation(topic_partition),}));})}+fn assign_hook(+ ledger: Arc<TopicOffsetLedger>,+ settlement: Arc<dyn Settlement>,+ assignment_epoch: AssignmentEpoch,+) -> PartitionHook {+ Box::new(move |partitions| {+ // Also covers a loss with no revoke callback (an error rebalance, a fenced member).+ forget_partitions(&ledger, &*settlement, partitions);+ assignment_epoch.bump();+ })+}++fn wire_rebalance_hooks(+ context: &RebalanceContext,+ revoker: Revoker,+ ledger: &Arc<TopicOffsetLedger>,+ settlement: &Arc<dyn Settlement>,+ assignment_epoch: AssignmentEpoch,+) -> Arc<Mutex<Vec<RevokedPartition>>> {+ let revoked_partitions = Arc::new(Mutex::new(Vec::new()));+ context.set_on_revoke(revoke_hook(+ revoker,+ Arc::clone(ledger),+ Arc::clone(settlement),+ Arc::clone(&revoked_partitions),+ ));+ context.set_on_assign(assign_hook(+ Arc::clone(ledger),+ Arc::clone(settlement),+ assignment_epoch,+ ));+ revoked_partitions+}+/// Remove revoked partitions from in-flight polls, and remove polls left/// empty. Only slices from before that partition's revoke generation go: a/// poll's kept or newly reassigned slices retain their counts and settle their@@ -278,31 +351,49 @@ fn strip_revoked_partitions(/// with any discarded offset counts as stale once.////// [`send_group_completions`]: crate::batcher-fn apply_completion(in_flight: &mut VecDeque<InFlightPoll>, completion: GroupCompletion) {+fn apply_completion(+ in_flight: &mut VecDeque<InFlightPoll>,+ completion: &GroupCompletion,+ mut settle: impl FnMut(&TopicPartition, u64, &[Offset]) -> anyhow::Result<()>,+) -> anyhow::Result<()> {+ let fully_accepted = completion.accepted as usize == completion.offsets.len();let mut accepted = completion.accepted;let mut unmatched: u64 = 0;- 'offsets: for offset in &completion.offsets {- let is_accepted = accepted > 0;- accepted = accepted.saturating_sub(1);+ let mut remaining = completion.offsets.as_slice();+ 'runs: while let Some(first) = remaining.first() {for poll in in_flight.iter_mut().filter(|poll| poll.assignment_epoch == completion.assignment_epoch){for (topic_partition, deliveries) in poll.partitions.iter_mut() {- if topic_partition.partition == completion.partition.0- && deliveries.span.first <= offset.0- && offset.0 <= deliveries.span.last+ let span = deliveries.span;+ if topic_partition.partition != completion.partition.0+ || first.0 < span.first+ || span.last < first.0{- poll.covered += 1;- poll.accepted += u32::from(is_accepted);- deliveries.covered += 1;- deliveries.accepted += u32::from(is_accepted);- deliveries.credit(offset.0);- continue 'offsets;+ continue;+ }+ let run_len = remaining+ .iter()+ .take_while(|offset| span.first <= offset.0 && offset.0 <= span.last)+ .count();+ let (run, rest) = remaining.split_at(run_len);+ let run_accepted = accepted.min(run_len as u32);+ accepted -= run_accepted;+ poll.covered += run_len as u32;+ poll.accepted += run_accepted;+ deliveries.covered += run_len as u32;+ deliveries.accepted += run_accepted;+ let settleable = deliveries.credit_run(run);+ if fully_accepted && !settleable.is_empty() {+ settle(topic_partition, deliveries.generation, &settleable)?;}+ remaining = rest;+ continue 'runs;}}unmatched += 1;+ remaining = &remaining[1..];}if unmatched > 0 {counter!("ingestion_consumer_stale_group_completions_total").increment(1);@@ -313,6 +404,7 @@ fn apply_completion(in_flight: &mut VecDeque<InFlightPoll>, completion: GroupCom"Discarding completion offsets that match no in-flight poll");}+ Ok(())}/// Pops the front poll for its commit. A complete poll commits only when@@ -350,13 +442,16 @@ pub struct IngestionConsumerOptions {pub group_id: String,/// Debug event recorder; `None` unless `DEBUG_API_ENABLED`.pub debug_recorder: Option<Arc<DebugRecorder>>,+ pub completion_granularity: CompletionGranularity,+ pub commit_interval: Duration,+ pub ledger: Arc<TopicOffsetLedger>,}/// The main consumer loop: reads from Kafka, demuxes each poll into groups,/// submits them to the [`Batcher`] (which routes, dispatches, and flushes),-/// and commits offsets once the batcher's completions cover a poll.+/// and commits offsets once the batcher's completions cover a poll or a group.pub struct IngestionConsumer {- consumer: Arc<StreamConsumer<SentinelContext>>,+ consumer: Arc<StreamConsumer<RebalanceContext>>,batcher: Batcher,/// Taken once by `process`.outputs: Option<BatcherOutputs>,@@ -368,27 +463,20 @@ pub struct IngestionConsumer {max_in_flight_batches: usize,handle: Handle,group_id: String,- /// Where settled frontiers go: the sentinel checks each one and passes- /// it to the pacer, which says when to commit. Shared with the- /// consumer's [`SentinelContext`], which tells it which partitions leave- /// the assignment.- commit_sentinel: Arc<CommitSentinel>,/// Debug event recorder; `None` unless `DEBUG_API_ENABLED`.debug_recorder: Option<Arc<DebugRecorder>>,- /// The per-partition offset ledger the commit path reads its frontiers- /// from. Shared with the consumer's [`SentinelContext`], which forgets- /// partitions on rebalance.topic_offset_ledger: Arc<TopicOffsetLedger>,/// Partitions revoked since the loop last looked, fed by the rebalance/// callback.revoked_partitions: Arc<Mutex<Vec<RevokedPartition>>>,+ settlement: Arc<dyn Settlement>,}impl IngestionConsumer {/// Constructs a consumer from pre-built parts. Useful in integration tests/// where the Kafka consumer is created and subscribed externally.pub fn from_parts(- consumer: StreamConsumer<SentinelContext>,+ consumer: StreamConsumer<RebalanceContext>,batcher: Batcher,outputs: BatcherOutputs,transport: Arc<GrpcTransport>,@@ -396,23 +484,25 @@ impl IngestionConsumer {options: IngestionConsumerOptions,handle: Handle,) -> Self {- // Share the context's ledger and sentinel so rebalance callbacks- // forget partitions on the same ones the commit path uses.- let topic_offset_ledger = consumer.context().topic_offset_ledger();- let commit_sentinel = consumer.context().commit_sentinel();- let revoked_partitions: Arc<Mutex<Vec<RevokedPartition>>> =- Arc::new(Mutex::new(Vec::new()));- consumer.context().set_revoke_hook(revoke_hook(- batcher.revoker(),+ let topic_offset_ledger = options.ledger;+ let settlement = settlement::for_granularity(+ options.completion_granularity,Arc::clone(&topic_offset_ledger),- Arc::clone(&revoked_partitions),- ));+ options.commit_interval,+ );+ let revoked_partitions = wire_rebalance_hooks(+ consumer.context(),+ batcher.revoker(),+ &topic_offset_ledger,+ &settlement,+ transport.assignment_epoch(),+ );let consumer = Arc::new(consumer);Self {- commit_sentinel,debug_recorder: options.debug_recorder,topic_offset_ledger,revoked_partitions,+ settlement,consumer,batcher,outputs: Some(outputs),@@ -454,25 +544,24 @@ impl IngestionConsumer {config.consumer_batch_size,config.consumer_batch_size_kb,);- let commit_sentinel = Arc::new(CommitSentinel::new(ImmediateCommitPacer::new()));- commit_sentinel.set_enabled(config.consumer_order_sentinel_enabled);batcher.key_order_sentinel().set_enabled(config.consumer_order_sentinel_enabled);let topic_offset_ledger = Arc::new(TopicOffsetLedger::new());- let mut context = SentinelContext::new(- Arc::clone(&commit_sentinel),+ let settlement = settlement::for_granularity(+ config.consumer_completion_granularity,Arc::clone(&topic_offset_ledger),+ Duration::from_millis(config.consumer_commit_interval_ms),);- context.set_assignment_epoch(transport.assignment_epoch());- let revoked_partitions: Arc<Mutex<Vec<RevokedPartition>>> =- Arc::new(Mutex::new(Vec::new()));- context.set_revoke_hook(revoke_hook(+ let context = RebalanceContext::new();+ let revoked_partitions = wire_rebalance_hooks(+ &context,batcher.revoker(),- Arc::clone(&topic_offset_ledger),- Arc::clone(&revoked_partitions),- ));- let consumer: StreamConsumer<SentinelContext> =+ &topic_offset_ledger,+ &settlement,+ transport.assignment_epoch(),+ );+ let consumer: StreamConsumer<RebalanceContext> =client_config.create_with_context(context)?;consumer.subscribe(&[&config.ingestion_consumer_consume_topic])?;let consumer = Arc::new(consumer);@@ -483,15 +572,16 @@ impl IngestionConsumer {workers = worker_urls.len(),batch_size = config.consumer_batch_size,batch_size_kb = config.consumer_batch_size_kb,+ completion_granularity = ?config.consumer_completion_granularity,"Kafka consumer subscribed");Ok(Self {- commit_sentinel,consumer,debug_recorder,topic_offset_ledger,revoked_partitions,+ settlement,batcher,outputs: Some(outputs),transport,@@ -537,13 +627,10 @@ impl IngestionConsumer {workers: self.worker_urls.clone(),});- let _commit_monitor = spawn_commit_monitor(- Arc::clone(&self.consumer),- Arc::clone(&self.commit_sentinel),- self.handle.clone(),- );-self.run(completions, errors).await;+ if let Err(err) = self.commit_all() {+ warn!(error = %err, "Could not submit pending frontiers on exit");+ }info!("Consumer loop stopped");}@@ -563,6 +650,13 @@ impl IngestionConsumer {// processed in parallel, bounded by `max_in_flight_batches`.gauge!("ingestion_consumer_in_flight_batches").set(in_flight_polls.len() as f64);+ // Not beside `collect_batch`: a tick there would drop a collection+ // in progress.+ if let Err(err) = self.commit_due() {+ self.fail_commit(err);+ return;+ }+if accepting_new_batches && in_flight_polls.len() < self.max_in_flight_batches {tokio::select! {_ = self.handle.shutdown_recv() => {@@ -621,9 +715,27 @@ impl IngestionConsumer {topic_partition.partition,rdkafka::Offset::Offset(next_to_read.0),)?;+ let topic: Arc<str> = Arc::from(topic_partition.topic.as_str());+ let partition: Arc<str> = Arc::from(topic_partition.partition.to_string());+ gauge!(+ "ingestion_consumer_committed_offset",+ "topic" => topic,+ "partition" => partition,+ )+ .set(next_to_read.0 as f64);}self.consumer.commit(&tpl, CommitMode::Async)?;counter!("ingestion_consumer_offset_commits_total").increment(1);+ record_if(&self.debug_recorder, || DebugEventKind::CommitSubmitted {+ offsets: offsets+ .iter()+ .map(|(topic_partition, offset)| CommitOffset {+ topic: topic_partition.topic.clone(),+ partition: topic_partition.partition,+ offset: offset.0,+ })+ .collect(),+ });Ok(())}@@ -671,11 +783,8 @@ impl IngestionConsumer {}}- /// Wait for completions to cover the oldest in-flight poll, then commit- /// it. Commits only the oldest poll: later completed polls stay- /// uncommitted behind any earlier one, preserving at-least-once delivery- /// across worker or pipeline failures. Completions for newer polls are- /// still credited while waiting.+ /// Wait for completions to cover the oldest in-flight poll, then retire+ /// it. Completions for newer polls are credited while waiting.async fn complete_oldest_poll(&self,in_flight_polls: &mut VecDeque<InFlightPoll>,@@ -696,7 +805,13 @@ impl IngestionConsumer {}tokio::select! {completion = completions.recv() => match completion {- Some(completion) => apply_completion(in_flight_polls, completion),+ Some(completion) => {+ apply_completion(in_flight_polls, &completion, |tp, generation, run| {+ self.settlement.settle_run(tp, generation, run)+ })+ .map_err(Failure::Batch)?;+ self.commit_due().map_err(Failure::Commit)?;+ }None => return Err(Failure::Batch(anyhow::anyhow!("batcher completion channel closed"))),@@ -707,18 +822,21 @@ impl IngestionConsumer {"batcher error channel closed"))),},- _ = heartbeat.tick() => self.handle.report_healthy(),+ _ = heartbeat.tick() => {+ self.handle.report_healthy();+ self.commit_due().map_err(Failure::Commit)?;+ }}}let poll = pop_committable(in_flight_polls).map_err(Failure::Batch)?;- self.settle_poll(&poll.partitions);- if let Some(offsets) = self.commit_sentinel.take_due() {- self.commit_offsets(&offsets).map_err(Failure::Commit)?;- }+ self.settlement+ .settle_poll(&poll.charged_slices())+ .map_err(Failure::Batch)?;+ self.commit_due().map_err(Failure::Commit)?;emit_latest_processed_timestamp_metrics(&poll.partitions, &self.group_id);- record_if(&self.debug_recorder, || DebugEventKind::BatchCommitted {+ record_if(&self.debug_recorder, || DebugEventKind::PollCompleted {batch_id: poll.poll_id.clone(),accepted: poll.accepted,duration_ms: poll.dispatched_at.elapsed().as_millis() as u64,@@ -752,6 +870,20 @@ impl IngestionConsumer {}}+ fn commit_due(&self) -> anyhow::Result<()> {+ match self.settlement.take_due(Instant::now()) {+ Some(offsets) => self.commit_offsets(&offsets),+ None => Ok(()),+ }+ }++ fn commit_all(&self) -> anyhow::Result<()> {+ match self.settlement.take_all() {+ Some(offsets) => self.commit_offsets(&offsets),+ None => Ok(()),+ }+ }+fn fail_batch_processing(&self, err: anyhow::Error) {error!(error = %err, "Batch processing failed");counter!("ingestion_consumer_batch_errors_total").increment(1);@@ -923,12 +1055,12 @@ impl IngestionConsumer {partition.generation,partition.charges.iter().copied(),) {- warn_rejection(+ report_rejection("charge",topic_partition,rejection,RejectedSlice::charged(&partition.charges),- );+ )?;}}@@ -938,122 +1070,6 @@ impl IngestionConsumer {stats,})}-- /// Settle the poll against the ledger and hand each partition's frontier- /// over for commit. A partition without a frontier stays on its last- /// commit.- fn settle_poll(&self, partitions: &HashMap<TopicPartition, PartitionDeliveries>) {- if partitions.is_empty() {- // Unreachable while batches require messages to be spawned; counted- // so "no empty commits" is a measurable guarantee, not an assumption.- counter!("ingestion_consumer_commit_violations_total", "kind" => "empty").increment(1);- warn!("Poll settled with no partitions");- return;- }-- let settlement =- settle_partitions(&self.topic_offset_ledger, &self.commit_sentinel, partitions);-- if settlement.advanced == 0 {- // `rejected`: the ledger dropped every slice, expected around a- // rebalance. `no_frontier`: a slice landed, but an earlier batch- // is still incomplete at the front of every window it settled.- let reason = if settlement.settled == 0 {- "rejected"- } else {- "no_frontier"- };- counter!("ingestion_consumer_commits_skipped_total", "reason" => reason).increment(1);- warn!(- reason,- "No ledger frontier available for completed offsets; skipping commit"- );- }- }-}--/// How a poll settled: how many partitions the ledger accepted, how many of-/// those reached a frontier that was handed over for commit, and what the-/// sentinel found on the way. The sentinel has already counted and logged-/// the violations; they are returned for tests.-struct PollSettlement {- settled: usize,- advanced: usize,- violations: Vec<CommitViolation>,-}--/// Settle each partition's slice against the ledger and hand every frontier-/// reached to the sentinel, with the span the poll delivered for it.-fn settle_partitions(- ledger: &TopicOffsetLedger,- sentinel: &CommitSentinel,- partitions: &HashMap<TopicPartition, PartitionDeliveries>,-) -> PollSettlement {- let mut settlement = PollSettlement {- settled: 0,- advanced: 0,- violations: Vec::new(),- };- for (topic_partition, partition) in partitions {- // A rejected slice is not committed, and the commit sentinel- // keeps its baseline. A stale slice belongs to an assignment the- // revoke callback already forgot on the sentinel, so the- // partition's next commit rebaselines. A violation reset the- // ledger and dropped what it held, so the partition's next commit- // can pass work still in flight; the sentinel reports that as the- // gap it is.- let Ok(frontier) = settle(ledger, topic_partition, partition) else {- continue;- };- settlement.settled += 1;- let Some(span) = frontier_span(&partition.span, frontier) else {- continue;- };- let Some(taken) = ledger.take_frontier(topic_partition) else {- continue;- };- settlement- .violations- .extend(sentinel.advance_frontier(topic_partition, span, taken));- settlement.advanced += 1;- }- settlement-}--/// Settle one partition's slice of a batch against the ledger and report-/// the frontier it reached. `Err` when the ledger rejected the slice,-/// which it has already counted and this logs.-fn settle(- ledger: &TopicOffsetLedger,- topic_partition: &TopicPartition,- partition: &PartitionDeliveries,-) -> Result<Option<Offset>, Rejection> {- ledger- .settle(- topic_partition,- partition.generation,- partition.charges.iter().map(|(offset, _)| *offset),- )- .inspect_err(|rejection| {- warn_rejection(- "settle",- topic_partition,- *rejection,- RejectedSlice::settled(&partition.span),- )- })-}--/// Map a settled frontier back to the span the sentinel checks: the-/// frontier is next-to-read and the span is last-processed, so the span-/// starts at the poll's first delivered offset and ends one before the-/// frontier. `None` for a partition that settled without a frontier; it-/// stays on its last commit.-fn frontier_span(span: &OffsetSpan, frontier: Option<Offset>) -> Option<OffsetSpan> {- frontier.map(|frontier| OffsetSpan {- first: span.first,- last: frontier.0 - 1,- })}/// Emit the per-poll parity metrics (received counts, batch sizes,@@ -1186,8 +1202,7 @@ fn emit_latest_processed_timestamp_metrics(partitions: &HashMap<TopicPartition, PartitionDeliveries>,group_id: &str,) {- // Per-partition latest committed timestamp — matches Node.js- // `latest_processed_timestamp_ms`.+ // Same name as the Node.js consumer's gauge, so one dashboard reads both.for (topic_partition, partition) in partitions {gauge!("latest_processed_timestamp_ms",@@ -1202,7 +1217,7 @@ fn emit_latest_processed_timestamp_metrics(#[cfg(test)]mod tests {use super::*;- use crate::commit_sentinel::CommitViolationKind;+ use crate::settlement::{charge_offsets, GroupSettlement};use common_kafka_consumer::Offset as MessageOffset;fn poll(epoch: u64, partition: i32, first: i64, last: i64, count: u32) -> InFlightPoll {@@ -1244,19 +1259,148 @@ mod tests {}}+ fn credit(in_flight: &mut VecDeque<InFlightPoll>, completion: GroupCompletion) {+ apply_completion(in_flight, &completion, |_, _, _| Ok(())).unwrap();+ }++ #[test]+ fn an_assignment_without_a_revoke_drops_the_partitions_pending_frontier() {+ let ledger = Arc::new(TopicOffsetLedger::new());+ let settlement: Arc<dyn Settlement> = Arc::new(GroupSettlement::new(+ Arc::clone(&ledger),+ Duration::from_secs(3600),+ ));+ let epoch = AssignmentEpoch::new();+ let on_assign = assign_hook(Arc::clone(&ledger), Arc::clone(&settlement), epoch.clone());+ let p0 = TopicPartition::new("events", 0);+ charge_offsets(&ledger, &p0, ledger.generation(&p0), &[10]);+ settlement+ .settle_run(&p0, ledger.generation(&p0), &[MessageOffset(10)])+ .unwrap();++ on_assign(std::slice::from_ref(&p0));++ assert!(settlement.take_all().is_none());+ assert_eq!(ledger.generation(&p0), 1);+ assert_eq!(epoch.current(), 2);+ }++ fn regained_poll(charged: i64, span_last: i64, delivered: u32) -> InFlightPoll {+ let mut poll = poll(1, 0, charged, charged, delivered);+ let deliveries = poll+ .partitions+ .get_mut(&TopicPartition::new("test", 0))+ .unwrap();+ deliveries.span.last = span_last;+ deliveries.generation = 2;+ poll+ }+#[test]- fn apply_completion_credits_the_poll_holding_the_offsets() {+ fn an_accepted_completion_settles_one_run_per_poll() {let mut in_flight = VecDeque::from([poll(1, 0, 0, 3, 4), poll(1, 0, 4, 7, 4)]);+ in_flight[1]+ .partitions+ .get_mut(&TopicPartition::new("test", 0))+ .unwrap()+ .generation = 1;+ let mut runs = Vec::new();- apply_completion(&mut in_flight, completion(1, 0, &[4, 6], 2));+ apply_completion(+ &mut in_flight,+ &completion(1, 0, &[2, 3, 4, 5], 4),+ |topic_partition, generation, offsets| {+ runs.push((topic_partition.partition, generation, offsets.to_vec()));+ Ok(())+ },+ )+ .unwrap();- assert_eq!(in_flight[0].covered, 0);- assert_eq!(in_flight[1].covered, 2);- assert_eq!(in_flight[1].accepted, 2);- assert!(!in_flight[1].is_complete());+ assert_eq!(+ runs,+ vec![+ (0, 0, vec![MessageOffset(2), MessageOffset(3)]),+ (0, 1, vec![MessageOffset(4), MessageOffset(5)]),+ ],+ "each run carries its own poll's generation"+ );+ }++ #[test]+ fn a_completion_spanning_two_polls_commits_once_the_prefix_is_whole() {+ let ledger = Arc::new(TopicOffsetLedger::new());+ let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO);+ let partition = TopicPartition::new("test", 0);+ charge_offsets(&ledger, &partition, 0, &[0, 1, 2, 3, 4, 5, 6, 7]);+ let mut in_flight = VecDeque::from([poll(1, 0, 0, 3, 4), poll(1, 0, 4, 7, 4)]);+ let mut settle = |topic_partition: &TopicPartition, generation: u64, offsets: &[Offset]| {+ settlement.settle_run(topic_partition, generation, offsets)+ };++ apply_completion(+ &mut in_flight,+ &completion(1, 0, &[2, 3, 4, 5], 4),+ &mut settle,+ )+ .unwrap();+ assert!(+ settlement.take_due(Instant::now()).is_none(),+ "offsets 0 and 1 are still in flight"+ );- apply_completion(&mut in_flight, completion(1, 0, &[5, 7], 2));- assert!(in_flight[1].is_complete());+ apply_completion(+ &mut in_flight,+ &completion(1, 0, &[0, 1, 6, 7], 4),+ &mut settle,+ )+ .unwrap();+ assert_eq!(+ settlement.take_due(Instant::now()),+ Some(HashMap::from([(partition, MessageOffset(8))]))+ );+ }++ #[test]+ fn a_short_accepted_completion_settles_nothing() {+ let mut in_flight = VecDeque::from([poll(1, 0, 0, 3, 4)]);+ let mut runs = 0;++ apply_completion(&mut in_flight, &completion(1, 0, &[0, 1], 1), |_, _, _| {+ runs += 1;+ Ok(())+ })+ .unwrap();++ assert_eq!(runs, 0, "the ledger must not commit past a dropped message");+ assert_eq!(in_flight[0].covered, 2);+ assert_eq!(in_flight[0].accepted, 1);+ }++ #[test]+ fn a_regained_poll_settles_each_charged_offset_once() {+ // Offsets 8..10, a regain, then offset 8 again: four deliveries, one charge.+ let mut in_flight = VecDeque::from([regained_poll(8, 10, 4)]);+ let mut runs = Vec::new();+ let mut settle = |_: &TopicPartition, generation: u64, offsets: &[Offset]| {+ runs.push((generation, offsets.to_vec()));+ Ok(())+ };++ apply_completion(+ &mut in_flight,+ &completion(1, 0, &[8, 9, 10], 3),+ &mut settle,+ )+ .unwrap();+ apply_completion(&mut in_flight, &completion(1, 0, &[8], 1), &mut settle).unwrap();++ assert_eq!(+ runs,+ vec![(2, vec![MessageOffset(8)])],+ "neither the uncharged old offsets nor the redelivered one reach the ledger"+ );+ assert!(in_flight[0].is_complete());+ assert!(pop_committable(&mut in_flight).is_ok());}#[test]@@ -1265,7 +1409,7 @@ mod tests {// send, so its completion spans both spans.let mut in_flight = VecDeque::from([poll(1, 0, 0, 3, 4), poll(1, 0, 4, 7, 4)]);- apply_completion(&mut in_flight, completion(1, 0, &[2, 3, 4, 5], 4));+ credit(&mut in_flight, completion(1, 0, &[2, 3, 4, 5], 4));assert_eq!(in_flight[0].covered, 2);assert_eq!(in_flight[0].accepted, 2);@@ -1277,7 +1421,7 @@ mod tests {fn apply_completion_under_report_shorts_the_tail_poll() {let mut in_flight = VecDeque::from([poll(1, 0, 0, 3, 4), poll(1, 0, 4, 7, 4)]);- apply_completion(&mut in_flight, completion(1, 0, &[2, 3, 4, 5], 3));+ credit(&mut in_flight, completion(1, 0, &[2, 3, 4, 5], 3));assert_eq!(in_flight[0].accepted, 2, "leading offsets are accepted");assert_eq!(@@ -1291,7 +1435,7 @@ mod tests {fn a_complete_poll_with_a_charged_offset_no_completion_covered_is_not_committable() {let mut in_flight = VecDeque::from([poll(1, 0, 10, 12, 3)]);- apply_completion(&mut in_flight, completion(1, 0, &[10, 12, 12], 3));+ credit(&mut in_flight, completion(1, 0, &[10, 12, 12], 3));assert!(in_flight[0].is_complete());let error = pop_committable(&mut in_flight)@@ -1315,7 +1459,7 @@ mod tests {.push((MessageOffset(i64::MAX), Charge::ZERO));let mut in_flight = VecDeque::from([gapped]);- apply_completion(&mut in_flight, completion(1, 0, &[0, i64::MAX], 2));+ credit(&mut in_flight, completion(1, 0, &[0, i64::MAX], 2));assert!(in_flight[0].is_complete());assert!(pop_committable(&mut in_flight).is_ok());@@ -1351,7 +1495,7 @@ mod tests {.delivered = 2;let mut in_flight = VecDeque::from([two]);// Partition 1 already had one message covered before the revoke.- apply_completion(&mut in_flight, completion(1, 1, &[10], 1));+ credit(&mut in_flight, completion(1, 1, &[10], 1));let stripped = strip_revoked_partitions(&mut in_flight,@@ -1368,9 +1512,9 @@ mod tests {// The kept partition completes the poll; a late completion for the// revoked partition is discarded, not credited.- apply_completion(&mut in_flight, completion(1, 1, &[11], 1));+ credit(&mut in_flight, completion(1, 1, &[11], 1));assert_eq!(in_flight[0].covered, 0);- apply_completion(&mut in_flight, completion(1, 0, &[0, 1], 2));+ credit(&mut in_flight, completion(1, 0, &[0, 1], 2));assert!(in_flight[0].is_complete());assert_eq!(in_flight[0].accepted, 2);assert!(pop_committable(&mut in_flight).is_ok());@@ -1461,7 +1605,7 @@ mod tests {assert_eq!(in_flight.len(), 1);assert_eq!(in_flight[0].poll_id, "reassigned");- apply_completion(&mut in_flight, completion(1, 0, &[10, 10], 2));+ credit(&mut in_flight, completion(1, 0, &[10, 10], 2));assert!(in_flight[0].is_complete());let reassigned = pop_committable(&mut in_flight).expect("an offset credited twice still counts as credited");@@ -1508,160 +1652,47 @@ mod tests {// completions in its own poll.let mut in_flight = VecDeque::from([poll(1, 0, 0, 3, 4), poll(2, 0, 0, 3, 4)]);- apply_completion(&mut in_flight, completion(2, 0, &[0, 1, 2, 3], 4));+ credit(&mut in_flight, completion(2, 0, &[0, 1, 2, 3], 4));assert_eq!(in_flight[0].covered, 0);assert_eq!(in_flight[1].covered, 4);}#[test]- fn apply_completion_discards_a_completion_matching_no_poll() {- let mut in_flight = VecDeque::from([poll(1, 0, 0, 3, 4)]);-- // Wrong partition, then wrong epoch: neither may be credited.- apply_completion(&mut in_flight, completion(1, 2, &[1], 1));- apply_completion(&mut in_flight, completion(9, 0, &[1], 1));-- assert_eq!(in_flight[0].covered, 0);- assert_eq!(in_flight[0].accepted, 0);- }-- /// A poll's slice of one partition, charged to the ledger under the- /// partition's current generation.- fn charged(- ledger: &TopicOffsetLedger,- topic_partition: &TopicPartition,- first: i64,- last: i64,- ) -> PartitionDeliveries {- let generation = ledger.generation(topic_partition);- let partition = PartitionDeliveries {- span: OffsetSpan { first, last },- generation,- generations_version_seen: 0,- charges: (first..=last)- .map(|offset| (MessageOffset(offset), Charge::ZERO))- .collect(),- latest_kafka_ts: 0,- max_lag_ms: None,- delivered: (last - first + 1) as u32,- covered: 0,- accepted: 0,- credited: HashSet::default(),+ fn a_covered_head_is_popped_before_a_redelivered_offset_is_matched() {+ let ledger = Arc::new(TopicOffsetLedger::new());+ let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO);+ let partition = TopicPartition::new("test", 0);+ ledger.forget_partitions([("test", 0)]);+ charge_offsets(&ledger, &partition, 1, &[0, 1]);+ let mut settle = |topic_partition: &TopicPartition, generation: u64, offsets: &[Offset]| {+ settlement.settle_run(topic_partition, generation, offsets)};- ledger- .charge(- topic_partition,- generation,- partition.charges.iter().copied(),- )- .expect("charge");- partition- }-- #[test]- fn a_settled_poll_hands_its_frontier_to_the_pacer() {- let ledger = TopicOffsetLedger::new();- let sentinel = CommitSentinel::new(ImmediateCommitPacer::new());- let tp = TopicPartition::new("test", 0);- let partitions = HashMap::from([(tp.clone(), charged(&ledger, &tp, 10, 11))]);-- let settlement = settle_partitions(&ledger, &sentinel, &partitions);- assert_eq!((settlement.settled, settlement.advanced), (1, 1));- assert_eq!(- sentinel.take_due(),- Some(HashMap::from([(tp, MessageOffset(12))]))- );- }-- #[test]- fn a_gap_in_what_kafka_delivered_fires_the_sentinel_and_still_commits() {- let ledger = TopicOffsetLedger::new();- let sentinel = CommitSentinel::new(ImmediateCommitPacer::new());- let tp = TopicPartition::new("test", 0);- let first = HashMap::from([(tp.clone(), charged(&ledger, &tp, 10, 11))]);- assert!(settle_partitions(&ledger, &sentinel, &first)- .violations- .is_empty());- assert_eq!(- sentinel.take_due(),- Some(HashMap::from([(tp.clone(), MessageOffset(12))]))- );-- // Offsets 12 and 13 never arrived. The ledger walks the gap, so its- // take chains from 12; the sentinel must see what was delivered.- let second = HashMap::from([(tp.clone(), charged(&ledger, &tp, 14, 15))]);- let settlement = settle_partitions(&ledger, &sentinel, &second);-- assert_eq!(settlement.violations.len(), 1);- assert_eq!(settlement.violations[0].kind, CommitViolationKind::Gap);- assert_eq!(settlement.violations[0].prev_committed, 12);- assert_eq!(settlement.violations[0].span.first, 14);- assert_eq!(- sentinel.take_due(),- Some(HashMap::from([(tp, MessageOffset(16))]))+ // Both polls carry the same offsets: a regain during collection.+ let mut old = poll(3, 0, 0, 1, 2);+ old.poll_id = "old".into();+ let mut replay = poll(3, 0, 0, 1, 2);+ replay.poll_id = "replay".into();+ replay.partitions.get_mut(&partition).unwrap().generation = 1;+ let mut in_flight = VecDeque::from([old, replay]);+ let event = completion(3, 0, &[0, 1], 2);++ apply_completion(&mut in_flight, &event, &mut settle).unwrap();+ assert_eq!(in_flight[0].poll_id, "old");+ assert!(in_flight[0].is_complete());+ assert!(+ settlement.take_due(Instant::now()).is_none(),+ "the old generation's settle is stale");- }-- #[test]- fn a_rejected_slice_hands_nothing_to_the_pacer() {- let ledger = TopicOffsetLedger::new();- let sentinel = CommitSentinel::new(ImmediateCommitPacer::new());- let tp = TopicPartition::new("test", 0);- let partitions = HashMap::from([(tp.clone(), charged(&ledger, &tp, 10, 11))]);- // The partition left and came back while the poll was in flight.- ledger.forget_partitions([("test", 0)]);-- let settlement = settle_partitions(&ledger, &sentinel, &partitions);-- assert_eq!((settlement.settled, settlement.advanced), (0, 0));- assert!(sentinel.take_due().is_none());- }-- #[test]- fn a_poll_behind_an_incomplete_one_hands_nothing_to_the_pacer() {- let ledger = TopicOffsetLedger::new();- let sentinel = CommitSentinel::new(ImmediateCommitPacer::new());- let tp = TopicPartition::new("test", 0);- let _older = charged(&ledger, &tp, 10, 11);- let newer = HashMap::from([(tp.clone(), charged(&ledger, &tp, 12, 13))]);-- let settlement = settle_partitions(&ledger, &sentinel, &newer);+ in_flight.pop_front();- assert_eq!((settlement.settled, settlement.advanced), (1, 0));- assert!(sentinel.take_due().is_none());- }-- #[test]- fn frontier_span_submits_the_frontier_verbatim() {- let span = OffsetSpan {- first: 10,- last: 11,- };- assert_eq!(- frontier_span(&span, Some(MessageOffset(12))),- Some(OffsetSpan {- first: 10,- last: 11- })- );+ apply_completion(&mut in_flight, &event, &mut settle).unwrap();+ assert_eq!(in_flight[0].poll_id, "replay");+ assert!(in_flight[0].is_complete());assert_eq!(- frontier_span(&span, Some(MessageOffset(11))),- Some(OffsetSpan {- first: 10,- last: 10- }),- "a frontier trailing the span wins"+ settlement.take_due(Instant::now()),+ Some(HashMap::from([(partition, MessageOffset(2))])));}-- #[test]- fn a_partition_without_a_frontier_is_not_committed() {- let span = OffsetSpan {- first: 20,- last: 21,- };- assert_eq!(frontier_span(&span, None), None);- }}diff --git a/rust/ingestion-consumer/src/debug_recorder.rs b/rust/ingestion-consumer/src/debug_recorder.rsindex d3ad9b974bfd..ac9a5e814f76 100644--- a/rust/ingestion-consumer/src/debug_recorder.rs+++ b/rust/ingestion-consumer/src/debug_recorder.rs@@ -78,6 +78,13 @@ pub fn partition_offsets(messages: &[SerializedKafkaMessage]) -> Vec<PartitionOfspans}+#[derive(Clone, Serialize)]+pub struct CommitOffset {+ pub topic: String,+ pub partition: i32,+ pub offset: i64,+}+/// The distinct event kinds recorded across the consumer's lifecycle. Serialized/// with an internal `type` tag so the UI can switch on it.#[derive(Clone, Serialize)]@@ -127,13 +134,14 @@ pub enum DebugEventKind {cause: &'static str,messages: usize,},- /// A batch's offsets were committed to Kafka.- BatchCommitted {+ PollCompleted {batch_id: String,accepted: u32,duration_ms: u64,partitions: Vec<PartitionOffset>,},+ /// An async offset commit was handed to the Kafka client.+ CommitSubmitted { offsets: Vec<CommitOffset> },/// Batch processing failed; the process will exit and restart.BatchFailed {batch_id: Option<String>,diff --git a/rust/ingestion-consumer/src/ledger_rejection.rs b/rust/ingestion-consumer/src/ledger_rejection.rsindex c00dc3033ccb..486f2554a958 100644--- a/rust/ingestion-consumer/src/ledger_rejection.rs+++ b/rust/ingestion-consumer/src/ledger_rejection.rs@@ -41,16 +41,17 @@ impl RejectedSlice {}}-/// Log one charge or settlement the ledger rejected. A stale slice is-/// expected around a rebalance and stays quiet; a violation is a bug in the-/// accounting and names the slice. Callers must build `slice` inside their-/// error arm so the happy path pays nothing for it.-pub(crate) fn warn_rejection(+/// Log one charge or settlement the ledger rejected, and fail on a violation:+/// the ledger reset the partition, so only a replay from the last commit keeps+/// its in-flight offsets. A stale slice is expected around a rebalance and stays+/// quiet. Callers must build `slice` inside their error arm so the happy path+/// pays nothing for it.+pub(crate) fn report_rejection(stage: &'static str,topic_partition: &TopicPartition,rejection: Rejection,slice: RejectedSlice,-) {+) -> anyhow::Result<()> {let Rejection::Violation {error,stamp,@@ -58,7 +59,7 @@ pub(crate) fn warn_rejection(held,} = rejectionelse {- return;+ return Ok(());};// A tracing field name is fixed at the call site, so each variant needs// its own `warn!`.@@ -95,4 +96,9 @@ pub(crate) fn warn_rejection("Offset ledger rejected a slice and reset its partition"),}+ Err(anyhow::anyhow!(+ "offset ledger rejected a {stage} on {}/{}: {error}",+ topic_partition.topic,+ topic_partition.partition+ ))}diff --git a/rust/ingestion-consumer/src/lib.rs b/rust/ingestion-consumer/src/lib.rsindex f8c1e6e1554e..32307723384f 100644--- a/rust/ingestion-consumer/src/lib.rs+++ b/rust/ingestion-consumer/src/lib.rs@@ -1,8 +1,6 @@pub mod aperture;pub mod batcher;-pub mod commit_monitor;-pub mod commit_pacer;-pub mod commit_sentinel;+mod commit_pacer;pub mod config;pub mod consumer;pub mod debug_recorder;@@ -12,7 +10,9 @@ pub mod kafka_stats;pub mod ledger_rejection;pub mod order_sentinel;pub mod readiness;+pub mod rebalance;pub mod routing;+mod settlement;pub mod transport;pub mod types;pub mod worker_registry;diff --git a/rust/ingestion-consumer/src/order_sentinel.rs b/rust/ingestion-consumer/src/order_sentinel.rsindex bb9073e1de3f..faf9999a339d 100644--- a/rust/ingestion-consumer/src/order_sentinel.rs+++ b/rust/ingestion-consumer/src/order_sentinel.rs@@ -1,5 +1,4 @@-//! The per-key send-order sentinel and the consumer's rdkafka context. The-//! commit sentinel lives in [`crate::commit_sentinel`].+//! The per-key send-order sentinel.//!//! **Per-key send order** ([`KeyOrderSentinel`]): for every routing key//! (the Kafka message key), messages must be handed to workers in Kafka offset@@ -15,25 +14,16 @@//! and offsets from different partitions are not comparable. Skipped messages//! are counted in `ingestion_consumer_key_sentinel_unkeyed_total`.//!-//! [`SentinelContext`] is the consumer's rdkafka context: it resets sentinel-//! baselines on rebalances, where Kafka legitimately re-deals partitions and-//! the invariants must re-baseline instead of firing false positives.-//!//! The sentinel is a pure observer: it never influences routing.use std::collections::HashMap;use std::sync::atomic::{AtomicBool, Ordering};-use std::sync::{Arc, Mutex, MutexGuard, OnceLock};+use std::sync::{Arc, Mutex, MutexGuard};use metrics::{counter, gauge};-use rdkafka::consumer::{BaseConsumer, ConsumerContext, Rebalance};-use rdkafka::{ClientContext, Statistics, TopicPartitionList};-use tracing::{info, warn};+use tracing::warn;-use crate::commit_pacer::ImmediateCommitPacer;-use crate::commit_sentinel::CommitSentinel;use crate::types::SerializedKafkaMessage;-use common_kafka_consumer::{AssignmentEpoch, TopicOffsetLedger, TopicPartition};/// The first and last Kafka offsets a batch holds for one topic-partition.#[derive(Clone, Copy, Debug, PartialEq, Eq)]@@ -405,166 +395,6 @@ pub fn ack_mark(.max()}-/// The consumer's rdkafka context: observes async commit results (a-/// fire-and-forget `CommitMode::Async` failure is otherwise invisible until-/// restart-time redelivery), forgets revoked partitions around rebalances, and-/// exports librdkafka's internal statistics (see [`crate::kafka_stats`]).-pub struct SentinelContext {- /// Where the consumer's frontiers go. Held here so the rebalance- /// callbacks tell it which partitions leave the assignment.- commit_sentinel: Arc<CommitSentinel>,- /// The offset ledger the commit path settles against. Owned here so the- /// rebalance callbacks forget partitions on the same ledger.- topic_offset_ledger: Arc<TopicOffsetLedger>,- /// Advanced once per assignment callback; the gRPC transport stamps it- /// on sub-batches so the worker's feed-order sentinel rebaselines across- /// rebalances. Distinct from the offset ledger's generations, which move- /// per partition and stamp offset accounting rather than stream order.- assignment_epoch: Option<AssignmentEpoch>,- /// Called with the revoked `(topic, partition)` pairs, so the batcher- /// can drop their queued messages before the new owner replays them.- revoke_hook: OnceLock<RevokeHook>,-}--pub type RevokeHook = Box<dyn Fn(&[(String, i32)]) + Send + Sync>;--impl SentinelContext {- pub fn new(- commit_sentinel: Arc<CommitSentinel>,- topic_offset_ledger: Arc<TopicOffsetLedger>,- ) -> Self {- Self {- commit_sentinel,- topic_offset_ledger,- assignment_epoch: None,- revoke_hook: OnceLock::new(),- }- }-- /// Wire the revocation hook; a second call is ignored. Settable after- /// construction because the rdkafka consumer only hands out `&self`.- pub fn set_revoke_hook(&self, hook: RevokeHook) {- let _ = self.revoke_hook.set(hook);- }-- /// Wire the process-wide assignment epoch. Call before the context is- /// handed to the Kafka consumer.- pub fn set_assignment_epoch(&mut self, epoch: AssignmentEpoch) {- self.assignment_epoch = Some(epoch);- }-- /// A context with its own free-standing sentinels, ledger, and commit- /// pacer, for tests and tools that build the Kafka consumer separately- /// from the batcher.- pub fn detached() -> Self {- Self::new(- Arc::new(CommitSentinel::new(ImmediateCommitPacer::new())),- Arc::new(TopicOffsetLedger::new()),- )- }-- pub fn commit_sentinel(&self) -> Arc<CommitSentinel> {- Arc::clone(&self.commit_sentinel)- }-- pub fn topic_offset_ledger(&self) -> Arc<TopicOffsetLedger> {- Arc::clone(&self.topic_offset_ledger)- }-- /// Start a new ledger generation for every partition in `tpl`, dropping- /// its window and any frontier it had ready to commit.- fn forget_ledger_partitions(&self, tpl: &TopicPartitionList) {- let elements = tpl.elements();- self.topic_offset_ledger- .forget_partitions(elements.iter().map(|e| (e.topic(), e.partition())));- let topic_partitions: Vec<TopicPartition> = elements- .iter()- .map(|e| TopicPartition::new(e.topic(), e.partition()))- .collect();- self.commit_sentinel.forget_partitions(&topic_partitions);- }-}--fn partition_names(tpl: &TopicPartitionList) -> Vec<String> {- let elements = tpl.elements();- let mut partitions: Vec<(&str, i32)> = elements- .iter()- .map(|element| (element.topic(), element.partition()))- .collect();- partitions.sort_unstable();- partitions- .into_iter()- .map(|(topic, partition)| format!("{topic}:{partition}"))- .collect()-}--impl ClientContext for SentinelContext {- /// Fired on a librdkafka thread every `statistics.interval.ms`; disabled- /// when that is 0.- fn stats(&self, stats: Statistics) {- crate::kafka_stats::export(&stats);- }-}--impl ConsumerContext for SentinelContext {- fn pre_rebalance(&self, _consumer: &BaseConsumer<Self>, rebalance: &Rebalance) {- match rebalance {- Rebalance::Revoke(tpl) => {- counter!("ingestion_consumer_rebalances_total", "event" => "revoke").increment(1);- info!(- partitions = tpl.count(),- topic_partitions = ?partition_names(tpl),- "Rebalance: partitions revoked"- );- self.forget_ledger_partitions(tpl);- // The hook must run after the ledger forget above: it stamps- // each revocation with the bumped generation so only older- // poll slices are stripped.- if let Some(hook) = self.revoke_hook.get() {- let partitions: Vec<(String, i32)> = tpl- .elements()- .iter()- .map(|e| (e.topic().to_string(), e.partition()))- .collect();- hook(&partitions);- }- }- Rebalance::Assign(_) => {}- Rebalance::Error(err) => {- counter!("ingestion_consumer_rebalances_total", "event" => "error").increment(1);- warn!(error = %err, "Rebalance error");- }- }- }-- fn post_rebalance(&self, _consumer: &BaseConsumer<Self>, rebalance: &Rebalance) {- if let Rebalance::Assign(tpl) = rebalance {- counter!("ingestion_consumer_rebalances_total", "event" => "assign").increment(1);- info!(- partitions = tpl.count(),- topic_partitions = ?partition_names(tpl),- "Rebalance: partitions assigned"- );- // An assign list names partitions that start a new assignment, so- // any surviving ledger for them is stale. The revoke callback- // normally dropped it already; this covers losses with no revoke- // callback (an error rebalance, a fenced member).- self.forget_ledger_partitions(tpl);- if let Some(epoch) = &self.assignment_epoch {- epoch.bump();- }- }- }-- // NOTE: `ConsumerContext::commit_callback` is deliberately not implemented.- // librdkafka only propagates a commit result to the application when a- // conf-level `offset_commit_cb` is registered (rust-rdkafka never does) or- // when the commit is synchronous (a replyq is attached); manual async- // commits silently drop their result (`rd_kafka_cgrp_propagate_commit_result`).- // Commit success is instead verified by the consumer's commit monitor via- // `CommitSentinel::observe_broker_committed`.-}-#[cfg(test)]mod tests {use super::*;@@ -868,17 +698,4 @@ mod tests {.note_sent("t:a", &[msg_at(0, 101)], SendKind::Fresh).is_empty());}-- #[test]- fn partition_names_sort_by_topic_then_partition_number() {- let mut tpl = TopicPartitionList::new();- tpl.add_partition("overflow", 2);- tpl.add_partition("events", 10);- tpl.add_partition("events", 9);-- assert_eq!(- partition_names(&tpl),- vec!["events:9", "events:10", "overflow:2"]- );- }}diff --git a/rust/ingestion-consumer/src/rebalance.rs b/rust/ingestion-consumer/src/rebalance.rsnew file mode 100644index 000000000000..6f3fb461c22f--- /dev/null+++ b/rust/ingestion-consumer/src/rebalance.rs@@ -0,0 +1,162 @@+//! The consumer's rdkafka context: rebalance callbacks into hooks, plus rebalance counters and client statistics.++use std::sync::OnceLock;++use common_kafka_consumer::TopicPartition;+use metrics::counter;+use rdkafka::consumer::{BaseConsumer, ConsumerContext, Rebalance};+use rdkafka::{ClientContext, Statistics, TopicPartitionList};+use tracing::{info, warn};++pub type PartitionHook = Box<dyn Fn(&[TopicPartition]) + Send + Sync>;++/// Hooks run inside the callback, before the next poll delivers the new assignment.+#[derive(Default)]+pub struct RebalanceContext {+ on_revoke: OnceLock<PartitionHook>,+ on_assign: OnceLock<PartitionHook>,+}++impl RebalanceContext {+ pub fn new() -> Self {+ Self::default()+ }++ /// Settable after construction because the rdkafka consumer only hands out `&self`.+ pub fn set_on_revoke(&self, hook: PartitionHook) {+ let _ = self.on_revoke.set(hook);+ }++ pub fn set_on_assign(&self, hook: PartitionHook) {+ let _ = self.on_assign.set(hook);+ }+}++fn topic_partitions(tpl: &TopicPartitionList) -> Vec<TopicPartition> {+ let mut partitions: Vec<TopicPartition> = tpl+ .elements()+ .iter()+ .map(|element| TopicPartition::new(element.topic(), element.partition()))+ .collect();+ partitions.sort_unstable_by(|a, b| (&a.topic, a.partition).cmp(&(&b.topic, b.partition)));+ partitions+}++fn partition_names(partitions: &[TopicPartition]) -> Vec<String> {+ partitions+ .iter()+ .map(|tp| format!("{}:{}", tp.topic, tp.partition))+ .collect()+}++impl ClientContext for RebalanceContext {+ /// Fired on a librdkafka thread every `statistics.interval.ms`; disabled+ /// when that is 0.+ fn stats(&self, stats: Statistics) {+ crate::kafka_stats::export(&stats);+ }+}++impl ConsumerContext for RebalanceContext {+ fn pre_rebalance(&self, _consumer: &BaseConsumer<Self>, rebalance: &Rebalance) {+ match rebalance {+ Rebalance::Revoke(tpl) => {+ counter!("ingestion_consumer_rebalances_total", "event" => "revoke").increment(1);+ let partitions = topic_partitions(tpl);+ info!(+ partitions = partitions.len(),+ topic_partitions = ?partition_names(&partitions),+ "Rebalance: partitions revoked"+ );+ if let Some(hook) = self.on_revoke.get() {+ hook(&partitions);+ }+ }+ Rebalance::Assign(_) => {}+ Rebalance::Error(err) => {+ counter!("ingestion_consumer_rebalances_total", "event" => "error").increment(1);+ warn!(error = %err, "Rebalance error");+ }+ }+ }++ fn post_rebalance(&self, _consumer: &BaseConsumer<Self>, rebalance: &Rebalance) {+ if let Rebalance::Assign(tpl) = rebalance {+ counter!("ingestion_consumer_rebalances_total", "event" => "assign").increment(1);+ let partitions = topic_partitions(tpl);+ info!(+ partitions = partitions.len(),+ topic_partitions = ?partition_names(&partitions),+ "Rebalance: partitions assigned"+ );+ if let Some(hook) = self.on_assign.get() {+ hook(&partitions);+ }+ }+ }++ // NOTE: `ConsumerContext::commit_callback` is deliberately not implemented.+ // librdkafka only propagates a commit result to the application when a+ // conf-level `offset_commit_cb` is registered (rust-rdkafka never does) or+ // when the commit is synchronous (a replyq is attached); manual async+ // commits silently drop their result (`rd_kafka_cgrp_propagate_commit_result`).+}++#[cfg(test)]+mod tests {+ use std::sync::{Arc, Mutex};++ use rdkafka::config::ClientConfig;+ use rdkafka::consumer::Consumer;++ use super::*;++ fn recording(seen: &Arc<Mutex<Vec<TopicPartition>>>) -> PartitionHook {+ let seen = Arc::clone(seen);+ Box::new(move |partitions| seen.lock().unwrap().extend_from_slice(partitions))+ }++ #[test]+ fn partitions_sort_by_topic_then_partition_number() {+ let mut tpl = TopicPartitionList::new();+ tpl.add_partition("overflow", 2);+ tpl.add_partition("events", 10);+ tpl.add_partition("events", 9);++ assert_eq!(+ partition_names(&topic_partitions(&tpl)),+ vec!["events:9", "events:10", "overflow:2"]+ );+ }++ #[test]+ fn each_callback_reaches_its_hook_with_the_partitions() {+ let revoked = Arc::new(Mutex::new(Vec::new()));+ let assigned = Arc::new(Mutex::new(Vec::new()));+ let context = RebalanceContext::new();+ context.set_on_revoke(recording(&revoked));+ context.set_on_assign(recording(&assigned));+ let consumer: BaseConsumer<RebalanceContext> = ClientConfig::new()+ .set("bootstrap.servers", "127.0.0.1:1")+ .set("group.id", "test")+ .create_with_context(context)+ .expect("a client needs no broker to be built");+ let mut tpl = TopicPartitionList::new();+ tpl.add_partition("events", 1);+ tpl.add_partition("events", 0);++ consumer+ .context()+ .pre_rebalance(&consumer, &Rebalance::Revoke(&tpl));+ consumer+ .context()+ .post_rebalance(&consumer, &Rebalance::Assign(&tpl));++ let expected = vec![+ TopicPartition::new("events", 0),+ TopicPartition::new("events", 1),+ ];+ assert_eq!(*revoked.lock().unwrap(), expected);+ assert_eq!(*assigned.lock().unwrap(), expected);+ }+}diff --git a/rust/ingestion-consumer/src/settlement/group.rs b/rust/ingestion-consumer/src/settlement/group.rsnew file mode 100644index 000000000000..7152349136ed--- /dev/null+++ b/rust/ingestion-consumer/src/settlement/group.rs@@ -0,0 +1,221 @@+use std::collections::HashMap;+use std::sync::Arc;+use std::time::{Duration, Instant};++use common_kafka_consumer::{Offset, TopicOffsetLedger, TopicPartition};++use super::{ChargedSlice, Settlement};+use crate::commit_pacer::CommitPacer;+use crate::ledger_rejection::{report_rejection, RejectedSlice};+use crate::order_sentinel::OffsetSpan;++pub(crate) struct GroupSettlement {+ ledger: Arc<TopicOffsetLedger>,+ pacer: Arc<CommitPacer>,+}++impl GroupSettlement {+ pub(crate) fn new(ledger: Arc<TopicOffsetLedger>, commit_interval: Duration) -> Self {+ Self {+ ledger,+ pacer: Arc::new(CommitPacer::every(commit_interval)),+ }+ }+}++impl Settlement for GroupSettlement {+ fn settle_run(+ &self,+ topic_partition: &TopicPartition,+ generation: u64,+ offsets: &[Offset],+ ) -> anyhow::Result<()> {+ match self+ .ledger+ .settle(topic_partition, generation, offsets.iter().copied())+ {+ Ok(Some(_)) => {+ if let Some(taken) = self.ledger.take_frontier(topic_partition) {+ self.pacer.advance_frontier(topic_partition, taken);+ }+ }+ Ok(None) => {}+ Err(rejection) => report_rejection(+ "settle",+ topic_partition,+ rejection,+ RejectedSlice::settled(&OffsetSpan {+ first: offsets[0].0,+ last: offsets[offsets.len() - 1].0,+ }),+ )?,+ }+ Ok(())+ }++ fn settle_poll(&self, _slices: &[ChargedSlice<'_>]) -> anyhow::Result<()> {+ Ok(())+ }++ fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, Offset>> {+ self.pacer.take_due(now)+ }++ fn take_all(&self) -> Option<HashMap<TopicPartition, Offset>> {+ self.pacer.take_all()+ }++ fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {+ self.pacer.forget_partitions(topic_partitions);+ }+}++#[cfg(test)]+mod tests {+ use super::super::charge_offsets;+ use super::*;++ fn settlement() -> (Arc<TopicOffsetLedger>, GroupSettlement) {+ let ledger = Arc::new(TopicOffsetLedger::new());+ let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO);+ (ledger, settlement)+ }++ #[test]+ fn a_partition_commits_only_once_its_completed_prefix_is_contiguous() {+ let (ledger, settlement) = settlement();+ let p0 = TopicPartition::new("events", 0);+ charge_offsets(&ledger, &p0, 0, &[10, 11, 12]);++ settlement.settle_run(&p0, 0, &[Offset(12)]).unwrap();+ assert!(+ settlement.take_due(Instant::now()).is_none(),+ "the window base is still incomplete"+ );++ settlement+ .settle_run(&p0, 0, &[Offset(10), Offset(11)])+ .unwrap();+ assert_eq!(+ settlement.take_due(Instant::now()),+ Some(HashMap::from([(p0, Offset(13))])),+ "the late completion releases the whole prefix"+ );+ }++ #[test]+ fn a_partition_with_a_held_prefix_does_not_hold_back_another() {+ let (ledger, settlement) = settlement();+ let p0 = TopicPartition::new("events", 0);+ let p1 = TopicPartition::new("events", 1);+ charge_offsets(&ledger, &p0, 0, &[10, 11]);+ charge_offsets(&ledger, &p1, 0, &[20, 21]);++ settlement+ .settle_run(&p0, 0, &[Offset(10), Offset(11)])+ .unwrap();+ settlement.settle_run(&p1, 0, &[Offset(21)]).unwrap();++ assert_eq!(+ settlement.take_due(Instant::now()),+ Some(HashMap::from([(p0, Offset(12))])),+ "partition 1 waits for offset 20"+ );+ }++ #[test]+ fn a_run_settled_twice_fails_and_keeps_the_frontier_taken_before_it() {+ let (ledger, settlement) = settlement();+ let p0 = TopicPartition::new("events", 0);+ charge_offsets(&ledger, &p0, 0, &[10, 11]);+ settlement.settle_run(&p0, 0, &[Offset(10)]).unwrap();++ assert!(settlement.settle_run(&p0, 0, &[Offset(10)]).is_err());++ assert_eq!(+ settlement.take_all(),+ Some(HashMap::from([(p0, Offset(11))])),+ "offset 10 completed before the violation"+ );+ }++ #[test]+ fn a_stale_stamp_run_commits_nothing() {+ let (ledger, settlement) = settlement();+ let p0 = TopicPartition::new("events", 0);+ charge_offsets(&ledger, &p0, 0, &[10]);++ ledger.forget_partitions([("events", 0)]);+ charge_offsets(&ledger, &p0, 1, &[10]);++ settlement.settle_run(&p0, 0, &[Offset(10)]).unwrap();++ assert!(settlement.take_due(Instant::now()).is_none());+ assert_eq!(+ ledger.held(&p0).offsets,+ 1,+ "the redelivered offset stays uncompleted"+ );+ }++ #[test]+ fn a_frontier_within_the_interval_waits_for_it_unless_everything_is_taken() {+ let ledger = Arc::new(TopicOffsetLedger::new());+ let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::from_secs(3600));+ let p0 = TopicPartition::new("events", 0);+ charge_offsets(&ledger, &p0, 0, &[10, 11]);+ let now = Instant::now();++ settlement.settle_run(&p0, 0, &[Offset(10)]).unwrap();+ assert_eq!(+ settlement.take_due(now),+ Some(HashMap::from([(p0.clone(), Offset(11))]))+ );++ settlement.settle_run(&p0, 0, &[Offset(11)]).unwrap();+ assert!(settlement.take_due(now).is_none());+ assert_eq!(+ settlement.take_all(),+ Some(HashMap::from([(p0, Offset(12))]))+ );+ }++ #[test]+ fn a_revoked_partitions_pending_frontier_is_not_committed() {+ let ledger = Arc::new(TopicOffsetLedger::new());+ let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::from_secs(3600));+ let p0 = TopicPartition::new("events", 0);+ charge_offsets(&ledger, &p0, 0, &[10]);+ settlement.settle_run(&p0, 0, &[Offset(10)]).unwrap();+ assert!(settlement.take_due(Instant::now()).is_some());++ charge_offsets(&ledger, &p0, 0, &[11]);+ settlement.settle_run(&p0, 0, &[Offset(11)]).unwrap();+ settlement.forget_partitions(&[p0]);++ assert!(settlement.take_all().is_none());+ }++ #[test]+ fn a_retired_poll_settles_nothing_more() {+ let (ledger, settlement) = settlement();+ let p0 = TopicPartition::new("events", 0);+ charge_offsets(&ledger, &p0, 0, &[10]);+ let charges = [(Offset(10), common_kafka_consumer::Charge::ZERO)];++ settlement+ .settle_poll(&[ChargedSlice {+ topic_partition: &p0,+ generation: 0,+ span: OffsetSpan {+ first: 10,+ last: 10,+ },+ charges: &charges,+ }])+ .unwrap();++ assert!(settlement.take_due(Instant::now()).is_none());+ assert_eq!(ledger.held(&p0).offsets, 1);+ }+}diff --git a/rust/ingestion-consumer/src/settlement/mod.rs b/rust/ingestion-consumer/src/settlement/mod.rsnew file mode 100644index 000000000000..d9c38a4f3da0--- /dev/null+++ b/rust/ingestion-consumer/src/settlement/mod.rs@@ -0,0 +1,69 @@+//! When accepted offsets reach the ledger and the commit pacer: per poll or per group.++mod group;+mod poll;++use std::collections::HashMap;+use std::sync::Arc;+use std::time::{Duration, Instant};++use common_kafka_consumer::{Charge, Offset, TopicOffsetLedger, TopicPartition};++use crate::config::CompletionGranularity;+use crate::order_sentinel::OffsetSpan;++pub(crate) use group::GroupSettlement;+pub(crate) use poll::PollSettlement;++pub(crate) struct ChargedSlice<'a> {+ pub(crate) topic_partition: &'a TopicPartition,+ pub(crate) generation: u64,+ pub(crate) span: OffsetSpan,+ pub(crate) charges: &'a [(Offset, Charge)],+}++pub(crate) trait Settlement: Send + Sync {+ /// Accepted offsets from one poll's slice of a partition, charged under `generation`.+ fn settle_run(+ &self,+ topic_partition: &TopicPartition,+ generation: u64,+ offsets: &[Offset],+ ) -> anyhow::Result<()>;++ /// The oldest poll retired with every message accepted.+ fn settle_poll(&self, slices: &[ChargedSlice<'_>]) -> anyhow::Result<()>;++ fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, Offset>>;++ fn take_all(&self) -> Option<HashMap<TopicPartition, Offset>>;++ fn forget_partitions(&self, topic_partitions: &[TopicPartition]);+}++pub(crate) fn for_granularity(+ granularity: CompletionGranularity,+ ledger: Arc<TopicOffsetLedger>,+ commit_interval: Duration,+) -> Arc<dyn Settlement> {+ match granularity {+ CompletionGranularity::Poll => Arc::new(PollSettlement::new(ledger)),+ CompletionGranularity::Group => Arc::new(GroupSettlement::new(ledger, commit_interval)),+ }+}++#[cfg(test)]+pub(crate) fn charge_offsets(+ ledger: &TopicOffsetLedger,+ topic_partition: &TopicPartition,+ generation: u64,+ offsets: &[i64],+) {+ ledger+ .charge(+ topic_partition,+ generation,+ offsets.iter().map(|&o| (Offset(o), Charge::ZERO)),+ )+ .expect("the test ledger accepts the charge");+}diff --git a/rust/ingestion-consumer/src/settlement/poll.rs b/rust/ingestion-consumer/src/settlement/poll.rsnew file mode 100644index 000000000000..7a58b81c332e--- /dev/null+++ b/rust/ingestion-consumer/src/settlement/poll.rs@@ -0,0 +1,234 @@+use std::collections::HashMap;+use std::sync::Arc;+use std::time::Instant;++use common_kafka_consumer::{Offset, TopicOffsetLedger, TopicPartition};+use metrics::counter;+use tracing::warn;++use super::{ChargedSlice, Settlement};+use crate::commit_pacer::CommitPacer;+use crate::ledger_rejection::{report_rejection, RejectedSlice};++pub(crate) struct PollSettlement {+ ledger: Arc<TopicOffsetLedger>,+ pacer: Arc<CommitPacer>,+}++impl PollSettlement {+ pub(crate) fn new(ledger: Arc<TopicOffsetLedger>) -> Self {+ Self {+ ledger,+ pacer: Arc::new(CommitPacer::immediate()),+ }+ }+}++impl Settlement for PollSettlement {+ fn settle_run(+ &self,+ _topic_partition: &TopicPartition,+ _generation: u64,+ _offsets: &[Offset],+ ) -> anyhow::Result<()> {+ Ok(())+ }++ fn settle_poll(&self, slices: &[ChargedSlice<'_>]) -> anyhow::Result<()> {+ if slices.is_empty() {+ // Unreachable: a batch always holds a message. Counted so that "no+ // empty commits" is measured, not assumed.+ counter!("ingestion_consumer_commit_violations_total", "kind" => "empty").increment(1);+ warn!("Poll settled with no partitions");+ return Ok(());+ }++ if let Some(no_frontier) = settle_slices(&self.ledger, &self.pacer, slices)? {+ let reason = no_frontier.reason();+ counter!("ingestion_consumer_commits_skipped_total", "reason" => reason).increment(1);+ warn!(reason, "Poll handed over no frontier; nothing to commit");+ }+ Ok(())+ }++ fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, Offset>> {+ self.pacer.take_due(now)+ }++ fn take_all(&self) -> Option<HashMap<TopicPartition, Offset>> {+ self.pacer.take_all()+ }++ fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {+ self.pacer.forget_partitions(topic_partitions);+ }+}++enum NoFrontier {+ EveryPartitionRejected,+ BehindAnEarlierPoll,+}++impl NoFrontier {+ fn reason(self) -> &'static str {+ match self {+ Self::EveryPartitionRejected => "rejected",+ Self::BehindAnEarlierPoll => "no_frontier",+ }+ }+}++fn settle_slices(+ ledger: &TopicOffsetLedger,+ pacer: &CommitPacer,+ slices: &[ChargedSlice<'_>],+) -> anyhow::Result<Option<NoFrontier>> {+ let mut settled = false;+ let mut handed_over = false;+ for slice in slices {+ let frontier = match ledger.settle(+ slice.topic_partition,+ slice.generation,+ slice.charges.iter().map(|(offset, _)| *offset),+ ) {+ Ok(frontier) => frontier,+ Err(rejection) => {+ report_rejection(+ "settle",+ slice.topic_partition,+ rejection,+ RejectedSlice::settled(&slice.span),+ )?;+ continue;+ }+ };+ settled = true;+ if frontier.is_none() {+ continue;+ }+ if let Some(taken) = ledger.take_frontier(slice.topic_partition) {+ pacer.advance_frontier(slice.topic_partition, taken);+ handed_over = true;+ }+ }+ Ok(match (handed_over, settled) {+ (true, _) => None,+ (false, true) => Some(NoFrontier::BehindAnEarlierPoll),+ (false, false) => Some(NoFrontier::EveryPartitionRejected),+ })+}++#[cfg(test)]+mod tests {+ use common_kafka_consumer::Charge;++ use super::*;+ use crate::order_sentinel::OffsetSpan;++ fn settlement() -> (Arc<TopicOffsetLedger>, PollSettlement) {+ let ledger = Arc::new(TopicOffsetLedger::new());+ let settlement = PollSettlement::new(Arc::clone(&ledger));+ (ledger, settlement)+ }++ struct Charged {+ topic_partition: TopicPartition,+ generation: u64,+ span: OffsetSpan,+ charges: Vec<(Offset, Charge)>,+ }++ impl Charged {+ fn slice(&self) -> ChargedSlice<'_> {+ ChargedSlice {+ topic_partition: &self.topic_partition,+ generation: self.generation,+ span: self.span,+ charges: &self.charges,+ }+ }+ }++ fn charged(ledger: &TopicOffsetLedger, partition: i32, first: i64, last: i64) -> Charged {+ let topic_partition = TopicPartition::new("test", partition);+ let generation = ledger.generation(&topic_partition);+ let charges: Vec<(Offset, Charge)> = (first..=last)+ .map(|offset| (Offset(offset), Charge::ZERO))+ .collect();+ ledger+ .charge(&topic_partition, generation, charges.iter().copied())+ .expect("charge");+ Charged {+ topic_partition,+ generation,+ span: OffsetSpan { first, last },+ charges,+ }+ }++ #[test]+ fn a_retired_poll_makes_each_partitions_frontier_due() {+ let (ledger, settlement) = settlement();+ let p0 = charged(&ledger, 0, 10, 11);+ let p1 = charged(&ledger, 1, 20, 21);++ settlement.settle_poll(&[p0.slice(), p1.slice()]).unwrap();++ assert_eq!(+ settlement.take_due(Instant::now()),+ Some(HashMap::from([+ (p0.topic_partition, Offset(12)),+ (p1.topic_partition, Offset(22)),+ ]))+ );+ }++ #[test]+ fn a_poll_behind_an_earlier_one_becomes_due_when_the_earlier_retires() {+ let (ledger, settlement) = settlement();+ let older = charged(&ledger, 0, 10, 11);+ let newer = charged(&ledger, 0, 12, 13);++ settlement.settle_poll(&[newer.slice()]).unwrap();+ assert!(settlement.take_due(Instant::now()).is_none());++ settlement.settle_poll(&[older.slice()]).unwrap();+ assert_eq!(+ settlement.take_due(Instant::now()),+ Some(HashMap::from([(older.topic_partition, Offset(14))]))+ );+ }++ #[test]+ fn a_stale_poll_makes_nothing_due() {+ let (ledger, settlement) = settlement();+ let poll = charged(&ledger, 0, 10, 11);+ // The partition left and came back while the poll was in flight.+ ledger.forget_partitions([("test", 0)]);++ settlement.settle_poll(&[poll.slice()]).unwrap();++ assert!(settlement.take_due(Instant::now()).is_none());+ }++ #[test]+ fn an_accepted_run_settles_nothing_before_its_poll_retires() {+ let (ledger, settlement) = settlement();+ let poll = charged(&ledger, 0, 10, 11);++ settlement+ .settle_run(+ &poll.topic_partition,+ poll.generation,+ &[Offset(10), Offset(11)],+ )+ .unwrap();+ assert!(settlement.take_due(Instant::now()).is_none());++ settlement.settle_poll(&[poll.slice()]).unwrap();+ assert_eq!(+ settlement.take_due(Instant::now()),+ Some(HashMap::from([(poll.topic_partition, Offset(12))]))+ );+ }+}diff --git a/rust/ingestion-consumer/tests/e2e_integration_test.rs b/rust/ingestion-consumer/tests/e2e_integration_test.rsindex 7d9975c759e5..175eb7f8b311 100644--- a/rust/ingestion-consumer/tests/e2e_integration_test.rs+++ b/rust/ingestion-consumer/tests/e2e_integration_test.rs@@ -33,10 +33,11 @@ use ingestion_consumer::batcher::packer::PackTargets;use ingestion_consumer::batcher::state_machine::RequestBudget;use ingestion_consumer::batcher::worker_pool::WorkerPoolSource;use ingestion_consumer::batcher::BatcherObserver;+use ingestion_consumer::config::CompletionGranularity;use ingestion_consumer::consumer::{IngestionConsumer, IngestionConsumerOptions};use ingestion_consumer::discovery::reconcile_membership;use ingestion_consumer::grpc_transport::{GrpcPort, GrpcTransport};-use ingestion_consumer::order_sentinel::SentinelContext;+use ingestion_consumer::rebalance::RebalanceContext;use ingestion_consumer::routing::RoutingStrategy;use ingestion_consumer::worker_registry::{WorkerId, WorkerRegistry, WorkerRegistryConfig, WorkerState,@@ -596,6 +597,8 @@ impl WorkerIngestService for FakeWorkerGrpc {// ── Test harness ───────────────────────────────────────────────────────────+const COMMIT_INTERVAL: Duration = Duration::from_millis(500);+struct Harness {pub workers: Vec<FakeWorker>,pub registry: Arc<WorkerRegistry>,@@ -614,6 +617,7 @@ struct Harness {deferred_flush_timeout: Duration,pack_targets: PackTargets,budget: Option<RequestBudget>,+ completion_granularity: CompletionGranularity,}/// Build a Kafka consumer subscribed to `topic` in `group_id`, configured like@@ -625,8 +629,8 @@ fn make_kafka_consumer(topic: &str,group_id: &str,instance_id: Option<&str>,-) -> StreamConsumer<SentinelContext> {- make_kafka_consumer_with_context(topic, group_id, instance_id, SentinelContext::detached())+) -> StreamConsumer<RebalanceContext> {+ make_kafka_consumer_with_context(topic, group_id, instance_id, RebalanceContext::new())}/// Like `make_kafka_consumer`, under a context the test keeps a hand on.@@ -634,8 +638,8 @@ fn make_kafka_consumer_with_context(topic: &str,group_id: &str,instance_id: Option<&str>,- context: SentinelContext,-) -> StreamConsumer<SentinelContext> {+ context: RebalanceContext,+) -> StreamConsumer<RebalanceContext> {let mut config = ClientConfig::new();config.set("bootstrap.servers", KAFKA_BROKERS)@@ -649,7 +653,7 @@ fn make_kafka_consumer_with_context(if let Some(id) = instance_id {config.set("group.instance.id", id);}- let kafka_consumer: StreamConsumer<SentinelContext> =+ let kafka_consumer: StreamConsumer<RebalanceContext> =config.create_with_context(context).expect("kafka consumer");kafka_consumer.subscribe(&[topic]).expect("subscribe");kafka_consumer@@ -712,6 +716,8 @@ impl Harness {ComponentOptions::new(),common::PRODUCTION_PACK_TARGETS,None,+ CompletionGranularity::Poll,+ COMMIT_INTERVAL,).await}@@ -742,6 +748,8 @@ impl Harness {soft_budget: Duration::from_millis(500),attempts: BUDGET_ATTEMPTS,}),+ CompletionGranularity::Poll,+ COMMIT_INTERVAL,).await}@@ -767,6 +775,8 @@ impl Harness {ComponentOptions::new(),pack_targets,None,+ CompletionGranularity::Poll,+ COMMIT_INTERVAL,).await}@@ -791,6 +801,8 @@ impl Harness {.with_stall_threshold(stall_threshold),common::ONE_MESSAGE_PER_REQUEST,None,+ CompletionGranularity::Poll,+ COMMIT_INTERVAL,).await}@@ -810,6 +822,51 @@ impl Harness {ComponentOptions::new(),common::PRODUCTION_PACK_TARGETS,None,+ CompletionGranularity::Poll,+ COMMIT_INTERVAL,+ )+ .await+ }++ async fn start_group_granularity(+ pack_targets: PackTargets,+ topic: &str,+ partitions: i32,+ worker_count: usize,+ max_in_flight: usize,+ ) -> Self {+ Self::start_group_granularity_committing_every(+ pack_targets,+ topic,+ partitions,+ worker_count,+ max_in_flight,+ COMMIT_INTERVAL,+ )+ .await+ }++ async fn start_group_granularity_committing_every(+ pack_targets: PackTargets,+ topic: &str,+ partitions: i32,+ worker_count: usize,+ max_in_flight: usize,+ commit_interval: Duration,+ ) -> Self {+ Self::start_inner(+ topic,+ partitions,+ worker_count,+ max_in_flight,+ Duration::from_secs(60),+ fast_registry_config(),+ 0,+ ComponentOptions::new(),+ pack_targets,+ None,+ CompletionGranularity::Group,+ commit_interval,).await}@@ -826,6 +883,8 @@ impl Harness {component_options: ComponentOptions,pack_targets: PackTargets,budget: Option<RequestBudget>,+ completion_granularity: CompletionGranularity,+ commit_interval: Duration,) -> Self {create_topic(topic, partitions).await;@@ -853,9 +912,9 @@ impl Harness {let _monitor = manager.monitor_background();let group_id = format!("e2e-{}", Uuid::new_v4());- let context = SentinelContext::detached();- let ledger = context.topic_offset_ledger();- let kafka_consumer = make_kafka_consumer_with_context(topic, &group_id, None, context);+ let ledger = Arc::new(TopicOffsetLedger::new());+ let kafka_consumer =+ make_kafka_consumer_with_context(topic, &group_id, None, RebalanceContext::new());let (batcher, outputs) = common::batcher(pool_source,@@ -878,6 +937,9 @@ impl Harness {max_in_flight_batches: max_in_flight,group_id: "e2e-test".to_string(),debug_recorder: None,+ completion_granularity,+ commit_interval,+ ledger: Arc::clone(&ledger),},handle,);@@ -909,6 +971,7 @@ impl Harness {deferred_flush_timeout,pack_targets,budget,+ completion_granularity,}}@@ -938,10 +1001,13 @@ impl Harness {let handle = manager.register("consumer", ComponentOptions::new());self.shutdown = handle.shutdown_token();- let context = SentinelContext::detached();- self.ledger = context.topic_offset_ledger();- let kafka_consumer =- make_kafka_consumer_with_context(&self.topic, &self.group_id, None, context);+ self.ledger = Arc::new(TopicOffsetLedger::new());+ let kafka_consumer = make_kafka_consumer_with_context(+ &self.topic,+ &self.group_id,+ None,+ RebalanceContext::new(),+ );let (batcher, outputs) = common::batcher(pool_source,Arc::clone(&transport),@@ -963,6 +1029,9 @@ impl Harness {max_in_flight_batches: self.max_in_flight,group_id: "e2e-test".to_string(),debug_recorder: None,+ completion_granularity: self.completion_granularity,+ commit_interval: COMMIT_INTERVAL,+ ledger: Arc::clone(&self.ledger),},handle,);@@ -993,20 +1062,7 @@ impl Harness {/// The group's committed offset for one partition of the harness topic;/// `None` while nothing is committed.fn committed_offset(&self, partition: i32) -> Option<i64> {- let probe: BaseConsumer = ClientConfig::new()- .set("bootstrap.servers", KAFKA_BROKERS)- .set("group.id", &self.group_id)- .create()- .expect("committed-offset probe");- let mut tpl = TopicPartitionList::new();- tpl.add_partition(&self.topic, partition);- let committed = probe- .committed_offsets(tpl, Duration::from_secs(2))- .expect("fetch committed offsets");- match committed.elements()[0].offset() {- rdkafka::Offset::Offset(offset) => Some(offset),- _ => None,- }+ broker_committed_offset(&self.group_id, &self.topic, partition)}async fn wait_for(&self, total: usize, timeout: Duration) {@@ -1064,6 +1120,9 @@ impl Harness {max_in_flight_batches: self.max_in_flight,group_id: "e2e-test".to_string(),debug_recorder: None,+ completion_granularity: self.completion_granularity,+ commit_interval: COMMIT_INTERVAL,+ ledger: Arc::new(TopicOffsetLedger::new()),},handle,);@@ -1107,6 +1166,23 @@ fn fast_registry_config() -> WorkerRegistryConfig {}/// Poll `cond` until it returns true or the timeout elapses (panics on timeout).+fn broker_committed_offset(group_id: &str, topic: &str, partition: i32) -> Option<i64> {+ let probe: BaseConsumer = ClientConfig::new()+ .set("bootstrap.servers", KAFKA_BROKERS)+ .set("group.id", group_id)+ .create()+ .expect("committed-offset probe");+ let mut tpl = TopicPartitionList::new();+ tpl.add_partition(topic, partition);+ let committed = probe+ .committed_offsets(tpl, Duration::from_secs(2))+ .expect("fetch committed offsets");+ match committed.elements()[0].offset() {+ rdkafka::Offset::Offset(offset) => Some(offset),+ _ => None,+ }+}+async fn wait_until(timeout: Duration, msg: &str, mut cond: impl FnMut() -> bool) {let deadline = tokio::time::Instant::now() + timeout;while !cond() {@@ -2475,21 +2551,8 @@ async fn consumer_crash_before_commit_redelivers_without_loss() {harness.stop().await;}-/// The broker-committed offset is the ledger frontier — one past everything-/// the workers accepted — and a restart resumes from it with no loss and no-/// redelivery.-#[tokio::test]-async fn the_committed_offset_is_the_ledger_frontier() {- let topic = format!("e2e-ledger-commit-{}", Uuid::new_v4());- let mut harness = Harness::start(- &topic,- 1,- 1,- 1,- Duration::from_secs(60),- fast_registry_config(),- )- .await;+async fn assert_commits_track_the_ledger_frontier(mut harness: Harness) {+ let topic = harness.topic.clone();let producer = make_producer();for seq in 0..10usize {@@ -2527,6 +2590,162 @@ async fn the_committed_offset_is_the_ledger_frontier() {harness.stop().await;}+#[tokio::test]+async fn the_committed_offset_is_the_ledger_frontier() {+ let topic = format!("e2e-ledger-commit-{}", Uuid::new_v4());+ assert_commits_track_the_ledger_frontier(+ Harness::start(+ &topic,+ 1,+ 1,+ 1,+ Duration::from_secs(60),+ fast_registry_config(),+ )+ .await,+ )+ .await;+}++#[tokio::test]+async fn group_granularity_commits_the_ledger_frontier() {+ let topic = format!("e2e-group-commit-{}", Uuid::new_v4());+ assert_commits_track_the_ledger_frontier(+ Harness::start_group_granularity(common::PRODUCTION_PACK_TARGETS, &topic, 1, 1, 1).await,+ )+ .await;+}++#[tokio::test]+async fn group_granularity_commits_the_pending_frontier_on_stop() {+ let topic = format!("e2e-group-stop-commit-{}", Uuid::new_v4());+ let harness = Harness::start_group_granularity_committing_every(+ common::PRODUCTION_PACK_TARGETS,+ &topic,+ 1,+ 1,+ 1,+ Duration::from_secs(3600),+ )+ .await;+ let p0 = TopicPartition::new(&topic, 0);+ let producer = make_producer();++ produce(&producer, &topic, 0, "tok", "user-a", 0).await;+ wait_until(+ Duration::from_secs(10),+ "the first frontier to commit",+ || harness.committed_offset(0) == Some(1),+ )+ .await;++ for seq in 1..10usize {+ produce(&producer, &topic, 0, "tok", "user-a", seq).await;+ }+ wait_until(Duration::from_secs(10), "every message to settle", || {+ harness.workers[0].count() == 10 && harness.ledger.held(&p0).offsets == 0+ })+ .await;+ assert_eq!(+ harness.committed_offset(0),+ Some(1),+ "the interval holds the later frontier"+ );++ let group_id = harness.group_id.clone();+ harness.stop().await;++ wait_until(Duration::from_secs(10), "the stop to commit", || {+ broker_committed_offset(&group_id, &topic, 0) == Some(10)+ })+ .await;+}++const STALL_KEYS: [(i32, &str); 2] = [(0, "user-a"), (1, "user-b")];++async fn stall_two_partitions_on_two_workers(+ harness: &Harness,+ topic: &str,+) -> Vec<Option<tokio::sync::OwnedMutexGuard<()>>> {+ let mut guards = Vec::new();+ for worker in &harness.workers {+ guards.push(Some(worker.block().await));+ }++ let producer = make_producer();+ let mut sends = Vec::new();+ for seq in 0..5usize {+ for (partition, key) in STALL_KEYS {+ sends.push(produce(&producer, topic, partition, "tok", key, seq));+ }+ }+ futures::future::join_all(sends).await;++ wait_until(+ Duration::from_secs(10),+ "both workers to hold a sub-batch",+ || harness.workers.iter().all(|w| w.arrived_count() > 0),+ )+ .await;++ guards+}++fn accepted_partition(harness: &Harness, idx: usize) -> i32 {+ let worker = &harness.workers[idx];+ assert_eq!(+ worker.count(),+ 5,+ "each worker must hold exactly one key's messages"+ );+ STALL_KEYS+ .iter()+ .find(|(_, key)| !worker.seqs_for(key).is_empty())+ .map(|(partition, _)| *partition)+ .expect("the released worker accepted neither key")+}++#[tokio::test]+async fn group_granularity_commits_a_partition_while_another_stalls() {+ let topic = format!("e2e-group-stall-{}", Uuid::new_v4());+ // One message per request, or the packer puts both keys on one worker.+ let harness =+ Harness::start_group_granularity(common::ONE_MESSAGE_PER_REQUEST, &topic, 2, 2, 2).await;+ let mut guards = stall_two_partitions_on_two_workers(&harness, &topic).await;++ guards[0] = None;+ wait_until(+ Duration::from_secs(10),+ "the released worker to accept its key",+ || harness.workers[0].count() == 5,+ )+ .await;+ let released = accepted_partition(&harness, 0);+ let stalled = 1 - released;++ wait_until(+ Duration::from_secs(10),+ "the released partition to commit",+ || harness.committed_offset(released) == Some(5),+ )+ .await;+ assert_eq!(+ harness.committed_offset(stalled),+ None,+ "the stalled send must hold its own partition back"+ );++ guards[1] = None;+ wait_until(+ Duration::from_secs(10),+ "the stalled partition to commit once released",+ || harness.committed_offset(stalled) == Some(5),+ )+ .await;++ harness.stop().await;+}+/// A worker that processes a batch but whose ACK is lost is indistinguishable/// from a failed send, so the consumer replays the batch: duplicates are the/// accepted cost of at-least-once, loss never is, and the replay lands behind@@ -3236,6 +3455,9 @@ async fn fenced_static_member_exits_on_fatal_error() {max_in_flight_batches: 1,group_id: "e2e-test".to_string(),debug_recorder: None,+ completion_granularity: CompletionGranularity::Poll,+ commit_interval: Duration::ZERO,+ ledger: Arc::new(TopicOffsetLedger::new()),},handle,);diff --git a/rust/ingestion-control-plane/src/ui/consumer_debug.html b/rust/ingestion-control-plane/src/ui/consumer_debug.htmlindex 7b76e0821b5a..74ede6f515e7 100644--- a/rust/ingestion-control-plane/src/ui/consumer_debug.html+++ b/rust/ingestion-control-plane/src/ui/consumer_debug.html@@ -53,7 +53,7 @@td.mono, th.mono { font-family: var(--mono); }td .bar { height: 6px; border-radius: 3px; background: var(--accent); min-width: 2px; }.pill { display: inline-block; padding: 1px 7px; border-radius: 10px; font-size: 11px; font-weight: 600; white-space: nowrap; }- .pill.healthy, .pill.committed { background: rgba(63,185,80,0.16); color: var(--good); }+ .pill.healthy, .pill.committed, .pill.completed { background: rgba(63,185,80,0.16); color: var(--good); }.pill.degraded, .pill.retrying, .pill.partial { background: rgba(210,153,34,0.16); color: var(--warn); }.pill.unhealthy, .pill.stalled, .pill.failed { background: rgba(248,81,73,0.16); color: var(--error); }.pill.draining, .pill.inflight { background: rgba(139,147,161,0.18); color: var(--muted); }@@ -128,7 +128,7 @@.tl .item.error::before { background: var(--error); }.tl .item .t { color: var(--muted); font-family: var(--mono); font-size: 11px; margin-right: 8px; }- /* commits */+ /* commit submissions */.banner { margin: 12px 14px 0; padding: 8px 12px; border-radius: 6px; font-size: 12px; }.banner.ok { background: rgba(63,185,80,0.12); color: var(--good); }.banner.bad { background: rgba(248,81,73,0.14); color: var(--error); font-weight: 600; }@@ -175,7 +175,7 @@<div class="tabs"><div class="tab active" data-tab="feed">Live feed</div><div class="tab" data-tab="polls">Polls <span class="badge" id="badge-polls"></span></div>- <div class="tab" data-tab="commits">Commits <span class="badge" id="badge-commits"></span></div>+ <div class="tab" data-tab="commits">Commit submissions <span class="badge" id="badge-commits"></span></div></div><div id="tab-feed">@@ -200,8 +200,9 @@</div><div id="tab-commits" style="display:none;">+ <div class="empty">Async commit submissions accepted by the Kafka client, not broker confirmations. Offsets are Kafka next-to-read positions and may span multiple polls.</div><div id="commit-banner"></div>- <div id="commits"><div class="empty">Waiting for commits…</div></div>+ <div id="commits"><div class="empty">Waiting for commit submissions…</div></div></div></section></div>@@ -225,7 +226,7 @@let requests = {};let requestOrder = [];let commits = [];- let partitionHigh = {}; // "topic/partition" -> highest committed offset+ let partitionHigh = {}; // "topic/partition" -> highest submitted next-to-read offsetlet oooCount = 0;let renderPending = false;@@ -362,7 +363,7 @@let p = polls[id];if (!p) {p = { id, ts: null, messages: 0, partitions: [], assignment_epoch: null, spans: [], requests: [],- committed: null, failed: null, events: [] };+ completed: null, failed: null, events: [] };polls[id] = p;pollOrder.push(id);while (pollOrder.length > MAX_POLLS) delete polls[pollOrder.shift()];@@ -444,14 +445,16 @@p.events.push(ev);break;}- case "BatchCommitted": {+ case "PollCompleted": {const p = pollFor(ev.batch_id);if (p.ts === null) p.ts = ev.ts_ms;- p.committed = { accepted: ev.accepted, duration_ms: ev.duration_ms, partitions: ev.partitions, ts: ev.ts_ms };+ p.completed = { accepted: ev.accepted, duration_ms: ev.duration_ms, partitions: ev.partitions, ts: ev.ts_ms };p.events.push(ev);- recordCommit(ev);break;}+ case "CommitSubmitted":+ recordCommit(ev);+ break;case "BatchFailed": {if (!ev.batch_id) break;const p = pollFor(ev.batch_id);@@ -486,16 +489,16 @@}function recordCommit(ev) {- const rows = ev.partitions.map((p) => {+ const rows = ev.offsets.map((p) => {const key = p.topic + "/" + p.partition;const prev = partitionHigh[key];- const outOfOrder = prev !== undefined && p.offset <= prev;+ const outOfOrder = prev !== undefined && p.offset < prev;if (prev === undefined || p.offset > prev) partitionHigh[key] = p.offset;return { topic: p.topic, partition: p.partition, offset: p.offset, prev, outOfOrder };});const bad = rows.some((r) => r.outOfOrder);if (bad) oooCount++;- commits.unshift({ ts: ev.ts_ms, batch_id: ev.batch_id, accepted: ev.accepted, duration_ms: ev.duration_ms, rows, outOfOrder: bad });+ commits.unshift({ ts: ev.ts_ms, rows, outOfOrder: bad });while (commits.length > MAX_COMMITS) commits.pop();}@@ -503,7 +506,7 @@const isPartial = (r) => r.resolved && r.resolved.unfinished > 0;function pollStatus(p) {- if (p.committed) return "committed";+ if (p.completed) return "completed";if (p.failed) return "failed";const rs = pollRequests(p);if (rs.some((r) => r.failed)) return "retrying";@@ -512,7 +515,7 @@}// A poll whose requests failed, came back partial, or were replays: the- // interesting ones, even if they ultimately committed cleanly.+ // interesting ones, even if they ultimately completed cleanly.function hadIssue(p) {return p.failed || pollRequests(p).some((r) => r.failed || r.replay || isPartial(r));}@@ -532,7 +535,8 @@? ["warn", `◐ request <code>#${ev.request_id}</code> partial — ${shortWorker(ev.worker)} acked ${ev.accepted}, ${ev.unfinished} unfinished`]: ["", `✓ request <code>#${ev.request_id}</code> acked — ${shortWorker(ev.worker)}, ${ev.accepted} msgs`];case "RequestFailed": return [ev.cause === "busy" ? "warn" : "error", `✗ request <code>#${ev.request_id}</code> failed [${ev.cause}] — ${shortWorker(ev.worker)}, ${ev.messages} msgs requeued`];- case "BatchCommitted": return ["good", `✓ committed poll <code>${shortId(ev.batch_id)}</code> — ${ev.accepted} msgs in ${ev.duration_ms}ms`];+ case "PollCompleted": return ["good", `✓ completed poll <code>${shortId(ev.batch_id)}</code> — ${ev.accepted} msgs in ${ev.duration_ms}ms (not a Kafka commit)`];+ case "CommitSubmitted": return ["", `async commit submitted — next-to-read offsets ${ev.offsets.map((p) => `${escapeXml(p.topic)}/${p.partition}@${p.offset}`).join(" ")} (not broker-confirmed)`];case "BatchFailed": return ["error", `✗ poll failed — ${ev.error}`];case "CommitFailed": return ["error", `✗ commit failed — ${ev.error}`];case "WorkerStateChanged": return [ev.to === "unhealthy" ? "error" : ev.to === "healthy" ? "good" : "warn", `worker ${shortWorker(ev.worker)} ${ev.from} → ${ev.to}`];@@ -686,7 +690,7 @@<span><i style="background:#4f7cff"></i>sent</span><span><i style="background:#d29922"></i>replay / partial</span><span><i style="background:#f85149"></i>failed</span>- <span><i style="background:#3fb950"></i>acked / committed</span>+ <span><i style="background:#3fb950"></i>acked / completed</span></div>${flow.html}<h2 style="border:none;padding:0 0 8px;">Requests</h2>@@ -700,7 +704,7 @@}// Build the node/edge model for a poll's flow graph. Nodes sit in four- // lanes (Kafka → batcher → workers → commit).+ // lanes (Kafka → batcher → workers → completion).function buildFlow(p, workers) {const partsShort = p.partitions.map((pp) => `${pp.partition}@${pp.offset}`).join(" ") || "—";const sourceNode = `<div class="node src" id="n-source"><div class="title">Kafka poll</div><div class="big">${p.messages}</div><div class="sub">msgs · p ${partsShort}</div></div>`;@@ -719,11 +723,11 @@return `<div class="node worker ${cls}" id="n-w-${i}"><div class="title">${shortWorker(w.worker)}</div><div class="big">${w.messages}</div><div class="sub">${w.requests} req · ${badges.join(" ") || "in flight"}</div></div>`;}).join("") || '<div class="node" id="n-w-none"><div class="sub">no request yet</div></div>';- const sinkNode = p.committed- ? `<div class="node sink-ok" id="n-sink"><div class="title">Committed</div><div class="big">${p.committed.accepted}</div><div class="sub">${p.committed.duration_ms}ms · ${p.committed.partitions.map((pp) => `${pp.partition}@${pp.offset}`).join(" ")}</div></div>`+ const sinkNode = p.completed+ ? `<div class="node sink-ok" id="n-sink"><div class="title">Poll completed</div><div class="big">${p.completed.accepted} msgs</div><div class="sub">${p.completed.duration_ms}ms · poll max offsets ${p.completed.partitions.map((pp) => `${pp.partition}@${pp.offset}`).join(" ")}</div><div class="sub">Not a Kafka commit</div></div>`: p.failed? `<div class="node worker stalled" id="n-sink"><div class="title">Failed</div><div class="sub">${escapeXml(p.failed.error)}</div></div>`- : `<div class="node sink-pending" id="n-sink"><div class="title">In-flight</div><div class="sub">not committed</div></div>`;+ : `<div class="node sink-pending" id="n-sink"><div class="title">In-flight</div><div class="sub">not completed</div></div>`;const html = `<div class="flow" id="flow"><div class="flow-main">@@ -743,7 +747,7 @@edges.push({ from: "n-batcher", to: `n-w-${i}`, label: `${w.requests} req · ${w.messages} msgs`, color });});- if (p.committed) {+ if (p.completed) {if (workers.length) workers.forEach((w, i) => { if (!w.failed) edges.push({ from: `n-w-${i}`, to: "n-sink", label: "acked", color: C.good }); });else edges.push({ from: "n-batcher", to: "n-sink", label: "", color: C.good });}@@ -807,31 +811,29 @@function renderCommits() {const banner = $("commit-banner");banner.innerHTML = oooCount- ? `<div class="banner bad">⚠ ${oooCount} out-of-order commit(s) detected — offsets went backwards for a partition</div>`- : commits.length ? `<div class="banner ok">✓ all ${commits.length} commits are in order (offsets monotonic per partition)</div>` : "";+ ? `<div class="banner bad">⚠ ${oooCount} out-of-order submission(s) detected — submitted offsets went backwards for a partition</div>`+ : commits.length ? `<div class="banner ok">✓ all ${commits.length} retained submissions are in order (submitted offsets nondecreasing per partition)</div>` : "";const el = $("commits");- if (!commits.length) { el.innerHTML = '<div class="empty">Waiting for commits…</div>'; return; }+ if (!commits.length) { el.innerHTML = '<div class="empty">Waiting for commit submissions…</div>'; return; }const rows = commits.map((c) => {const chips = c.rows.slice().sort((a, b) => a.partition - b.partition).map((r) =>- `<span class="offchip ${r.outOfOrder ? "bad" : ""}" title="${r.topic}">- <span class="pl">P${r.partition}</span>+ `<span class="offchip ${r.outOfOrder ? "bad" : ""}">+ <span class="pl">${escapeXml(r.topic)}/P${r.partition}</span><span class="ov">${r.offset}</span>${r.outOfOrder ? `<span class="was">↓ was ${r.prev}</span>` : ""}</span>`).join("");return `<tr class="${c.outOfOrder ? "ooo" : ""}"><td class="mono">${fmtTime(c.ts)}</td>- <td class="mono">${shortId(c.batch_id)}</td>- <td class="mono">${c.accepted}</td><td><div class="offchips">${chips}</div></td><td>${c.outOfOrder ? '<span class="pill failed">out of order</span>' : ""}</td></tr>`;}).join("");- setHTML(el, `<table><thead><tr><th>Time</th><th>Batch</th><th>Msgs</th><th>Committed offset per partition</th><th></th></tr></thead><tbody>${rows}</tbody></table>`, "prepend");+ setHTML(el, `<table><thead><tr><th>Submitted at</th><th>Submitted next-to-read offset per partition</th><th></th></tr></thead><tbody>${rows}</tbody></table>`, "prepend");}// ---------- tabs ----------