:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(ingestion-consumer): pause a partition at its budget and resume it on commit (#118697)
Paweł Ledwoń1ad5c63c0aa3
diff --git a/rust/ingestion-consumer/README.md b/rust/ingestion-consumer/README.mdindex 7c91dafdd58e..2b989da80774 100644--- a/rust/ingestion-consumer/README.md+++ b/rust/ingestion-consumer/README.md@@ -50,6 +50,18 @@ The ledger emits its own counters, so any consumer built on the crate reports th`kafka_consumer_ledger_stale_slices_total{stage}` counts charges and settlements dropped because their partition was reassigned while they were in flight; a few around a rebalance are expected.`kafka_consumer_ledger_errors_total{stage,kind}` counts contract violations in the ledger's accounting; it must stay 0. A violation stops the consumer, and the restart replays from the last commit: the ledger resets the partition's window, so offsets in flight below its next delivery would never complete and a later commit would move past them. The consumer logs the rejected slice with the batch and ledger generations and the window depth before it stops.+## Partition budget++`CONSUMER_PARTITION_BUDGET_EVENTS` and `CONSUMER_PARTITION_BUDGET_BYTES` bound each partition's replay exposure: the events and bytes the settlement reports as uncommitted for it.+A budget needs `CONSUMER_COMPLETION_GRANULARITY=group`. At `poll` a stalled partition holds back the commits of every partition in its poll, so their budgets would fill and pause together. The consumer ignores a budget at `poll` and logs a warning, so switching a lane back to `poll` rolls it back without unsetting the budget.+When either axis reaches its limit after a poll, the consumer pauses that partition's fetch through the Kafka client; when a commit brings both axes back under `CONSUMER_PARTITION_BUDGET_LOW_WATERMARK` (a fraction of the budget, default 0.8), it resumes.+The hysteresis matters because a pause purges the partition's fetched messages from librdkafka's queue and a resume refetches them.+A pause stops only the partition's fetch: the consumer keeps polling, so rebalance callbacks still run and `max.poll.interval.ms` stays satisfied, except while the loop waits on its oldest in-flight poll.+A revoked partition is forgotten. librdkafka keeps a partition's pause across a revoke and a reassignment, so the consumer resumes every partition it is assigned, and the partition starts with a fresh ledger.+Both limits default to `0`, which sets no budget; the admission cap (`CONSUMER_MAX_BACKGROUND_TASKS`) then governs alone.+The budget is a fixed amount per partition, not a share of a pod-wide amount, so a rebalance changes how many partitions a pod holds, not how much each may expose.+`ingestion_consumer_budget_paused_partitions` gauges how many partitions are paused and `ingestion_consumer_budget_pauses_total{topic,partition}` counts pauses; read them against `ingestion_lag_ms`.+## BatcherThe batcher state machine in `src/batcher/` orders runs and places them on workers.diff --git a/rust/ingestion-consumer/docs/driver-model-plan.md b/rust/ingestion-consumer/docs/driver-model-plan.mdindex a701273d8540..826d098c3ccc 100644--- a/rust/ingestion-consumer/docs/driver-model-plan.md+++ b/rust/ingestion-consumer/docs/driver-model-plan.md@@ -101,7 +101,7 @@ A change that adds metrics lists their names.| 22 | 8 decoupled commits | Cleanup | Delete the per-poll completion path || 23 | 9 bounded replay | Implement | Add the budget accounting || 24 | 9 bounded replay | Switchover | Enable the budget |-| 25 | 9 bounded replay | Cleanup | Remove the in-flight admission cap |+| 25 | 9 bounded replay | Structure | Poll continuously and remove the in-flight admission cap || 26 | 10 clean handoff | Prepare | Add revoke and drained markers || 27 | 10 clean handoff | Implement | Build the drain || 28 | 10 clean handoff | Switchover | Enable the drain |@@ -601,7 +601,8 @@ Exit criterion at the canary: zero ledger errors and gap walks, and a stalled pa**Interfaces:**-- Remove the ordered `InFlightBatch` queue, the per-poll aggregation from change 7, and the completion granularity from `Config`.+- Remove `PollSettlement` and the completion granularity from `Config`.+- Keep the in-flight poll queue and its per-poll aggregation from change 7. The loop waits on the oldest poll until change 25 replaces it.## Cycle 9: replay exposure is bounded per partition@@ -641,7 +642,7 @@ The budget is per partition, and so is the pause. One budget for the whole assig- A charts values PR sets the budget lane by lane. The admission cap and the budget both bound work during the transition.- Derive the first values from the current config: the cap times the batch size, divided by the partitions a pod holds, for events; the byte bound likewise for bytes. The per-pod bound then does not change at the switchover.- Pause a partition when its outstanding charge reaches the budget on either axis. Resume it only when its charge is back at or under the low watermark on both axes. A pause purges that partition's fetched messages from librdkafka's queue and a resume refetches them, so a gate that reopens on the first refund refetches on every crossing.-- Keep polling while partitions are paused. A paused partition still serves rebalance callbacks and keeps `max.poll.interval.ms` alive; an unpolled consumer does neither. An assignment resets librdkafka's pause flags, so a partition that arrives starts unpaused, with a fresh ledger generation and zero charge.+- Keep polling while some partitions are paused. A paused partition still serves rebalance callbacks and keeps `max.poll.interval.ms` alive; an unpolled consumer does neither. While every partition is paused, the loop waits on completions without polling Kafka, as it already does when the cap is full; change 25 removes that wait. librdkafka keeps a partition's pause across a revoke and a reassignment, so the consumer resumes every partition it is assigned: a partition that arrives starts unpaused, with a fresh ledger generation and zero charge.- Watch the paused-partitions gauge and `ingestion_lag_ms`. Rollback is unsetting the values.- The budget is a fixed amount per partition, not a share of a pod-wide amount. A rebalance changes how many partitions a pod holds, not how much each one may expose, so the value does not move at rebalances.@@ -655,17 +656,22 @@ The budget is per partition, and so is the pause. One budget for the whole assig- Add `ingestion_consumer_budget_paused_partitions` (gauge) and `ingestion_consumer_budget_pauses_total{topic,partition}` (counter).-### 25. Remove the in-flight admission cap (cleanup)+### 25. Poll continuously and remove the in-flight admission cap (structure)-**Task:** Remove `max_in_flight_batches` after the budget is stable on all lanes.+**Task:** After the budget is stable on all lanes, replace the poll-and-wait loop with one loop that polls Kafka continuously and applies completions as they arrive. Remove `max_in_flight_batches`.-**Goal:** The budget is the only admission bound.+**Goal:** The budget is the only admission bound. No wait on one poll holds back fetching, completions, or commits.+- The loop and the cap change together. The loop applies completions only while it waits for the oldest poll, and under steady traffic a full cap is what makes it wait. Without the cap and with the old loop, completions stay unread until every partition pauses at its budget.+- One `select!` serves the Kafka poll, the completions, the batcher's errors, the commit tick, and shutdown. Collection must be safe to cancel: the loop keeps the poll it is building in its own state and adds each message when `recv()` returns it, so a branch that wins the race drops no message.+- The commit tick fires when the pacer is next due, so a refund and the resume it allows go out on time.+- Kafka is polled while every partition is paused, so rebalance callbacks and `max.poll.interval.ms` no longer depend on completions.- Only admission control goes. The collection bounds — batch size and batch timeout — stay: they shape one poll, not the amount of admitted work.-- The in-flight poll count goes with the cap. After change 22 the polls exist only to be counted.+- Nothing tracks a poll after it is submitted. The budget bounds the admitted work, so the loop does not count polls or wait for one to finish.**Interfaces:**+- Modify `IngestionConsumer::run`: one select loop over collection, completions, errors, the commit tick, and shutdown. Remove `complete_oldest_poll`, the in-flight poll queue, and the per-poll aggregation.- Remove `max_in_flight_batches` from `Config` and the admission bookkeeping from `IngestionConsumer`.- Remove `ingestion_consumer_in_flight_batches`. The budget gauges replace it.diff --git a/rust/ingestion-consumer/src/commit_pacer.rs b/rust/ingestion-consumer/src/commit_pacer.rsindex 4a46756a79cc..8712e075ce32 100644--- a/rust/ingestion-consumer/src/commit_pacer.rs+++ b/rust/ingestion-consumer/src/commit_pacer.rs@@ -190,6 +190,16 @@ mod tests {);}+ #[test]+ fn an_empty_take_does_not_start_the_interval() {+ let mut pacer = CommitPacer::every(Duration::from_millis(500));+ let start = Instant::now();+ assert!(pacer.take_due(start).is_none());++ pacer.advance_frontier(&tp(0), taken(0, 10));+ assert!(pacer.take_due(start + Duration::from_millis(100)).is_some());+ }+#[test]fn take_all_ignores_the_interval() {let mut pacer = CommitPacer::every(Duration::from_secs(3600));diff --git a/rust/ingestion-consumer/src/config.rs b/rust/ingestion-consumer/src/config.rsindex 6c3dd2b39b67..5966cffd4ab0 100644--- a/rust/ingestion-consumer/src/config.rs+++ b/rust/ingestion-consumer/src/config.rs@@ -1,4 +1,4 @@-use std::num::{NonZeroU32, NonZeroUsize};+use std::num::{NonZeroU32, NonZeroU64, NonZeroUsize};use std::str::FromStr;use std::time::{Duration, Instant};@@ -13,6 +13,7 @@ use crate::batcher::retry_policy::RetryPolicy;use crate::batcher::state_machine::{BatcherStateMachine, RequestBudget};use crate::batcher::worker_assigner::WorkerAssigner;use crate::discovery::DiscoveryMode;+use crate::partition_budget::BudgetLimits;use crate::routing::{Router, RoutingStrategy};use common_kafka_consumer::config::ConsumerConfigBuilder;@@ -255,6 +256,22 @@ pub struct Config {#[envconfig(from = "CONSUMER_COMMIT_INTERVAL_MS", default = "500")]pub consumer_commit_interval_ms: u64,+ /// Uncommitted events at which a partition's fetch pauses. `0` (default)+ /// sets no event limit. Not read at `poll` granularity.+ #[envconfig(from = "CONSUMER_PARTITION_BUDGET_EVENTS", default = "0")]+ pub consumer_partition_budget_events: u64,++ /// Uncommitted key-plus-value-plus-header bytes at which a partition's+ /// fetch pauses. `0` (default) sets no byte limit. Not read at `poll`+ /// granularity.+ #[envconfig(from = "CONSUMER_PARTITION_BUDGET_BYTES", default = "0")]+ pub consumer_partition_budget_bytes: u64,++ /// Fraction of the budget a paused partition must be back under, on+ /// every set axis, before its fetch resumes. Above 0 and at most 1.+ #[envconfig(from = "CONSUMER_PARTITION_BUDGET_LOW_WATERMARK", default = "0.8")]+ pub consumer_partition_budget_low_watermark: f64,+// ---- Debug API ----/// Serve the real-time debug API (`/debug/load`, `/debug/state`,/// `/debug/events` SSE) on the health server, recording structured events@@ -503,6 +520,14 @@ impl Config {.with_request_budget(self.request_budget()?))}+ pub fn partition_budget(&self) -> Result<BudgetLimits, String> {+ BudgetLimits::new(+ NonZeroU64::new(self.consumer_partition_budget_events),+ NonZeroU64::new(self.consumer_partition_budget_bytes),+ self.consumer_partition_budget_low_watermark,+ )+ }+const MIN_REQUEST_SOFT_BUDGET_MS: u64 = 100;pub fn request_budget(&self) -> Result<Option<RequestBudget>, String> {@@ -698,6 +723,23 @@ mod tests {assert!(config(vars).request_budget().is_err());}+ #[rstest]+ #[case::unset(&[], false)]+ #[case::events_only(&[("CONSUMER_PARTITION_BUDGET_EVENTS", "5000")], true)]+ #[case::bytes_only(&[("CONSUMER_PARTITION_BUDGET_BYTES", "1048576")], true)]+ fn a_valid_partition_budget_is_read(#[case] vars: &[(&str, &str)], #[case] is_set: bool) {+ assert_eq!(config(vars).partition_budget().unwrap().is_set(), is_set);+ }++ #[test]+ fn a_partition_budget_with_an_invalid_watermark_is_rejected() {+ let vars = &[+ ("CONSUMER_PARTITION_BUDGET_EVENTS", "5000"),+ ("CONSUMER_PARTITION_BUDGET_LOW_WATERMARK", "1.2"),+ ];+ assert!(config(vars).partition_budget().is_err());+ }+#[test]fn completion_granularity_parses_known_values() {assert_eq!(diff --git a/rust/ingestion-consumer/src/consumer.rs b/rust/ingestion-consumer/src/consumer.rsindex e7cb1f5ac850..07b77244e19c 100644--- a/rust/ingestion-consumer/src/consumer.rs+++ b/rust/ingestion-consumer/src/consumer.rs@@ -1,4 +1,5 @@use std::collections::{HashMap, HashSet, VecDeque};+use std::sync::atomic::{AtomicBool, Ordering};use std::sync::{Arc, Mutex};use std::time::{Duration, Instant};@@ -23,8 +24,9 @@ use crate::discovery::DiscoveryMode;use crate::grpc_transport::GrpcTransport;use crate::ledger_rejection::{report_rejection, RejectedSlice};use crate::order_sentinel::OffsetSpan;-use crate::rebalance::{PartitionHook, RebalanceContext};-use crate::settlement::{self, ChargedSlice, Settlement};+use crate::partition_budget::BudgetLimits;+use crate::rebalance::{partition_error, PartitionHook, RebalanceContext};+use crate::settlement::{self, ChargedSlice, Commit, Pause, Settlement};use crate::types::{Accumulator, SerializedKafkaMessage};/// Batch-wide statistics gathered while collecting, used to emit parity@@ -229,6 +231,67 @@ struct RevokedPartition {generation: u64,}+#[derive(Default)]+struct Partitions {+ fetching: HashSet<TopicPartition>,+ paused: HashSet<TopicPartition>,+}++#[derive(Default)]+struct FetchingPartitions {+ partitions: Mutex<Partitions>,+ all_paused: AtomicBool,+}++impl FetchingPartitions {+ fn assigned(&self, topic_partitions: &[TopicPartition]) {+ self.update(|partitions| {+ for topic_partition in topic_partitions {+ partitions.paused.remove(topic_partition);+ partitions.fetching.insert(topic_partition.clone());+ }+ });+ }++ fn revoked(&self, topic_partitions: &[TopicPartition]) {+ self.update(|partitions| {+ for topic_partition in topic_partitions {+ partitions.paused.remove(topic_partition);+ partitions.fetching.remove(topic_partition);+ }+ });+ }++ fn paused(&self, topic_partition: &TopicPartition) {+ self.update(|partitions| {+ if partitions.fetching.remove(topic_partition) {+ partitions.paused.insert(topic_partition.clone());+ }+ });+ }++ fn resumed(&self, topic_partition: &TopicPartition) {+ self.update(|partitions| {+ if partitions.paused.remove(topic_partition) {+ partitions.fetching.insert(topic_partition.clone());+ }+ });+ }++ fn all_paused(&self) -> bool {+ self.all_paused.load(Ordering::Relaxed)+ }++ fn update(&self, change: impl FnOnce(&mut Partitions)) {+ let mut partitions = self.partitions.lock().unwrap();+ change(&mut partitions);+ self.all_paused.store(+ !partitions.paused.is_empty() && partitions.fetching.is_empty(),+ Ordering::Relaxed,+ );+ }+}+/// The consumer's revocation hook: purge the batcher, and record the revoked/// partitions, because the purge means their polls never complete.fn revoke_hook(@@ -236,9 +299,11 @@ fn revoke_hook(ledger: Arc<TopicOffsetLedger>,settlement: Arc<dyn Settlement>,revoked_partitions: Arc<Mutex<Vec<RevokedPartition>>>,+ fetching: Arc<FetchingPartitions>,) -> PartitionHook {Box::new(move |partitions| {settlement.forget_partitions(partitions);+ fetching.revoked(partitions);let pairs: Vec<(String, i32)> = partitions.iter().map(|tp| (tp.topic.clone(), tp.partition))@@ -256,11 +321,13 @@ fn revoke_hook(fn assign_hook(settlement: Arc<dyn Settlement>,+ fetching: Arc<FetchingPartitions>,assignment_epoch: AssignmentEpoch,) -> PartitionHook {Box::new(move |partitions| {// Also covers a loss with no revoke callback (an error rebalance, a fenced member).settlement.forget_partitions(partitions);+ fetching.assigned(partitions);assignment_epoch.bump();})}@@ -271,16 +338,22 @@ fn wire_rebalance_hooks(ledger: &Arc<TopicOffsetLedger>,settlement: &Arc<dyn Settlement>,assignment_epoch: AssignmentEpoch,-) -> Arc<Mutex<Vec<RevokedPartition>>> {+) -> (Arc<Mutex<Vec<RevokedPartition>>>, Arc<FetchingPartitions>) {let revoked_partitions = Arc::new(Mutex::new(Vec::new()));+ let fetching = Arc::new(FetchingPartitions::default());context.set_on_revoke(revoke_hook(revoker,Arc::clone(ledger),Arc::clone(settlement),Arc::clone(&revoked_partitions),+ Arc::clone(&fetching),));- context.set_on_assign(assign_hook(Arc::clone(settlement), assignment_epoch));- revoked_partitions+ context.set_on_assign(assign_hook(+ Arc::clone(settlement),+ Arc::clone(&fetching),+ assignment_epoch,+ ));+ (revoked_partitions, fetching)}/// Remove revoked partitions from in-flight polls, and remove polls left@@ -412,6 +485,18 @@ fn pop_committable(in_flight: &mut VecDeque<InFlightPoll>) -> anyhow::Result<InFOk(poll)}+fn commit_tick(granularity: CompletionGranularity, commit_interval: Duration) -> Duration {+ const HEALTH_TICK: Duration = Duration::from_secs(1);+ // A commit can resume a partition, so a tick longer than the commit+ // interval delays the resume.+ match granularity {+ CompletionGranularity::Group if !commit_interval.is_zero() => {+ commit_interval.min(HEALTH_TICK)+ }+ _ => HEALTH_TICK,+ }+}+/// Options for constructing an [`IngestionConsumer`] from pre-built parts./// Used in integration tests where the Kafka consumer is created externally.pub struct IngestionConsumerOptions {@@ -427,6 +512,7 @@ pub struct IngestionConsumerOptions {pub completion_granularity: CompletionGranularity,pub commit_interval: Duration,pub ledger: Arc<TopicOffsetLedger>,+ pub partition_budget: BudgetLimits,}/// The main consumer loop: reads from Kafka, demuxes each poll into groups,@@ -443,6 +529,7 @@ pub struct IngestionConsumer {batch_size_bytes: usize,batch_timeout: Duration,max_in_flight_batches: usize,+ commit_tick: Duration,handle: Handle,group_id: String,/// Debug event recorder; `None` unless `DEBUG_API_ENABLED`.@@ -451,6 +538,7 @@ pub struct IngestionConsumer {/// Partitions revoked since the loop last looked, fed by the rebalance/// callback.revoked_partitions: Arc<Mutex<Vec<RevokedPartition>>>,+ fetching: Arc<FetchingPartitions>,settlement: Arc<dyn Settlement>,}@@ -471,8 +559,9 @@ impl IngestionConsumer {options.completion_granularity,Arc::clone(&topic_offset_ledger),options.commit_interval,+ options.partition_budget,);- let revoked_partitions = wire_rebalance_hooks(+ let (revoked_partitions, fetching) = wire_rebalance_hooks(consumer.context(),batcher.revoker(),&topic_offset_ledger,@@ -484,6 +573,7 @@ impl IngestionConsumer {debug_recorder: options.debug_recorder,topic_offset_ledger,revoked_partitions,+ fetching,settlement,consumer,batcher,@@ -494,6 +584,7 @@ impl IngestionConsumer {batch_size_bytes: options.batch_size_bytes,batch_timeout: options.batch_timeout,max_in_flight_batches: options.max_in_flight_batches.max(1),+ commit_tick: commit_tick(options.completion_granularity, options.commit_interval),handle,group_id: options.group_id,}@@ -534,9 +625,12 @@ impl IngestionConsumer {config.consumer_completion_granularity,Arc::clone(&topic_offset_ledger),Duration::from_millis(config.consumer_commit_interval_ms),+ config+ .partition_budget()+ .map_err(|err| anyhow::anyhow!("invalid partition budget: {err}"))?,);let context = RebalanceContext::new();- let revoked_partitions = wire_rebalance_hooks(+ let (revoked_partitions, fetching) = wire_rebalance_hooks(&context,batcher.revoker(),&topic_offset_ledger,@@ -563,6 +657,7 @@ impl IngestionConsumer {debug_recorder,topic_offset_ledger,revoked_partitions,+ fetching,settlement,batcher,outputs: Some(outputs),@@ -572,6 +667,10 @@ impl IngestionConsumer {batch_size_bytes: config.consumer_batch_size_kb.saturating_mul(1024),batch_timeout: Duration::from_millis(config.consumer_batch_timeout_ms),max_in_flight_batches: config.consumer_max_background_tasks.max(1),+ commit_tick: commit_tick(+ config.consumer_completion_granularity,+ Duration::from_millis(config.consumer_commit_interval_ms),+ ),handle,group_id: config.ingestion_consumer_group_id.clone(),})@@ -639,7 +738,10 @@ impl IngestionConsumer {return;}- if accepting_new_batches && in_flight_polls.len() < self.max_in_flight_batches {+ let collect = accepting_new_batches+ && in_flight_polls.len() < self.max_in_flight_batches+ && (in_flight_polls.is_empty() || !self.every_partition_paused());+ if collect {tokio::select! {_ = self.handle.shutdown_recv() => {info!(@@ -773,7 +875,10 @@ impl IngestionConsumer {completions: &mut mpsc::UnboundedReceiver<GroupCompletion>,errors: &mut mpsc::UnboundedReceiver<String>,) -> Result<(), Failure> {- let mut heartbeat = tokio::time::interval(Duration::from_secs(1));+ let mut heartbeat = tokio::time::interval_at(+ tokio::time::Instant::now() + self.commit_tick,+ self.commit_tick,+ );loop {// A rebalance can drop the front poll (its partitions were// revoked and its purged messages will never complete), so@@ -792,7 +897,10 @@ impl IngestionConsumer {self.settlement.settle_run(tp, generation, run)}).map_err(Failure::Batch)?;- self.commit_due().map_err(Failure::Commit)?;+ // A resumed partition can fill a poll, so collect instead of waiting here.+ if self.commit_due().map_err(Failure::Commit)? {+ return Ok(());+ }}None => return Err(Failure::Batch(anyhow::anyhow!("batcher completion channel closed"@@ -806,7 +914,12 @@ impl IngestionConsumer {},_ = heartbeat.tick() => {self.handle.report_healthy();- self.commit_due().map_err(Failure::Commit)?;+ if self.commit_due().map_err(Failure::Commit)?+ || (in_flight_polls.len() < self.max_in_flight_batches+ && !self.every_partition_paused())+ {+ return Ok(());+ }}}}@@ -852,18 +965,67 @@ impl IngestionConsumer {}}- fn commit_due(&self) -> anyhow::Result<()> {+ /// Paused partitions fill no poll, so a collection would wait out the batch+ /// timeout before the completions whose commits resume them.+ fn every_partition_paused(&self) -> bool {+ self.fetching.all_paused()+ }++ fn pause_partition(&self, topic_partition: &TopicPartition) {+ let tpl = partition_list(topic_partition);+ match self+ .consumer+ .pause(&tpl)+ .and_then(|()| partition_error(&tpl))+ {+ Ok(()) => self.fetching.paused(topic_partition),+ Err(err) => error!(+ topic = %topic_partition.topic,+ partition = topic_partition.partition,+ error = %err,+ "Could not pause a partition at its budget"+ ),+ }+ }++ fn resume_partition(&self, topic_partition: &TopicPartition) {+ let tpl = partition_list(topic_partition);+ match self+ .consumer+ .resume(&tpl)+ .and_then(|()| partition_error(&tpl))+ {+ Ok(()) => self.fetching.resumed(topic_partition),+ Err(err) => error!(+ topic = %topic_partition.topic,+ partition = topic_partition.partition,+ error = %err,+ "Could not resume a partition under its budget"+ ),+ }+ }++ /// Returns whether the commit resumed a partition.+ fn commit_due(&self) -> anyhow::Result<bool> {match self.settlement.take_due(Instant::now()) {- Some(offsets) => self.commit_offsets(&offsets),- None => Ok(()),+ Some(commit) => self.commit(commit),+ None => Ok(false),}}- fn commit_all(&self) -> anyhow::Result<()> {+ fn commit_all(&self) -> anyhow::Result<bool> {match self.settlement.take_all() {- Some(offsets) => self.commit_offsets(&offsets),- None => Ok(()),+ Some(commit) => self.commit(commit),+ None => Ok(false),+ }+ }++ fn commit(&self, commit: Commit) -> anyhow::Result<bool> {+ self.commit_offsets(&commit.offsets)?;+ for topic_partition in &commit.resume {+ self.resume_partition(topic_partition);}+ Ok(!commit.resume.is_empty())}fn fail_batch_processing(&self, err: anyhow::Error) {@@ -1031,16 +1193,18 @@ impl IngestionConsumer {for (topic_partition, partition) in &partitions {// The ledger counts a rejection; it is logged here, where the// slice is still known.- if let Err(rejection) =- self.settlement- .charge(topic_partition, partition.generation, &partition.charges)+ match self+ .settlement+ .charge(topic_partition, partition.generation, &partition.charges){- report_rejection(+ Ok(Some(Pause)) => self.pause_partition(topic_partition),+ Ok(None) => {}+ Err(rejection) => report_rejection("charge",topic_partition,rejection,RejectedSlice::charged(&partition.charges),- )?;+ )?,}}@@ -1052,6 +1216,12 @@ impl IngestionConsumer {}}+fn partition_list(topic_partition: &TopicPartition) -> TopicPartitionList {+ let mut tpl = TopicPartitionList::new();+ tpl.add_partition(&topic_partition.topic, topic_partition.partition);+ tpl+}+/// Emit the per-poll parity metrics (received counts, batch sizes,/// utilization, and lag) right after collection, before the poll is/// submitted.@@ -1197,6 +1367,8 @@ fn emit_latest_processed_timestamp_metrics(#[cfg(test)]mod tests {use super::*;+ use std::num::NonZeroU64;+use crate::settlement::{charge_offsets, GroupSettlement};use common_kafka_consumer::Offset as MessageOffset;@@ -1243,26 +1415,107 @@ mod tests {apply_completion(in_flight, &completion, |_, _, _| Ok(())).unwrap();}+ fn partitions(numbers: &[i32]) -> Vec<TopicPartition> {+ numbers+ .iter()+ .map(|&number| TopicPartition::new("events", number))+ .collect()+ }++ #[rstest::rstest]+ #[case::group_below_a_second(CompletionGranularity::Group, 200, 200)]+ #[case::group_above_a_second(CompletionGranularity::Group, 5_000, 1_000)]+ #[case::group_immediate(CompletionGranularity::Group, 0, 1_000)]+ #[case::poll(CompletionGranularity::Poll, 200, 1_000)]+ fn the_commit_tick_follows_the_commit_interval(+ #[case] granularity: CompletionGranularity,+ #[case] commit_interval_ms: u64,+ #[case] expected_ms: u64,+ ) {+ assert_eq!(+ commit_tick(granularity, Duration::from_millis(commit_interval_ms)),+ Duration::from_millis(expected_ms)+ );+ }++ #[test]+ fn every_partition_is_paused_only_while_none_is_fetching() {+ let fetching = FetchingPartitions::default();+ assert!(!fetching.all_paused(), "nothing is assigned");++ fetching.assigned(&partitions(&[0, 1]));+ fetching.paused(&TopicPartition::new("events", 0));+ assert!(!fetching.all_paused(), "partition 1 still fetches");++ fetching.paused(&TopicPartition::new("events", 1));+ assert!(fetching.all_paused());++ fetching.resumed(&TopicPartition::new("events", 1));+ assert!(!fetching.all_paused());+ }++ #[test]+ fn a_rebalance_changes_which_partitions_must_be_paused() {+ let fetching = FetchingPartitions::default();+ fetching.assigned(&partitions(&[0, 1]));+ fetching.paused(&TopicPartition::new("events", 0));++ fetching.revoked(&partitions(&[1]));+ assert!(fetching.all_paused(), "only the paused partition is left");++ fetching.assigned(&partitions(&[0]));+ assert!(+ !fetching.all_paused(),+ "a reassigned partition starts fetching"+ );+ }++ #[test]+ fn a_partition_that_is_not_assigned_neither_pauses_nor_resumes() {+ let fetching = FetchingPartitions::default();++ fetching.paused(&TopicPartition::new("events", 0));+ assert!(!fetching.all_paused());++ fetching.resumed(&TopicPartition::new("events", 0));+ assert!(!fetching.all_paused());+ }+#[test]fn an_assignment_without_a_revoke_drops_the_partitions_pending_frontier() {let ledger = Arc::new(TopicOffsetLedger::new());let settlement: Arc<dyn Settlement> = Arc::new(GroupSettlement::new(Arc::clone(&ledger),Duration::from_secs(3600),+ BudgetLimits::new(NonZeroU64::new(1), None, 0.5).expect("valid limits"),));let epoch = AssignmentEpoch::new();- let on_assign = assign_hook(Arc::clone(&settlement), epoch.clone());+ let on_assign = assign_hook(+ Arc::clone(&settlement),+ Arc::new(FetchingPartitions::default()),+ epoch.clone(),+ );let p0 = TopicPartition::new("events", 0);- charge_offsets(&ledger, &p0, ledger.generation(&p0), &[10]);- settlement- .settle_run(&p0, ledger.generation(&p0), &[MessageOffset(10)])- .unwrap();+ let one_event = [(+ MessageOffset(10),+ Charge {+ events: 1,+ bytes: 0,+ },+ )];+ assert_eq!(settlement.charge(&p0, 0, &one_event), Ok(Some(Pause)));+ settlement.settle_run(&p0, 0, &[MessageOffset(10)]).unwrap();on_assign(std::slice::from_ref(&p0));assert!(settlement.take_all().is_none());assert_eq!(ledger.generation(&p0), 1);assert_eq!(epoch.current(), 2);+ assert_eq!(+ settlement.charge(&p0, 1, &one_event),+ Ok(Some(Pause)),+ "the regained partition starts unpaused"+ );}fn regained_poll(charged: i64, span_last: i64, delivered: u32) -> InFlightPoll {@@ -1309,7 +1562,8 @@ mod tests {#[test]fn a_completion_spanning_two_polls_commits_once_the_prefix_is_whole() {let ledger = Arc::new(TopicOffsetLedger::new());- let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO);+ let settlement =+ GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO, BudgetLimits::unset());let partition = TopicPartition::new("test", 0);charge_offsets(&ledger, &partition, 0, &[0, 1, 2, 3, 4, 5, 6, 7]);let mut in_flight = VecDeque::from([poll(1, 0, 0, 3, 4), poll(1, 0, 4, 7, 4)]);@@ -1335,7 +1589,9 @@ mod tests {).unwrap();assert_eq!(- settlement.take_due(Instant::now()),+ settlement+ .take_due(Instant::now())+ .map(|commit| commit.offsets),Some(HashMap::from([(partition, MessageOffset(8))])));}@@ -1641,7 +1897,8 @@ mod tests {#[test]fn a_covered_head_is_popped_before_a_redelivered_offset_is_matched() {let ledger = Arc::new(TopicOffsetLedger::new());- let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO);+ let settlement =+ GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO, BudgetLimits::unset());let partition = TopicPartition::new("test", 0);ledger.forget_partitions([("test", 0)]);charge_offsets(&ledger, &partition, 1, &[0, 1]);@@ -1671,7 +1928,9 @@ mod tests {assert_eq!(in_flight[0].poll_id, "replay");assert!(in_flight[0].is_complete());assert_eq!(- settlement.take_due(Instant::now()),+ settlement+ .take_due(Instant::now())+ .map(|commit| commit.offsets),Some(HashMap::from([(partition, MessageOffset(2))])));}diff --git a/rust/ingestion-consumer/src/lib.rs b/rust/ingestion-consumer/src/lib.rsindex 32307723384f..81893274b8ce 100644--- a/rust/ingestion-consumer/src/lib.rs+++ b/rust/ingestion-consumer/src/lib.rs@@ -9,6 +9,7 @@ pub mod grpc_transport;pub mod kafka_stats;pub mod ledger_rejection;pub mod order_sentinel;+pub mod partition_budget;pub mod readiness;pub mod rebalance;pub mod routing;diff --git a/rust/ingestion-consumer/src/main.rs b/rust/ingestion-consumer/src/main.rsindex 6cc20bee9631..416489c7486a 100644--- a/rust/ingestion-consumer/src/main.rs+++ b/rust/ingestion-consumer/src/main.rs@@ -50,6 +50,9 @@ fn main() -> Result<()> {config.request_budget().map_err(|err| anyhow::anyhow!("invalid request budget: {err}"))?;+ config+ .partition_budget()+ .map_err(|err| anyhow::anyhow!("invalid partition budget: {err}"))?;let runtime = tokio::runtime::Builder::new_multi_thread().enable_all()diff --git a/rust/ingestion-consumer/src/partition_budget.rs b/rust/ingestion-consumer/src/partition_budget.rsnew file mode 100644index 000000000000..e735fcf03cf1--- /dev/null+++ b/rust/ingestion-consumer/src/partition_budget.rs@@ -0,0 +1,174 @@+use std::num::NonZeroU64;++use common_kafka_consumer::Charge;++#[derive(Debug, Clone, Copy)]+struct Threshold {+ events: Option<u64>,+ bytes: Option<u64>,+}++impl Threshold {+ fn reached(&self, charge: Charge) -> bool {+ self.events.is_some_and(|limit| charge.events >= limit)+ || self.bytes.is_some_and(|limit| charge.bytes >= limit)+ }++ fn scaled(&self, fraction: f64) -> Self {+ let scale = |limit: u64| ((limit as f64 * fraction) as u64).clamp(1, limit);+ Self {+ events: self.events.map(scale),+ bytes: self.bytes.map(scale),+ }+ }+}++#[derive(Debug, Clone, Copy)]+struct Limits {+ pause_at: Threshold,+ resume_at: Threshold,+}++#[derive(Debug, Clone, Copy)]+pub struct BudgetLimits {+ limits: Option<Limits>,+}++impl BudgetLimits {+ pub fn unset() -> Self {+ Self { limits: None }+ }++ /// With neither axis set the budget is unset.+ pub fn new(+ events: Option<NonZeroU64>,+ bytes: Option<NonZeroU64>,+ low_watermark: f64,+ ) -> Result<Self, String> {+ if !(low_watermark > 0.0 && low_watermark <= 1.0) {+ return Err(format!(+ "the low watermark must be above 0 and at most 1, got {low_watermark}"+ ));+ }+ if events.is_none() && bytes.is_none() {+ return Ok(Self::unset());+ }+ let pause_at = Threshold {+ events: events.map(NonZeroU64::get),+ bytes: bytes.map(NonZeroU64::get),+ };+ Ok(Self {+ limits: Some(Limits {+ pause_at,+ resume_at: pause_at.scaled(low_watermark),+ }),+ })+ }++ pub fn is_set(&self) -> bool {+ self.limits.is_some()+ }++ pub(crate) fn pauses(&self, uncommitted: Charge) -> bool {+ self.limits+ .is_some_and(|limits| limits.pause_at.reached(uncommitted))+ }++ pub(crate) fn resumes(&self, uncommitted: Charge) -> bool {+ self.limits+ .is_none_or(|limits| !limits.resume_at.reached(uncommitted))+ }+}++#[cfg(test)]+mod tests {+ use rstest::rstest;++ use super::*;++ fn limits(events: u64, bytes: u64, low_watermark: f64) -> BudgetLimits {+ BudgetLimits::new(+ NonZeroU64::new(events),+ NonZeroU64::new(bytes),+ low_watermark,+ )+ .expect("valid limits")+ }++ fn charge(events: u64, bytes: u64) -> Charge {+ Charge { events, bytes }+ }++ struct Observed {+ uncommitted: Charge,+ expected: bool,+ }++ #[rstest]+ #[case::below_both_limits(Observed { uncommitted: charge(9, 999), expected: false })]+ #[case::at_the_event_limit(Observed { uncommitted: charge(10, 0), expected: true })]+ #[case::at_the_byte_limit(Observed { uncommitted: charge(0, 1000), expected: true })]+ fn a_partition_pauses_when_either_axis_reaches_its_limit(#[case] observed: Observed) {+ assert_eq!(+ limits(10, 1000, 0.8).pauses(observed.uncommitted),+ observed.expected+ );+ }++ #[rstest]+ #[case::between_the_watermark_and_the_limit(Observed { uncommitted: charge(9, 0), expected: false })]+ #[case::at_the_watermark(Observed { uncommitted: charge(8, 0), expected: false })]+ #[case::one_axis_still_at_the_watermark(Observed { uncommitted: charge(7, 800), expected: false })]+ #[case::under_the_watermark_on_both_axes(Observed { uncommitted: charge(7, 799), expected: true })]+ fn a_partition_resumes_under_the_watermark_on_every_axis(#[case] observed: Observed) {+ assert_eq!(+ limits(10, 1000, 0.8).resumes(observed.uncommitted),+ observed.expected+ );+ }++ #[test]+ fn a_full_watermark_resumes_only_below_the_limit() {+ let limits = limits(10, 0, 1.0);++ assert!(limits.pauses(charge(10, 0)));+ assert!(!limits.resumes(charge(10, 0)));+ assert!(limits.resumes(charge(9, 0)));+ }++ #[test]+ fn a_limit_above_float_precision_resumes_only_below_the_limit() {+ let limit = (1 << 53) + 3;+ let limits = limits(limit, 0, 1.0);++ assert!(limits.pauses(charge(limit, 0)));+ assert!(!limits.resumes(charge(limit, 0)));+ }++ #[rstest]+ #[case::events(limits(1, 0, 0.8), charge(1, 0))]+ #[case::bytes(limits(0, 1, 0.8), charge(0, 1))]+ fn a_limit_of_one_resumes_once_fully_refunded(+ #[case] limits: BudgetLimits,+ #[case] at_the_limit: Charge,+ ) {+ assert!(limits.pauses(at_the_limit));+ assert!(limits.resumes(Charge::ZERO));+ }++ #[test]+ fn unset_limits_never_pause() {+ let limits = BudgetLimits::new(None, None, 0.8).expect("unset is valid");+ assert!(!limits.is_set());+ assert!(!limits.pauses(charge(u64::MAX, u64::MAX)));+ }++ #[rstest]+ #[case::zero(0.0)]+ #[case::above_one(1.5)]+ #[case::negative(-0.5)]+ #[case::nan(f64::NAN)]+ fn a_watermark_outside_the_unit_interval_is_rejected(#[case] low_watermark: f64) {+ assert!(BudgetLimits::new(NonZeroU64::new(10), None, low_watermark).is_err());+ }+}diff --git a/rust/ingestion-consumer/src/rebalance.rs b/rust/ingestion-consumer/src/rebalance.rsindex 6f3fb461c22f..76d0e135aa1e 100644--- a/rust/ingestion-consumer/src/rebalance.rs+++ b/rust/ingestion-consumer/src/rebalance.rs@@ -4,9 +4,10 @@ use std::sync::OnceLock;use common_kafka_consumer::TopicPartition;use metrics::counter;-use rdkafka::consumer::{BaseConsumer, ConsumerContext, Rebalance};+use rdkafka::consumer::{BaseConsumer, Consumer, ConsumerContext, Rebalance};+use rdkafka::error::KafkaResult;use rdkafka::{ClientContext, Statistics, TopicPartitionList};-use tracing::{info, warn};+use tracing::{error, info, warn};pub type PartitionHook = Box<dyn Fn(&[TopicPartition]) + Send + Sync>;@@ -42,6 +43,14 @@ fn topic_partitions(tpl: &TopicPartitionList) -> Vec<TopicPartition> {partitions}+/// librdkafka's pause and resume return success when they cannot change a+/// partition. They set the error on that partition's list element.+pub(crate) fn partition_error(tpl: &TopicPartitionList) -> KafkaResult<()> {+ tpl.elements()+ .iter()+ .try_for_each(|element| element.error())+}+fn partition_names(partitions: &[TopicPartition]) -> Vec<String> {partitions.iter()@@ -80,7 +89,7 @@ impl ConsumerContext for RebalanceContext {}}- fn post_rebalance(&self, _consumer: &BaseConsumer<Self>, rebalance: &Rebalance) {+ fn post_rebalance(&self, consumer: &BaseConsumer<Self>, rebalance: &Rebalance) {if let Rebalance::Assign(tpl) = rebalance {counter!("ingestion_consumer_rebalances_total", "event" => "assign").increment(1);let partitions = topic_partitions(tpl);@@ -89,6 +98,11 @@ impl ConsumerContext for RebalanceContext {topic_partitions = ?partition_names(&partitions),"Rebalance: partitions assigned");+ // librdkafka keeps a partition's pause across a revoke and a+ // reassignment, and the assign hook forgets the budget that paused it.+ if let Err(err) = consumer.resume(tpl).and_then(|()| partition_error(tpl)) {+ error!(error = %err, "Could not resume the assigned partitions");+ }if let Some(hook) = self.on_assign.get() {hook(&partitions);}@@ -159,4 +173,18 @@ mod tests {assert_eq!(*revoked.lock().unwrap(), expected);assert_eq!(*assigned.lock().unwrap(), expected);}++ #[test]+ fn pausing_an_unassigned_partition_is_an_error() {+ let consumer: BaseConsumer = ClientConfig::new()+ .set("bootstrap.servers", "127.0.0.1:1")+ .set("group.id", "test")+ .create()+ .expect("a client needs no broker to be built");+ let mut tpl = TopicPartitionList::new();+ tpl.add_partition("events", 0);++ assert!(consumer.pause(&tpl).is_ok());+ assert!(partition_error(&tpl).is_err());+ }}diff --git a/rust/ingestion-consumer/src/settlement/budget.rs b/rust/ingestion-consumer/src/settlement/budget.rsnew file mode 100644index 000000000000..b163b91c4e1e--- /dev/null+++ b/rust/ingestion-consumer/src/settlement/budget.rs@@ -0,0 +1,248 @@+use std::collections::{HashMap, HashSet};+use std::sync::{Arc, Mutex};++use common_kafka_consumer::{Charge, TopicPartition};+use metrics::{counter, gauge};+use tracing::info;++use super::{Admission, HandedOut, Pause};+use crate::partition_budget::BudgetLimits;++const PAUSED_PARTITIONS: &str = "ingestion_consumer_budget_paused_partitions";+const PAUSES: &str = "ingestion_consumer_budget_pauses_total";++pub(crate) struct PartitionBudget {+ limits: BudgetLimits,+ paused: Mutex<HashSet<TopicPartition>>,+}++impl PartitionBudget {+ pub(super) fn new(limits: BudgetLimits) -> Self {+ gauge!(PAUSED_PARTITIONS).set(0.0);+ Self {+ limits,+ paused: Mutex::new(HashSet::new()),+ }+ }+}++impl Admission for PartitionBudget {+ fn charged(&self, topic_partition: &TopicPartition, uncommitted: Charge) -> Option<Pause> {+ if !self.limits.pauses(uncommitted) {+ return None;+ }+ let mut paused = self.paused.lock().unwrap();+ if !paused.insert(topic_partition.clone()) {+ return None;+ }+ gauge!(PAUSED_PARTITIONS).set(paused.len() as f64);+ let topic: Arc<str> = Arc::from(topic_partition.topic.as_str());+ let partition: Arc<str> = Arc::from(topic_partition.partition.to_string());+ counter!(PAUSES, "topic" => topic, "partition" => partition).increment(1);+ log_transition(topic_partition, "pause", uncommitted);+ Some(Pause)+ }++ fn handed_out(&self, handed_out: &HashMap<TopicPartition, HandedOut>) -> Vec<TopicPartition> {+ let mut paused = self.paused.lock().unwrap();+ let mut resume = Vec::new();+ for (topic_partition, handed_out) in handed_out {+ let uncommitted = handed_out.remaining.charge;+ if paused.contains(topic_partition) && self.limits.resumes(uncommitted) {+ paused.remove(topic_partition);+ log_transition(topic_partition, "resume", uncommitted);+ resume.push(topic_partition.clone());+ }+ }+ if !resume.is_empty() {+ gauge!(PAUSED_PARTITIONS).set(paused.len() as f64);+ }+ resume+ }++ fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {+ let mut paused = self.paused.lock().unwrap();+ for topic_partition in topic_partitions {+ paused.remove(topic_partition);+ }+ gauge!(PAUSED_PARTITIONS).set(paused.len() as f64);+ }+}++fn log_transition(topic_partition: &TopicPartition, transition: &str, uncommitted: Charge) {+ info!(+ topic = %topic_partition.topic,+ partition = topic_partition.partition,+ uncommitted_events = uncommitted.events,+ uncommitted_bytes = uncommitted.bytes,+ transition,+ "Partition budget"+ );+}++#[cfg(test)]+mod tests {+ use std::num::NonZeroU64;++ use common_kafka_consumer::{Held, Offset};+ use metrics_util::debugging::{DebugValue, DebuggingRecorder, Snapshotter};++ use super::*;++ fn p0() -> TopicPartition {+ TopicPartition::new("events", 0)+ }++ fn p1() -> TopicPartition {+ TopicPartition::new("events", 1)+ }++ fn events(events: u64) -> Charge {+ Charge { events, bytes: 0 }+ }++ fn budget_of_10_events_resuming_under_8() -> PartitionBudget {+ PartitionBudget::new(+ BudgetLimits::new(NonZeroU64::new(10), None, 0.8).expect("valid limits"),+ )+ }++ #[rstest::rstest]+ #[case::below_the_limit(9, None)]+ #[case::at_the_limit(10, Some(Pause))]+ fn a_charge_pauses_a_partition_that_reaches_its_limit(+ #[case] uncommitted: u64,+ #[case] expected: Option<Pause>,+ ) {+ assert_eq!(+ budget_of_10_events_resuming_under_8().charged(&p0(), events(uncommitted)),+ expected+ );+ }++ #[test]+ fn a_paused_partition_pauses_once() {+ let budget = budget_of_10_events_resuming_under_8();+ assert_eq!(budget.charged(&p0(), events(10)), Some(Pause));++ assert_eq!(budget.charged(&p0(), events(12)), None);+ }++ fn handed_out(remaining_events: u64) -> HandedOut {+ HandedOut {+ offset: Offset(1),+ remaining: Held {+ offsets: remaining_events as usize,+ charge: events(remaining_events),+ },+ }+ }++ fn hand_out_leaving(budget: &PartitionBudget, remaining_events: u64) -> Vec<TopicPartition> {+ budget.handed_out(&HashMap::from([(p0(), handed_out(remaining_events))]))+ }++ #[rstest::rstest]+ #[case::above_the_watermark(9, false)]+ #[case::at_the_watermark(8, false)]+ #[case::under_the_watermark(7, true)]+ fn a_hand_out_resumes_a_paused_partition_under_the_watermark(+ #[case] remaining: u64,+ #[case] expected_resume: bool,+ ) {+ let budget = budget_of_10_events_resuming_under_8();+ budget.charged(&p0(), events(10));++ assert_eq!(+ hand_out_leaving(&budget, remaining).contains(&p0()),+ expected_resume+ );+ }++ #[test]+ fn a_hand_out_never_resumes_a_partition_that_is_not_paused() {+ let budget = budget_of_10_events_resuming_under_8();++ assert!(hand_out_leaving(&budget, 0).is_empty());+ }++ #[test]+ fn a_resumed_partition_pauses_again_at_its_limit() {+ let budget = budget_of_10_events_resuming_under_8();+ budget.charged(&p0(), events(10));+ hand_out_leaving(&budget, 0);++ assert_eq!(budget.charged(&p0(), events(10)), Some(Pause));+ }++ #[test]+ fn a_forgotten_partition_starts_unpaused() {+ let budget = budget_of_10_events_resuming_under_8();+ budget.charged(&p0(), events(10));++ budget.forget_partitions(&[p0()]);++ assert_eq!(budget.charged(&p0(), events(10)), Some(Pause));+ }++ #[derive(Debug, PartialEq)]+ struct Recorded {+ paused_partitions: f64,+ pauses: HashMap<String, u64>,+ }++ fn recorded(snapshotter: &Snapshotter) -> Recorded {+ let mut recorded = Recorded {+ paused_partitions: f64::NAN,+ pauses: HashMap::new(),+ };+ for (key, _, _, value) in snapshotter.snapshot().into_vec() {+ match (key.key().name(), value) {+ (PAUSED_PARTITIONS, DebugValue::Gauge(value)) => {+ recorded.paused_partitions = value.into_inner();+ }+ (PAUSES, DebugValue::Counter(count)) => {+ let partition = key+ .key()+ .labels()+ .find(|label| label.key() == "partition")+ .expect("a partition label")+ .value()+ .to_string();+ recorded.pauses.insert(partition, count);+ }+ _ => {}+ }+ }+ recorded+ }++ #[test]+ fn the_metrics_follow_pauses_resumes_and_revokes() {+ let recorder = DebuggingRecorder::new();+ let snapshotter = recorder.snapshotter();+ let _guard = metrics::set_default_local_recorder(&recorder);+ let budget = budget_of_10_events_resuming_under_8();++ budget.charged(&p0(), events(10));+ budget.charged(&p1(), events(10));+ budget.charged(&p0(), events(12));+ assert_eq!(+ recorded(&snapshotter),+ Recorded {+ paused_partitions: 2.0,+ pauses: HashMap::from([("0".to_string(), 1), ("1".to_string(), 1)]),+ }+ );++ let resume = budget.handed_out(&HashMap::from([+ (p0(), handed_out(0)),+ (p1(), handed_out(9)),+ ]));+ assert_eq!(resume, vec![p0()]);+ assert_eq!(recorded(&snapshotter).paused_partitions, 1.0);++ budget.forget_partitions(&[p1()]);+ assert_eq!(recorded(&snapshotter).paused_partitions, 0.0);+ }+}diff --git a/rust/ingestion-consumer/src/settlement/fakes.rs b/rust/ingestion-consumer/src/settlement/fakes.rsnew file mode 100644index 000000000000..93e57c4cbcb1--- /dev/null+++ b/rust/ingestion-consumer/src/settlement/fakes.rs@@ -0,0 +1,162 @@+use std::collections::{HashMap, VecDeque};+use std::sync::{Arc, Mutex};+use std::time::Instant;++use common_kafka_consumer::{Charge, Held, Offset, Rejection, TopicPartition};++use super::{Accounting, Admission, HandedOut, Pause};++#[derive(Debug, Clone, PartialEq, Eq)]+pub(super) struct Settled {+ pub(super) topic_partition: TopicPartition,+ pub(super) generation: u64,+ pub(super) offsets: Vec<Offset>,+}++struct Script<T>(Arc<Mutex<VecDeque<T>>>);++impl<T> Default for Script<T> {+ fn default() -> Self {+ Self(Arc::new(Mutex::new(VecDeque::new())))+ }+}++impl<T> Clone for Script<T> {+ fn clone(&self) -> Self {+ Self(Arc::clone(&self.0))+ }+}++impl<T> Script<T> {+ fn push(&self, result: T) {+ self.0.lock().unwrap().push_back(result);+ }++ fn next(&self) -> Option<T> {+ self.0.lock().unwrap().pop_front()+ }+}++#[derive(Default, Clone)]+pub(super) struct FakeAccounting {+ charges: Script<Result<Held, Rejection>>,+ settles: Script<Result<Option<Offset>, Rejection>>,+ takes: Script<HashMap<TopicPartition, HandedOut>>,+ settled: Arc<Mutex<Vec<Settled>>>,+ forgotten: Arc<Mutex<Vec<TopicPartition>>>,+}++impl FakeAccounting {+ pub(super) fn charge_returns(&self, result: Result<Held, Rejection>) {+ self.charges.push(result);+ }++ pub(super) fn settle_returns(&self, result: Result<Option<Offset>, Rejection>) {+ self.settles.push(result);+ }++ pub(super) fn take_hands_out(&self, handed_out: HashMap<TopicPartition, HandedOut>) {+ self.takes.push(handed_out);+ }++ pub(super) fn settles_seen(&self) -> Vec<Settled> {+ self.settled.lock().unwrap().clone()+ }++ pub(super) fn forgotten(&self) -> Vec<TopicPartition> {+ self.forgotten.lock().unwrap().clone()+ }+}++impl Accounting for FakeAccounting {+ fn charge(+ &self,+ _topic_partition: &TopicPartition,+ _generation: u64,+ _charges: &[(Offset, Charge)],+ ) -> Result<Held, Rejection> {+ self.charges.next().unwrap_or(Ok(Held::default()))+ }++ fn settled(+ &self,+ topic_partition: &TopicPartition,+ generation: u64,+ offsets: impl IntoIterator<Item = Offset>,+ ) -> Result<Option<Offset>, Rejection> {+ self.settled.lock().unwrap().push(Settled {+ topic_partition: topic_partition.clone(),+ generation,+ offsets: offsets.into_iter().collect(),+ });+ self.settles.next().unwrap_or(Ok(None))+ }++ fn take_due(&self, _now: Instant) -> Option<HashMap<TopicPartition, HandedOut>> {+ self.takes.next()+ }++ fn take_all(&self) -> Option<HashMap<TopicPartition, HandedOut>> {+ self.takes.next()+ }++ fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {+ self.forgotten+ .lock()+ .unwrap()+ .extend_from_slice(topic_partitions);+ }+}++#[derive(Default, Clone)]+pub(super) struct FakeAdmission {+ pauses: Script<Option<Pause>>,+ resumes: Script<Vec<TopicPartition>>,+ charged: Arc<Mutex<Vec<(TopicPartition, Charge)>>>,+ handed_out: Arc<Mutex<Vec<HashMap<TopicPartition, HandedOut>>>>,+ forgotten: Arc<Mutex<Vec<TopicPartition>>>,+}++impl FakeAdmission {+ pub(super) fn charged_returns(&self, decision: Option<Pause>) {+ self.pauses.push(decision);+ }++ pub(super) fn handed_out_returns(&self, resume: Vec<TopicPartition>) {+ self.resumes.push(resume);+ }++ pub(super) fn charges_seen(&self) -> Vec<(TopicPartition, Charge)> {+ self.charged.lock().unwrap().clone()+ }++ pub(super) fn hand_outs_seen(&self) -> Vec<HashMap<TopicPartition, HandedOut>> {+ self.handed_out.lock().unwrap().clone()+ }++ pub(super) fn forgotten(&self) -> Vec<TopicPartition> {+ self.forgotten.lock().unwrap().clone()+ }+}++impl Admission for FakeAdmission {+ fn charged(&self, topic_partition: &TopicPartition, uncommitted: Charge) -> Option<Pause> {+ self.charged+ .lock()+ .unwrap()+ .push((topic_partition.clone(), uncommitted));+ self.pauses.next().flatten()+ }++ fn handed_out(&self, handed_out: &HashMap<TopicPartition, HandedOut>) -> Vec<TopicPartition> {+ self.handed_out.lock().unwrap().push(handed_out.clone());+ self.resumes.next().unwrap_or_default()+ }++ fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {+ self.forgotten+ .lock()+ .unwrap()+ .extend_from_slice(topic_partitions);+ }+}diff --git a/rust/ingestion-consumer/src/settlement/group.rs b/rust/ingestion-consumer/src/settlement/group.rsindex 3f925098cf56..e202a0630751 100644--- a/rust/ingestion-consumer/src/settlement/group.rs+++ b/rust/ingestion-consumer/src/settlement/group.rs@@ -1,35 +1,46 @@-use std::collections::HashMap;use std::sync::Arc;use std::time::{Duration, Instant};-use common_kafka_consumer::{Charge, Held, Offset, Rejection, TopicOffsetLedger, TopicPartition};+use common_kafka_consumer::{Charge, Offset, Rejection, TopicOffsetLedger, TopicPartition};-use super::{ChargedSlice, Settlement, Uncommitted};+use super::{+ offsets, Accounting, Admission, ChargedSlice, Commit, PartitionBudget, Pause, Settlement,+ Uncommitted,+};use crate::commit_pacer::CommitPacer;use crate::ledger_rejection::{report_rejection, RejectedSlice};use crate::order_sentinel::OffsetSpan;+use crate::partition_budget::BudgetLimits;-pub(crate) struct GroupSettlement {- uncommitted: Uncommitted,+pub(crate) struct GroupSettlement<A: Accounting = Uncommitted, D: Admission = PartitionBudget> {+ accounting: A,+ admission: D,}impl GroupSettlement {- pub(crate) fn new(ledger: Arc<TopicOffsetLedger>, commit_interval: Duration) -> Self {+ pub(crate) fn new(+ ledger: Arc<TopicOffsetLedger>,+ commit_interval: Duration,+ limits: BudgetLimits,+ ) -> Self {Self {- uncommitted: Uncommitted::new(ledger, CommitPacer::every(commit_interval)),+ accounting: Uncommitted::new(ledger, CommitPacer::every(commit_interval)),+ admission: PartitionBudget::new(limits),}}}-impl Settlement for GroupSettlement {+impl<A: Accounting + 'static, D: Admission + 'static> Settlement for GroupSettlement<A, D> {fn charge(&self,topic_partition: &TopicPartition,generation: u64,charges: &[(Offset, Charge)],- ) -> Result<Held, Rejection> {- self.uncommitted- .charge(topic_partition, generation, charges)+ ) -> Result<Option<Pause>, Rejection> {+ let uncommitted = self+ .accounting+ .charge(topic_partition, generation, charges)?;+ Ok(self.admission.charged(topic_partition, uncommitted.charge))}fn settle_run(@@ -38,8 +49,11 @@ impl Settlement for GroupSettlement {generation: u64,offsets: &[Offset],) -> anyhow::Result<()> {+ let (Some(first), Some(last)) = (offsets.first(), offsets.last()) else {+ return Ok(());+ };if let Err(rejection) =- self.uncommitted+ self.accounting.settled(topic_partition, generation, offsets.iter().copied()){report_rejection(@@ -47,8 +61,8 @@ impl Settlement for GroupSettlement {topic_partition,rejection,RejectedSlice::settled(&OffsetSpan {- first: offsets[0].0,- last: offsets[offsets.len() - 1].0,+ first: first.0,+ last: last.0,}),)?;}@@ -59,27 +73,225 @@ impl Settlement for GroupSettlement {Ok(())}- fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, Offset>> {- self.uncommitted.take_due(now)+ fn take_due(&self, now: Instant) -> Option<Commit> {+ let handed_out = self.accounting.take_due(now)?;+ Some(Commit {+ resume: self.admission.handed_out(&handed_out),+ offsets: offsets(handed_out),+ })}- fn take_all(&self) -> Option<HashMap<TopicPartition, Offset>> {- self.uncommitted.take_all()+ fn take_all(&self) -> Option<Commit> {+ let handed_out = self.accounting.take_all()?;+ Some(Commit {+ resume: self.admission.handed_out(&handed_out),+ offsets: offsets(handed_out),+ })}fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {- self.uncommitted.forget_partitions(topic_partitions);+ self.accounting.forget_partitions(topic_partitions);+ self.admission.forget_partitions(topic_partitions);}}#[cfg(test)]mod tests {+ use std::collections::HashMap;++ use common_kafka_consumer::{Held, LedgerError};++ use super::super::fakes::{FakeAccounting, FakeAdmission, Settled};+ use super::super::HandedOut;+ use super::*;++ fn p(partition: i32) -> TopicPartition {+ TopicPartition::new("events", partition)+ }++ fn events(events: u64) -> Charge {+ Charge { events, bytes: 0 }+ }++ fn uncommitted_events(events: u64) -> Held {+ Held {+ offsets: events as usize,+ charge: Charge { events, bytes: 0 },+ }+ }++ fn handed_out(offset: i64, remaining_events: u64) -> HandedOut {+ HandedOut {+ offset: Offset(offset),+ remaining: uncommitted_events(remaining_events),+ }+ }++ fn settlement() -> (+ GroupSettlement<FakeAccounting, FakeAdmission>,+ FakeAccounting,+ FakeAdmission,+ ) {+ let accounting = FakeAccounting::default();+ let admission = FakeAdmission::default();+ let settlement = GroupSettlement {+ accounting: accounting.clone(),+ admission: admission.clone(),+ };+ (settlement, accounting, admission)+ }++ #[test]+ fn a_charge_asks_the_admission_with_the_uncommitted_charge() {+ let (settlement, accounting, admission) = settlement();+ accounting.charge_returns(Ok(uncommitted_events(10)));+ admission.charged_returns(Some(Pause));++ assert_eq!(settlement.charge(&p(0), 0, &[]), Ok(Some(Pause)));+ assert_eq!(admission.charges_seen(), vec![(p(0), events(10))]);+ }++ #[test]+ fn a_rejected_charge_asks_the_admission_nothing() {+ let (settlement, accounting, admission) = settlement();+ let stale = Rejection::Stale {+ stamp: 0,+ generation: 1,+ };+ accounting.charge_returns(Err(stale));++ assert_eq!(settlement.charge(&p(0), 0, &[]), Err(stale));+ assert!(admission.charges_seen().is_empty());+ }++ #[test]+ fn a_hand_out_commits_its_offsets_and_resumes_what_the_admission_decides() {+ let (settlement, accounting, admission) = settlement();+ let hand_out = HashMap::from([(p(0), handed_out(11, 3)), (p(1), handed_out(21, 0))]);+ accounting.take_hands_out(hand_out.clone());+ admission.handed_out_returns(vec![p(0)]);++ let commit = settlement.take_due(Instant::now()).expect("handed out");++ assert_eq!(+ commit.offsets,+ HashMap::from([(p(0), Offset(11)), (p(1), Offset(21))])+ );+ assert_eq!(commit.resume, vec![p(0)]);+ assert_eq!(admission.hand_outs_seen(), vec![hand_out]);+ }++ #[test]+ fn take_all_asks_the_admission_like_take_due() {+ let (settlement, accounting, admission) = settlement();+ accounting.take_hands_out(HashMap::from([(p(0), handed_out(11, 0))]));+ admission.handed_out_returns(vec![p(0)]);++ assert_eq!(+ settlement.take_all().expect("handed out").resume,+ vec![p(0)]+ );+ }++ #[test]+ fn a_run_reaches_the_accounting_under_its_generation() {+ let (settlement, accounting, _) = settlement();++ settlement+ .settle_run(&p(0), 3, &[Offset(10), Offset(11)])+ .unwrap();++ assert_eq!(+ accounting.settles_seen(),+ vec![Settled {+ topic_partition: p(0),+ generation: 3,+ offsets: vec![Offset(10), Offset(11)],+ }]+ );+ }++ #[test]+ fn an_empty_run_settles_nothing() {+ let (settlement, accounting, _) = settlement();++ settlement.settle_run(&p(0), 0, &[]).unwrap();++ assert!(accounting.settles_seen().is_empty());+ }++ struct RejectedRun {+ rejection: Rejection,+ expected_error: bool,+ }++ #[rstest::rstest]+ #[case::stale(RejectedRun {+ rejection: Rejection::Stale { stamp: 0, generation: 1 },+ expected_error: false,+ })]+ #[case::violation(RejectedRun {+ rejection: Rejection::Violation {+ error: LedgerError::CompletedTwice { offset: Offset(10) },+ stamp: 0,+ generation: 0,+ held: Held::default(),+ },+ expected_error: true,+ })]+ fn only_a_violating_run_fails(#[case] run: RejectedRun) {+ let (settlement, accounting, _) = settlement();+ accounting.settle_returns(Err(run.rejection));++ assert_eq!(+ settlement.settle_run(&p(0), 0, &[Offset(10)]).is_err(),+ run.expected_error+ );+ }++ #[test]+ fn a_retired_poll_settles_nothing() {+ let (settlement, accounting, _) = settlement();+ let charges = [(Offset(10), Charge::ZERO)];++ settlement+ .settle_poll(&[ChargedSlice {+ topic_partition: &p(0),+ generation: 0,+ span: OffsetSpan {+ first: 10,+ last: 10,+ },+ charges: &charges,+ }])+ .unwrap();++ assert!(accounting.settles_seen().is_empty());+ }++ #[test]+ fn a_revoke_reaches_both_the_accounting_and_the_admission() {+ let (settlement, accounting, admission) = settlement();++ settlement.forget_partitions(&[p(0)]);++ assert_eq!(accounting.forgotten(), vec![p(0)]);+ assert_eq!(admission.forgotten(), vec![p(0)]);+ }+}++#[cfg(test)]+mod integration_tests {+ use std::collections::HashMap;+ use std::num::NonZeroU64;+use super::super::charge_offsets;use super::*;fn settlement() -> (Arc<TopicOffsetLedger>, GroupSettlement) {let ledger = Arc::new(TopicOffsetLedger::new());- let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO);+ let settlement =+ GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO, BudgetLimits::unset());(ledger, settlement)}@@ -99,7 +311,9 @@ mod tests {.settle_run(&p0, 0, &[Offset(10), Offset(11)]).unwrap();assert_eq!(- settlement.take_due(Instant::now()),+ settlement+ .take_due(Instant::now())+ .map(|commit| commit.offsets),Some(HashMap::from([(p0, Offset(13))])),"the late completion releases the whole prefix");@@ -119,7 +333,9 @@ mod tests {settlement.settle_run(&p1, 0, &[Offset(21)]).unwrap();assert_eq!(- settlement.take_due(Instant::now()),+ settlement+ .take_due(Instant::now())+ .map(|commit| commit.offsets),Some(HashMap::from([(p0, Offset(12))])),"partition 1 waits for offset 20");@@ -135,7 +351,7 @@ mod tests {assert!(settlement.settle_run(&p0, 0, &[Offset(10)]).is_err());assert_eq!(- settlement.take_all(),+ settlement.take_all().map(|commit| commit.offsets),Some(HashMap::from([(p0, Offset(11))])),"offset 10 completed before the violation");@@ -163,21 +379,25 @@ mod tests {#[test]fn a_frontier_within_the_interval_waits_for_it_unless_everything_is_taken() {let ledger = Arc::new(TopicOffsetLedger::new());- let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::from_secs(3600));+ let settlement = GroupSettlement::new(+ Arc::clone(&ledger),+ Duration::from_secs(3600),+ BudgetLimits::unset(),+ );let p0 = TopicPartition::new("events", 0);charge_offsets(&ledger, &p0, 0, &[10, 11]);let now = Instant::now();settlement.settle_run(&p0, 0, &[Offset(10)]).unwrap();assert_eq!(- settlement.take_due(now),+ settlement.take_due(now).map(|commit| commit.offsets),Some(HashMap::from([(p0.clone(), Offset(11))])));settlement.settle_run(&p0, 0, &[Offset(11)]).unwrap();assert!(settlement.take_due(now).is_none());assert_eq!(- settlement.take_all(),+ settlement.take_all().map(|commit| commit.offsets),Some(HashMap::from([(p0, Offset(12))])));}@@ -185,7 +405,11 @@ mod tests {#[test]fn a_revoked_partitions_pending_frontier_is_not_committed() {let ledger = Arc::new(TopicOffsetLedger::new());- let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::from_secs(3600));+ let settlement = GroupSettlement::new(+ Arc::clone(&ledger),+ Duration::from_secs(3600),+ BudgetLimits::unset(),+ );let p0 = TopicPartition::new("events", 0);charge_offsets(&ledger, &p0, 0, &[10]);settlement.settle_run(&p0, 0, &[Offset(10)]).unwrap();@@ -198,26 +422,110 @@ mod tests {assert!(settlement.take_all().is_none());}+ const HOUR: Duration = Duration::from_secs(3600);++ fn p0() -> TopicPartition {+ TopicPartition::new("events", 0)+ }++ fn budgeted_hourly_settlement_of_10_events_resuming_under_8() -> GroupSettlement {+ GroupSettlement::new(+ Arc::new(TopicOffsetLedger::new()),+ HOUR,+ BudgetLimits::new(NonZeroU64::new(10), None, 0.8).expect("valid limits"),+ )+ }++ fn charge(+ settlement: &GroupSettlement,+ topic_partition: &TopicPartition,+ offset: i64,+ events: u64,+ ) -> Option<Pause> {+ settlement+ .charge(+ topic_partition,+ 0,+ &[(Offset(offset), Charge { events, bytes: 0 })],+ )+ .expect("the ledger accepts the charge")+ }+#[test]- fn a_retired_poll_settles_nothing_more() {- let (ledger, settlement) = settlement();- let p0 = TopicPartition::new("events", 0);- charge_offsets(&ledger, &p0, 0, &[10]);- let charges = [(Offset(10), common_kafka_consumer::Charge::ZERO)];+ fn a_take_the_interval_holds_counts_toward_the_pause() {+ let settlement = budgeted_hourly_settlement_of_10_events_resuming_under_8();+ charge(&settlement, &p0(), 10, 6);+ settlement.settle_run(&p0(), 0, &[Offset(10)]).unwrap();++ assert_eq!(charge(&settlement, &p0(), 11, 4), Some(Pause));+ }+ enum HandOut {+ DueAfterTheInterval,+ All,+ }++ #[rstest::rstest]+ #[case::due_after_the_interval(HandOut::DueAfterTheInterval)]+ #[case::all(HandOut::All)]+ fn a_paused_partition_resumes_only_when_its_frontier_is_handed_out(#[case] hand_out: HandOut) {+ let settlement = budgeted_hourly_settlement_of_10_events_resuming_under_8();+ let now = Instant::now();+ charge(&settlement, &p0(), 9, 1);+ settlement.settle_run(&p0(), 0, &[Offset(9)]).unwrap();settlement- .settle_poll(&[ChargedSlice {- topic_partition: &p0,- generation: 0,- span: OffsetSpan {- first: 10,- last: 10,- },- charges: &charges,- }])- .unwrap();+ .take_due(now)+ .expect("the first frontier is due at once");+ assert_eq!(charge(&settlement, &p0(), 10, 10), Some(Pause));- assert!(settlement.take_due(Instant::now()).is_none());- assert_eq!(ledger.held(&p0).offsets, 1);+ settlement.settle_run(&p0(), 0, &[Offset(10)]).unwrap();+ assert!(settlement.take_due(now).is_none());++ let commit = match hand_out {+ HandOut::DueAfterTheInterval => settlement.take_due(now + HOUR),+ HandOut::All => settlement.take_all(),+ }+ .expect("the frontier is handed out");+ assert_eq!(commit.resume, vec![p0()]);+ }++ #[test]+ fn a_held_prefix_keeps_its_partition_paused_and_another_is_not_resumed() {+ let settlement = budgeted_hourly_settlement_of_10_events_resuming_under_8();+ let p1 = TopicPartition::new("events", 1);+ charge(&settlement, &p0(), 10, 5);+ assert_eq!(charge(&settlement, &p0(), 11, 5), Some(Pause));+ charge(&settlement, &p1, 20, 1);++ settlement.settle_run(&p0(), 0, &[Offset(11)]).unwrap();+ settlement.settle_run(&p1, 0, &[Offset(20)]).unwrap();+ let commit = settlement.take_all().expect("partition 1 is due");+ assert_eq!(commit.offsets, HashMap::from([(p1, Offset(21))]));+ assert!(+ commit.resume.is_empty(),+ "offset 10 still holds partition 0"+ );++ settlement.settle_run(&p0(), 0, &[Offset(10)]).unwrap();+ assert_eq!(+ settlement.take_all().expect("partition 0 is due").resume,+ vec![p0()]+ );+ }++ #[test]+ fn a_revoked_paused_partition_is_never_resumed() {+ let settlement = budgeted_hourly_settlement_of_10_events_resuming_under_8();+ assert_eq!(charge(&settlement, &p0(), 10, 10), Some(Pause));+ settlement.settle_run(&p0(), 0, &[Offset(10)]).unwrap();++ settlement.forget_partitions(&[p0()]);++ assert!(settlement.take_all().is_none());+ settlement.settle_run(&p0(), 0, &[Offset(10)]).unwrap();+ assert!(+ settlement.take_all().is_none(),+ "the stale run moves nothing"+ );}}diff --git a/rust/ingestion-consumer/src/settlement/mod.rs b/rust/ingestion-consumer/src/settlement/mod.rsindex 890cd3913748..2bbf4d15608f 100644--- a/rust/ingestion-consumer/src/settlement/mod.rs+++ b/rust/ingestion-consumer/src/settlement/mod.rs@@ -1,5 +1,8 @@//! When accepted offsets reach the ledger and the commit pacer: per poll or per group.+mod budget;+#[cfg(test)]+mod fakes;mod group;mod poll;mod uncommitted;@@ -10,9 +13,13 @@ use std::time::{Duration, Instant};use common_kafka_consumer::{Charge, Held, Offset, Rejection, TopicOffsetLedger, TopicPartition};+use tracing::warn;+use crate::config::CompletionGranularity;use crate::order_sentinel::OffsetSpan;+use crate::partition_budget::BudgetLimits;+use budget::PartitionBudget;pub(crate) use group::GroupSettlement;pub(crate) use poll::PollSettlement;use uncommitted::Uncommitted;@@ -30,7 +37,7 @@ pub(crate) trait Settlement: Send + Sync {topic_partition: &TopicPartition,generation: u64,charges: &[(Offset, Charge)],- ) -> Result<Held, Rejection>;+ ) -> Result<Option<Pause>, Rejection>;/// Accepted offsets from one poll's slice of a partition, charged under `generation`.fn settle_run(@@ -43,21 +50,77 @@ pub(crate) trait Settlement: Send + Sync {/// The oldest poll retired with every message accepted.fn settle_poll(&self, slices: &[ChargedSlice<'_>]) -> anyhow::Result<()>;- fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, Offset>>;+ fn take_due(&self, now: Instant) -> Option<Commit>;++ fn take_all(&self) -> Option<Commit>;++ fn forget_partitions(&self, topic_partitions: &[TopicPartition]);+}++pub(crate) trait Accounting: Send + Sync {+ fn charge(+ &self,+ topic_partition: &TopicPartition,+ generation: u64,+ charges: &[(Offset, Charge)],+ ) -> Result<Held, Rejection>;+ fn settled(+ &self,+ topic_partition: &TopicPartition,+ generation: u64,+ offsets: impl IntoIterator<Item = Offset>,+ ) -> Result<Option<Offset>, Rejection>;+ fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, HandedOut>>;+ fn take_all(&self) -> Option<HashMap<TopicPartition, HandedOut>>;+ fn forget_partitions(&self, topic_partitions: &[TopicPartition]);+}++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub(crate) struct HandedOut {+ pub(crate) offset: Offset,+ pub(crate) remaining: Held,+}- fn take_all(&self) -> Option<HashMap<TopicPartition, Offset>>;+pub(crate) trait Admission: Send + Sync {+ fn charged(&self, topic_partition: &TopicPartition, uncommitted: Charge) -> Option<Pause>;+ fn handed_out(&self, handed_out: &HashMap<TopicPartition, HandedOut>) -> Vec<TopicPartition>;fn forget_partitions(&self, topic_partitions: &[TopicPartition]);}+#[derive(Debug, PartialEq, Eq)]+pub(crate) struct Pause;++pub(crate) struct Commit {+ pub(crate) offsets: HashMap<TopicPartition, Offset>,+ pub(crate) resume: Vec<TopicPartition>,+}++fn offsets(handed_out: HashMap<TopicPartition, HandedOut>) -> HashMap<TopicPartition, Offset> {+ handed_out+ .into_iter()+ .map(|(topic_partition, handed_out)| (topic_partition, handed_out.offset))+ .collect()+}+pub(crate) fn for_granularity(granularity: CompletionGranularity,ledger: Arc<TopicOffsetLedger>,commit_interval: Duration,+ limits: BudgetLimits,) -> Arc<dyn Settlement> {match granularity {- CompletionGranularity::Poll => Arc::new(PollSettlement::new(ledger)),- CompletionGranularity::Group => Arc::new(GroupSettlement::new(ledger, commit_interval)),+ CompletionGranularity::Poll => {+ if limits.is_set() {+ warn!(+ "Ignoring the partition budget: it needs CONSUMER_COMPLETION_GRANULARITY=group"+ );+ }+ Arc::new(PollSettlement::new(ledger))+ }+ CompletionGranularity::Group => {+ Arc::new(GroupSettlement::new(ledger, commit_interval, limits))+ }}}diff --git a/rust/ingestion-consumer/src/settlement/poll.rs b/rust/ingestion-consumer/src/settlement/poll.rsindex 1ac43e2a071f..7346a7b53568 100644--- a/rust/ingestion-consumer/src/settlement/poll.rs+++ b/rust/ingestion-consumer/src/settlement/poll.rs@@ -1,17 +1,16 @@-use std::collections::HashMap;use std::sync::Arc;use std::time::Instant;-use common_kafka_consumer::{Charge, Held, Offset, Rejection, TopicOffsetLedger, TopicPartition};+use common_kafka_consumer::{Charge, Offset, Rejection, TopicOffsetLedger, TopicPartition};use metrics::counter;use tracing::warn;-use super::{ChargedSlice, Settlement, Uncommitted};+use super::{offsets, Accounting, ChargedSlice, Commit, Pause, Settlement, Uncommitted};use crate::commit_pacer::CommitPacer;use crate::ledger_rejection::{report_rejection, RejectedSlice};-pub(crate) struct PollSettlement {- uncommitted: Uncommitted,+pub(crate) struct PollSettlement<A: Accounting = Uncommitted> {+ uncommitted: A,}impl PollSettlement {@@ -22,15 +21,16 @@ impl PollSettlement {}}-impl Settlement for PollSettlement {+impl<A: Accounting + 'static> Settlement for PollSettlement<A> {fn charge(&self,topic_partition: &TopicPartition,generation: u64,charges: &[(Offset, Charge)],- ) -> Result<Held, Rejection> {+ ) -> Result<Option<Pause>, Rejection> {self.uncommitted.charge(topic_partition, generation, charges)+ .map(|_| None)}fn settle_run(@@ -59,12 +59,20 @@ impl Settlement for PollSettlement {Ok(())}- fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, Offset>> {- self.uncommitted.take_due(now)+ fn take_due(&self, now: Instant) -> Option<Commit> {+ let handed_out = self.uncommitted.take_due(now)?;+ Some(Commit {+ offsets: offsets(handed_out),+ resume: Vec::new(),+ })}- fn take_all(&self) -> Option<HashMap<TopicPartition, Offset>> {- self.uncommitted.take_all()+ fn take_all(&self) -> Option<Commit> {+ let handed_out = self.uncommitted.take_all()?;+ Some(Commit {+ offsets: offsets(handed_out),+ resume: Vec::new(),+ })}fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {@@ -87,7 +95,7 @@ impl NoFrontier {}fn settle_slices(- uncommitted: &Uncommitted,+ uncommitted: &impl Accounting,slices: &[ChargedSlice<'_>],) -> anyhow::Result<Option<NoFrontier>> {let mut settled = false;@@ -121,6 +129,209 @@ fn settle_slices(#[cfg(test)]mod tests {+ use std::collections::HashMap;++ use common_kafka_consumer::{Held, LedgerError};+ use metrics_util::debugging::{DebugValue, DebuggingRecorder};++ use super::super::fakes::{FakeAccounting, Settled};+ use super::super::HandedOut;+ use super::*;+ use crate::order_sentinel::OffsetSpan;++ fn p(partition: i32) -> TopicPartition {+ TopicPartition::new("events", partition)+ }++ fn settlement() -> (PollSettlement<FakeAccounting>, FakeAccounting) {+ let accounting = FakeAccounting::default();+ let settlement = PollSettlement {+ uncommitted: accounting.clone(),+ };+ (settlement, accounting)+ }++ struct Slice {+ topic_partition: TopicPartition,+ charges: Vec<(Offset, Charge)>,+ }++ impl Slice {+ fn of(partition: i32, offsets: &[i64]) -> Self {+ Self {+ topic_partition: p(partition),+ charges: offsets+ .iter()+ .map(|&offset| (Offset(offset), Charge::ZERO))+ .collect(),+ }+ }++ fn charged(&self) -> ChargedSlice<'_> {+ ChargedSlice {+ topic_partition: &self.topic_partition,+ generation: 2,+ span: OffsetSpan {+ first: self.charges[0].0 .0,+ last: self.charges[self.charges.len() - 1].0 .0,+ },+ charges: &self.charges,+ }+ }+ }++ fn stale() -> Rejection {+ Rejection::Stale {+ stamp: 0,+ generation: 1,+ }+ }++ fn settle_poll_counting_skips(+ settlement: &PollSettlement<FakeAccounting>,+ slices: &[Slice],+ ) -> (anyhow::Result<()>, Vec<String>) {+ let recorder = DebuggingRecorder::new();+ let snapshotter = recorder.snapshotter();+ let charged: Vec<ChargedSlice<'_>> = slices.iter().map(Slice::charged).collect();+ let result = metrics::with_local_recorder(&recorder, || settlement.settle_poll(&charged));+ let skipped = snapshotter+ .snapshot()+ .into_vec()+ .into_iter()+ .filter(|(key, _, _, value)| {+ key.key().name() == "ingestion_consumer_commits_skipped_total"+ && matches!(value, DebugValue::Counter(count) if *count > 0)+ })+ .filter_map(|(key, _, _, _)| {+ key.key()+ .labels()+ .find(|label| label.key() == "reason")+ .map(|label| label.value().to_string())+ })+ .collect();+ (result, skipped)+ }++ #[test]+ fn a_retired_poll_settles_each_slice_under_its_generation() {+ let (settlement, accounting) = settlement();++ settle_poll_counting_skips(&settlement, &[Slice::of(0, &[10, 11]), Slice::of(1, &[20])])+ .0+ .unwrap();++ assert_eq!(+ accounting.settles_seen(),+ vec![+ Settled {+ topic_partition: p(0),+ generation: 2,+ offsets: vec![Offset(10), Offset(11)],+ },+ Settled {+ topic_partition: p(1),+ generation: 2,+ offsets: vec![Offset(20)],+ },+ ]+ );+ }++ struct RetiredPoll {+ settles: Vec<Result<Option<Offset>, Rejection>>,+ expected_skip: Option<&'static str>,+ }++ #[rstest::rstest]+ #[case::a_frontier(RetiredPoll { settles: vec![Ok(Some(Offset(12)))], expected_skip: None })]+ #[case::behind_an_earlier_poll(RetiredPoll { settles: vec![Ok(None)], expected_skip: Some("no_frontier") })]+ #[case::every_slice_stale(RetiredPoll { settles: vec![Err(stale())], expected_skip: Some("rejected") })]+ #[case::one_stale_one_frontier(RetiredPoll { settles: vec![Err(stale()), Ok(Some(Offset(21)))], expected_skip: None })]+ #[case::one_stale_one_behind(RetiredPoll { settles: vec![Err(stale()), Ok(None)], expected_skip: Some("no_frontier") })]+ fn a_retired_poll_that_hands_over_no_frontier_counts_why(#[case] poll: RetiredPoll) {+ let (settlement, accounting) = settlement();+ let slices: Vec<Slice> = (0..poll.settles.len() as i32)+ .map(|partition| Slice::of(partition, &[10]))+ .collect();+ for result in poll.settles {+ accounting.settle_returns(result);+ }++ let (result, skipped) = settle_poll_counting_skips(&settlement, &slices);++ result.unwrap();+ assert_eq!(+ skipped,+ poll.expected_skip+ .into_iter()+ .map(String::from)+ .collect::<Vec<_>>()+ );+ }++ #[test]+ fn a_violating_slice_fails_the_poll_and_stops_settling() {+ let (settlement, accounting) = settlement();+ accounting.settle_returns(Err(Rejection::Violation {+ error: LedgerError::CompletedTwice { offset: Offset(10) },+ stamp: 2,+ generation: 2,+ held: Held::default(),+ }));++ let (result, _) =+ settle_poll_counting_skips(&settlement, &[Slice::of(0, &[10]), Slice::of(1, &[20])]);++ assert!(result.is_err());+ assert_eq!(accounting.settles_seen().len(), 1);+ }++ #[test]+ fn a_run_settles_nothing_before_its_poll_retires() {+ let (settlement, accounting) = settlement();++ settlement.settle_run(&p(0), 2, &[Offset(10)]).unwrap();++ assert!(accounting.settles_seen().is_empty());+ }++ #[test]+ fn a_charge_never_pauses() {+ let (settlement, accounting) = settlement();+ accounting.charge_returns(Ok(Held {+ offsets: usize::MAX,+ charge: Charge {+ events: u64::MAX,+ bytes: u64::MAX,+ },+ }));++ assert_eq!(settlement.charge(&p(0), 2, &[]), Ok(None));+ }++ #[test]+ fn a_hand_out_commits_without_resuming() {+ let (settlement, accounting) = settlement();+ accounting.take_hands_out(HashMap::from([(+ p(0),+ HandedOut {+ offset: Offset(12),+ remaining: Held::default(),+ },+ )]));++ let commit = settlement.take_due(Instant::now()).expect("handed out");++ assert_eq!(commit.offsets, HashMap::from([(p(0), Offset(12))]));+ assert!(commit.resume.is_empty());+ }+}++#[cfg(test)]+mod integration_tests {+ use std::collections::HashMap;+use common_kafka_consumer::Charge;use super::*;@@ -176,7 +387,9 @@ mod tests {settlement.settle_poll(&[p0.slice(), p1.slice()]).unwrap();assert_eq!(- settlement.take_due(Instant::now()),+ settlement+ .take_due(Instant::now())+ .map(|commit| commit.offsets),Some(HashMap::from([(p0.topic_partition, Offset(12)),(p1.topic_partition, Offset(22)),@@ -195,7 +408,9 @@ mod tests {settlement.settle_poll(&[older.slice()]).unwrap();assert_eq!(- settlement.take_due(Instant::now()),+ settlement+ .take_due(Instant::now())+ .map(|commit| commit.offsets),Some(HashMap::from([(older.topic_partition, Offset(14))])));}@@ -211,25 +426,4 @@ mod tests {assert!(settlement.take_due(Instant::now()).is_none());}-- #[test]- fn an_accepted_run_settles_nothing_before_its_poll_retires() {- let (ledger, settlement) = settlement();- let poll = charged(&ledger, 0, 10, 11);-- settlement- .settle_run(- &poll.topic_partition,- poll.generation,- &[Offset(10), Offset(11)],- )- .unwrap();- assert!(settlement.take_due(Instant::now()).is_none());-- settlement.settle_poll(&[poll.slice()]).unwrap();- assert_eq!(- settlement.take_due(Instant::now()),- Some(HashMap::from([(poll.topic_partition, Offset(12))]))- );- }}diff --git a/rust/ingestion-consumer/src/settlement/uncommitted.rs b/rust/ingestion-consumer/src/settlement/uncommitted.rsindex c6c62a0ecab2..b1cf6bb87897 100644--- a/rust/ingestion-consumer/src/settlement/uncommitted.rs+++ b/rust/ingestion-consumer/src/settlement/uncommitted.rs@@ -5,6 +5,7 @@ use std::time::Instant;use common_kafka_consumer::{Charge, Held, Offset, Rejection, TopicOffsetLedger, TopicPartition};use metrics::gauge;+use super::{Accounting, HandedOut};use crate::commit_pacer::CommitPacer;const UNCOMMITTED_OFFSETS: &str = "kafka_consumer_ledger_uncommitted_offsets";@@ -17,7 +18,7 @@ const UNCOMMITTED_BYTES: &str = "kafka_consumer_ledger_uncommitted_bytes";////// Every call runs on the consumer loop task, one at a time, so a move between/// the ledger and the pacer never interleaves with a read of both.-pub(super) struct Uncommitted {+pub(crate) struct Uncommitted {ledger: Arc<TopicOffsetLedger>,pacer: Mutex<CommitPacer>,}@@ -30,7 +31,19 @@ impl Uncommitted {}}- pub(super) fn charge(+ fn hand_out(&self, due: HashMap<TopicPartition, Offset>) -> HashMap<TopicPartition, HandedOut> {+ due.into_iter()+ .map(|(topic_partition, offset)| {+ let remaining = self.ledger.held(&topic_partition);+ publish(&topic_partition, remaining);+ (topic_partition, HandedOut { offset, remaining })+ })+ .collect()+ }+}++impl Accounting for Uncommitted {+ fn charge(&self,topic_partition: &TopicPartition,generation: u64,@@ -44,11 +57,7 @@ impl Uncommitted {Ok(uncommitted)}- /// Called after a group run or a poll settles: mark its offsets complete- /// and move the frontier that results, with its charge, to the pacer.- /// Returns that frontier, `None` while the front of the window is still- /// incomplete.- pub(super) fn settled(+ fn settled(&self,topic_partition: &TopicPartition,generation: u64,@@ -66,22 +75,17 @@ impl Uncommitted {Ok(frontier)}- /// Refunds the charge of every frontier handed out; the consumer submits- /// their commit right after.- pub(super) fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, Offset>> {+ fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, HandedOut>> {let due = self.pacer.lock().unwrap().take_due(now)?;- self.publish_window(&due);- Some(due)+ Some(self.hand_out(due))}- /// Like `take_due`, without waiting for the interval.- pub(super) fn take_all(&self) -> Option<HashMap<TopicPartition, Offset>> {+ fn take_all(&self) -> Option<HashMap<TopicPartition, HandedOut>> {let due = self.pacer.lock().unwrap().take_all()?;- self.publish_window(&due);- Some(due)+ Some(self.hand_out(due))}- pub(super) fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {+ fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {self.ledger.forget_partitions(topic_partitions.iter()@@ -95,12 +99,6 @@ impl Uncommitted {publish(topic_partition, Held::default());}}-- fn publish_window(&self, due: &HashMap<TopicPartition, Offset>) {- for topic_partition in due.keys() {- publish(topic_partition, self.ledger.held(topic_partition));- }- }}fn publish(topic_partition: &TopicPartition, uncommitted: Held) {@@ -155,12 +153,23 @@ mod tests {Uncommitted::new(Arc::new(TopicOffsetLedger::new()), CommitPacer::immediate())}- fn published(snapshotter: &Snapshotter) -> Held {+ fn published(snapshotter: &Snapshotter, topic_partition: &TopicPartition) -> Held {+ let partition = topic_partition.partition.to_string();let mut published = Held::default();for (key, _, _, value) in snapshotter.snapshot().into_vec() {let DebugValue::Gauge(value) = value else {continue;};+ let labelled =+ key.key()+ .labels()+ .any(|label| label.key() == "partition" && label.value() == partition)+ && key.key().labels().any(|label| {+ label.key() == "topic" && label.value() == topic_partition.topic+ });+ if !labelled {+ continue;+ }let value = value.into_inner();match key.key().name() {UNCOMMITTED_OFFSETS => published.offsets = value as usize,@@ -193,7 +202,17 @@ mod tests {assert_eq!(charge(&uncommitted, 0, 11), held(2));- uncommitted.take_due(Instant::now()).expect("due");+ let handed_out = uncommitted.take_due(Instant::now()).expect("due");+ assert_eq!(+ handed_out,+ HashMap::from([(+ p0(),+ HandedOut {+ offset: Offset(11),+ remaining: held(1),+ }+ )])+ );assert_eq!(charge(&uncommitted, 0, 12), held(2));}@@ -228,6 +247,48 @@ mod tests {assert_eq!(charge_p1(22), held(3));}+ #[test]+ fn a_hand_out_carries_each_partitions_own_remaining_charge() {+ let recorder = DebuggingRecorder::new();+ let snapshotter = recorder.snapshotter();+ let _guard = metrics::set_default_local_recorder(&recorder);+ let uncommitted = uncommitted();+ let p1 = TopicPartition::new("events", 1);+ charge(&uncommitted, 0, 10);+ charge(&uncommitted, 0, 11);+ for offset in [20, 21, 22] {+ uncommitted+ .charge(&p1, 0, &[(Offset(offset), EACH)])+ .expect("the ledger accepts the charge");+ }+ settled(&uncommitted, &[10]);+ uncommitted+ .settled(&p1, 0, [Offset(20)])+ .expect("the ledger settles");++ assert_eq!(+ uncommitted.take_due(Instant::now()).expect("due"),+ HashMap::from([+ (+ p0(),+ HandedOut {+ offset: Offset(11),+ remaining: held(1),+ }+ ),+ (+ p1.clone(),+ HandedOut {+ offset: Offset(21),+ remaining: held(2),+ }+ ),+ ])+ );+ assert_eq!(published(&snapshotter, &p0()), held(1));+ assert_eq!(published(&snapshotter, &p1), held(2));+ }+#[test]fn a_forgotten_partition_drops_its_window_and_its_pending_take() {let uncommitted = uncommitted();@@ -274,15 +335,19 @@ mod tests {charge(&uncommitted, 0, 10);charge(&uncommitted, 0, 11);- assert_eq!(published(&snapshotter), held(2));+ assert_eq!(published(&snapshotter, &p0()), held(2));settled(&uncommitted, &[10]);- assert_eq!(published(&snapshotter), held(2), "a take moves the charge");+ assert_eq!(+ published(&snapshotter, &p0()),+ held(2),+ "a take moves the charge"+ );uncommitted.take_due(Instant::now()).expect("due");- assert_eq!(published(&snapshotter), held(1));+ assert_eq!(published(&snapshotter, &p0()), held(1));uncommitted.forget_partitions(&[p0()]);- assert_eq!(published(&snapshotter), Held::default());+ assert_eq!(published(&snapshotter, &p0()), Held::default());}}diff --git a/rust/ingestion-consumer/tests/e2e_integration_test.rs b/rust/ingestion-consumer/tests/e2e_integration_test.rsindex 6378827308b9..d28543db0aa3 100644--- a/rust/ingestion-consumer/tests/e2e_integration_test.rs+++ b/rust/ingestion-consumer/tests/e2e_integration_test.rs@@ -6,7 +6,7 @@mod common;use std::collections::HashSet;-use std::num::{NonZeroU32, NonZeroUsize};+use std::num::{NonZeroU32, NonZeroU64, NonZeroUsize};use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};use std::sync::{Arc, Mutex, OnceLock};use std::time::Duration;@@ -38,6 +38,7 @@ use ingestion_consumer::config::CompletionGranularity;use ingestion_consumer::consumer::{IngestionConsumer, IngestionConsumerOptions};use ingestion_consumer::discovery::reconcile_membership;use ingestion_consumer::grpc_transport::{GrpcPort, GrpcTransport};+use ingestion_consumer::partition_budget::BudgetLimits;use ingestion_consumer::rebalance::RebalanceContext;use ingestion_consumer::routing::RoutingStrategy;use ingestion_consumer::worker_registry::{@@ -719,6 +720,7 @@ impl Harness {None,CompletionGranularity::Poll,COMMIT_INTERVAL,+ BudgetLimits::unset(),).await}@@ -751,6 +753,7 @@ impl Harness {}),CompletionGranularity::Poll,COMMIT_INTERVAL,+ BudgetLimits::unset(),).await}@@ -778,6 +781,7 @@ impl Harness {None,CompletionGranularity::Poll,COMMIT_INTERVAL,+ BudgetLimits::unset(),).await}@@ -804,6 +808,7 @@ impl Harness {None,CompletionGranularity::Poll,COMMIT_INTERVAL,+ BudgetLimits::unset(),).await}@@ -825,6 +830,7 @@ impl Harness {None,CompletionGranularity::Poll,COMMIT_INTERVAL,+ BudgetLimits::unset(),).await}@@ -868,6 +874,51 @@ impl Harness {None,CompletionGranularity::Group,commit_interval,+ BudgetLimits::unset(),+ )+ .await+ }++ async fn start_with_partition_budget(+ pack_targets: PackTargets,+ topic: &str,+ partitions: i32,+ worker_count: usize,+ partition_budget: BudgetLimits,+ ) -> Self {+ Self::start_with_partition_budget_committing_every(+ pack_targets,+ topic,+ partitions,+ worker_count,+ partition_budget,+ COMMIT_INTERVAL,+ )+ .await+ }++ async fn start_with_partition_budget_committing_every(+ pack_targets: PackTargets,+ topic: &str,+ partitions: i32,+ worker_count: usize,+ partition_budget: BudgetLimits,+ commit_interval: Duration,+ ) -> Self {+ Self::start_inner(+ topic,+ partitions,+ worker_count,+ 8,+ Duration::from_secs(60),+ fast_registry_config(),+ 0,+ ComponentOptions::new(),+ pack_targets,+ None,+ CompletionGranularity::Group,+ commit_interval,+ partition_budget,).await}@@ -886,6 +937,7 @@ impl Harness {budget: Option<RequestBudget>,completion_granularity: CompletionGranularity,commit_interval: Duration,+ partition_budget: BudgetLimits,) -> Self {create_topic(topic, partitions).await;@@ -941,6 +993,7 @@ impl Harness {completion_granularity,commit_interval,ledger: Arc::clone(&ledger),+ partition_budget,},handle,);@@ -1033,6 +1086,7 @@ impl Harness {completion_granularity: self.completion_granularity,commit_interval: COMMIT_INTERVAL,ledger: Arc::clone(&self.ledger),+ partition_budget: BudgetLimits::unset(),},handle,);@@ -1124,6 +1178,7 @@ impl Harness {completion_granularity: self.completion_granularity,commit_interval: COMMIT_INTERVAL,ledger: Arc::new(TopicOffsetLedger::new()),+ partition_budget: BudgetLimits::unset(),},handle,);@@ -2790,6 +2845,165 @@ async fn group_granularity_commits_a_partition_while_another_stalls() {harness.stop().await;}+#[tokio::test]+async fn a_partition_at_its_budget_pauses_until_a_commit_frees_it() {+ let topic = format!("e2e-partition-budget-{}", Uuid::new_v4());+ let budget = BudgetLimits::new(NonZeroU64::new(5), None, 0.5).expect("a valid budget");+ let harness =+ Harness::start_with_partition_budget(common::PRODUCTION_PACK_TARGETS, &topic, 1, 1, budget)+ .await;+ let p0 = TopicPartition::new(&topic, 0);+ let guard = harness.workers[0].block().await;+ let producer = make_producer();++ for seq in 0..5usize {+ produce(&producer, &topic, 0, "tok", "user-a", seq).await;+ }+ wait_until(Duration::from_secs(10), "the budget to fill", || {+ harness.ledger.held(&p0).offsets == 5+ })+ .await;++ for seq in 5..20usize {+ produce(&producer, &topic, 0, "tok", "user-a", seq).await;+ }+ tokio::time::sleep(Duration::from_secs(1)).await;+ assert_eq!(+ harness.ledger.held(&p0).offsets,+ 5,+ "a paused partition fetches nothing more"+ );++ drop(guard);+ wait_until(+ Duration::from_secs(10),+ "the partition to resume and drain",+ || harness.committed_offset(0) == Some(20),+ )+ .await;++ harness.stop().await;+}++#[tokio::test]+async fn a_partition_at_its_budget_does_not_pause_another() {+ let topic = format!("e2e-partition-budget-two-{}", Uuid::new_v4());+ let budget = BudgetLimits::new(NonZeroU64::new(5), None, 0.5).expect("a valid budget");+ // One message per request, or the packer puts both keys on one worker.+ let harness =+ Harness::start_with_partition_budget(common::ONE_MESSAGE_PER_REQUEST, &topic, 2, 2, budget)+ .await;+ let mut guards = stall_two_partitions_on_two_workers(&harness, &topic).await;++ guards[0] = None;+ wait_until(+ Duration::from_secs(10),+ "the released worker to accept its key",+ || harness.workers[0].count() == 5,+ )+ .await;+ let released = accepted_partition(&harness, 0);+ let stalled = 1 - released;+ let stalled_partition = TopicPartition::new(&topic, stalled);++ let producer = make_producer();+ for seq in 5..10usize {+ for (partition, key) in STALL_KEYS {+ produce(&producer, &topic, partition, "tok", key, seq).await;+ }+ }+ wait_until(+ Duration::from_secs(10),+ "the released partition to resume and drain",+ || harness.committed_offset(released) == Some(10),+ )+ .await;+ assert_eq!(+ harness.ledger.held(&stalled_partition).offsets,+ 5,+ "the partition at its budget fetches nothing more"+ );+ assert_eq!(harness.committed_offset(stalled), None);++ guards[1] = None;+ wait_until(+ Duration::from_secs(10),+ "the stalled partition to resume and drain",+ || harness.committed_offset(stalled) == Some(10),+ )+ .await;++ harness.stop().await;+}++#[tokio::test]+async fn a_partition_paused_at_its_budget_fetches_again_after_a_rebalance_returns_it() {+ let topic = format!("e2e-partition-budget-rebalance-{}", Uuid::new_v4());+ let budget = BudgetLimits::new(NonZeroU64::new(5), None, 0.5).expect("a valid budget");+ let harness = Harness::start_with_partition_budget_committing_every(+ common::PRODUCTION_PACK_TARGETS,+ &topic,+ 1,+ 1,+ budget,+ Duration::from_secs(20),+ )+ .await;+ let p0 = TopicPartition::new(&topic, 0);+ let producer = make_producer();++ produce(&producer, &topic, 0, "tok", "user-a", 0).await;+ wait_until(+ Duration::from_secs(10),+ "the first frontier to commit",+ || harness.committed_offset(0) == Some(1),+ )+ .await;+ for seq in 1..6usize {+ produce(&producer, &topic, 0, "tok", "user-a", seq).await;+ }+ wait_until(+ Duration::from_secs(10),+ "the interval to hold the budget's worth of settled offsets",+ || harness.workers[0].count() == 6 && harness.ledger.held(&p0).offsets == 0,+ )+ .await;++ let generation_before = harness.ledger.generation(&p0);+ let second = harness.join_second_consumer().await;+ wait_until(+ Duration::from_secs(15),+ "the rebalance to revoke the partition",+ || harness.ledger.generation(&p0) > generation_before,+ )+ .await;+ assert_eq!(+ harness.committed_offset(0),+ Some(1),+ "the partition must still be at its budget when it is revoked"+ );+ second.stop().await;++ for seq in 6..10usize {+ produce(&producer, &topic, 0, "tok", "user-a", seq).await;+ }+ wait_until(+ Duration::from_secs(40),+ "the returned partition to fetch again",+ || {+ let delivered: HashSet<usize> = harness+ .workers+ .iter()+ .flat_map(|w| w.seqs_for("user-a"))+ .collect();+ (6..10).all(|seq| delivered.contains(&seq))+ },+ )+ .await;++ harness.stop().await;+}+/// A worker that processes a batch but whose ACK is lost is indistinguishable/// from a failed send, so the consumer replays the batch: duplicates are the/// accepted cost of at-least-once, loss never is, and the replay lands behind@@ -3502,6 +3716,7 @@ async fn fenced_static_member_exits_on_fatal_error() {completion_granularity: CompletionGranularity::Poll,commit_interval: Duration::ZERO,ledger: Arc::new(TopicOffsetLedger::new()),+ partition_budget: BudgetLimits::unset(),},handle,);