Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(ingestion-consumer): pause a partition at its budget and resume it on commit (#118697)

Paweł Ledwoń1ad5c63c0aa3
diff --git a/rust/ingestion-consumer/README.md b/rust/ingestion-consumer/README.md
index 7c91dafdd58e..2b989da80774 100644
--- a/rust/ingestion-consumer/README.md
+++ b/rust/ingestion-consumer/README.md
@@ -50,6 +50,18 @@ The ledger emits its own counters, so any consumer built on the crate reports th
`kafka_consumer_ledger_stale_slices_total{stage}` counts charges and settlements dropped because their partition was reassigned while they were in flight; a few around a rebalance are expected.
`kafka_consumer_ledger_errors_total{stage,kind}` counts contract violations in the ledger's accounting; it must stay 0. A violation stops the consumer, and the restart replays from the last commit: the ledger resets the partition's window, so offsets in flight below its next delivery would never complete and a later commit would move past them. The consumer logs the rejected slice with the batch and ledger generations and the window depth before it stops.
+## Partition budget
+
+`CONSUMER_PARTITION_BUDGET_EVENTS` and `CONSUMER_PARTITION_BUDGET_BYTES` bound each partition's replay exposure: the events and bytes the settlement reports as uncommitted for it.
+A budget needs `CONSUMER_COMPLETION_GRANULARITY=group`. At `poll` a stalled partition holds back the commits of every partition in its poll, so their budgets would fill and pause together. The consumer ignores a budget at `poll` and logs a warning, so switching a lane back to `poll` rolls it back without unsetting the budget.
+When either axis reaches its limit after a poll, the consumer pauses that partition's fetch through the Kafka client; when a commit brings both axes back under `CONSUMER_PARTITION_BUDGET_LOW_WATERMARK` (a fraction of the budget, default 0.8), it resumes.
+The hysteresis matters because a pause purges the partition's fetched messages from librdkafka's queue and a resume refetches them.
+A pause stops only the partition's fetch: the consumer keeps polling, so rebalance callbacks still run and `max.poll.interval.ms` stays satisfied, except while the loop waits on its oldest in-flight poll.
+A revoked partition is forgotten. librdkafka keeps a partition's pause across a revoke and a reassignment, so the consumer resumes every partition it is assigned, and the partition starts with a fresh ledger.
+Both limits default to `0`, which sets no budget; the admission cap (`CONSUMER_MAX_BACKGROUND_TASKS`) then governs alone.
+The budget is a fixed amount per partition, not a share of a pod-wide amount, so a rebalance changes how many partitions a pod holds, not how much each may expose.
+`ingestion_consumer_budget_paused_partitions` gauges how many partitions are paused and `ingestion_consumer_budget_pauses_total{topic,partition}` counts pauses; read them against `ingestion_lag_ms`.
+
## Batcher
The batcher state machine in `src/batcher/` orders runs and places them on workers.
diff --git a/rust/ingestion-consumer/docs/driver-model-plan.md b/rust/ingestion-consumer/docs/driver-model-plan.md
index a701273d8540..826d098c3ccc 100644
--- a/rust/ingestion-consumer/docs/driver-model-plan.md
+++ b/rust/ingestion-consumer/docs/driver-model-plan.md
@@ -101,7 +101,7 @@ A change that adds metrics lists their names.
| 22 | 8 decoupled commits | Cleanup | Delete the per-poll completion path |
| 23 | 9 bounded replay | Implement | Add the budget accounting |
| 24 | 9 bounded replay | Switchover | Enable the budget |
-| 25 | 9 bounded replay | Cleanup | Remove the in-flight admission cap |
+| 25 | 9 bounded replay | Structure | Poll continuously and remove the in-flight admission cap |
| 26 | 10 clean handoff | Prepare | Add revoke and drained markers |
| 27 | 10 clean handoff | Implement | Build the drain |
| 28 | 10 clean handoff | Switchover | Enable the drain |
@@ -601,7 +601,8 @@ Exit criterion at the canary: zero ledger errors and gap walks, and a stalled pa
**Interfaces:**
-- Remove the ordered `InFlightBatch` queue, the per-poll aggregation from change 7, and the completion granularity from `Config`.
+- Remove `PollSettlement` and the completion granularity from `Config`.
+- Keep the in-flight poll queue and its per-poll aggregation from change 7. The loop waits on the oldest poll until change 25 replaces it.
## Cycle 9: replay exposure is bounded per partition
@@ -641,7 +642,7 @@ The budget is per partition, and so is the pause. One budget for the whole assig
- A charts values PR sets the budget lane by lane. The admission cap and the budget both bound work during the transition.
- Derive the first values from the current config: the cap times the batch size, divided by the partitions a pod holds, for events; the byte bound likewise for bytes. The per-pod bound then does not change at the switchover.
- Pause a partition when its outstanding charge reaches the budget on either axis. Resume it only when its charge is back at or under the low watermark on both axes. A pause purges that partition's fetched messages from librdkafka's queue and a resume refetches them, so a gate that reopens on the first refund refetches on every crossing.
-- Keep polling while partitions are paused. A paused partition still serves rebalance callbacks and keeps `max.poll.interval.ms` alive; an unpolled consumer does neither. An assignment resets librdkafka's pause flags, so a partition that arrives starts unpaused, with a fresh ledger generation and zero charge.
+- Keep polling while some partitions are paused. A paused partition still serves rebalance callbacks and keeps `max.poll.interval.ms` alive; an unpolled consumer does neither. While every partition is paused, the loop waits on completions without polling Kafka, as it already does when the cap is full; change 25 removes that wait. librdkafka keeps a partition's pause across a revoke and a reassignment, so the consumer resumes every partition it is assigned: a partition that arrives starts unpaused, with a fresh ledger generation and zero charge.
- Watch the paused-partitions gauge and `ingestion_lag_ms`. Rollback is unsetting the values.
- The budget is a fixed amount per partition, not a share of a pod-wide amount. A rebalance changes how many partitions a pod holds, not how much each one may expose, so the value does not move at rebalances.
@@ -655,17 +656,22 @@ The budget is per partition, and so is the pause. One budget for the whole assig
- Add `ingestion_consumer_budget_paused_partitions` (gauge) and `ingestion_consumer_budget_pauses_total{topic,partition}` (counter).
-### 25. Remove the in-flight admission cap (cleanup)
+### 25. Poll continuously and remove the in-flight admission cap (structure)
-**Task:** Remove `max_in_flight_batches` after the budget is stable on all lanes.
+**Task:** After the budget is stable on all lanes, replace the poll-and-wait loop with one loop that polls Kafka continuously and applies completions as they arrive. Remove `max_in_flight_batches`.
-**Goal:** The budget is the only admission bound.
+**Goal:** The budget is the only admission bound. No wait on one poll holds back fetching, completions, or commits.
+- The loop and the cap change together. The loop applies completions only while it waits for the oldest poll, and under steady traffic a full cap is what makes it wait. Without the cap and with the old loop, completions stay unread until every partition pauses at its budget.
+- One `select!` serves the Kafka poll, the completions, the batcher's errors, the commit tick, and shutdown. Collection must be safe to cancel: the loop keeps the poll it is building in its own state and adds each message when `recv()` returns it, so a branch that wins the race drops no message.
+- The commit tick fires when the pacer is next due, so a refund and the resume it allows go out on time.
+- Kafka is polled while every partition is paused, so rebalance callbacks and `max.poll.interval.ms` no longer depend on completions.
- Only admission control goes. The collection bounds — batch size and batch timeout — stay: they shape one poll, not the amount of admitted work.
-- The in-flight poll count goes with the cap. After change 22 the polls exist only to be counted.
+- Nothing tracks a poll after it is submitted. The budget bounds the admitted work, so the loop does not count polls or wait for one to finish.
**Interfaces:**
+- Modify `IngestionConsumer::run`: one select loop over collection, completions, errors, the commit tick, and shutdown. Remove `complete_oldest_poll`, the in-flight poll queue, and the per-poll aggregation.
- Remove `max_in_flight_batches` from `Config` and the admission bookkeeping from `IngestionConsumer`.
- Remove `ingestion_consumer_in_flight_batches`. The budget gauges replace it.
diff --git a/rust/ingestion-consumer/src/commit_pacer.rs b/rust/ingestion-consumer/src/commit_pacer.rs
index 4a46756a79cc..8712e075ce32 100644
--- a/rust/ingestion-consumer/src/commit_pacer.rs
+++ b/rust/ingestion-consumer/src/commit_pacer.rs
@@ -190,6 +190,16 @@ mod tests {
);
}
+ #[test]
+ fn an_empty_take_does_not_start_the_interval() {
+ let mut pacer = CommitPacer::every(Duration::from_millis(500));
+ let start = Instant::now();
+ assert!(pacer.take_due(start).is_none());
+
+ pacer.advance_frontier(&tp(0), taken(0, 10));
+ assert!(pacer.take_due(start + Duration::from_millis(100)).is_some());
+ }
+
#[test]
fn take_all_ignores_the_interval() {
let mut pacer = CommitPacer::every(Duration::from_secs(3600));
diff --git a/rust/ingestion-consumer/src/config.rs b/rust/ingestion-consumer/src/config.rs
index 6c3dd2b39b67..5966cffd4ab0 100644
--- a/rust/ingestion-consumer/src/config.rs
+++ b/rust/ingestion-consumer/src/config.rs
@@ -1,4 +1,4 @@
-use std::num::{NonZeroU32, NonZeroUsize};
+use std::num::{NonZeroU32, NonZeroU64, NonZeroUsize};
use std::str::FromStr;
use std::time::{Duration, Instant};
@@ -13,6 +13,7 @@ use crate::batcher::retry_policy::RetryPolicy;
use crate::batcher::state_machine::{BatcherStateMachine, RequestBudget};
use crate::batcher::worker_assigner::WorkerAssigner;
use crate::discovery::DiscoveryMode;
+use crate::partition_budget::BudgetLimits;
use crate::routing::{Router, RoutingStrategy};
use common_kafka_consumer::config::ConsumerConfigBuilder;
@@ -255,6 +256,22 @@ pub struct Config {
#[envconfig(from = "CONSUMER_COMMIT_INTERVAL_MS", default = "500")]
pub consumer_commit_interval_ms: u64,
+ /// Uncommitted events at which a partition's fetch pauses. `0` (default)
+ /// sets no event limit. Not read at `poll` granularity.
+ #[envconfig(from = "CONSUMER_PARTITION_BUDGET_EVENTS", default = "0")]
+ pub consumer_partition_budget_events: u64,
+
+ /// Uncommitted key-plus-value-plus-header bytes at which a partition's
+ /// fetch pauses. `0` (default) sets no byte limit. Not read at `poll`
+ /// granularity.
+ #[envconfig(from = "CONSUMER_PARTITION_BUDGET_BYTES", default = "0")]
+ pub consumer_partition_budget_bytes: u64,
+
+ /// Fraction of the budget a paused partition must be back under, on
+ /// every set axis, before its fetch resumes. Above 0 and at most 1.
+ #[envconfig(from = "CONSUMER_PARTITION_BUDGET_LOW_WATERMARK", default = "0.8")]
+ pub consumer_partition_budget_low_watermark: f64,
+
// ---- Debug API ----
/// Serve the real-time debug API (`/debug/load`, `/debug/state`,
/// `/debug/events` SSE) on the health server, recording structured events
@@ -503,6 +520,14 @@ impl Config {
.with_request_budget(self.request_budget()?))
}
+ pub fn partition_budget(&self) -> Result<BudgetLimits, String> {
+ BudgetLimits::new(
+ NonZeroU64::new(self.consumer_partition_budget_events),
+ NonZeroU64::new(self.consumer_partition_budget_bytes),
+ self.consumer_partition_budget_low_watermark,
+ )
+ }
+
const MIN_REQUEST_SOFT_BUDGET_MS: u64 = 100;
pub fn request_budget(&self) -> Result<Option<RequestBudget>, String> {
@@ -698,6 +723,23 @@ mod tests {
assert!(config(vars).request_budget().is_err());
}
+ #[rstest]
+ #[case::unset(&[], false)]
+ #[case::events_only(&[("CONSUMER_PARTITION_BUDGET_EVENTS", "5000")], true)]
+ #[case::bytes_only(&[("CONSUMER_PARTITION_BUDGET_BYTES", "1048576")], true)]
+ fn a_valid_partition_budget_is_read(#[case] vars: &[(&str, &str)], #[case] is_set: bool) {
+ assert_eq!(config(vars).partition_budget().unwrap().is_set(), is_set);
+ }
+
+ #[test]
+ fn a_partition_budget_with_an_invalid_watermark_is_rejected() {
+ let vars = &[
+ ("CONSUMER_PARTITION_BUDGET_EVENTS", "5000"),
+ ("CONSUMER_PARTITION_BUDGET_LOW_WATERMARK", "1.2"),
+ ];
+ assert!(config(vars).partition_budget().is_err());
+ }
+
#[test]
fn completion_granularity_parses_known_values() {
assert_eq!(
diff --git a/rust/ingestion-consumer/src/consumer.rs b/rust/ingestion-consumer/src/consumer.rs
index e7cb1f5ac850..07b77244e19c 100644
--- a/rust/ingestion-consumer/src/consumer.rs
+++ b/rust/ingestion-consumer/src/consumer.rs
@@ -1,4 +1,5 @@
use std::collections::{HashMap, HashSet, VecDeque};
+use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
@@ -23,8 +24,9 @@ use crate::discovery::DiscoveryMode;
use crate::grpc_transport::GrpcTransport;
use crate::ledger_rejection::{report_rejection, RejectedSlice};
use crate::order_sentinel::OffsetSpan;
-use crate::rebalance::{PartitionHook, RebalanceContext};
-use crate::settlement::{self, ChargedSlice, Settlement};
+use crate::partition_budget::BudgetLimits;
+use crate::rebalance::{partition_error, PartitionHook, RebalanceContext};
+use crate::settlement::{self, ChargedSlice, Commit, Pause, Settlement};
use crate::types::{Accumulator, SerializedKafkaMessage};
/// Batch-wide statistics gathered while collecting, used to emit parity
@@ -229,6 +231,67 @@ struct RevokedPartition {
generation: u64,
}
+#[derive(Default)]
+struct Partitions {
+ fetching: HashSet<TopicPartition>,
+ paused: HashSet<TopicPartition>,
+}
+
+#[derive(Default)]
+struct FetchingPartitions {
+ partitions: Mutex<Partitions>,
+ all_paused: AtomicBool,
+}
+
+impl FetchingPartitions {
+ fn assigned(&self, topic_partitions: &[TopicPartition]) {
+ self.update(|partitions| {
+ for topic_partition in topic_partitions {
+ partitions.paused.remove(topic_partition);
+ partitions.fetching.insert(topic_partition.clone());
+ }
+ });
+ }
+
+ fn revoked(&self, topic_partitions: &[TopicPartition]) {
+ self.update(|partitions| {
+ for topic_partition in topic_partitions {
+ partitions.paused.remove(topic_partition);
+ partitions.fetching.remove(topic_partition);
+ }
+ });
+ }
+
+ fn paused(&self, topic_partition: &TopicPartition) {
+ self.update(|partitions| {
+ if partitions.fetching.remove(topic_partition) {
+ partitions.paused.insert(topic_partition.clone());
+ }
+ });
+ }
+
+ fn resumed(&self, topic_partition: &TopicPartition) {
+ self.update(|partitions| {
+ if partitions.paused.remove(topic_partition) {
+ partitions.fetching.insert(topic_partition.clone());
+ }
+ });
+ }
+
+ fn all_paused(&self) -> bool {
+ self.all_paused.load(Ordering::Relaxed)
+ }
+
+ fn update(&self, change: impl FnOnce(&mut Partitions)) {
+ let mut partitions = self.partitions.lock().unwrap();
+ change(&mut partitions);
+ self.all_paused.store(
+ !partitions.paused.is_empty() && partitions.fetching.is_empty(),
+ Ordering::Relaxed,
+ );
+ }
+}
+
/// The consumer's revocation hook: purge the batcher, and record the revoked
/// partitions, because the purge means their polls never complete.
fn revoke_hook(
@@ -236,9 +299,11 @@ fn revoke_hook(
ledger: Arc<TopicOffsetLedger>,
settlement: Arc<dyn Settlement>,
revoked_partitions: Arc<Mutex<Vec<RevokedPartition>>>,
+ fetching: Arc<FetchingPartitions>,
) -> PartitionHook {
Box::new(move |partitions| {
settlement.forget_partitions(partitions);
+ fetching.revoked(partitions);
let pairs: Vec<(String, i32)> = partitions
.iter()
.map(|tp| (tp.topic.clone(), tp.partition))
@@ -256,11 +321,13 @@ fn revoke_hook(
fn assign_hook(
settlement: Arc<dyn Settlement>,
+ fetching: Arc<FetchingPartitions>,
assignment_epoch: AssignmentEpoch,
) -> PartitionHook {
Box::new(move |partitions| {
// Also covers a loss with no revoke callback (an error rebalance, a fenced member).
settlement.forget_partitions(partitions);
+ fetching.assigned(partitions);
assignment_epoch.bump();
})
}
@@ -271,16 +338,22 @@ fn wire_rebalance_hooks(
ledger: &Arc<TopicOffsetLedger>,
settlement: &Arc<dyn Settlement>,
assignment_epoch: AssignmentEpoch,
-) -> Arc<Mutex<Vec<RevokedPartition>>> {
+) -> (Arc<Mutex<Vec<RevokedPartition>>>, Arc<FetchingPartitions>) {
let revoked_partitions = Arc::new(Mutex::new(Vec::new()));
+ let fetching = Arc::new(FetchingPartitions::default());
context.set_on_revoke(revoke_hook(
revoker,
Arc::clone(ledger),
Arc::clone(settlement),
Arc::clone(&revoked_partitions),
+ Arc::clone(&fetching),
));
- context.set_on_assign(assign_hook(Arc::clone(settlement), assignment_epoch));
- revoked_partitions
+ context.set_on_assign(assign_hook(
+ Arc::clone(settlement),
+ Arc::clone(&fetching),
+ assignment_epoch,
+ ));
+ (revoked_partitions, fetching)
}
/// Remove revoked partitions from in-flight polls, and remove polls left
@@ -412,6 +485,18 @@ fn pop_committable(in_flight: &mut VecDeque<InFlightPoll>) -> anyhow::Result<InF
Ok(poll)
}
+fn commit_tick(granularity: CompletionGranularity, commit_interval: Duration) -> Duration {
+ const HEALTH_TICK: Duration = Duration::from_secs(1);
+ // A commit can resume a partition, so a tick longer than the commit
+ // interval delays the resume.
+ match granularity {
+ CompletionGranularity::Group if !commit_interval.is_zero() => {
+ commit_interval.min(HEALTH_TICK)
+ }
+ _ => HEALTH_TICK,
+ }
+}
+
/// Options for constructing an [`IngestionConsumer`] from pre-built parts.
/// Used in integration tests where the Kafka consumer is created externally.
pub struct IngestionConsumerOptions {
@@ -427,6 +512,7 @@ pub struct IngestionConsumerOptions {
pub completion_granularity: CompletionGranularity,
pub commit_interval: Duration,
pub ledger: Arc<TopicOffsetLedger>,
+ pub partition_budget: BudgetLimits,
}
/// The main consumer loop: reads from Kafka, demuxes each poll into groups,
@@ -443,6 +529,7 @@ pub struct IngestionConsumer {
batch_size_bytes: usize,
batch_timeout: Duration,
max_in_flight_batches: usize,
+ commit_tick: Duration,
handle: Handle,
group_id: String,
/// Debug event recorder; `None` unless `DEBUG_API_ENABLED`.
@@ -451,6 +538,7 @@ pub struct IngestionConsumer {
/// Partitions revoked since the loop last looked, fed by the rebalance
/// callback.
revoked_partitions: Arc<Mutex<Vec<RevokedPartition>>>,
+ fetching: Arc<FetchingPartitions>,
settlement: Arc<dyn Settlement>,
}
@@ -471,8 +559,9 @@ impl IngestionConsumer {
options.completion_granularity,
Arc::clone(&topic_offset_ledger),
options.commit_interval,
+ options.partition_budget,
);
- let revoked_partitions = wire_rebalance_hooks(
+ let (revoked_partitions, fetching) = wire_rebalance_hooks(
consumer.context(),
batcher.revoker(),
&topic_offset_ledger,
@@ -484,6 +573,7 @@ impl IngestionConsumer {
debug_recorder: options.debug_recorder,
topic_offset_ledger,
revoked_partitions,
+ fetching,
settlement,
consumer,
batcher,
@@ -494,6 +584,7 @@ impl IngestionConsumer {
batch_size_bytes: options.batch_size_bytes,
batch_timeout: options.batch_timeout,
max_in_flight_batches: options.max_in_flight_batches.max(1),
+ commit_tick: commit_tick(options.completion_granularity, options.commit_interval),
handle,
group_id: options.group_id,
}
@@ -534,9 +625,12 @@ impl IngestionConsumer {
config.consumer_completion_granularity,
Arc::clone(&topic_offset_ledger),
Duration::from_millis(config.consumer_commit_interval_ms),
+ config
+ .partition_budget()
+ .map_err(|err| anyhow::anyhow!("invalid partition budget: {err}"))?,
);
let context = RebalanceContext::new();
- let revoked_partitions = wire_rebalance_hooks(
+ let (revoked_partitions, fetching) = wire_rebalance_hooks(
&context,
batcher.revoker(),
&topic_offset_ledger,
@@ -563,6 +657,7 @@ impl IngestionConsumer {
debug_recorder,
topic_offset_ledger,
revoked_partitions,
+ fetching,
settlement,
batcher,
outputs: Some(outputs),
@@ -572,6 +667,10 @@ impl IngestionConsumer {
batch_size_bytes: config.consumer_batch_size_kb.saturating_mul(1024),
batch_timeout: Duration::from_millis(config.consumer_batch_timeout_ms),
max_in_flight_batches: config.consumer_max_background_tasks.max(1),
+ commit_tick: commit_tick(
+ config.consumer_completion_granularity,
+ Duration::from_millis(config.consumer_commit_interval_ms),
+ ),
handle,
group_id: config.ingestion_consumer_group_id.clone(),
})
@@ -639,7 +738,10 @@ impl IngestionConsumer {
return;
}
- if accepting_new_batches && in_flight_polls.len() < self.max_in_flight_batches {
+ let collect = accepting_new_batches
+ && in_flight_polls.len() < self.max_in_flight_batches
+ && (in_flight_polls.is_empty() || !self.every_partition_paused());
+ if collect {
tokio::select! {
_ = self.handle.shutdown_recv() => {
info!(
@@ -773,7 +875,10 @@ impl IngestionConsumer {
completions: &mut mpsc::UnboundedReceiver<GroupCompletion>,
errors: &mut mpsc::UnboundedReceiver<String>,
) -> Result<(), Failure> {
- let mut heartbeat = tokio::time::interval(Duration::from_secs(1));
+ let mut heartbeat = tokio::time::interval_at(
+ tokio::time::Instant::now() + self.commit_tick,
+ self.commit_tick,
+ );
loop {
// A rebalance can drop the front poll (its partitions were
// revoked and its purged messages will never complete), so
@@ -792,7 +897,10 @@ impl IngestionConsumer {
self.settlement.settle_run(tp, generation, run)
})
.map_err(Failure::Batch)?;
- self.commit_due().map_err(Failure::Commit)?;
+ // A resumed partition can fill a poll, so collect instead of waiting here.
+ if self.commit_due().map_err(Failure::Commit)? {
+ return Ok(());
+ }
}
None => return Err(Failure::Batch(anyhow::anyhow!(
"batcher completion channel closed"
@@ -806,7 +914,12 @@ impl IngestionConsumer {
},
_ = heartbeat.tick() => {
self.handle.report_healthy();
- self.commit_due().map_err(Failure::Commit)?;
+ if self.commit_due().map_err(Failure::Commit)?
+ || (in_flight_polls.len() < self.max_in_flight_batches
+ && !self.every_partition_paused())
+ {
+ return Ok(());
+ }
}
}
}
@@ -852,18 +965,67 @@ impl IngestionConsumer {
}
}
- fn commit_due(&self) -> anyhow::Result<()> {
+ /// Paused partitions fill no poll, so a collection would wait out the batch
+ /// timeout before the completions whose commits resume them.
+ fn every_partition_paused(&self) -> bool {
+ self.fetching.all_paused()
+ }
+
+ fn pause_partition(&self, topic_partition: &TopicPartition) {
+ let tpl = partition_list(topic_partition);
+ match self
+ .consumer
+ .pause(&tpl)
+ .and_then(|()| partition_error(&tpl))
+ {
+ Ok(()) => self.fetching.paused(topic_partition),
+ Err(err) => error!(
+ topic = %topic_partition.topic,
+ partition = topic_partition.partition,
+ error = %err,
+ "Could not pause a partition at its budget"
+ ),
+ }
+ }
+
+ fn resume_partition(&self, topic_partition: &TopicPartition) {
+ let tpl = partition_list(topic_partition);
+ match self
+ .consumer
+ .resume(&tpl)
+ .and_then(|()| partition_error(&tpl))
+ {
+ Ok(()) => self.fetching.resumed(topic_partition),
+ Err(err) => error!(
+ topic = %topic_partition.topic,
+ partition = topic_partition.partition,
+ error = %err,
+ "Could not resume a partition under its budget"
+ ),
+ }
+ }
+
+ /// Returns whether the commit resumed a partition.
+ fn commit_due(&self) -> anyhow::Result<bool> {
match self.settlement.take_due(Instant::now()) {
- Some(offsets) => self.commit_offsets(&offsets),
- None => Ok(()),
+ Some(commit) => self.commit(commit),
+ None => Ok(false),
}
}
- fn commit_all(&self) -> anyhow::Result<()> {
+ fn commit_all(&self) -> anyhow::Result<bool> {
match self.settlement.take_all() {
- Some(offsets) => self.commit_offsets(&offsets),
- None => Ok(()),
+ Some(commit) => self.commit(commit),
+ None => Ok(false),
+ }
+ }
+
+ fn commit(&self, commit: Commit) -> anyhow::Result<bool> {
+ self.commit_offsets(&commit.offsets)?;
+ for topic_partition in &commit.resume {
+ self.resume_partition(topic_partition);
}
+ Ok(!commit.resume.is_empty())
}
fn fail_batch_processing(&self, err: anyhow::Error) {
@@ -1031,16 +1193,18 @@ impl IngestionConsumer {
for (topic_partition, partition) in &partitions {
// The ledger counts a rejection; it is logged here, where the
// slice is still known.
- if let Err(rejection) =
- self.settlement
- .charge(topic_partition, partition.generation, &partition.charges)
+ match self
+ .settlement
+ .charge(topic_partition, partition.generation, &partition.charges)
{
- report_rejection(
+ Ok(Some(Pause)) => self.pause_partition(topic_partition),
+ Ok(None) => {}
+ Err(rejection) => report_rejection(
"charge",
topic_partition,
rejection,
RejectedSlice::charged(&partition.charges),
- )?;
+ )?,
}
}
@@ -1052,6 +1216,12 @@ impl IngestionConsumer {
}
}
+fn partition_list(topic_partition: &TopicPartition) -> TopicPartitionList {
+ let mut tpl = TopicPartitionList::new();
+ tpl.add_partition(&topic_partition.topic, topic_partition.partition);
+ tpl
+}
+
/// Emit the per-poll parity metrics (received counts, batch sizes,
/// utilization, and lag) right after collection, before the poll is
/// submitted.
@@ -1197,6 +1367,8 @@ fn emit_latest_processed_timestamp_metrics(
#[cfg(test)]
mod tests {
use super::*;
+ use std::num::NonZeroU64;
+
use crate::settlement::{charge_offsets, GroupSettlement};
use common_kafka_consumer::Offset as MessageOffset;
@@ -1243,26 +1415,107 @@ mod tests {
apply_completion(in_flight, &completion, |_, _, _| Ok(())).unwrap();
}
+ fn partitions(numbers: &[i32]) -> Vec<TopicPartition> {
+ numbers
+ .iter()
+ .map(|&number| TopicPartition::new("events", number))
+ .collect()
+ }
+
+ #[rstest::rstest]
+ #[case::group_below_a_second(CompletionGranularity::Group, 200, 200)]
+ #[case::group_above_a_second(CompletionGranularity::Group, 5_000, 1_000)]
+ #[case::group_immediate(CompletionGranularity::Group, 0, 1_000)]
+ #[case::poll(CompletionGranularity::Poll, 200, 1_000)]
+ fn the_commit_tick_follows_the_commit_interval(
+ #[case] granularity: CompletionGranularity,
+ #[case] commit_interval_ms: u64,
+ #[case] expected_ms: u64,
+ ) {
+ assert_eq!(
+ commit_tick(granularity, Duration::from_millis(commit_interval_ms)),
+ Duration::from_millis(expected_ms)
+ );
+ }
+
+ #[test]
+ fn every_partition_is_paused_only_while_none_is_fetching() {
+ let fetching = FetchingPartitions::default();
+ assert!(!fetching.all_paused(), "nothing is assigned");
+
+ fetching.assigned(&partitions(&[0, 1]));
+ fetching.paused(&TopicPartition::new("events", 0));
+ assert!(!fetching.all_paused(), "partition 1 still fetches");
+
+ fetching.paused(&TopicPartition::new("events", 1));
+ assert!(fetching.all_paused());
+
+ fetching.resumed(&TopicPartition::new("events", 1));
+ assert!(!fetching.all_paused());
+ }
+
+ #[test]
+ fn a_rebalance_changes_which_partitions_must_be_paused() {
+ let fetching = FetchingPartitions::default();
+ fetching.assigned(&partitions(&[0, 1]));
+ fetching.paused(&TopicPartition::new("events", 0));
+
+ fetching.revoked(&partitions(&[1]));
+ assert!(fetching.all_paused(), "only the paused partition is left");
+
+ fetching.assigned(&partitions(&[0]));
+ assert!(
+ !fetching.all_paused(),
+ "a reassigned partition starts fetching"
+ );
+ }
+
+ #[test]
+ fn a_partition_that_is_not_assigned_neither_pauses_nor_resumes() {
+ let fetching = FetchingPartitions::default();
+
+ fetching.paused(&TopicPartition::new("events", 0));
+ assert!(!fetching.all_paused());
+
+ fetching.resumed(&TopicPartition::new("events", 0));
+ assert!(!fetching.all_paused());
+ }
+
#[test]
fn an_assignment_without_a_revoke_drops_the_partitions_pending_frontier() {
let ledger = Arc::new(TopicOffsetLedger::new());
let settlement: Arc<dyn Settlement> = Arc::new(GroupSettlement::new(
Arc::clone(&ledger),
Duration::from_secs(3600),
+ BudgetLimits::new(NonZeroU64::new(1), None, 0.5).expect("valid limits"),
));
let epoch = AssignmentEpoch::new();
- let on_assign = assign_hook(Arc::clone(&settlement), epoch.clone());
+ let on_assign = assign_hook(
+ Arc::clone(&settlement),
+ Arc::new(FetchingPartitions::default()),
+ epoch.clone(),
+ );
let p0 = TopicPartition::new("events", 0);
- charge_offsets(&ledger, &p0, ledger.generation(&p0), &[10]);
- settlement
- .settle_run(&p0, ledger.generation(&p0), &[MessageOffset(10)])
- .unwrap();
+ let one_event = [(
+ MessageOffset(10),
+ Charge {
+ events: 1,
+ bytes: 0,
+ },
+ )];
+ assert_eq!(settlement.charge(&p0, 0, &one_event), Ok(Some(Pause)));
+ settlement.settle_run(&p0, 0, &[MessageOffset(10)]).unwrap();
on_assign(std::slice::from_ref(&p0));
assert!(settlement.take_all().is_none());
assert_eq!(ledger.generation(&p0), 1);
assert_eq!(epoch.current(), 2);
+ assert_eq!(
+ settlement.charge(&p0, 1, &one_event),
+ Ok(Some(Pause)),
+ "the regained partition starts unpaused"
+ );
}
fn regained_poll(charged: i64, span_last: i64, delivered: u32) -> InFlightPoll {
@@ -1309,7 +1562,8 @@ mod tests {
#[test]
fn a_completion_spanning_two_polls_commits_once_the_prefix_is_whole() {
let ledger = Arc::new(TopicOffsetLedger::new());
- let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO);
+ let settlement =
+ GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO, BudgetLimits::unset());
let partition = TopicPartition::new("test", 0);
charge_offsets(&ledger, &partition, 0, &[0, 1, 2, 3, 4, 5, 6, 7]);
let mut in_flight = VecDeque::from([poll(1, 0, 0, 3, 4), poll(1, 0, 4, 7, 4)]);
@@ -1335,7 +1589,9 @@ mod tests {
)
.unwrap();
assert_eq!(
- settlement.take_due(Instant::now()),
+ settlement
+ .take_due(Instant::now())
+ .map(|commit| commit.offsets),
Some(HashMap::from([(partition, MessageOffset(8))]))
);
}
@@ -1641,7 +1897,8 @@ mod tests {
#[test]
fn a_covered_head_is_popped_before_a_redelivered_offset_is_matched() {
let ledger = Arc::new(TopicOffsetLedger::new());
- let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO);
+ let settlement =
+ GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO, BudgetLimits::unset());
let partition = TopicPartition::new("test", 0);
ledger.forget_partitions([("test", 0)]);
charge_offsets(&ledger, &partition, 1, &[0, 1]);
@@ -1671,7 +1928,9 @@ mod tests {
assert_eq!(in_flight[0].poll_id, "replay");
assert!(in_flight[0].is_complete());
assert_eq!(
- settlement.take_due(Instant::now()),
+ settlement
+ .take_due(Instant::now())
+ .map(|commit| commit.offsets),
Some(HashMap::from([(partition, MessageOffset(2))]))
);
}
diff --git a/rust/ingestion-consumer/src/lib.rs b/rust/ingestion-consumer/src/lib.rs
index 32307723384f..81893274b8ce 100644
--- a/rust/ingestion-consumer/src/lib.rs
+++ b/rust/ingestion-consumer/src/lib.rs
@@ -9,6 +9,7 @@ pub mod grpc_transport;
pub mod kafka_stats;
pub mod ledger_rejection;
pub mod order_sentinel;
+pub mod partition_budget;
pub mod readiness;
pub mod rebalance;
pub mod routing;
diff --git a/rust/ingestion-consumer/src/main.rs b/rust/ingestion-consumer/src/main.rs
index 6cc20bee9631..416489c7486a 100644
--- a/rust/ingestion-consumer/src/main.rs
+++ b/rust/ingestion-consumer/src/main.rs
@@ -50,6 +50,9 @@ fn main() -> Result<()> {
config
.request_budget()
.map_err(|err| anyhow::anyhow!("invalid request budget: {err}"))?;
+ config
+ .partition_budget()
+ .map_err(|err| anyhow::anyhow!("invalid partition budget: {err}"))?;
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
diff --git a/rust/ingestion-consumer/src/partition_budget.rs b/rust/ingestion-consumer/src/partition_budget.rs
new file mode 100644
index 000000000000..e735fcf03cf1
--- /dev/null
+++ b/rust/ingestion-consumer/src/partition_budget.rs
@@ -0,0 +1,174 @@
+use std::num::NonZeroU64;
+
+use common_kafka_consumer::Charge;
+
+#[derive(Debug, Clone, Copy)]
+struct Threshold {
+ events: Option<u64>,
+ bytes: Option<u64>,
+}
+
+impl Threshold {
+ fn reached(&self, charge: Charge) -> bool {
+ self.events.is_some_and(|limit| charge.events >= limit)
+ || self.bytes.is_some_and(|limit| charge.bytes >= limit)
+ }
+
+ fn scaled(&self, fraction: f64) -> Self {
+ let scale = |limit: u64| ((limit as f64 * fraction) as u64).clamp(1, limit);
+ Self {
+ events: self.events.map(scale),
+ bytes: self.bytes.map(scale),
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy)]
+struct Limits {
+ pause_at: Threshold,
+ resume_at: Threshold,
+}
+
+#[derive(Debug, Clone, Copy)]
+pub struct BudgetLimits {
+ limits: Option<Limits>,
+}
+
+impl BudgetLimits {
+ pub fn unset() -> Self {
+ Self { limits: None }
+ }
+
+ /// With neither axis set the budget is unset.
+ pub fn new(
+ events: Option<NonZeroU64>,
+ bytes: Option<NonZeroU64>,
+ low_watermark: f64,
+ ) -> Result<Self, String> {
+ if !(low_watermark > 0.0 && low_watermark <= 1.0) {
+ return Err(format!(
+ "the low watermark must be above 0 and at most 1, got {low_watermark}"
+ ));
+ }
+ if events.is_none() && bytes.is_none() {
+ return Ok(Self::unset());
+ }
+ let pause_at = Threshold {
+ events: events.map(NonZeroU64::get),
+ bytes: bytes.map(NonZeroU64::get),
+ };
+ Ok(Self {
+ limits: Some(Limits {
+ pause_at,
+ resume_at: pause_at.scaled(low_watermark),
+ }),
+ })
+ }
+
+ pub fn is_set(&self) -> bool {
+ self.limits.is_some()
+ }
+
+ pub(crate) fn pauses(&self, uncommitted: Charge) -> bool {
+ self.limits
+ .is_some_and(|limits| limits.pause_at.reached(uncommitted))
+ }
+
+ pub(crate) fn resumes(&self, uncommitted: Charge) -> bool {
+ self.limits
+ .is_none_or(|limits| !limits.resume_at.reached(uncommitted))
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use rstest::rstest;
+
+ use super::*;
+
+ fn limits(events: u64, bytes: u64, low_watermark: f64) -> BudgetLimits {
+ BudgetLimits::new(
+ NonZeroU64::new(events),
+ NonZeroU64::new(bytes),
+ low_watermark,
+ )
+ .expect("valid limits")
+ }
+
+ fn charge(events: u64, bytes: u64) -> Charge {
+ Charge { events, bytes }
+ }
+
+ struct Observed {
+ uncommitted: Charge,
+ expected: bool,
+ }
+
+ #[rstest]
+ #[case::below_both_limits(Observed { uncommitted: charge(9, 999), expected: false })]
+ #[case::at_the_event_limit(Observed { uncommitted: charge(10, 0), expected: true })]
+ #[case::at_the_byte_limit(Observed { uncommitted: charge(0, 1000), expected: true })]
+ fn a_partition_pauses_when_either_axis_reaches_its_limit(#[case] observed: Observed) {
+ assert_eq!(
+ limits(10, 1000, 0.8).pauses(observed.uncommitted),
+ observed.expected
+ );
+ }
+
+ #[rstest]
+ #[case::between_the_watermark_and_the_limit(Observed { uncommitted: charge(9, 0), expected: false })]
+ #[case::at_the_watermark(Observed { uncommitted: charge(8, 0), expected: false })]
+ #[case::one_axis_still_at_the_watermark(Observed { uncommitted: charge(7, 800), expected: false })]
+ #[case::under_the_watermark_on_both_axes(Observed { uncommitted: charge(7, 799), expected: true })]
+ fn a_partition_resumes_under_the_watermark_on_every_axis(#[case] observed: Observed) {
+ assert_eq!(
+ limits(10, 1000, 0.8).resumes(observed.uncommitted),
+ observed.expected
+ );
+ }
+
+ #[test]
+ fn a_full_watermark_resumes_only_below_the_limit() {
+ let limits = limits(10, 0, 1.0);
+
+ assert!(limits.pauses(charge(10, 0)));
+ assert!(!limits.resumes(charge(10, 0)));
+ assert!(limits.resumes(charge(9, 0)));
+ }
+
+ #[test]
+ fn a_limit_above_float_precision_resumes_only_below_the_limit() {
+ let limit = (1 << 53) + 3;
+ let limits = limits(limit, 0, 1.0);
+
+ assert!(limits.pauses(charge(limit, 0)));
+ assert!(!limits.resumes(charge(limit, 0)));
+ }
+
+ #[rstest]
+ #[case::events(limits(1, 0, 0.8), charge(1, 0))]
+ #[case::bytes(limits(0, 1, 0.8), charge(0, 1))]
+ fn a_limit_of_one_resumes_once_fully_refunded(
+ #[case] limits: BudgetLimits,
+ #[case] at_the_limit: Charge,
+ ) {
+ assert!(limits.pauses(at_the_limit));
+ assert!(limits.resumes(Charge::ZERO));
+ }
+
+ #[test]
+ fn unset_limits_never_pause() {
+ let limits = BudgetLimits::new(None, None, 0.8).expect("unset is valid");
+ assert!(!limits.is_set());
+ assert!(!limits.pauses(charge(u64::MAX, u64::MAX)));
+ }
+
+ #[rstest]
+ #[case::zero(0.0)]
+ #[case::above_one(1.5)]
+ #[case::negative(-0.5)]
+ #[case::nan(f64::NAN)]
+ fn a_watermark_outside_the_unit_interval_is_rejected(#[case] low_watermark: f64) {
+ assert!(BudgetLimits::new(NonZeroU64::new(10), None, low_watermark).is_err());
+ }
+}
diff --git a/rust/ingestion-consumer/src/rebalance.rs b/rust/ingestion-consumer/src/rebalance.rs
index 6f3fb461c22f..76d0e135aa1e 100644
--- a/rust/ingestion-consumer/src/rebalance.rs
+++ b/rust/ingestion-consumer/src/rebalance.rs
@@ -4,9 +4,10 @@ use std::sync::OnceLock;
use common_kafka_consumer::TopicPartition;
use metrics::counter;
-use rdkafka::consumer::{BaseConsumer, ConsumerContext, Rebalance};
+use rdkafka::consumer::{BaseConsumer, Consumer, ConsumerContext, Rebalance};
+use rdkafka::error::KafkaResult;
use rdkafka::{ClientContext, Statistics, TopicPartitionList};
-use tracing::{info, warn};
+use tracing::{error, info, warn};
pub type PartitionHook = Box<dyn Fn(&[TopicPartition]) + Send + Sync>;
@@ -42,6 +43,14 @@ fn topic_partitions(tpl: &TopicPartitionList) -> Vec<TopicPartition> {
partitions
}
+/// librdkafka's pause and resume return success when they cannot change a
+/// partition. They set the error on that partition's list element.
+pub(crate) fn partition_error(tpl: &TopicPartitionList) -> KafkaResult<()> {
+ tpl.elements()
+ .iter()
+ .try_for_each(|element| element.error())
+}
+
fn partition_names(partitions: &[TopicPartition]) -> Vec<String> {
partitions
.iter()
@@ -80,7 +89,7 @@ impl ConsumerContext for RebalanceContext {
}
}
- fn post_rebalance(&self, _consumer: &BaseConsumer<Self>, rebalance: &Rebalance) {
+ fn post_rebalance(&self, consumer: &BaseConsumer<Self>, rebalance: &Rebalance) {
if let Rebalance::Assign(tpl) = rebalance {
counter!("ingestion_consumer_rebalances_total", "event" => "assign").increment(1);
let partitions = topic_partitions(tpl);
@@ -89,6 +98,11 @@ impl ConsumerContext for RebalanceContext {
topic_partitions = ?partition_names(&partitions),
"Rebalance: partitions assigned"
);
+ // librdkafka keeps a partition's pause across a revoke and a
+ // reassignment, and the assign hook forgets the budget that paused it.
+ if let Err(err) = consumer.resume(tpl).and_then(|()| partition_error(tpl)) {
+ error!(error = %err, "Could not resume the assigned partitions");
+ }
if let Some(hook) = self.on_assign.get() {
hook(&partitions);
}
@@ -159,4 +173,18 @@ mod tests {
assert_eq!(*revoked.lock().unwrap(), expected);
assert_eq!(*assigned.lock().unwrap(), expected);
}
+
+ #[test]
+ fn pausing_an_unassigned_partition_is_an_error() {
+ let consumer: BaseConsumer = ClientConfig::new()
+ .set("bootstrap.servers", "127.0.0.1:1")
+ .set("group.id", "test")
+ .create()
+ .expect("a client needs no broker to be built");
+ let mut tpl = TopicPartitionList::new();
+ tpl.add_partition("events", 0);
+
+ assert!(consumer.pause(&tpl).is_ok());
+ assert!(partition_error(&tpl).is_err());
+ }
}
diff --git a/rust/ingestion-consumer/src/settlement/budget.rs b/rust/ingestion-consumer/src/settlement/budget.rs
new file mode 100644
index 000000000000..b163b91c4e1e
--- /dev/null
+++ b/rust/ingestion-consumer/src/settlement/budget.rs
@@ -0,0 +1,248 @@
+use std::collections::{HashMap, HashSet};
+use std::sync::{Arc, Mutex};
+
+use common_kafka_consumer::{Charge, TopicPartition};
+use metrics::{counter, gauge};
+use tracing::info;
+
+use super::{Admission, HandedOut, Pause};
+use crate::partition_budget::BudgetLimits;
+
+const PAUSED_PARTITIONS: &str = "ingestion_consumer_budget_paused_partitions";
+const PAUSES: &str = "ingestion_consumer_budget_pauses_total";
+
+pub(crate) struct PartitionBudget {
+ limits: BudgetLimits,
+ paused: Mutex<HashSet<TopicPartition>>,
+}
+
+impl PartitionBudget {
+ pub(super) fn new(limits: BudgetLimits) -> Self {
+ gauge!(PAUSED_PARTITIONS).set(0.0);
+ Self {
+ limits,
+ paused: Mutex::new(HashSet::new()),
+ }
+ }
+}
+
+impl Admission for PartitionBudget {
+ fn charged(&self, topic_partition: &TopicPartition, uncommitted: Charge) -> Option<Pause> {
+ if !self.limits.pauses(uncommitted) {
+ return None;
+ }
+ let mut paused = self.paused.lock().unwrap();
+ if !paused.insert(topic_partition.clone()) {
+ return None;
+ }
+ gauge!(PAUSED_PARTITIONS).set(paused.len() as f64);
+ let topic: Arc<str> = Arc::from(topic_partition.topic.as_str());
+ let partition: Arc<str> = Arc::from(topic_partition.partition.to_string());
+ counter!(PAUSES, "topic" => topic, "partition" => partition).increment(1);
+ log_transition(topic_partition, "pause", uncommitted);
+ Some(Pause)
+ }
+
+ fn handed_out(&self, handed_out: &HashMap<TopicPartition, HandedOut>) -> Vec<TopicPartition> {
+ let mut paused = self.paused.lock().unwrap();
+ let mut resume = Vec::new();
+ for (topic_partition, handed_out) in handed_out {
+ let uncommitted = handed_out.remaining.charge;
+ if paused.contains(topic_partition) && self.limits.resumes(uncommitted) {
+ paused.remove(topic_partition);
+ log_transition(topic_partition, "resume", uncommitted);
+ resume.push(topic_partition.clone());
+ }
+ }
+ if !resume.is_empty() {
+ gauge!(PAUSED_PARTITIONS).set(paused.len() as f64);
+ }
+ resume
+ }
+
+ fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {
+ let mut paused = self.paused.lock().unwrap();
+ for topic_partition in topic_partitions {
+ paused.remove(topic_partition);
+ }
+ gauge!(PAUSED_PARTITIONS).set(paused.len() as f64);
+ }
+}
+
+fn log_transition(topic_partition: &TopicPartition, transition: &str, uncommitted: Charge) {
+ info!(
+ topic = %topic_partition.topic,
+ partition = topic_partition.partition,
+ uncommitted_events = uncommitted.events,
+ uncommitted_bytes = uncommitted.bytes,
+ transition,
+ "Partition budget"
+ );
+}
+
+#[cfg(test)]
+mod tests {
+ use std::num::NonZeroU64;
+
+ use common_kafka_consumer::{Held, Offset};
+ use metrics_util::debugging::{DebugValue, DebuggingRecorder, Snapshotter};
+
+ use super::*;
+
+ fn p0() -> TopicPartition {
+ TopicPartition::new("events", 0)
+ }
+
+ fn p1() -> TopicPartition {
+ TopicPartition::new("events", 1)
+ }
+
+ fn events(events: u64) -> Charge {
+ Charge { events, bytes: 0 }
+ }
+
+ fn budget_of_10_events_resuming_under_8() -> PartitionBudget {
+ PartitionBudget::new(
+ BudgetLimits::new(NonZeroU64::new(10), None, 0.8).expect("valid limits"),
+ )
+ }
+
+ #[rstest::rstest]
+ #[case::below_the_limit(9, None)]
+ #[case::at_the_limit(10, Some(Pause))]
+ fn a_charge_pauses_a_partition_that_reaches_its_limit(
+ #[case] uncommitted: u64,
+ #[case] expected: Option<Pause>,
+ ) {
+ assert_eq!(
+ budget_of_10_events_resuming_under_8().charged(&p0(), events(uncommitted)),
+ expected
+ );
+ }
+
+ #[test]
+ fn a_paused_partition_pauses_once() {
+ let budget = budget_of_10_events_resuming_under_8();
+ assert_eq!(budget.charged(&p0(), events(10)), Some(Pause));
+
+ assert_eq!(budget.charged(&p0(), events(12)), None);
+ }
+
+ fn handed_out(remaining_events: u64) -> HandedOut {
+ HandedOut {
+ offset: Offset(1),
+ remaining: Held {
+ offsets: remaining_events as usize,
+ charge: events(remaining_events),
+ },
+ }
+ }
+
+ fn hand_out_leaving(budget: &PartitionBudget, remaining_events: u64) -> Vec<TopicPartition> {
+ budget.handed_out(&HashMap::from([(p0(), handed_out(remaining_events))]))
+ }
+
+ #[rstest::rstest]
+ #[case::above_the_watermark(9, false)]
+ #[case::at_the_watermark(8, false)]
+ #[case::under_the_watermark(7, true)]
+ fn a_hand_out_resumes_a_paused_partition_under_the_watermark(
+ #[case] remaining: u64,
+ #[case] expected_resume: bool,
+ ) {
+ let budget = budget_of_10_events_resuming_under_8();
+ budget.charged(&p0(), events(10));
+
+ assert_eq!(
+ hand_out_leaving(&budget, remaining).contains(&p0()),
+ expected_resume
+ );
+ }
+
+ #[test]
+ fn a_hand_out_never_resumes_a_partition_that_is_not_paused() {
+ let budget = budget_of_10_events_resuming_under_8();
+
+ assert!(hand_out_leaving(&budget, 0).is_empty());
+ }
+
+ #[test]
+ fn a_resumed_partition_pauses_again_at_its_limit() {
+ let budget = budget_of_10_events_resuming_under_8();
+ budget.charged(&p0(), events(10));
+ hand_out_leaving(&budget, 0);
+
+ assert_eq!(budget.charged(&p0(), events(10)), Some(Pause));
+ }
+
+ #[test]
+ fn a_forgotten_partition_starts_unpaused() {
+ let budget = budget_of_10_events_resuming_under_8();
+ budget.charged(&p0(), events(10));
+
+ budget.forget_partitions(&[p0()]);
+
+ assert_eq!(budget.charged(&p0(), events(10)), Some(Pause));
+ }
+
+ #[derive(Debug, PartialEq)]
+ struct Recorded {
+ paused_partitions: f64,
+ pauses: HashMap<String, u64>,
+ }
+
+ fn recorded(snapshotter: &Snapshotter) -> Recorded {
+ let mut recorded = Recorded {
+ paused_partitions: f64::NAN,
+ pauses: HashMap::new(),
+ };
+ for (key, _, _, value) in snapshotter.snapshot().into_vec() {
+ match (key.key().name(), value) {
+ (PAUSED_PARTITIONS, DebugValue::Gauge(value)) => {
+ recorded.paused_partitions = value.into_inner();
+ }
+ (PAUSES, DebugValue::Counter(count)) => {
+ let partition = key
+ .key()
+ .labels()
+ .find(|label| label.key() == "partition")
+ .expect("a partition label")
+ .value()
+ .to_string();
+ recorded.pauses.insert(partition, count);
+ }
+ _ => {}
+ }
+ }
+ recorded
+ }
+
+ #[test]
+ fn the_metrics_follow_pauses_resumes_and_revokes() {
+ let recorder = DebuggingRecorder::new();
+ let snapshotter = recorder.snapshotter();
+ let _guard = metrics::set_default_local_recorder(&recorder);
+ let budget = budget_of_10_events_resuming_under_8();
+
+ budget.charged(&p0(), events(10));
+ budget.charged(&p1(), events(10));
+ budget.charged(&p0(), events(12));
+ assert_eq!(
+ recorded(&snapshotter),
+ Recorded {
+ paused_partitions: 2.0,
+ pauses: HashMap::from([("0".to_string(), 1), ("1".to_string(), 1)]),
+ }
+ );
+
+ let resume = budget.handed_out(&HashMap::from([
+ (p0(), handed_out(0)),
+ (p1(), handed_out(9)),
+ ]));
+ assert_eq!(resume, vec![p0()]);
+ assert_eq!(recorded(&snapshotter).paused_partitions, 1.0);
+
+ budget.forget_partitions(&[p1()]);
+ assert_eq!(recorded(&snapshotter).paused_partitions, 0.0);
+ }
+}
diff --git a/rust/ingestion-consumer/src/settlement/fakes.rs b/rust/ingestion-consumer/src/settlement/fakes.rs
new file mode 100644
index 000000000000..93e57c4cbcb1
--- /dev/null
+++ b/rust/ingestion-consumer/src/settlement/fakes.rs
@@ -0,0 +1,162 @@
+use std::collections::{HashMap, VecDeque};
+use std::sync::{Arc, Mutex};
+use std::time::Instant;
+
+use common_kafka_consumer::{Charge, Held, Offset, Rejection, TopicPartition};
+
+use super::{Accounting, Admission, HandedOut, Pause};
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub(super) struct Settled {
+ pub(super) topic_partition: TopicPartition,
+ pub(super) generation: u64,
+ pub(super) offsets: Vec<Offset>,
+}
+
+struct Script<T>(Arc<Mutex<VecDeque<T>>>);
+
+impl<T> Default for Script<T> {
+ fn default() -> Self {
+ Self(Arc::new(Mutex::new(VecDeque::new())))
+ }
+}
+
+impl<T> Clone for Script<T> {
+ fn clone(&self) -> Self {
+ Self(Arc::clone(&self.0))
+ }
+}
+
+impl<T> Script<T> {
+ fn push(&self, result: T) {
+ self.0.lock().unwrap().push_back(result);
+ }
+
+ fn next(&self) -> Option<T> {
+ self.0.lock().unwrap().pop_front()
+ }
+}
+
+#[derive(Default, Clone)]
+pub(super) struct FakeAccounting {
+ charges: Script<Result<Held, Rejection>>,
+ settles: Script<Result<Option<Offset>, Rejection>>,
+ takes: Script<HashMap<TopicPartition, HandedOut>>,
+ settled: Arc<Mutex<Vec<Settled>>>,
+ forgotten: Arc<Mutex<Vec<TopicPartition>>>,
+}
+
+impl FakeAccounting {
+ pub(super) fn charge_returns(&self, result: Result<Held, Rejection>) {
+ self.charges.push(result);
+ }
+
+ pub(super) fn settle_returns(&self, result: Result<Option<Offset>, Rejection>) {
+ self.settles.push(result);
+ }
+
+ pub(super) fn take_hands_out(&self, handed_out: HashMap<TopicPartition, HandedOut>) {
+ self.takes.push(handed_out);
+ }
+
+ pub(super) fn settles_seen(&self) -> Vec<Settled> {
+ self.settled.lock().unwrap().clone()
+ }
+
+ pub(super) fn forgotten(&self) -> Vec<TopicPartition> {
+ self.forgotten.lock().unwrap().clone()
+ }
+}
+
+impl Accounting for FakeAccounting {
+ fn charge(
+ &self,
+ _topic_partition: &TopicPartition,
+ _generation: u64,
+ _charges: &[(Offset, Charge)],
+ ) -> Result<Held, Rejection> {
+ self.charges.next().unwrap_or(Ok(Held::default()))
+ }
+
+ fn settled(
+ &self,
+ topic_partition: &TopicPartition,
+ generation: u64,
+ offsets: impl IntoIterator<Item = Offset>,
+ ) -> Result<Option<Offset>, Rejection> {
+ self.settled.lock().unwrap().push(Settled {
+ topic_partition: topic_partition.clone(),
+ generation,
+ offsets: offsets.into_iter().collect(),
+ });
+ self.settles.next().unwrap_or(Ok(None))
+ }
+
+ fn take_due(&self, _now: Instant) -> Option<HashMap<TopicPartition, HandedOut>> {
+ self.takes.next()
+ }
+
+ fn take_all(&self) -> Option<HashMap<TopicPartition, HandedOut>> {
+ self.takes.next()
+ }
+
+ fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {
+ self.forgotten
+ .lock()
+ .unwrap()
+ .extend_from_slice(topic_partitions);
+ }
+}
+
+#[derive(Default, Clone)]
+pub(super) struct FakeAdmission {
+ pauses: Script<Option<Pause>>,
+ resumes: Script<Vec<TopicPartition>>,
+ charged: Arc<Mutex<Vec<(TopicPartition, Charge)>>>,
+ handed_out: Arc<Mutex<Vec<HashMap<TopicPartition, HandedOut>>>>,
+ forgotten: Arc<Mutex<Vec<TopicPartition>>>,
+}
+
+impl FakeAdmission {
+ pub(super) fn charged_returns(&self, decision: Option<Pause>) {
+ self.pauses.push(decision);
+ }
+
+ pub(super) fn handed_out_returns(&self, resume: Vec<TopicPartition>) {
+ self.resumes.push(resume);
+ }
+
+ pub(super) fn charges_seen(&self) -> Vec<(TopicPartition, Charge)> {
+ self.charged.lock().unwrap().clone()
+ }
+
+ pub(super) fn hand_outs_seen(&self) -> Vec<HashMap<TopicPartition, HandedOut>> {
+ self.handed_out.lock().unwrap().clone()
+ }
+
+ pub(super) fn forgotten(&self) -> Vec<TopicPartition> {
+ self.forgotten.lock().unwrap().clone()
+ }
+}
+
+impl Admission for FakeAdmission {
+ fn charged(&self, topic_partition: &TopicPartition, uncommitted: Charge) -> Option<Pause> {
+ self.charged
+ .lock()
+ .unwrap()
+ .push((topic_partition.clone(), uncommitted));
+ self.pauses.next().flatten()
+ }
+
+ fn handed_out(&self, handed_out: &HashMap<TopicPartition, HandedOut>) -> Vec<TopicPartition> {
+ self.handed_out.lock().unwrap().push(handed_out.clone());
+ self.resumes.next().unwrap_or_default()
+ }
+
+ fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {
+ self.forgotten
+ .lock()
+ .unwrap()
+ .extend_from_slice(topic_partitions);
+ }
+}
diff --git a/rust/ingestion-consumer/src/settlement/group.rs b/rust/ingestion-consumer/src/settlement/group.rs
index 3f925098cf56..e202a0630751 100644
--- a/rust/ingestion-consumer/src/settlement/group.rs
+++ b/rust/ingestion-consumer/src/settlement/group.rs
@@ -1,35 +1,46 @@
-use std::collections::HashMap;
use std::sync::Arc;
use std::time::{Duration, Instant};
-use common_kafka_consumer::{Charge, Held, Offset, Rejection, TopicOffsetLedger, TopicPartition};
+use common_kafka_consumer::{Charge, Offset, Rejection, TopicOffsetLedger, TopicPartition};
-use super::{ChargedSlice, Settlement, Uncommitted};
+use super::{
+ offsets, Accounting, Admission, ChargedSlice, Commit, PartitionBudget, Pause, Settlement,
+ Uncommitted,
+};
use crate::commit_pacer::CommitPacer;
use crate::ledger_rejection::{report_rejection, RejectedSlice};
use crate::order_sentinel::OffsetSpan;
+use crate::partition_budget::BudgetLimits;
-pub(crate) struct GroupSettlement {
- uncommitted: Uncommitted,
+pub(crate) struct GroupSettlement<A: Accounting = Uncommitted, D: Admission = PartitionBudget> {
+ accounting: A,
+ admission: D,
}
impl GroupSettlement {
- pub(crate) fn new(ledger: Arc<TopicOffsetLedger>, commit_interval: Duration) -> Self {
+ pub(crate) fn new(
+ ledger: Arc<TopicOffsetLedger>,
+ commit_interval: Duration,
+ limits: BudgetLimits,
+ ) -> Self {
Self {
- uncommitted: Uncommitted::new(ledger, CommitPacer::every(commit_interval)),
+ accounting: Uncommitted::new(ledger, CommitPacer::every(commit_interval)),
+ admission: PartitionBudget::new(limits),
}
}
}
-impl Settlement for GroupSettlement {
+impl<A: Accounting + 'static, D: Admission + 'static> Settlement for GroupSettlement<A, D> {
fn charge(
&self,
topic_partition: &TopicPartition,
generation: u64,
charges: &[(Offset, Charge)],
- ) -> Result<Held, Rejection> {
- self.uncommitted
- .charge(topic_partition, generation, charges)
+ ) -> Result<Option<Pause>, Rejection> {
+ let uncommitted = self
+ .accounting
+ .charge(topic_partition, generation, charges)?;
+ Ok(self.admission.charged(topic_partition, uncommitted.charge))
}
fn settle_run(
@@ -38,8 +49,11 @@ impl Settlement for GroupSettlement {
generation: u64,
offsets: &[Offset],
) -> anyhow::Result<()> {
+ let (Some(first), Some(last)) = (offsets.first(), offsets.last()) else {
+ return Ok(());
+ };
if let Err(rejection) =
- self.uncommitted
+ self.accounting
.settled(topic_partition, generation, offsets.iter().copied())
{
report_rejection(
@@ -47,8 +61,8 @@ impl Settlement for GroupSettlement {
topic_partition,
rejection,
RejectedSlice::settled(&OffsetSpan {
- first: offsets[0].0,
- last: offsets[offsets.len() - 1].0,
+ first: first.0,
+ last: last.0,
}),
)?;
}
@@ -59,27 +73,225 @@ impl Settlement for GroupSettlement {
Ok(())
}
- fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, Offset>> {
- self.uncommitted.take_due(now)
+ fn take_due(&self, now: Instant) -> Option<Commit> {
+ let handed_out = self.accounting.take_due(now)?;
+ Some(Commit {
+ resume: self.admission.handed_out(&handed_out),
+ offsets: offsets(handed_out),
+ })
}
- fn take_all(&self) -> Option<HashMap<TopicPartition, Offset>> {
- self.uncommitted.take_all()
+ fn take_all(&self) -> Option<Commit> {
+ let handed_out = self.accounting.take_all()?;
+ Some(Commit {
+ resume: self.admission.handed_out(&handed_out),
+ offsets: offsets(handed_out),
+ })
}
fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {
- self.uncommitted.forget_partitions(topic_partitions);
+ self.accounting.forget_partitions(topic_partitions);
+ self.admission.forget_partitions(topic_partitions);
}
}
#[cfg(test)]
mod tests {
+ use std::collections::HashMap;
+
+ use common_kafka_consumer::{Held, LedgerError};
+
+ use super::super::fakes::{FakeAccounting, FakeAdmission, Settled};
+ use super::super::HandedOut;
+ use super::*;
+
+ fn p(partition: i32) -> TopicPartition {
+ TopicPartition::new("events", partition)
+ }
+
+ fn events(events: u64) -> Charge {
+ Charge { events, bytes: 0 }
+ }
+
+ fn uncommitted_events(events: u64) -> Held {
+ Held {
+ offsets: events as usize,
+ charge: Charge { events, bytes: 0 },
+ }
+ }
+
+ fn handed_out(offset: i64, remaining_events: u64) -> HandedOut {
+ HandedOut {
+ offset: Offset(offset),
+ remaining: uncommitted_events(remaining_events),
+ }
+ }
+
+ fn settlement() -> (
+ GroupSettlement<FakeAccounting, FakeAdmission>,
+ FakeAccounting,
+ FakeAdmission,
+ ) {
+ let accounting = FakeAccounting::default();
+ let admission = FakeAdmission::default();
+ let settlement = GroupSettlement {
+ accounting: accounting.clone(),
+ admission: admission.clone(),
+ };
+ (settlement, accounting, admission)
+ }
+
+ #[test]
+ fn a_charge_asks_the_admission_with_the_uncommitted_charge() {
+ let (settlement, accounting, admission) = settlement();
+ accounting.charge_returns(Ok(uncommitted_events(10)));
+ admission.charged_returns(Some(Pause));
+
+ assert_eq!(settlement.charge(&p(0), 0, &[]), Ok(Some(Pause)));
+ assert_eq!(admission.charges_seen(), vec![(p(0), events(10))]);
+ }
+
+ #[test]
+ fn a_rejected_charge_asks_the_admission_nothing() {
+ let (settlement, accounting, admission) = settlement();
+ let stale = Rejection::Stale {
+ stamp: 0,
+ generation: 1,
+ };
+ accounting.charge_returns(Err(stale));
+
+ assert_eq!(settlement.charge(&p(0), 0, &[]), Err(stale));
+ assert!(admission.charges_seen().is_empty());
+ }
+
+ #[test]
+ fn a_hand_out_commits_its_offsets_and_resumes_what_the_admission_decides() {
+ let (settlement, accounting, admission) = settlement();
+ let hand_out = HashMap::from([(p(0), handed_out(11, 3)), (p(1), handed_out(21, 0))]);
+ accounting.take_hands_out(hand_out.clone());
+ admission.handed_out_returns(vec![p(0)]);
+
+ let commit = settlement.take_due(Instant::now()).expect("handed out");
+
+ assert_eq!(
+ commit.offsets,
+ HashMap::from([(p(0), Offset(11)), (p(1), Offset(21))])
+ );
+ assert_eq!(commit.resume, vec![p(0)]);
+ assert_eq!(admission.hand_outs_seen(), vec![hand_out]);
+ }
+
+ #[test]
+ fn take_all_asks_the_admission_like_take_due() {
+ let (settlement, accounting, admission) = settlement();
+ accounting.take_hands_out(HashMap::from([(p(0), handed_out(11, 0))]));
+ admission.handed_out_returns(vec![p(0)]);
+
+ assert_eq!(
+ settlement.take_all().expect("handed out").resume,
+ vec![p(0)]
+ );
+ }
+
+ #[test]
+ fn a_run_reaches_the_accounting_under_its_generation() {
+ let (settlement, accounting, _) = settlement();
+
+ settlement
+ .settle_run(&p(0), 3, &[Offset(10), Offset(11)])
+ .unwrap();
+
+ assert_eq!(
+ accounting.settles_seen(),
+ vec![Settled {
+ topic_partition: p(0),
+ generation: 3,
+ offsets: vec![Offset(10), Offset(11)],
+ }]
+ );
+ }
+
+ #[test]
+ fn an_empty_run_settles_nothing() {
+ let (settlement, accounting, _) = settlement();
+
+ settlement.settle_run(&p(0), 0, &[]).unwrap();
+
+ assert!(accounting.settles_seen().is_empty());
+ }
+
+ struct RejectedRun {
+ rejection: Rejection,
+ expected_error: bool,
+ }
+
+ #[rstest::rstest]
+ #[case::stale(RejectedRun {
+ rejection: Rejection::Stale { stamp: 0, generation: 1 },
+ expected_error: false,
+ })]
+ #[case::violation(RejectedRun {
+ rejection: Rejection::Violation {
+ error: LedgerError::CompletedTwice { offset: Offset(10) },
+ stamp: 0,
+ generation: 0,
+ held: Held::default(),
+ },
+ expected_error: true,
+ })]
+ fn only_a_violating_run_fails(#[case] run: RejectedRun) {
+ let (settlement, accounting, _) = settlement();
+ accounting.settle_returns(Err(run.rejection));
+
+ assert_eq!(
+ settlement.settle_run(&p(0), 0, &[Offset(10)]).is_err(),
+ run.expected_error
+ );
+ }
+
+ #[test]
+ fn a_retired_poll_settles_nothing() {
+ let (settlement, accounting, _) = settlement();
+ let charges = [(Offset(10), Charge::ZERO)];
+
+ settlement
+ .settle_poll(&[ChargedSlice {
+ topic_partition: &p(0),
+ generation: 0,
+ span: OffsetSpan {
+ first: 10,
+ last: 10,
+ },
+ charges: &charges,
+ }])
+ .unwrap();
+
+ assert!(accounting.settles_seen().is_empty());
+ }
+
+ #[test]
+ fn a_revoke_reaches_both_the_accounting_and_the_admission() {
+ let (settlement, accounting, admission) = settlement();
+
+ settlement.forget_partitions(&[p(0)]);
+
+ assert_eq!(accounting.forgotten(), vec![p(0)]);
+ assert_eq!(admission.forgotten(), vec![p(0)]);
+ }
+}
+
+#[cfg(test)]
+mod integration_tests {
+ use std::collections::HashMap;
+ use std::num::NonZeroU64;
+
use super::super::charge_offsets;
use super::*;
fn settlement() -> (Arc<TopicOffsetLedger>, GroupSettlement) {
let ledger = Arc::new(TopicOffsetLedger::new());
- let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO);
+ let settlement =
+ GroupSettlement::new(Arc::clone(&ledger), Duration::ZERO, BudgetLimits::unset());
(ledger, settlement)
}
@@ -99,7 +311,9 @@ mod tests {
.settle_run(&p0, 0, &[Offset(10), Offset(11)])
.unwrap();
assert_eq!(
- settlement.take_due(Instant::now()),
+ settlement
+ .take_due(Instant::now())
+ .map(|commit| commit.offsets),
Some(HashMap::from([(p0, Offset(13))])),
"the late completion releases the whole prefix"
);
@@ -119,7 +333,9 @@ mod tests {
settlement.settle_run(&p1, 0, &[Offset(21)]).unwrap();
assert_eq!(
- settlement.take_due(Instant::now()),
+ settlement
+ .take_due(Instant::now())
+ .map(|commit| commit.offsets),
Some(HashMap::from([(p0, Offset(12))])),
"partition 1 waits for offset 20"
);
@@ -135,7 +351,7 @@ mod tests {
assert!(settlement.settle_run(&p0, 0, &[Offset(10)]).is_err());
assert_eq!(
- settlement.take_all(),
+ settlement.take_all().map(|commit| commit.offsets),
Some(HashMap::from([(p0, Offset(11))])),
"offset 10 completed before the violation"
);
@@ -163,21 +379,25 @@ mod tests {
#[test]
fn a_frontier_within_the_interval_waits_for_it_unless_everything_is_taken() {
let ledger = Arc::new(TopicOffsetLedger::new());
- let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::from_secs(3600));
+ let settlement = GroupSettlement::new(
+ Arc::clone(&ledger),
+ Duration::from_secs(3600),
+ BudgetLimits::unset(),
+ );
let p0 = TopicPartition::new("events", 0);
charge_offsets(&ledger, &p0, 0, &[10, 11]);
let now = Instant::now();
settlement.settle_run(&p0, 0, &[Offset(10)]).unwrap();
assert_eq!(
- settlement.take_due(now),
+ settlement.take_due(now).map(|commit| commit.offsets),
Some(HashMap::from([(p0.clone(), Offset(11))]))
);
settlement.settle_run(&p0, 0, &[Offset(11)]).unwrap();
assert!(settlement.take_due(now).is_none());
assert_eq!(
- settlement.take_all(),
+ settlement.take_all().map(|commit| commit.offsets),
Some(HashMap::from([(p0, Offset(12))]))
);
}
@@ -185,7 +405,11 @@ mod tests {
#[test]
fn a_revoked_partitions_pending_frontier_is_not_committed() {
let ledger = Arc::new(TopicOffsetLedger::new());
- let settlement = GroupSettlement::new(Arc::clone(&ledger), Duration::from_secs(3600));
+ let settlement = GroupSettlement::new(
+ Arc::clone(&ledger),
+ Duration::from_secs(3600),
+ BudgetLimits::unset(),
+ );
let p0 = TopicPartition::new("events", 0);
charge_offsets(&ledger, &p0, 0, &[10]);
settlement.settle_run(&p0, 0, &[Offset(10)]).unwrap();
@@ -198,26 +422,110 @@ mod tests {
assert!(settlement.take_all().is_none());
}
+ const HOUR: Duration = Duration::from_secs(3600);
+
+ fn p0() -> TopicPartition {
+ TopicPartition::new("events", 0)
+ }
+
+ fn budgeted_hourly_settlement_of_10_events_resuming_under_8() -> GroupSettlement {
+ GroupSettlement::new(
+ Arc::new(TopicOffsetLedger::new()),
+ HOUR,
+ BudgetLimits::new(NonZeroU64::new(10), None, 0.8).expect("valid limits"),
+ )
+ }
+
+ fn charge(
+ settlement: &GroupSettlement,
+ topic_partition: &TopicPartition,
+ offset: i64,
+ events: u64,
+ ) -> Option<Pause> {
+ settlement
+ .charge(
+ topic_partition,
+ 0,
+ &[(Offset(offset), Charge { events, bytes: 0 })],
+ )
+ .expect("the ledger accepts the charge")
+ }
+
#[test]
- fn a_retired_poll_settles_nothing_more() {
- let (ledger, settlement) = settlement();
- let p0 = TopicPartition::new("events", 0);
- charge_offsets(&ledger, &p0, 0, &[10]);
- let charges = [(Offset(10), common_kafka_consumer::Charge::ZERO)];
+ fn a_take_the_interval_holds_counts_toward_the_pause() {
+ let settlement = budgeted_hourly_settlement_of_10_events_resuming_under_8();
+ charge(&settlement, &p0(), 10, 6);
+ settlement.settle_run(&p0(), 0, &[Offset(10)]).unwrap();
+
+ assert_eq!(charge(&settlement, &p0(), 11, 4), Some(Pause));
+ }
+ enum HandOut {
+ DueAfterTheInterval,
+ All,
+ }
+
+ #[rstest::rstest]
+ #[case::due_after_the_interval(HandOut::DueAfterTheInterval)]
+ #[case::all(HandOut::All)]
+ fn a_paused_partition_resumes_only_when_its_frontier_is_handed_out(#[case] hand_out: HandOut) {
+ let settlement = budgeted_hourly_settlement_of_10_events_resuming_under_8();
+ let now = Instant::now();
+ charge(&settlement, &p0(), 9, 1);
+ settlement.settle_run(&p0(), 0, &[Offset(9)]).unwrap();
settlement
- .settle_poll(&[ChargedSlice {
- topic_partition: &p0,
- generation: 0,
- span: OffsetSpan {
- first: 10,
- last: 10,
- },
- charges: &charges,
- }])
- .unwrap();
+ .take_due(now)
+ .expect("the first frontier is due at once");
+ assert_eq!(charge(&settlement, &p0(), 10, 10), Some(Pause));
- assert!(settlement.take_due(Instant::now()).is_none());
- assert_eq!(ledger.held(&p0).offsets, 1);
+ settlement.settle_run(&p0(), 0, &[Offset(10)]).unwrap();
+ assert!(settlement.take_due(now).is_none());
+
+ let commit = match hand_out {
+ HandOut::DueAfterTheInterval => settlement.take_due(now + HOUR),
+ HandOut::All => settlement.take_all(),
+ }
+ .expect("the frontier is handed out");
+ assert_eq!(commit.resume, vec![p0()]);
+ }
+
+ #[test]
+ fn a_held_prefix_keeps_its_partition_paused_and_another_is_not_resumed() {
+ let settlement = budgeted_hourly_settlement_of_10_events_resuming_under_8();
+ let p1 = TopicPartition::new("events", 1);
+ charge(&settlement, &p0(), 10, 5);
+ assert_eq!(charge(&settlement, &p0(), 11, 5), Some(Pause));
+ charge(&settlement, &p1, 20, 1);
+
+ settlement.settle_run(&p0(), 0, &[Offset(11)]).unwrap();
+ settlement.settle_run(&p1, 0, &[Offset(20)]).unwrap();
+ let commit = settlement.take_all().expect("partition 1 is due");
+ assert_eq!(commit.offsets, HashMap::from([(p1, Offset(21))]));
+ assert!(
+ commit.resume.is_empty(),
+ "offset 10 still holds partition 0"
+ );
+
+ settlement.settle_run(&p0(), 0, &[Offset(10)]).unwrap();
+ assert_eq!(
+ settlement.take_all().expect("partition 0 is due").resume,
+ vec![p0()]
+ );
+ }
+
+ #[test]
+ fn a_revoked_paused_partition_is_never_resumed() {
+ let settlement = budgeted_hourly_settlement_of_10_events_resuming_under_8();
+ assert_eq!(charge(&settlement, &p0(), 10, 10), Some(Pause));
+ settlement.settle_run(&p0(), 0, &[Offset(10)]).unwrap();
+
+ settlement.forget_partitions(&[p0()]);
+
+ assert!(settlement.take_all().is_none());
+ settlement.settle_run(&p0(), 0, &[Offset(10)]).unwrap();
+ assert!(
+ settlement.take_all().is_none(),
+ "the stale run moves nothing"
+ );
}
}
diff --git a/rust/ingestion-consumer/src/settlement/mod.rs b/rust/ingestion-consumer/src/settlement/mod.rs
index 890cd3913748..2bbf4d15608f 100644
--- a/rust/ingestion-consumer/src/settlement/mod.rs
+++ b/rust/ingestion-consumer/src/settlement/mod.rs
@@ -1,5 +1,8 @@
//! When accepted offsets reach the ledger and the commit pacer: per poll or per group.
+mod budget;
+#[cfg(test)]
+mod fakes;
mod group;
mod poll;
mod uncommitted;
@@ -10,9 +13,13 @@ use std::time::{Duration, Instant};
use common_kafka_consumer::{Charge, Held, Offset, Rejection, TopicOffsetLedger, TopicPartition};
+use tracing::warn;
+
use crate::config::CompletionGranularity;
use crate::order_sentinel::OffsetSpan;
+use crate::partition_budget::BudgetLimits;
+use budget::PartitionBudget;
pub(crate) use group::GroupSettlement;
pub(crate) use poll::PollSettlement;
use uncommitted::Uncommitted;
@@ -30,7 +37,7 @@ pub(crate) trait Settlement: Send + Sync {
topic_partition: &TopicPartition,
generation: u64,
charges: &[(Offset, Charge)],
- ) -> Result<Held, Rejection>;
+ ) -> Result<Option<Pause>, Rejection>;
/// Accepted offsets from one poll's slice of a partition, charged under `generation`.
fn settle_run(
@@ -43,21 +50,77 @@ pub(crate) trait Settlement: Send + Sync {
/// The oldest poll retired with every message accepted.
fn settle_poll(&self, slices: &[ChargedSlice<'_>]) -> anyhow::Result<()>;
- fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, Offset>>;
+ fn take_due(&self, now: Instant) -> Option<Commit>;
+
+ fn take_all(&self) -> Option<Commit>;
+
+ fn forget_partitions(&self, topic_partitions: &[TopicPartition]);
+}
+
+pub(crate) trait Accounting: Send + Sync {
+ fn charge(
+ &self,
+ topic_partition: &TopicPartition,
+ generation: u64,
+ charges: &[(Offset, Charge)],
+ ) -> Result<Held, Rejection>;
+ fn settled(
+ &self,
+ topic_partition: &TopicPartition,
+ generation: u64,
+ offsets: impl IntoIterator<Item = Offset>,
+ ) -> Result<Option<Offset>, Rejection>;
+ fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, HandedOut>>;
+ fn take_all(&self) -> Option<HashMap<TopicPartition, HandedOut>>;
+ fn forget_partitions(&self, topic_partitions: &[TopicPartition]);
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) struct HandedOut {
+ pub(crate) offset: Offset,
+ pub(crate) remaining: Held,
+}
- fn take_all(&self) -> Option<HashMap<TopicPartition, Offset>>;
+pub(crate) trait Admission: Send + Sync {
+ fn charged(&self, topic_partition: &TopicPartition, uncommitted: Charge) -> Option<Pause>;
+ fn handed_out(&self, handed_out: &HashMap<TopicPartition, HandedOut>) -> Vec<TopicPartition>;
fn forget_partitions(&self, topic_partitions: &[TopicPartition]);
}
+#[derive(Debug, PartialEq, Eq)]
+pub(crate) struct Pause;
+
+pub(crate) struct Commit {
+ pub(crate) offsets: HashMap<TopicPartition, Offset>,
+ pub(crate) resume: Vec<TopicPartition>,
+}
+
+fn offsets(handed_out: HashMap<TopicPartition, HandedOut>) -> HashMap<TopicPartition, Offset> {
+ handed_out
+ .into_iter()
+ .map(|(topic_partition, handed_out)| (topic_partition, handed_out.offset))
+ .collect()
+}
+
pub(crate) fn for_granularity(
granularity: CompletionGranularity,
ledger: Arc<TopicOffsetLedger>,
commit_interval: Duration,
+ limits: BudgetLimits,
) -> Arc<dyn Settlement> {
match granularity {
- CompletionGranularity::Poll => Arc::new(PollSettlement::new(ledger)),
- CompletionGranularity::Group => Arc::new(GroupSettlement::new(ledger, commit_interval)),
+ CompletionGranularity::Poll => {
+ if limits.is_set() {
+ warn!(
+ "Ignoring the partition budget: it needs CONSUMER_COMPLETION_GRANULARITY=group"
+ );
+ }
+ Arc::new(PollSettlement::new(ledger))
+ }
+ CompletionGranularity::Group => {
+ Arc::new(GroupSettlement::new(ledger, commit_interval, limits))
+ }
}
}
diff --git a/rust/ingestion-consumer/src/settlement/poll.rs b/rust/ingestion-consumer/src/settlement/poll.rs
index 1ac43e2a071f..7346a7b53568 100644
--- a/rust/ingestion-consumer/src/settlement/poll.rs
+++ b/rust/ingestion-consumer/src/settlement/poll.rs
@@ -1,17 +1,16 @@
-use std::collections::HashMap;
use std::sync::Arc;
use std::time::Instant;
-use common_kafka_consumer::{Charge, Held, Offset, Rejection, TopicOffsetLedger, TopicPartition};
+use common_kafka_consumer::{Charge, Offset, Rejection, TopicOffsetLedger, TopicPartition};
use metrics::counter;
use tracing::warn;
-use super::{ChargedSlice, Settlement, Uncommitted};
+use super::{offsets, Accounting, ChargedSlice, Commit, Pause, Settlement, Uncommitted};
use crate::commit_pacer::CommitPacer;
use crate::ledger_rejection::{report_rejection, RejectedSlice};
-pub(crate) struct PollSettlement {
- uncommitted: Uncommitted,
+pub(crate) struct PollSettlement<A: Accounting = Uncommitted> {
+ uncommitted: A,
}
impl PollSettlement {
@@ -22,15 +21,16 @@ impl PollSettlement {
}
}
-impl Settlement for PollSettlement {
+impl<A: Accounting + 'static> Settlement for PollSettlement<A> {
fn charge(
&self,
topic_partition: &TopicPartition,
generation: u64,
charges: &[(Offset, Charge)],
- ) -> Result<Held, Rejection> {
+ ) -> Result<Option<Pause>, Rejection> {
self.uncommitted
.charge(topic_partition, generation, charges)
+ .map(|_| None)
}
fn settle_run(
@@ -59,12 +59,20 @@ impl Settlement for PollSettlement {
Ok(())
}
- fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, Offset>> {
- self.uncommitted.take_due(now)
+ fn take_due(&self, now: Instant) -> Option<Commit> {
+ let handed_out = self.uncommitted.take_due(now)?;
+ Some(Commit {
+ offsets: offsets(handed_out),
+ resume: Vec::new(),
+ })
}
- fn take_all(&self) -> Option<HashMap<TopicPartition, Offset>> {
- self.uncommitted.take_all()
+ fn take_all(&self) -> Option<Commit> {
+ let handed_out = self.uncommitted.take_all()?;
+ Some(Commit {
+ offsets: offsets(handed_out),
+ resume: Vec::new(),
+ })
}
fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {
@@ -87,7 +95,7 @@ impl NoFrontier {
}
fn settle_slices(
- uncommitted: &Uncommitted,
+ uncommitted: &impl Accounting,
slices: &[ChargedSlice<'_>],
) -> anyhow::Result<Option<NoFrontier>> {
let mut settled = false;
@@ -121,6 +129,209 @@ fn settle_slices(
#[cfg(test)]
mod tests {
+ use std::collections::HashMap;
+
+ use common_kafka_consumer::{Held, LedgerError};
+ use metrics_util::debugging::{DebugValue, DebuggingRecorder};
+
+ use super::super::fakes::{FakeAccounting, Settled};
+ use super::super::HandedOut;
+ use super::*;
+ use crate::order_sentinel::OffsetSpan;
+
+ fn p(partition: i32) -> TopicPartition {
+ TopicPartition::new("events", partition)
+ }
+
+ fn settlement() -> (PollSettlement<FakeAccounting>, FakeAccounting) {
+ let accounting = FakeAccounting::default();
+ let settlement = PollSettlement {
+ uncommitted: accounting.clone(),
+ };
+ (settlement, accounting)
+ }
+
+ struct Slice {
+ topic_partition: TopicPartition,
+ charges: Vec<(Offset, Charge)>,
+ }
+
+ impl Slice {
+ fn of(partition: i32, offsets: &[i64]) -> Self {
+ Self {
+ topic_partition: p(partition),
+ charges: offsets
+ .iter()
+ .map(|&offset| (Offset(offset), Charge::ZERO))
+ .collect(),
+ }
+ }
+
+ fn charged(&self) -> ChargedSlice<'_> {
+ ChargedSlice {
+ topic_partition: &self.topic_partition,
+ generation: 2,
+ span: OffsetSpan {
+ first: self.charges[0].0 .0,
+ last: self.charges[self.charges.len() - 1].0 .0,
+ },
+ charges: &self.charges,
+ }
+ }
+ }
+
+ fn stale() -> Rejection {
+ Rejection::Stale {
+ stamp: 0,
+ generation: 1,
+ }
+ }
+
+ fn settle_poll_counting_skips(
+ settlement: &PollSettlement<FakeAccounting>,
+ slices: &[Slice],
+ ) -> (anyhow::Result<()>, Vec<String>) {
+ let recorder = DebuggingRecorder::new();
+ let snapshotter = recorder.snapshotter();
+ let charged: Vec<ChargedSlice<'_>> = slices.iter().map(Slice::charged).collect();
+ let result = metrics::with_local_recorder(&recorder, || settlement.settle_poll(&charged));
+ let skipped = snapshotter
+ .snapshot()
+ .into_vec()
+ .into_iter()
+ .filter(|(key, _, _, value)| {
+ key.key().name() == "ingestion_consumer_commits_skipped_total"
+ && matches!(value, DebugValue::Counter(count) if *count > 0)
+ })
+ .filter_map(|(key, _, _, _)| {
+ key.key()
+ .labels()
+ .find(|label| label.key() == "reason")
+ .map(|label| label.value().to_string())
+ })
+ .collect();
+ (result, skipped)
+ }
+
+ #[test]
+ fn a_retired_poll_settles_each_slice_under_its_generation() {
+ let (settlement, accounting) = settlement();
+
+ settle_poll_counting_skips(&settlement, &[Slice::of(0, &[10, 11]), Slice::of(1, &[20])])
+ .0
+ .unwrap();
+
+ assert_eq!(
+ accounting.settles_seen(),
+ vec![
+ Settled {
+ topic_partition: p(0),
+ generation: 2,
+ offsets: vec![Offset(10), Offset(11)],
+ },
+ Settled {
+ topic_partition: p(1),
+ generation: 2,
+ offsets: vec![Offset(20)],
+ },
+ ]
+ );
+ }
+
+ struct RetiredPoll {
+ settles: Vec<Result<Option<Offset>, Rejection>>,
+ expected_skip: Option<&'static str>,
+ }
+
+ #[rstest::rstest]
+ #[case::a_frontier(RetiredPoll { settles: vec![Ok(Some(Offset(12)))], expected_skip: None })]
+ #[case::behind_an_earlier_poll(RetiredPoll { settles: vec![Ok(None)], expected_skip: Some("no_frontier") })]
+ #[case::every_slice_stale(RetiredPoll { settles: vec![Err(stale())], expected_skip: Some("rejected") })]
+ #[case::one_stale_one_frontier(RetiredPoll { settles: vec![Err(stale()), Ok(Some(Offset(21)))], expected_skip: None })]
+ #[case::one_stale_one_behind(RetiredPoll { settles: vec![Err(stale()), Ok(None)], expected_skip: Some("no_frontier") })]
+ fn a_retired_poll_that_hands_over_no_frontier_counts_why(#[case] poll: RetiredPoll) {
+ let (settlement, accounting) = settlement();
+ let slices: Vec<Slice> = (0..poll.settles.len() as i32)
+ .map(|partition| Slice::of(partition, &[10]))
+ .collect();
+ for result in poll.settles {
+ accounting.settle_returns(result);
+ }
+
+ let (result, skipped) = settle_poll_counting_skips(&settlement, &slices);
+
+ result.unwrap();
+ assert_eq!(
+ skipped,
+ poll.expected_skip
+ .into_iter()
+ .map(String::from)
+ .collect::<Vec<_>>()
+ );
+ }
+
+ #[test]
+ fn a_violating_slice_fails_the_poll_and_stops_settling() {
+ let (settlement, accounting) = settlement();
+ accounting.settle_returns(Err(Rejection::Violation {
+ error: LedgerError::CompletedTwice { offset: Offset(10) },
+ stamp: 2,
+ generation: 2,
+ held: Held::default(),
+ }));
+
+ let (result, _) =
+ settle_poll_counting_skips(&settlement, &[Slice::of(0, &[10]), Slice::of(1, &[20])]);
+
+ assert!(result.is_err());
+ assert_eq!(accounting.settles_seen().len(), 1);
+ }
+
+ #[test]
+ fn a_run_settles_nothing_before_its_poll_retires() {
+ let (settlement, accounting) = settlement();
+
+ settlement.settle_run(&p(0), 2, &[Offset(10)]).unwrap();
+
+ assert!(accounting.settles_seen().is_empty());
+ }
+
+ #[test]
+ fn a_charge_never_pauses() {
+ let (settlement, accounting) = settlement();
+ accounting.charge_returns(Ok(Held {
+ offsets: usize::MAX,
+ charge: Charge {
+ events: u64::MAX,
+ bytes: u64::MAX,
+ },
+ }));
+
+ assert_eq!(settlement.charge(&p(0), 2, &[]), Ok(None));
+ }
+
+ #[test]
+ fn a_hand_out_commits_without_resuming() {
+ let (settlement, accounting) = settlement();
+ accounting.take_hands_out(HashMap::from([(
+ p(0),
+ HandedOut {
+ offset: Offset(12),
+ remaining: Held::default(),
+ },
+ )]));
+
+ let commit = settlement.take_due(Instant::now()).expect("handed out");
+
+ assert_eq!(commit.offsets, HashMap::from([(p(0), Offset(12))]));
+ assert!(commit.resume.is_empty());
+ }
+}
+
+#[cfg(test)]
+mod integration_tests {
+ use std::collections::HashMap;
+
use common_kafka_consumer::Charge;
use super::*;
@@ -176,7 +387,9 @@ mod tests {
settlement.settle_poll(&[p0.slice(), p1.slice()]).unwrap();
assert_eq!(
- settlement.take_due(Instant::now()),
+ settlement
+ .take_due(Instant::now())
+ .map(|commit| commit.offsets),
Some(HashMap::from([
(p0.topic_partition, Offset(12)),
(p1.topic_partition, Offset(22)),
@@ -195,7 +408,9 @@ mod tests {
settlement.settle_poll(&[older.slice()]).unwrap();
assert_eq!(
- settlement.take_due(Instant::now()),
+ settlement
+ .take_due(Instant::now())
+ .map(|commit| commit.offsets),
Some(HashMap::from([(older.topic_partition, Offset(14))]))
);
}
@@ -211,25 +426,4 @@ mod tests {
assert!(settlement.take_due(Instant::now()).is_none());
}
-
- #[test]
- fn an_accepted_run_settles_nothing_before_its_poll_retires() {
- let (ledger, settlement) = settlement();
- let poll = charged(&ledger, 0, 10, 11);
-
- settlement
- .settle_run(
- &poll.topic_partition,
- poll.generation,
- &[Offset(10), Offset(11)],
- )
- .unwrap();
- assert!(settlement.take_due(Instant::now()).is_none());
-
- settlement.settle_poll(&[poll.slice()]).unwrap();
- assert_eq!(
- settlement.take_due(Instant::now()),
- Some(HashMap::from([(poll.topic_partition, Offset(12))]))
- );
- }
}
diff --git a/rust/ingestion-consumer/src/settlement/uncommitted.rs b/rust/ingestion-consumer/src/settlement/uncommitted.rs
index c6c62a0ecab2..b1cf6bb87897 100644
--- a/rust/ingestion-consumer/src/settlement/uncommitted.rs
+++ b/rust/ingestion-consumer/src/settlement/uncommitted.rs
@@ -5,6 +5,7 @@ use std::time::Instant;
use common_kafka_consumer::{Charge, Held, Offset, Rejection, TopicOffsetLedger, TopicPartition};
use metrics::gauge;
+use super::{Accounting, HandedOut};
use crate::commit_pacer::CommitPacer;
const UNCOMMITTED_OFFSETS: &str = "kafka_consumer_ledger_uncommitted_offsets";
@@ -17,7 +18,7 @@ const UNCOMMITTED_BYTES: &str = "kafka_consumer_ledger_uncommitted_bytes";
///
/// Every call runs on the consumer loop task, one at a time, so a move between
/// the ledger and the pacer never interleaves with a read of both.
-pub(super) struct Uncommitted {
+pub(crate) struct Uncommitted {
ledger: Arc<TopicOffsetLedger>,
pacer: Mutex<CommitPacer>,
}
@@ -30,7 +31,19 @@ impl Uncommitted {
}
}
- pub(super) fn charge(
+ fn hand_out(&self, due: HashMap<TopicPartition, Offset>) -> HashMap<TopicPartition, HandedOut> {
+ due.into_iter()
+ .map(|(topic_partition, offset)| {
+ let remaining = self.ledger.held(&topic_partition);
+ publish(&topic_partition, remaining);
+ (topic_partition, HandedOut { offset, remaining })
+ })
+ .collect()
+ }
+}
+
+impl Accounting for Uncommitted {
+ fn charge(
&self,
topic_partition: &TopicPartition,
generation: u64,
@@ -44,11 +57,7 @@ impl Uncommitted {
Ok(uncommitted)
}
- /// Called after a group run or a poll settles: mark its offsets complete
- /// and move the frontier that results, with its charge, to the pacer.
- /// Returns that frontier, `None` while the front of the window is still
- /// incomplete.
- pub(super) fn settled(
+ fn settled(
&self,
topic_partition: &TopicPartition,
generation: u64,
@@ -66,22 +75,17 @@ impl Uncommitted {
Ok(frontier)
}
- /// Refunds the charge of every frontier handed out; the consumer submits
- /// their commit right after.
- pub(super) fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, Offset>> {
+ fn take_due(&self, now: Instant) -> Option<HashMap<TopicPartition, HandedOut>> {
let due = self.pacer.lock().unwrap().take_due(now)?;
- self.publish_window(&due);
- Some(due)
+ Some(self.hand_out(due))
}
- /// Like `take_due`, without waiting for the interval.
- pub(super) fn take_all(&self) -> Option<HashMap<TopicPartition, Offset>> {
+ fn take_all(&self) -> Option<HashMap<TopicPartition, HandedOut>> {
let due = self.pacer.lock().unwrap().take_all()?;
- self.publish_window(&due);
- Some(due)
+ Some(self.hand_out(due))
}
- pub(super) fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {
+ fn forget_partitions(&self, topic_partitions: &[TopicPartition]) {
self.ledger.forget_partitions(
topic_partitions
.iter()
@@ -95,12 +99,6 @@ impl Uncommitted {
publish(topic_partition, Held::default());
}
}
-
- fn publish_window(&self, due: &HashMap<TopicPartition, Offset>) {
- for topic_partition in due.keys() {
- publish(topic_partition, self.ledger.held(topic_partition));
- }
- }
}
fn publish(topic_partition: &TopicPartition, uncommitted: Held) {
@@ -155,12 +153,23 @@ mod tests {
Uncommitted::new(Arc::new(TopicOffsetLedger::new()), CommitPacer::immediate())
}
- fn published(snapshotter: &Snapshotter) -> Held {
+ fn published(snapshotter: &Snapshotter, topic_partition: &TopicPartition) -> Held {
+ let partition = topic_partition.partition.to_string();
let mut published = Held::default();
for (key, _, _, value) in snapshotter.snapshot().into_vec() {
let DebugValue::Gauge(value) = value else {
continue;
};
+ let labelled =
+ key.key()
+ .labels()
+ .any(|label| label.key() == "partition" && label.value() == partition)
+ && key.key().labels().any(|label| {
+ label.key() == "topic" && label.value() == topic_partition.topic
+ });
+ if !labelled {
+ continue;
+ }
let value = value.into_inner();
match key.key().name() {
UNCOMMITTED_OFFSETS => published.offsets = value as usize,
@@ -193,7 +202,17 @@ mod tests {
assert_eq!(charge(&uncommitted, 0, 11), held(2));
- uncommitted.take_due(Instant::now()).expect("due");
+ let handed_out = uncommitted.take_due(Instant::now()).expect("due");
+ assert_eq!(
+ handed_out,
+ HashMap::from([(
+ p0(),
+ HandedOut {
+ offset: Offset(11),
+ remaining: held(1),
+ }
+ )])
+ );
assert_eq!(charge(&uncommitted, 0, 12), held(2));
}
@@ -228,6 +247,48 @@ mod tests {
assert_eq!(charge_p1(22), held(3));
}
+ #[test]
+ fn a_hand_out_carries_each_partitions_own_remaining_charge() {
+ let recorder = DebuggingRecorder::new();
+ let snapshotter = recorder.snapshotter();
+ let _guard = metrics::set_default_local_recorder(&recorder);
+ let uncommitted = uncommitted();
+ let p1 = TopicPartition::new("events", 1);
+ charge(&uncommitted, 0, 10);
+ charge(&uncommitted, 0, 11);
+ for offset in [20, 21, 22] {
+ uncommitted
+ .charge(&p1, 0, &[(Offset(offset), EACH)])
+ .expect("the ledger accepts the charge");
+ }
+ settled(&uncommitted, &[10]);
+ uncommitted
+ .settled(&p1, 0, [Offset(20)])
+ .expect("the ledger settles");
+
+ assert_eq!(
+ uncommitted.take_due(Instant::now()).expect("due"),
+ HashMap::from([
+ (
+ p0(),
+ HandedOut {
+ offset: Offset(11),
+ remaining: held(1),
+ }
+ ),
+ (
+ p1.clone(),
+ HandedOut {
+ offset: Offset(21),
+ remaining: held(2),
+ }
+ ),
+ ])
+ );
+ assert_eq!(published(&snapshotter, &p0()), held(1));
+ assert_eq!(published(&snapshotter, &p1), held(2));
+ }
+
#[test]
fn a_forgotten_partition_drops_its_window_and_its_pending_take() {
let uncommitted = uncommitted();
@@ -274,15 +335,19 @@ mod tests {
charge(&uncommitted, 0, 10);
charge(&uncommitted, 0, 11);
- assert_eq!(published(&snapshotter), held(2));
+ assert_eq!(published(&snapshotter, &p0()), held(2));
settled(&uncommitted, &[10]);
- assert_eq!(published(&snapshotter), held(2), "a take moves the charge");
+ assert_eq!(
+ published(&snapshotter, &p0()),
+ held(2),
+ "a take moves the charge"
+ );
uncommitted.take_due(Instant::now()).expect("due");
- assert_eq!(published(&snapshotter), held(1));
+ assert_eq!(published(&snapshotter, &p0()), held(1));
uncommitted.forget_partitions(&[p0()]);
- assert_eq!(published(&snapshotter), Held::default());
+ assert_eq!(published(&snapshotter, &p0()), Held::default());
}
}
diff --git a/rust/ingestion-consumer/tests/e2e_integration_test.rs b/rust/ingestion-consumer/tests/e2e_integration_test.rs
index 6378827308b9..d28543db0aa3 100644
--- a/rust/ingestion-consumer/tests/e2e_integration_test.rs
+++ b/rust/ingestion-consumer/tests/e2e_integration_test.rs
@@ -6,7 +6,7 @@
mod common;
use std::collections::HashSet;
-use std::num::{NonZeroU32, NonZeroUsize};
+use std::num::{NonZeroU32, NonZeroU64, NonZeroUsize};
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::{Arc, Mutex, OnceLock};
use std::time::Duration;
@@ -38,6 +38,7 @@ use ingestion_consumer::config::CompletionGranularity;
use ingestion_consumer::consumer::{IngestionConsumer, IngestionConsumerOptions};
use ingestion_consumer::discovery::reconcile_membership;
use ingestion_consumer::grpc_transport::{GrpcPort, GrpcTransport};
+use ingestion_consumer::partition_budget::BudgetLimits;
use ingestion_consumer::rebalance::RebalanceContext;
use ingestion_consumer::routing::RoutingStrategy;
use ingestion_consumer::worker_registry::{
@@ -719,6 +720,7 @@ impl Harness {
None,
CompletionGranularity::Poll,
COMMIT_INTERVAL,
+ BudgetLimits::unset(),
)
.await
}
@@ -751,6 +753,7 @@ impl Harness {
}),
CompletionGranularity::Poll,
COMMIT_INTERVAL,
+ BudgetLimits::unset(),
)
.await
}
@@ -778,6 +781,7 @@ impl Harness {
None,
CompletionGranularity::Poll,
COMMIT_INTERVAL,
+ BudgetLimits::unset(),
)
.await
}
@@ -804,6 +808,7 @@ impl Harness {
None,
CompletionGranularity::Poll,
COMMIT_INTERVAL,
+ BudgetLimits::unset(),
)
.await
}
@@ -825,6 +830,7 @@ impl Harness {
None,
CompletionGranularity::Poll,
COMMIT_INTERVAL,
+ BudgetLimits::unset(),
)
.await
}
@@ -868,6 +874,51 @@ impl Harness {
None,
CompletionGranularity::Group,
commit_interval,
+ BudgetLimits::unset(),
+ )
+ .await
+ }
+
+ async fn start_with_partition_budget(
+ pack_targets: PackTargets,
+ topic: &str,
+ partitions: i32,
+ worker_count: usize,
+ partition_budget: BudgetLimits,
+ ) -> Self {
+ Self::start_with_partition_budget_committing_every(
+ pack_targets,
+ topic,
+ partitions,
+ worker_count,
+ partition_budget,
+ COMMIT_INTERVAL,
+ )
+ .await
+ }
+
+ async fn start_with_partition_budget_committing_every(
+ pack_targets: PackTargets,
+ topic: &str,
+ partitions: i32,
+ worker_count: usize,
+ partition_budget: BudgetLimits,
+ commit_interval: Duration,
+ ) -> Self {
+ Self::start_inner(
+ topic,
+ partitions,
+ worker_count,
+ 8,
+ Duration::from_secs(60),
+ fast_registry_config(),
+ 0,
+ ComponentOptions::new(),
+ pack_targets,
+ None,
+ CompletionGranularity::Group,
+ commit_interval,
+ partition_budget,
)
.await
}
@@ -886,6 +937,7 @@ impl Harness {
budget: Option<RequestBudget>,
completion_granularity: CompletionGranularity,
commit_interval: Duration,
+ partition_budget: BudgetLimits,
) -> Self {
create_topic(topic, partitions).await;
@@ -941,6 +993,7 @@ impl Harness {
completion_granularity,
commit_interval,
ledger: Arc::clone(&ledger),
+ partition_budget,
},
handle,
);
@@ -1033,6 +1086,7 @@ impl Harness {
completion_granularity: self.completion_granularity,
commit_interval: COMMIT_INTERVAL,
ledger: Arc::clone(&self.ledger),
+ partition_budget: BudgetLimits::unset(),
},
handle,
);
@@ -1124,6 +1178,7 @@ impl Harness {
completion_granularity: self.completion_granularity,
commit_interval: COMMIT_INTERVAL,
ledger: Arc::new(TopicOffsetLedger::new()),
+ partition_budget: BudgetLimits::unset(),
},
handle,
);
@@ -2790,6 +2845,165 @@ async fn group_granularity_commits_a_partition_while_another_stalls() {
harness.stop().await;
}
+#[tokio::test]
+async fn a_partition_at_its_budget_pauses_until_a_commit_frees_it() {
+ let topic = format!("e2e-partition-budget-{}", Uuid::new_v4());
+ let budget = BudgetLimits::new(NonZeroU64::new(5), None, 0.5).expect("a valid budget");
+ let harness =
+ Harness::start_with_partition_budget(common::PRODUCTION_PACK_TARGETS, &topic, 1, 1, budget)
+ .await;
+ let p0 = TopicPartition::new(&topic, 0);
+ let guard = harness.workers[0].block().await;
+ let producer = make_producer();
+
+ for seq in 0..5usize {
+ produce(&producer, &topic, 0, "tok", "user-a", seq).await;
+ }
+ wait_until(Duration::from_secs(10), "the budget to fill", || {
+ harness.ledger.held(&p0).offsets == 5
+ })
+ .await;
+
+ for seq in 5..20usize {
+ produce(&producer, &topic, 0, "tok", "user-a", seq).await;
+ }
+ tokio::time::sleep(Duration::from_secs(1)).await;
+ assert_eq!(
+ harness.ledger.held(&p0).offsets,
+ 5,
+ "a paused partition fetches nothing more"
+ );
+
+ drop(guard);
+ wait_until(
+ Duration::from_secs(10),
+ "the partition to resume and drain",
+ || harness.committed_offset(0) == Some(20),
+ )
+ .await;
+
+ harness.stop().await;
+}
+
+#[tokio::test]
+async fn a_partition_at_its_budget_does_not_pause_another() {
+ let topic = format!("e2e-partition-budget-two-{}", Uuid::new_v4());
+ let budget = BudgetLimits::new(NonZeroU64::new(5), None, 0.5).expect("a valid budget");
+ // One message per request, or the packer puts both keys on one worker.
+ let harness =
+ Harness::start_with_partition_budget(common::ONE_MESSAGE_PER_REQUEST, &topic, 2, 2, budget)
+ .await;
+ let mut guards = stall_two_partitions_on_two_workers(&harness, &topic).await;
+
+ guards[0] = None;
+ wait_until(
+ Duration::from_secs(10),
+ "the released worker to accept its key",
+ || harness.workers[0].count() == 5,
+ )
+ .await;
+ let released = accepted_partition(&harness, 0);
+ let stalled = 1 - released;
+ let stalled_partition = TopicPartition::new(&topic, stalled);
+
+ let producer = make_producer();
+ for seq in 5..10usize {
+ for (partition, key) in STALL_KEYS {
+ produce(&producer, &topic, partition, "tok", key, seq).await;
+ }
+ }
+ wait_until(
+ Duration::from_secs(10),
+ "the released partition to resume and drain",
+ || harness.committed_offset(released) == Some(10),
+ )
+ .await;
+ assert_eq!(
+ harness.ledger.held(&stalled_partition).offsets,
+ 5,
+ "the partition at its budget fetches nothing more"
+ );
+ assert_eq!(harness.committed_offset(stalled), None);
+
+ guards[1] = None;
+ wait_until(
+ Duration::from_secs(10),
+ "the stalled partition to resume and drain",
+ || harness.committed_offset(stalled) == Some(10),
+ )
+ .await;
+
+ harness.stop().await;
+}
+
+#[tokio::test]
+async fn a_partition_paused_at_its_budget_fetches_again_after_a_rebalance_returns_it() {
+ let topic = format!("e2e-partition-budget-rebalance-{}", Uuid::new_v4());
+ let budget = BudgetLimits::new(NonZeroU64::new(5), None, 0.5).expect("a valid budget");
+ let harness = Harness::start_with_partition_budget_committing_every(
+ common::PRODUCTION_PACK_TARGETS,
+ &topic,
+ 1,
+ 1,
+ budget,
+ Duration::from_secs(20),
+ )
+ .await;
+ let p0 = TopicPartition::new(&topic, 0);
+ let producer = make_producer();
+
+ produce(&producer, &topic, 0, "tok", "user-a", 0).await;
+ wait_until(
+ Duration::from_secs(10),
+ "the first frontier to commit",
+ || harness.committed_offset(0) == Some(1),
+ )
+ .await;
+ for seq in 1..6usize {
+ produce(&producer, &topic, 0, "tok", "user-a", seq).await;
+ }
+ wait_until(
+ Duration::from_secs(10),
+ "the interval to hold the budget's worth of settled offsets",
+ || harness.workers[0].count() == 6 && harness.ledger.held(&p0).offsets == 0,
+ )
+ .await;
+
+ let generation_before = harness.ledger.generation(&p0);
+ let second = harness.join_second_consumer().await;
+ wait_until(
+ Duration::from_secs(15),
+ "the rebalance to revoke the partition",
+ || harness.ledger.generation(&p0) > generation_before,
+ )
+ .await;
+ assert_eq!(
+ harness.committed_offset(0),
+ Some(1),
+ "the partition must still be at its budget when it is revoked"
+ );
+ second.stop().await;
+
+ for seq in 6..10usize {
+ produce(&producer, &topic, 0, "tok", "user-a", seq).await;
+ }
+ wait_until(
+ Duration::from_secs(40),
+ "the returned partition to fetch again",
+ || {
+ let delivered: HashSet<usize> = harness
+ .workers
+ .iter()
+ .flat_map(|w| w.seqs_for("user-a"))
+ .collect();
+ (6..10).all(|seq| delivered.contains(&seq))
+ },
+ )
+ .await;
+
+ harness.stop().await;
+}
+
/// A worker that processes a batch but whose ACK is lost is indistinguishable
/// from a failed send, so the consumer replays the batch: duplicates are the
/// accepted cost of at-least-once, loss never is, and the replay lands behind
@@ -3502,6 +3716,7 @@ async fn fenced_static_member_exits_on_fatal_error() {
completion_granularity: CompletionGranularity::Poll,
commit_interval: Duration::ZERO,
ledger: Arc::new(TopicOffsetLedger::new()),
+ partition_budget: BudgetLimits::unset(),
},
handle,
);