Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(ingestion): give every worker the same share of aperture slices (#129711)

Paweł Ledwoń6a523ffeaa6b
diff --git a/rust/ingestion-consumer/src/aperture.rs b/rust/ingestion-consumer/src/aperture.rs
index 1162ea48746c..3d21dfacd538 100644
--- a/rust/ingestion-consumer/src/aperture.rs
+++ b/rust/ingestion-consumer/src/aperture.rs
@@ -1,33 +1,45 @@
-//! Deterministic aperture: ring-slice candidate selection.
+//! Deterministic aperture: weighted ring-slice candidate selection.
//!
-//! Every consumer sorts the full worker set into the same ring, takes the
-//! slice starting at `peer_index * ring_len / peer_count`, and places its
-//! requests only within that slice (via the configured selection
-//! strategy). Because peers agree on both the ring order (same EndpointSlices,
-//! same sort) and their own indices ([`k8s_awareness::PeerTracker`]), the
-//! slices tile the ring without any coordination, so each consumer talks to
-//! a handful of workers instead of the whole pool, which is what keeps
-//! sub-batches consolidated. Modeled on Finagle's deterministic aperture.
+//! Every consumer sorts the workers into the same ring, takes an arc
+//! of it, and places its requests only on the workers under that arc (via the
+//! configured selection strategy). Because peers agree on both the ring order
+//! (same EndpointSlices, same sort) and their own indices
+//! ([`k8s_awareness::PeerTracker`]), the arcs need no coordination, so each
+//! consumer talks to a handful of workers instead of the whole pool, which is
+//! what keeps sub-batches consolidated. Modeled on Finagle's deterministic
+//! aperture.
//!
-//! The configured width is a *minimum*: it's floored at
-//! `ceil(ring_len / peer_count)` — the narrowest equal slices that can cover
-//! the ring — so the union of the fleet's slices covers every worker even
-//! when workers outnumber `width x dispatchers`. Every peer computes the same
-//! floor from the same inputs, so the property stays coordination-free.
+//! The ring is measured in units: each worker spans `peer_count` units. Peer
+//! `i`'s arc starts at unit `i x ring_len`, so the arcs start at equal
+//! distances, and its length is a multiple of `ring_len` units. So every unit
+//! is under the same number of arcs. A worker's weight is the number of its
+//! units under the arc, so every worker gets the same total weight from the
+//! fleet. With 7 workers, 3 peers, and arcs of 7 units:
//!
-//! Unroutable ring positions (unhealthy or draining workers) are skipped and
-//! the slice extends further around the ring, so the *effective* width holds
-//! under partial failure.
+//! | peer | arc | weights |
+//! |------|-------|------------------|
+//! | 0 | 0-7 | w0 3, w1 3, w2 1 |
+//! | 1 | 7-14 | w2 2, w3 3, w4 2 |
+//! | 2 | 14-21 | w4 1, w5 3, w6 3 |
+//!
+//! The ring holds the workers that discovery lists, so a draining worker
+//! leaves every peer's ring and the arcs move to share its units. A worker
+//! that only this consumer finds unhealthy leaves only this slice, and the
+//! other workers of the slice take its share. When the routable workers hold
+//! less than half of the arc's weight, the consumer takes its arc of a ring of
+//! the routable workers instead, so that a large failure spreads over the
+//! whole routable pool and does not fall on the few workers left in a slice.
use std::collections::HashSet;
+use std::iter;
+use std::num::NonZeroU64;
+use crate::routing::Candidate;
use crate::worker_registry::WorkerId;
-/// The candidate workers for one assignment round: walk the ring from this
-/// dispatcher's slice start, collecting healthy workers until the effective
-/// width (`width` floored at `ceil(ring_len / peer_count)` for coverage) is
-/// reached or the ring is exhausted. `ring` is the canonically sorted full
-/// worker set; `healthy` the subset this dispatcher may route to.
+/// The weighted candidate workers for one assignment round. `ring` is the
+/// canonically sorted set of workers that discovery lists; `healthy` the
+/// subset this dispatcher may route to.
///
/// Returns `None` (caller falls back to the full healthy pool) when the
/// slice can't be computed: no peers known yet, an out-of-range index, or an
@@ -37,38 +49,125 @@ pub fn ring_slice(
healthy: &[WorkerId],
peer_index: Option<usize>,
peer_count: usize,
- width: usize,
-) -> Option<Vec<WorkerId>> {
+ min_width: usize,
+) -> Option<Vec<Candidate>> {
let index = peer_index?;
- if ring.is_empty() || peer_count == 0 || index >= peer_count || width == 0 {
+ if ring.is_empty() || peer_count == 0 || index >= peer_count || min_width == 0 {
return None;
}
+ let healthy: HashSet<&WorkerId> = healthy.iter().collect();
- // Coverage floor: consecutive slice starts are at most `ceil(len/peers)`
- // apart, so any narrower width would leave ring positions no dispatcher
- // ever routes fresh keys to. Flooring the width there guarantees the
- // union of the fleet's slices covers the whole ring for any pool/peer
- // ratio, at the cost of slight slice overlap when the division is uneven.
- let width = width.max(ring.len().div_ceil(peer_count));
+ let arc = arc_slice(ring, index, peer_count, min_width);
+ let arc_weight = total_weight(&arc);
+ let slice: Vec<Candidate> = arc
+ .into_iter()
+ .filter(|candidate| healthy.contains(&candidate.worker))
+ .collect();
+ if total_weight(&slice) * 2 >= arc_weight {
+ return Some(slice);
+ }
+ let routable_ring: Vec<WorkerId> = ring
+ .iter()
+ .filter(|worker| healthy.contains(worker))
+ .cloned()
+ .collect();
+ (!routable_ring.is_empty()).then(|| arc_slice(&routable_ring, index, peer_count, min_width))
+}
- let healthy_set: HashSet<&WorkerId> = healthy.iter().collect();
- let start = index * ring.len() / peer_count;
+fn total_weight(slice: &[Candidate]) -> u64 {
+ slice.iter().map(|candidate| candidate.weight.get()).sum()
+}
- let mut candidates = Vec::with_capacity(width.min(ring.len()));
- for offset in 0..ring.len() {
- let worker = &ring[(start + offset) % ring.len()];
- if healthy_set.contains(worker) {
- candidates.push(worker.clone());
- if candidates.len() == width {
- break;
- }
+/// Every worker under peer `index`'s arc of `ring`, weighted by its units
+/// under the arc. `ring` and `peer_count` must not be empty or zero.
+fn arc_slice(
+ ring: &[WorkerId],
+ index: usize,
+ peer_count: usize,
+ min_width: usize,
+) -> Vec<Candidate> {
+ let arc = Arc::new(ring.len(), peer_count, index, min_width);
+ let full_weight = NonZeroU64::new(arc.units_per_worker).expect("peer_count is above zero");
+ let covered = take_units(arc.spans(ring.len()), arc.len);
+ sum_by_position(covered, ring.len())
+ .into_iter()
+ .map(|(position, units)| {
+ let weight = NonZeroU64::new(units).expect("take_units yields no empty span");
+ Candidate::new(ring[position].clone(), weight, full_weight)
+ })
+ .collect()
+}
+
+/// The spans in order, cut so that their units add up to at most `units`.
+fn take_units(
+ spans: impl Iterator<Item = (usize, u64)>,
+ mut units: u64,
+) -> impl Iterator<Item = (usize, u64)> {
+ spans.map_while(move |(position, span)| {
+ let taken = span.min(units);
+ units -= taken;
+ (taken > 0).then_some((position, taken))
+ })
+}
+
+/// The units of each position, in the order of each position's first span.
+/// Every span must hold at least one unit.
+fn sum_by_position(
+ spans: impl Iterator<Item = (usize, u64)>,
+ ring_len: usize,
+) -> Vec<(usize, u64)> {
+ let mut units_at = vec![0u64; ring_len];
+ let mut order = Vec::new();
+ for (position, units) in spans {
+ if units_at[position] == 0 {
+ order.push(position);
}
+ units_at[position] += units;
}
+ order
+ .into_iter()
+ .map(|position| (position, units_at[position]))
+ .collect()
+}
- if candidates.is_empty() {
- return None;
+/// One peer's arc of the ring, in units.
+struct Arc {
+ units_per_worker: u64,
+ start: u64,
+ len: u64,
+}
+
+impl Arc {
+ /// The arc is the smallest multiple of `ring_len` units that covers
+ /// `min_width` workers. `spans` visits each unit of the ring once, so a
+ /// longer arc would still cover the ring only once. The clamp keeps the
+ /// numbers small.
+ fn new(ring_len: usize, peer_count: usize, index: usize, min_width: usize) -> Self {
+ let units_per_worker = peer_count as u64;
+ let spacing = ring_len as u64;
+ let multiple = (min_width as u64 * units_per_worker)
+ .div_ceil(spacing)
+ .clamp(1, units_per_worker);
+ Self {
+ units_per_worker,
+ start: index as u64 * spacing,
+ len: multiple * spacing,
+ }
+ }
+
+ /// The ring positions in walk order from the worker that holds `start`,
+ /// each with its units from the walk point on. That worker comes again
+ /// last with its units before `start`. Only an arc as long as the ring
+ /// gets to them.
+ fn spans(&self, ring_len: usize) -> impl Iterator<Item = (usize, u64)> + '_ {
+ let first = (self.start / self.units_per_worker) as usize;
+ let before_start = self.start % self.units_per_worker;
+ let whole =
+ (1..ring_len).map(move |offset| ((first + offset) % ring_len, self.units_per_worker));
+ iter::once((first, self.units_per_worker - before_start))
+ .chain(whole)
+ .chain(iter::once((first, before_start)))
}
- Some(candidates)
}
/// The canonical worker ring: the full registered set, sorted. All dispatchers
@@ -80,6 +179,8 @@ pub fn sorted_ring(mut workers: Vec<WorkerId>) -> Vec<WorkerId> {
#[cfg(test)]
mod tests {
+ use std::collections::HashMap;
+
use super::*;
fn wid(n: usize) -> WorkerId {
@@ -90,72 +191,206 @@ mod tests {
sorted_ring((1..=n).map(wid).collect())
}
+ fn weights(slice: &[Candidate]) -> Vec<(WorkerId, u64)> {
+ slice
+ .iter()
+ .map(|c| (c.worker.clone(), c.weight.get()))
+ .collect()
+ }
+
+ fn fleet_weights(ring: &[WorkerId], peers: usize, min_width: usize) -> HashMap<WorkerId, u64> {
+ let mut total = HashMap::new();
+ for index in 0..peers {
+ for candidate in ring_slice(ring, ring, Some(index), peers, min_width).unwrap() {
+ *total.entry(candidate.worker).or_insert(0) += candidate.weight.get();
+ }
+ }
+ total
+ }
+
#[test]
fn test_slices_tile_the_ring_across_peers() {
- // 3 peers x width 2 over 6 workers: each peer gets a distinct pair and
- // the union covers the whole ring — the coordination-free guarantee
- // the whole design rests on.
let ring = ring(6);
let mut seen = Vec::new();
for index in 0..3 {
let slice = ring_slice(&ring, &ring, Some(index), 3, 2).unwrap();
assert_eq!(slice.len(), 2, "peer {index}");
- seen.extend(slice);
+ seen.extend(slice.into_iter().map(|c| c.worker));
}
- let mut sorted = seen.clone();
- sorted.sort_unstable();
- sorted.dedup();
- assert_eq!(sorted.len(), 6, "slices must tile the ring without overlap");
+ seen.sort_unstable();
+ seen.dedup();
+ assert_eq!(seen.len(), 6, "slices must tile the ring without overlap");
}
#[test]
- fn test_width_floored_so_slices_cover_an_oversized_ring() {
- // 2 peers x configured width 3 over 8 workers: unfloored slices would
- // be spaced 4 apart and leave positions 3 and 7 with no traffic. The ceil(8/2)=4 floor widens each slice just enough that
- // the union covers the whole ring.
- let ring = ring(8);
- let mut seen = HashSet::new();
- for index in 0..2 {
- let slice = ring_slice(&ring, &ring, Some(index), 2, 3).unwrap();
- assert_eq!(slice.len(), 4, "peer {index} floored to ceil(8/2)");
- seen.extend(slice);
+ fn test_every_worker_gets_the_same_fleet_weight_when_peers_do_not_divide_the_ring() {
+ struct Case {
+ workers: usize,
+ peers: usize,
+ min_width: usize,
+ }
+ let cases = [
+ Case {
+ workers: 7,
+ peers: 3,
+ min_width: 1,
+ },
+ Case {
+ workers: 100,
+ peers: 17,
+ min_width: 3,
+ },
+ Case {
+ workers: 250,
+ peers: 30,
+ min_width: 3,
+ },
+ Case {
+ workers: 250,
+ peers: 30,
+ min_width: 32,
+ },
+ Case {
+ workers: 50,
+ peers: 8,
+ min_width: 48,
+ },
+ Case {
+ workers: 3,
+ peers: 5,
+ min_width: 1,
+ },
+ Case {
+ workers: 5,
+ peers: 3,
+ min_width: 10,
+ },
+ ];
+ for case in cases {
+ let ring = ring(case.workers);
+ let total = fleet_weights(&ring, case.peers, case.min_width);
+ assert_eq!(total.len(), case.workers, "every worker is covered");
+ let distinct: HashSet<u64> = total.values().copied().collect();
+ assert_eq!(
+ distinct.len(),
+ 1,
+ "{} workers, {} peers, min width {}: {distinct:?}",
+ case.workers,
+ case.peers,
+ case.min_width
+ );
}
- assert_eq!(seen.len(), 8, "every ring position must be covered");
}
#[test]
- fn test_width_floor_overlaps_slices_on_uneven_division() {
- // 3 peers over 7 workers: starts 0, 2, 4 with floor ceil(7/3)=3 give
- // slices {0,1,2}, {2,3,4}, {4,5,6} — full coverage, with boundary
- // positions shared by two peers rather than left uncovered.
- let ring = ring(7);
- let mut seen = HashSet::new();
- for index in 0..3 {
- seen.extend(ring_slice(&ring, &ring, Some(index), 3, 1).unwrap());
+ fn test_every_worker_gets_the_same_fleet_slots_when_arcs_end_inside_workers() {
+ let ring = ring(6);
+ let mut slots: HashMap<WorkerId, usize> = HashMap::new();
+ for index in 0..4 {
+ for candidate in ring_slice(&ring, &ring, Some(index), 4, 3).unwrap() {
+ *slots.entry(candidate.worker.clone()).or_insert(0) += candidate.slots(8);
+ }
}
- assert_eq!(seen.len(), 7, "uneven division must still cover the ring");
+ assert_eq!(slots.len(), 6);
+ assert!(slots.values().all(|n| *n == 16), "{slots:?}");
+ }
+
+ #[test]
+ fn test_arc_ends_get_the_covered_part_of_the_worker() {
+ let ring = ring(7);
+ let slice = ring_slice(&ring, &ring, Some(1), 3, 1).unwrap();
+ assert_eq!(
+ weights(&slice),
+ vec![
+ (ring[2].clone(), 2),
+ (ring[3].clone(), 3),
+ (ring[4].clone(), 2),
+ ]
+ );
+ }
+
+ #[test]
+ fn test_width_is_at_least_the_minimum() {
+ let ring = ring(8);
+ let slice = ring_slice(&ring, &ring, Some(0), 2, 5).unwrap();
+ assert_eq!(slice.len(), 8);
}
#[test]
fn test_slice_wraps_around_the_ring() {
- // Last peer's slice starts near the ring end and wraps to the front.
let ring = ring(5);
let slice = ring_slice(&ring, &ring, Some(2), 3, 3).unwrap();
- // start = 2 * 5 / 3 = 3 → positions 3, 4, 0.
assert_eq!(
- slice,
- vec![ring[3].clone(), ring[4].clone(), ring[0].clone()]
+ weights(&slice),
+ vec![
+ (ring[3].clone(), 2),
+ (ring[4].clone(), 3),
+ (ring[0].clone(), 3),
+ (ring[1].clone(), 2),
+ ]
);
}
#[test]
- fn test_slice_extends_past_unhealthy_workers() {
- // An unroutable worker inside the slice must not shrink the effective
- // width — the slice walks on around the ring.
+ fn test_an_arc_over_the_whole_ring_gives_every_worker_the_same_weight() {
+ let ring = ring(4);
+ let slice = ring_slice(&ring, &ring, Some(1), 3, 10).unwrap();
+ assert_eq!(slice.len(), 4);
+ assert!(slice.iter().all(|c| c.weight.get() == 3), "{slice:?}");
+ }
+
+ #[test]
+ fn test_with_exactly_half_the_arc_healthy_the_unhealthy_worker_leaves_the_slice() {
let ring = ring(6);
let healthy: Vec<WorkerId> = ring.iter().filter(|w| **w != ring[1]).cloned().collect();
let slice = ring_slice(&ring, &healthy, Some(0), 3, 2).unwrap();
- assert_eq!(slice, vec![ring[0].clone(), ring[2].clone()]);
+ assert_eq!(weights(&slice), vec![(ring[0].clone(), 3)]);
+ }
+
+ #[test]
+ fn test_with_less_than_half_the_arc_healthy_the_peer_takes_its_arc_of_the_healthy_ring() {
+ let ring = ring(7);
+ let arc = ring_slice(&ring, &ring, Some(1), 3, 1).unwrap();
+ assert_eq!(
+ weights(&arc),
+ vec![
+ (ring[2].clone(), 2),
+ (ring[3].clone(), 3),
+ (ring[4].clone(), 2)
+ ]
+ );
+ let healthy: Vec<WorkerId> = ring
+ .iter()
+ .filter(|w| **w != ring[2] && **w != ring[4])
+ .cloned()
+ .collect();
+
+ let slice = ring_slice(&ring, &healthy, Some(1), 3, 1).unwrap();
+
+ assert_eq!(
+ slice,
+ ring_slice(&healthy, &healthy, Some(1), 3, 1).unwrap()
+ );
+ assert_ne!(weights(&slice), vec![(ring[3].clone(), 3)]);
+ }
+
+ #[test]
+ fn test_with_no_healthy_worker_under_the_arc_the_peer_takes_its_arc_of_the_healthy_ring() {
+ let ring = ring(6);
+ let healthy: Vec<WorkerId> = ring[2..].to_vec();
+ let slice = ring_slice(&ring, &healthy, Some(0), 3, 2).unwrap();
+ assert_eq!(
+ slice,
+ ring_slice(&healthy, &healthy, Some(0), 3, 2).unwrap()
+ );
+ assert_eq!(
+ weights(&slice),
+ vec![
+ (ring[2].clone(), 3),
+ (ring[3].clone(), 3),
+ (ring[4].clone(), 2)
+ ]
+ );
}
#[test]
diff --git a/rust/ingestion-consumer/src/batcher/driver.rs b/rust/ingestion-consumer/src/batcher/driver.rs
index f059a0903686..9dc315a1a03c 100644
--- a/rust/ingestion-consumer/src/batcher/driver.rs
+++ b/rust/ingestion-consumer/src/batcher/driver.rs
@@ -24,6 +24,7 @@ use super::{make_batch_id, BatcherCompletion, BatcherOutputs, RevokeId};
use crate::debug_recorder::{partition_offsets, record_if, DebugEventKind, DebugRecorder};
use crate::grpc_transport::{GrpcTransport, SendOptions};
use crate::order_sentinel::{KeyOrderSentinel, SendKind, SentinelBatch};
+use crate::routing::Candidate;
use crate::transport::{SendError, SendOutcome};
use crate::types::{Accumulator, SerializedKafkaMessage};
use crate::worker_registry::WorkerId;
@@ -90,7 +91,7 @@ pub(super) trait RequestSender: Send + Sync + 'static {
}
pub(super) trait Workers: Send + 'static {
- fn candidates(&self) -> Vec<WorkerId>;
+ fn candidates(&self) -> Vec<Candidate>;
fn record_outcome(&self, worker: &WorkerId, fault: bool);
@@ -116,7 +117,7 @@ impl RequestSender for GrpcTransport {
}
impl Workers for WorkerPoolSource {
- fn candidates(&self) -> Vec<WorkerId> {
+ fn candidates(&self) -> Vec<Candidate> {
WorkerPoolSource::candidates(self)
}
@@ -572,7 +573,7 @@ mod tests {
#[derive(Default)]
struct WorkersState {
- candidates: Vec<WorkerId>,
+ candidates: Vec<Candidate>,
outcomes: Vec<(WorkerId, bool)>,
idle: Vec<WorkerId>,
}
@@ -581,7 +582,7 @@ mod tests {
struct FakeWorkers(Arc<Mutex<WorkersState>>);
impl Workers for FakeWorkers {
- fn candidates(&self) -> Vec<WorkerId> {
+ fn candidates(&self) -> Vec<Candidate> {
self.0.lock().unwrap().candidates.clone()
}
@@ -720,8 +721,10 @@ mod tests {
impl FakeWorkers {
fn set_candidates(&self, workers: &[&str]) {
- self.0.lock().unwrap().candidates =
- workers.iter().map(|w| WorkerId::from(*w)).collect();
+ self.0.lock().unwrap().candidates = workers
+ .iter()
+ .map(|w| Candidate::uniform(WorkerId::from(*w)))
+ .collect();
}
fn outcomes(&self) -> Vec<(WorkerId, bool)> {
diff --git a/rust/ingestion-consumer/src/batcher/fleet_tests.rs b/rust/ingestion-consumer/src/batcher/fleet_tests.rs
new file mode 100644
index 000000000000..d2409bd1608a
--- /dev/null
+++ b/rust/ingestion-consumer/src/batcher/fleet_tests.rs
@@ -0,0 +1,263 @@
+use std::collections::{HashMap, HashSet};
+use std::num::NonZeroUsize;
+use std::time::{Duration, Instant};
+
+use super::governor::{Adaptation, GovernorSettings};
+use super::key_queues::KeyRun;
+use super::packer::{PackTargets, Packer};
+use super::retry_policy::RetryPolicy;
+use super::state_machine::BatcherStateMachine;
+use super::test_support::message;
+use super::worker_assigner::WorkerAssigner;
+use crate::aperture::{ring_slice, sorted_ring};
+use crate::routing::{Candidate, Router, RoutingStrategy};
+use crate::worker_registry::WorkerId;
+
+const LATENCY_BUDGET: Duration = Duration::from_secs(1);
+const SLOW_RTT: Duration = Duration::from_secs(2);
+
+#[derive(Debug)]
+struct Fleet {
+ workers: usize,
+ peers: usize,
+ min_width: usize,
+ max_per_worker: usize,
+ min_permits: usize,
+ draining: Vec<usize>,
+ unhealthy: Vec<usize>,
+}
+
+impl Fleet {
+ fn new(workers: usize, peers: usize, min_width: usize, max: usize, min: usize) -> Self {
+ Self {
+ workers,
+ peers,
+ min_width,
+ max_per_worker: max,
+ min_permits: min,
+ draining: Vec::new(),
+ unhealthy: Vec::new(),
+ }
+ }
+
+ fn draining(self, positions: impl IntoIterator<Item = usize>) -> Self {
+ Self {
+ draining: positions.into_iter().collect(),
+ ..self
+ }
+ }
+
+ fn unhealthy(self, positions: impl IntoIterator<Item = usize>) -> Self {
+ Self {
+ unhealthy: positions.into_iter().collect(),
+ ..self
+ }
+ }
+
+ fn worker(position: usize) -> WorkerId {
+ WorkerId::from(format!("http://10.0.{}.{}:6738", position / 200, position % 200).as_str())
+ }
+
+ fn ring(&self) -> Vec<WorkerId> {
+ sorted_ring(
+ (0..self.workers)
+ .filter(|p| !self.draining.contains(p))
+ .map(Self::worker)
+ .collect(),
+ )
+ }
+
+ fn healthy(&self) -> Vec<WorkerId> {
+ let unhealthy: HashSet<WorkerId> =
+ self.unhealthy.iter().map(|p| Self::worker(*p)).collect();
+ self.ring()
+ .into_iter()
+ .filter(|w| !unhealthy.contains(w))
+ .collect()
+ }
+
+ fn slices(&self) -> Vec<Vec<Candidate>> {
+ let (ring, healthy) = (self.ring(), self.healthy());
+ (0..self.peers)
+ .map(|index| {
+ let slice = ring_slice(&ring, &healthy, Some(index), self.peers, self.min_width)
+ .expect("every peer gets a slice");
+ assert!(
+ !slice.is_empty(),
+ "{self:?}: peer {index} got an empty slice"
+ );
+ slice
+ })
+ .collect()
+ }
+
+ fn batcher(&self, now: Instant) -> BatcherStateMachine {
+ let packer = Packer::new(PackTargets {
+ events: NonZeroUsize::new(1),
+ bytes: None,
+ latency_budget: Duration::ZERO,
+ });
+ let assigner = WorkerAssigner::new(
+ Router::with_seed(RoutingStrategy::Aperture, 0),
+ self.max_per_worker,
+ )
+ .expect("valid request cap");
+ let retry = RetryPolicy::new(
+ Duration::from_millis(200),
+ Duration::from_millis(20),
+ Duration::from_millis(100),
+ )
+ .expect("valid retry policy");
+ BatcherStateMachine::new(packer, assigner, retry, Duration::from_secs(60), now)
+ .expect("valid batcher")
+ .with_governor(GovernorSettings {
+ adaptation: Adaptation::On,
+ latency_budget: Some(LATENCY_BUDGET),
+ min_permits_per_worker: self.min_permits,
+ ..GovernorSettings::observing()
+ })
+ .expect("valid governor settings")
+ }
+}
+
+fn backlog(requests: usize) -> Vec<KeyRun> {
+ (0..requests)
+ .map(|n| {
+ let key = format!("k{n}");
+ KeyRun {
+ messages: vec![message(&key, 0, n as i64)],
+ routing_key: key.into(),
+ }
+ })
+ .collect()
+}
+
+fn requests_held_under_a_backlog(fleet: &Fleet) -> HashMap<WorkerId, usize> {
+ let now = Instant::now();
+ let mut held = HashMap::new();
+ for slice in fleet.slices() {
+ let requests = slice.len() * fleet.max_per_worker + 1;
+ let (_, effects) = fleet
+ .batcher(now)
+ .on_groups(now, &slice, 0, backlog(requests));
+ for send in effects.sends {
+ *held.entry(send.worker).or_insert(0) += 1;
+ }
+ }
+ held
+}
+
+fn traffic_shares(fleet: &Fleet) -> HashMap<WorkerId, f64> {
+ let mut share = HashMap::new();
+ for slice in fleet.slices() {
+ let total: u64 = slice.iter().map(|c| c.weight.get()).sum();
+ for candidate in slice {
+ *share.entry(candidate.worker).or_insert(0.0) +=
+ candidate.weight.get() as f64 / total as f64;
+ }
+ }
+ share
+}
+
+fn fleets() -> Vec<Fleet> {
+ vec![
+ Fleet::new(6, 4, 3, 4, 3),
+ Fleet::new(6, 4, 3, 1, 1),
+ Fleet::new(7, 3, 1, 8, 7),
+ Fleet::new(2, 4, 1, 4, 3),
+ Fleet::new(3, 5, 1, 2, 1),
+ Fleet::new(12, 30, 3, 4, 2),
+ Fleet::new(60, 7, 3, 8, 4),
+ Fleet::new(100, 17, 3, 8, 7).draining(25..50),
+ Fleet::new(40, 40, 3, 4, 3).draining((0..40).step_by(5)),
+ Fleet::new(100, 17, 3, 8, 4).unhealthy([10]),
+ Fleet::new(60, 7, 3, 8, 4).unhealthy([3, 4, 5]),
+ ]
+}
+
+#[test]
+fn every_healthy_worker_is_in_some_slice() {
+ for fleet in fleets() {
+ let covered: HashSet<WorkerId> = fleet
+ .slices()
+ .into_iter()
+ .flatten()
+ .map(|c| c.worker)
+ .collect();
+ let healthy: HashSet<WorkerId> = fleet.healthy().into_iter().collect();
+ assert_eq!(covered, healthy, "{fleet:?}");
+ }
+}
+
+#[test]
+fn without_unhealthy_workers_every_worker_gets_the_same_share_of_the_traffic() {
+ for fleet in fleets().into_iter().filter(|f| f.unhealthy.is_empty()) {
+ let expected = fleet.peers as f64 / fleet.ring().len() as f64;
+ for (worker, got) in traffic_shares(&fleet) {
+ assert!(
+ (got - expected).abs() < 1e-9,
+ "{fleet:?}: {worker} gets {got}"
+ );
+ }
+ }
+}
+
+#[test]
+fn with_a_large_unhealthy_block_no_worker_gets_more_than_two_and_a_half_times_the_mean_share() {
+ let blocks = [
+ Fleet::new(464, 77, 3, 8, 4).unhealthy(100..216),
+ Fleet::new(334, 18, 3, 8, 4).unhealthy(0..84),
+ Fleet::new(100, 17, 3, 8, 4).unhealthy(40..65),
+ Fleet::new(60, 7, 3, 8, 4).unhealthy(10..25),
+ ];
+ for fleet in blocks {
+ let mean = fleet.peers as f64 / fleet.healthy().len() as f64;
+ let hottest = traffic_shares(&fleet).values().cloned().fold(0.0, f64::max);
+ assert!(
+ hottest <= 2.5 * mean,
+ "{fleet:?}: {:.2} x the mean",
+ hottest / mean
+ );
+ }
+}
+
+#[test]
+fn under_a_backlog_no_worker_holds_more_than_one_extra_slot_in_max_per_worker() {
+ for fleet in fleets() {
+ let held = requests_held_under_a_backlog(&fleet);
+ let most = *held.values().max().unwrap();
+ let least = *held.values().min().unwrap();
+ assert!(
+ most * fleet.max_per_worker <= least * (fleet.max_per_worker + 1),
+ "{fleet:?}: {least} to {most} requests per worker"
+ );
+ }
+}
+
+#[test]
+fn a_slow_response_shrinks_every_peer_that_has_room_to_shrink() {
+ let now = Instant::now();
+ for fleet in fleets()
+ .into_iter()
+ .filter(|f| f.min_permits < f.max_per_worker)
+ {
+ for slice in fleet.slices() {
+ let requests = slice.len() * fleet.max_per_worker;
+ let (batcher, effects) =
+ fleet
+ .batcher(now)
+ .on_groups(now, &slice, 0, backlog(requests));
+ let ceiling = batcher.snapshot().governor.ceiling;
+ if ceiling < 2 {
+ continue;
+ }
+ let first = effects.sends[0].request;
+ let (batcher, _) = batcher.on_request_succeeded(now + SLOW_RTT, &slice, first, 1);
+ assert_eq!(
+ batcher.snapshot().governor.permits,
+ ceiling - 1,
+ "{fleet:?}: {slice:?}"
+ );
+ }
+ }
+}
diff --git a/rust/ingestion-consumer/src/batcher/governor.rs b/rust/ingestion-consumer/src/batcher/governor.rs
index 0d292c7001af..d2517d6186b1 100644
--- a/rust/ingestion-consumer/src/batcher/governor.rs
+++ b/rust/ingestion-consumer/src/batcher/governor.rs
@@ -1,9 +1,8 @@
//! Governs how many requests the batcher keeps in flight from request
//! round-trip time.
//!
-//! The permit bounds follow the routable workers: the ceiling is the
-//! per-worker cap times the pool, so at the ceiling the permits are the free
-//! worker slots. Permits move by one at most once per step interval, because
+//! The permit bounds follow the routable workers: the ceiling is the slots of
+//! the pool, so at the ceiling the permits are the free worker slots. Permits move by one at most once per step interval, because
//! a step per settlement walks the pool to the floor in one round trip and
//! then sawtooths against the average. With adaptation off the pool is a
//! shadow: it moves by the law and reports what it would have done, while the
@@ -12,6 +11,8 @@
use std::str::FromStr;
use std::time::{Duration, Instant};
+use crate::routing::Candidate;
+
const RTT_TIME_CONSTANT: Duration = Duration::from_secs(10);
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
@@ -212,11 +213,21 @@ impl Governor {
}
}
- /// `in_flight` counts the requests on those workers only: a request on a
+ /// The ceiling is the slots of `pool`. The floor is the min permits for
+ /// each worker that `pool` covers, and stays at least one below the
+ /// ceiling when the min is below the max, so the law can always shrink the
+ /// pool. `in_flight` counts the requests on `pool` only: a request on a
/// worker that left the pool holds none of the pool's permits.
- pub fn set_pool(&mut self, workers: usize, in_flight: usize) {
- self.ceiling = self.max_permits_per_worker * workers;
- self.floor = self.settings.min_permits_per_worker * workers;
+ pub fn set_pool(&mut self, pool: &[Candidate], in_flight: usize) {
+ self.ceiling = pool
+ .iter()
+ .map(|candidate| candidate.slots(self.max_permits_per_worker))
+ .sum();
+ let mut floor = permits_for_covered_workers(pool, self.settings.min_permits_per_worker);
+ if self.settings.min_permits_per_worker < self.max_permits_per_worker {
+ floor = floor.min(self.ceiling.saturating_sub(1));
+ }
+ self.floor = floor.max(1).min(self.ceiling);
self.in_flight = in_flight;
}
@@ -260,11 +271,24 @@ impl Governor {
}
}
+/// `per_worker` permits for each worker that `pool` covers, rounded up once
+/// for the whole pool. The candidates of one pool share one full weight.
+fn permits_for_covered_workers(pool: &[Candidate], per_worker: usize) -> usize {
+ let Some(first) = pool.first() else {
+ return 0;
+ };
+ debug_assert!(pool.iter().all(|c| c.full_weight == first.full_weight));
+ let weight: u128 = pool.iter().map(|c| u128::from(c.weight.get())).sum();
+ (per_worker as u128 * weight).div_ceil(u128::from(first.full_weight.get())) as usize
+}
+
#[cfg(test)]
mod tests {
use rstest::rstest;
use super::*;
+ use crate::aperture::{ring_slice, sorted_ring};
+ use crate::worker_registry::WorkerId;
const STEP: Duration = Duration::from_secs(1);
const BUDGET: Duration = Duration::from_secs(1);
@@ -281,10 +305,16 @@ mod tests {
}
}
+ fn workers(count: usize) -> Vec<Candidate> {
+ (0..count)
+ .map(|n| Candidate::uniform(WorkerId::from(format!("w{n}").as_str())))
+ .collect()
+ }
+
/// Two workers: a ceiling of 8 and a floor of 2.
fn governor(adaptation: Adaptation) -> Governor {
let mut governor = Governor::new(settings(adaptation), MAX_PER_WORKER).expect("valid");
- governor.set_pool(2, 0);
+ governor.set_pool(&workers(2), 0);
governor
}
@@ -388,11 +418,11 @@ mod tests {
assert_eq!(governor.observe(SLOW, now), applied(Direction::Shrink));
assert_eq!(governor.permits(), 7);
- governor.set_pool(1, 0);
+ governor.set_pool(&workers(1), 0);
assert_eq!(governor.ceiling(), 4);
assert_eq!(governor.permits(), 3, "the ceiling dropped by one worker");
- governor.set_pool(3, 0);
+ governor.set_pool(&workers(3), 0);
assert_eq!(governor.ceiling(), 12);
assert_eq!(
governor.permits(),
@@ -404,10 +434,56 @@ mod tests {
governor.observe(SLOW, now + STEP * step);
}
assert_eq!(governor.permits(), 3, "at the floor of one per worker");
- governor.set_pool(1, 0);
+ governor.set_pool(&workers(1), 0);
assert_eq!(governor.permits(), 1, "the floor dropped with the pool");
}
+ #[test]
+ fn a_slice_with_partly_covered_workers_shrinks_to_its_weighted_floor() {
+ let ring = sorted_ring(
+ (1..=6)
+ .map(|n| WorkerId::from(format!("w{n}").as_str()))
+ .collect(),
+ );
+ let slice = ring_slice(&ring, &ring, Some(1), 4, 3).unwrap();
+ assert_eq!(slice.len(), 4, "two whole workers and two halves");
+ let settings = GovernorSettings {
+ min_permits_per_worker: 3,
+ ..settings(Adaptation::On)
+ };
+ let mut governor = Governor::new(settings, MAX_PER_WORKER).expect("valid");
+ governor.set_pool(&slice, 0);
+ assert_eq!(governor.ceiling(), 12);
+
+ let now = Instant::now();
+ for step in 0..3 {
+ assert_eq!(
+ governor.observe(SLOW, now + STEP * step),
+ applied(Direction::Shrink)
+ );
+ }
+ assert_eq!(governor.observe(SLOW, now + STEP * 3), None);
+ assert_eq!(governor.permits(), 9, "3 permits x 3 covered workers");
+ }
+
+ #[test]
+ fn a_slice_of_two_half_workers_can_still_shrink() {
+ let ring = sorted_ring(vec![WorkerId::from("w1"), WorkerId::from("w2")]);
+ let slice = ring_slice(&ring, &ring, Some(1), 4, 1).unwrap();
+ assert_eq!(slice.len(), 2);
+ let settings = GovernorSettings {
+ min_permits_per_worker: 3,
+ ..settings(Adaptation::On)
+ };
+ let mut governor = Governor::new(settings, MAX_PER_WORKER).expect("valid");
+ governor.set_pool(&slice, 0);
+ assert_eq!(governor.ceiling(), 4);
+
+ let now = Instant::now();
+ assert_eq!(governor.observe(SLOW, now), applied(Direction::Shrink));
+ assert_eq!(governor.permits(), 3);
+ }
+
#[test]
fn an_empty_pool_keeps_the_shrink_until_workers_return() {
let now = Instant::now();
@@ -419,7 +495,7 @@ mod tests {
);
assert_eq!(governor.permits(), 6);
- governor.set_pool(0, 0);
+ governor.set_pool(&workers(0), 0);
assert_eq!((governor.permits(), governor.free()), (0, 0));
assert_eq!(
governor.observe(SLOW, now + STEP * 2),
@@ -427,7 +503,7 @@ mod tests {
"nothing to govern"
);
- governor.set_pool(2, 0);
+ governor.set_pool(&workers(2), 0);
assert_eq!(governor.permits(), 6);
}
@@ -435,12 +511,12 @@ mod tests {
fn with_adaptation_off_steps_are_unapplied_and_the_shadow_pool_still_moves() {
let now = Instant::now();
let mut shadow = governor(Adaptation::Off);
- shadow.set_pool(2, 8);
+ shadow.set_pool(&workers(2), 8);
assert_eq!(shadow.free(), 0);
assert_eq!(shadow.observe(SLOW, now), unapplied(Direction::Shrink));
assert_eq!(shadow.permits(), 7);
- shadow.set_pool(2, 7);
+ shadow.set_pool(&workers(2), 7);
assert_eq!(
shadow.free(),
0,
diff --git a/rust/ingestion-consumer/src/batcher/mod.rs b/rust/ingestion-consumer/src/batcher/mod.rs
index e7657de9f5bb..a71780cfa325 100644
--- a/rust/ingestion-consumer/src/batcher/mod.rs
+++ b/rust/ingestion-consumer/src/batcher/mod.rs
@@ -17,6 +17,8 @@
mod drains;
mod driver;
+#[cfg(test)]
+mod fleet_tests;
pub mod governor;
pub mod in_flight;
pub mod key_queues;
diff --git a/rust/ingestion-consumer/src/batcher/state_machine.rs b/rust/ingestion-consumer/src/batcher/state_machine.rs
index 96eb1d068774..e69b8c9c5709 100644
--- a/rust/ingestion-consumer/src/batcher/state_machine.rs
+++ b/rust/ingestion-consumer/src/batcher/state_machine.rs
@@ -31,6 +31,7 @@ use super::retry_policy::{RetryPolicy, RetryReason};
use super::worker_assigner::{WorkerAssigner, WorkerInFlight};
use super::RevokeId;
use crate::order_sentinel::ack_mark;
+use crate::routing::Candidate;
use crate::types::SerializedKafkaMessage;
use crate::worker_registry::WorkerId;
@@ -180,7 +181,7 @@ impl BatcherStateMachine {
pub fn on_groups(
self,
now: Instant,
- workers: &[WorkerId],
+ workers: &[Candidate],
assignment_epoch: u64,
runs: Vec<KeyRun>,
) -> (Self, Effects) {
@@ -199,7 +200,7 @@ impl BatcherStateMachine {
pub fn on_request_succeeded(
self,
now: Instant,
- workers: &[WorkerId],
+ workers: &[Candidate],
request: RequestId,
accepted: u32,
) -> (Self, Effects) {
@@ -214,7 +215,7 @@ impl BatcherStateMachine {
pub fn on_request_partial(
self,
now: Instant,
- workers: &[WorkerId],
+ workers: &[Candidate],
request: RequestId,
accepted: u32,
unfinished: Vec<SerializedKafkaMessage>,
@@ -229,7 +230,7 @@ impl BatcherStateMachine {
pub fn on_request_failed(
self,
now: Instant,
- workers: &[WorkerId],
+ workers: &[Candidate],
request: RequestId,
cause: FailureCause,
messages: Vec<SerializedKafkaMessage>,
@@ -239,7 +240,7 @@ impl BatcherStateMachine {
})
}
- pub fn on_wakeup(self, now: Instant, workers: &[WorkerId]) -> (Self, Effects) {
+ pub fn on_wakeup(self, now: Instant, workers: &[Candidate]) -> (Self, Effects) {
self.act(|active, draining| {
let mut effects = Effects::default();
active.advance(now, workers, draining, &mut effects)?;
@@ -263,7 +264,7 @@ impl BatcherStateMachine {
/// Takes no new groups from now on, and stops once nothing is pending or
/// in flight.
- pub fn on_shutdown(self, now: Instant, workers: &[WorkerId]) -> (Self, Effects) {
+ pub fn on_shutdown(self, now: Instant, workers: &[Candidate]) -> (Self, Effects) {
match self {
BatcherStateMachine::Running(active) | BatcherStateMachine::Draining(active) => {
BatcherStateMachine::Draining(active).on_wakeup(now, workers)
@@ -394,7 +395,7 @@ impl ActiveState {
fn on_groups(
&mut self,
now: Instant,
- workers: &[WorkerId],
+ workers: &[Candidate],
assignment_epoch: u64,
runs: Vec<KeyRun>,
) -> Result<Effects, String> {
@@ -411,7 +412,7 @@ impl ActiveState {
fn on_request_succeeded(
&mut self,
now: Instant,
- workers: &[WorkerId],
+ workers: &[Candidate],
draining: bool,
request: RequestId,
accepted: u32,
@@ -454,7 +455,7 @@ impl ActiveState {
fn on_request_partial(
&mut self,
now: Instant,
- workers: &[WorkerId],
+ workers: &[Candidate],
draining: bool,
request: RequestId,
accepted: u32,
@@ -505,7 +506,7 @@ impl ActiveState {
fn on_request_failed(
&mut self,
now: Instant,
- workers: &[WorkerId],
+ workers: &[Candidate],
draining: bool,
request: RequestId,
cause: FailureCause,
@@ -602,14 +603,14 @@ impl ActiveState {
/// A request on a worker that left the pool holds none of the pool's
/// permits, so the governor counts only the requests on `workers`.
- fn sync_governor(&mut self, workers: &[WorkerId]) {
+ fn sync_governor(&mut self, workers: &[Candidate]) {
let in_flight = workers
.iter()
- .map(|worker| self.assigner.requests_on(worker))
+ .map(|candidate| self.assigner.requests_on(&candidate.worker))
.sum();
- self.governor.set_pool(workers.len(), in_flight);
+ self.governor.set_pool(workers, in_flight);
self.reconnecting
- .retain(|worker, _| workers.contains(worker));
+ .retain(|worker, _| workers.iter().any(|candidate| candidate.worker == *worker));
}
fn settle_key(
@@ -635,7 +636,7 @@ impl ActiveState {
fn advance(
&mut self,
now: Instant,
- workers: &[WorkerId],
+ workers: &[Candidate],
draining: bool,
effects: &mut Effects,
) -> Result<(), String> {
@@ -653,7 +654,7 @@ impl ActiveState {
fn place(
&mut self,
now: Instant,
- workers: &[WorkerId],
+ workers: &[Candidate],
draining: bool,
effects: &mut Effects,
) -> Result<(), String> {
@@ -901,8 +902,11 @@ mod tests {
.expect("valid governor settings")
}
- fn pool(workers: &[&str]) -> Vec<WorkerId> {
- workers.iter().map(|w| WorkerId::from(*w)).collect()
+ fn pool(workers: &[&str]) -> Vec<Candidate> {
+ workers
+ .iter()
+ .map(|w| Candidate::uniform(WorkerId::from(*w)))
+ .collect()
}
fn run(key: &str, offsets: &[i64]) -> KeyRun {
@@ -1106,6 +1110,29 @@ mod tests {
assert_eq!(shape(&effects.sends[0]), vec![("k9", vec![9])]);
}
+ #[test]
+ fn a_backlog_fills_each_worker_to_its_weighted_slots() {
+ let now = Instant::now();
+ let quarters = |w: &str, n: u64| {
+ Candidate::new(
+ WorkerId::from(w),
+ std::num::NonZeroU64::new(n).unwrap(),
+ std::num::NonZeroU64::new(4).unwrap(),
+ )
+ };
+ let workers = vec![quarters("whole", 4), quarters("half", 2)];
+ let runs = (1..=10).map(|n| run(&format!("k{n}"), &[n])).collect();
+ let (_, effects) = batcher(4, now).on_groups(now, &workers, 0, runs);
+ let mut sent: HashMap<WorkerId, usize> = HashMap::new();
+ for send in &effects.sends {
+ *sent.entry(send.worker.clone()).or_insert(0) += 1;
+ }
+ assert_eq!(
+ sent,
+ HashMap::from([(WorkerId::from("whole"), 4), (WorkerId::from("half"), 2)])
+ );
+ }
+
#[test]
fn each_key_run_is_its_own_request_and_its_next_run_waits_for_the_response() {
let now = Instant::now();
diff --git a/rust/ingestion-consumer/src/batcher/worker_assigner.rs b/rust/ingestion-consumer/src/batcher/worker_assigner.rs
index 415b5cc03181..455fbe9cd95c 100644
--- a/rust/ingestion-consumer/src/batcher/worker_assigner.rs
+++ b/rust/ingestion-consumer/src/batcher/worker_assigner.rs
@@ -2,7 +2,7 @@
use std::collections::HashMap;
-use crate::routing::{Router, WorkerLoad};
+use crate::routing::{Candidate, Router, WorkerLoad};
use crate::worker_registry::WorkerId;
pub struct WorkerAssigner {
@@ -52,18 +52,24 @@ impl WorkerAssigner {
.collect()
}
- pub fn free_slots(&self, pool: &[WorkerId]) -> usize {
+ pub fn free_slots(&self, pool: &[Candidate]) -> usize {
pool.iter()
- .map(|worker| self.max_requests_per_worker - self.requests_on(worker))
+ .map(|candidate| {
+ candidate
+ .slots(self.max_requests_per_worker)
+ .saturating_sub(self.requests_on(&candidate.worker))
+ })
.sum()
}
- pub fn assign(&mut self, pool: &[WorkerId], message_count: usize) -> Option<WorkerId> {
+ pub fn assign(&mut self, pool: &[Candidate], message_count: usize) -> Option<WorkerId> {
// A worker at the cap is skipped, so one slow worker cannot hold
// every send slot.
- let open: Vec<WorkerId> = pool
+ let open: Vec<Candidate> = pool
.iter()
- .filter(|worker| self.requests_on(worker) < self.max_requests_per_worker)
+ .filter(|candidate| {
+ self.requests_on(&candidate.worker) < candidate.slots(self.max_requests_per_worker)
+ })
.cloned()
.collect();
let worker = self.router.select(&open, &self.message_load)?;
@@ -98,6 +104,8 @@ impl WorkerAssigner {
#[cfg(test)]
mod tests {
+ use std::num::NonZeroU64;
+
use super::*;
use crate::routing::RoutingStrategy;
@@ -105,6 +113,10 @@ mod tests {
WorkerId::from(s)
}
+ fn pool(workers: &[&str]) -> Vec<Candidate> {
+ workers.iter().map(|w| Candidate::uniform(wid(w))).collect()
+ }
+
#[test]
fn a_zero_request_cap_is_rejected() {
assert!(WorkerAssigner::new(Router::new(RoutingStrategy::P2c), 0).is_err());
@@ -112,7 +124,7 @@ mod tests {
#[test]
fn a_worker_at_the_request_cap_is_skipped_until_a_request_settles() {
- let pool = vec![wid("a"), wid("b")];
+ let pool = pool(&["a", "b"]);
let mut assigner =
WorkerAssigner::new(Router::with_seed(RoutingStrategy::P2c, 0), 1).unwrap();
@@ -126,9 +138,31 @@ mod tests {
assert_eq!(assigner.assign(&pool, 10), Some(first));
}
+ #[test]
+ fn a_candidate_covering_part_of_a_worker_gets_that_part_of_its_slots() {
+ let quarters = |w: &str, n: u64| {
+ Candidate::new(
+ wid(w),
+ NonZeroU64::new(n).unwrap(),
+ NonZeroU64::new(4).unwrap(),
+ )
+ };
+ let pool = vec![quarters("whole", 4), quarters("half", 2)];
+ let mut assigner =
+ WorkerAssigner::new(Router::with_seed(RoutingStrategy::P2c, 0), 4).unwrap();
+ assert_eq!(assigner.free_slots(&pool), 6);
+
+ let mut sent = HashMap::new();
+ while let Some(worker) = assigner.assign(&pool, 1) {
+ *sent.entry(worker).or_insert(0) += 1;
+ }
+ assert_eq!(sent, HashMap::from([(wid("whole"), 4), (wid("half"), 2)]));
+ assert_eq!(assigner.free_slots(&pool), 0);
+ }
+
#[test]
fn placement_follows_the_in_flight_message_load() {
- let pool = vec![wid("a"), wid("b")];
+ let pool = pool(&["a", "b"]);
let mut assigner =
WorkerAssigner::new(Router::with_seed(RoutingStrategy::P2c, 0), 10).unwrap();
diff --git a/rust/ingestion-consumer/src/batcher/worker_pool.rs b/rust/ingestion-consumer/src/batcher/worker_pool.rs
index 01b12e4d389e..8522518d37a1 100644
--- a/rust/ingestion-consumer/src/batcher/worker_pool.rs
+++ b/rust/ingestion-consumer/src/batcher/worker_pool.rs
@@ -3,16 +3,16 @@ use std::sync::Arc;
use k8s_awareness::PeerTracker;
use crate::aperture;
-use crate::routing::RoutingStrategy;
+use crate::routing::{Candidate, RoutingStrategy};
use crate::worker_registry::{WorkerId, WorkerRegistry};
/// The routable workers for one action.
#[derive(Clone, Debug, Default)]
pub struct WorkerPool {
pub healthy: Vec<WorkerId>,
- /// Candidates for fresh requests: the aperture ring slice when narrowing
- /// applies, otherwise the healthy pool.
- pub candidates: Vec<WorkerId>,
+ /// Candidates for fresh requests: the weighted aperture ring slice when
+ /// narrowing applies, otherwise the healthy pool with uniform weights.
+ pub candidates: Vec<Candidate>,
}
/// Captures a [`WorkerPool`] from the registry and the peer set.
@@ -33,7 +33,7 @@ impl WorkerPoolSource {
}
/// Narrow fresh candidates to this consumer's slice of the worker ring,
- /// `min_aperture` wide. Only consulted under [`RoutingStrategy::Aperture`].
+ /// at least `min_aperture` wide. Only consulted under [`RoutingStrategy::Aperture`].
pub fn set_aperture(&mut self, tracker: Arc<PeerTracker>, min_aperture: usize) {
self.aperture = Some((tracker, min_aperture.max(1)));
}
@@ -53,7 +53,7 @@ impl WorkerPoolSource {
/// The ring slice of `healthy`, or `None` when narrowing does not apply.
/// The slice falls back to `None` while the peer set is unknown, at
/// startup or with peer awareness disabled.
- pub fn slice(&self, ring: &[WorkerId], healthy: &[WorkerId]) -> Option<Vec<WorkerId>> {
+ pub fn slice(&self, ring: &[WorkerId], healthy: &[WorkerId]) -> Option<Vec<Candidate>> {
let (tracker, width) = self.active_aperture()?;
let peers = tracker.snapshot();
aperture::ring_slice(ring, healthy, peers.self_index, peers.peer_count(), *width)
@@ -61,16 +61,19 @@ impl WorkerPoolSource {
pub fn pool(&self) -> WorkerPool {
let healthy = self.registry.healthy_workers();
- let candidates = self.narrow(&healthy).unwrap_or_else(|| healthy.clone());
+ let candidates = self
+ .narrow(&healthy)
+ .unwrap_or_else(|| uniform(healthy.iter().cloned()));
WorkerPool {
healthy,
candidates,
}
}
- pub fn candidates(&self) -> Vec<WorkerId> {
+ pub fn candidates(&self) -> Vec<Candidate> {
let healthy = self.registry.healthy_workers();
- self.narrow(&healthy).unwrap_or(healthy)
+ self.narrow(&healthy)
+ .unwrap_or_else(|| uniform(healthy.into_iter()))
}
fn active_aperture(&self) -> Option<&(Arc<PeerTracker>, usize)> {
@@ -81,11 +84,13 @@ impl WorkerPoolSource {
}
}
- fn narrow(&self, healthy: &[WorkerId]) -> Option<Vec<WorkerId>> {
+ fn narrow(&self, healthy: &[WorkerId]) -> Option<Vec<Candidate>> {
self.active_aperture()?;
- let ring = aperture::sorted_ring(self.registry.workers());
- // The fleet's slices tile the pool, so each consumer's requests
- // consolidate onto few workers.
+ let ring = aperture::sorted_ring(self.registry.members());
self.slice(&ring, healthy)
}
}
+
+fn uniform(workers: impl Iterator<Item = WorkerId>) -> Vec<Candidate> {
+ workers.map(Candidate::uniform).collect()
+}
diff --git a/rust/ingestion-consumer/src/config.rs b/rust/ingestion-consumer/src/config.rs
index d5b8bd17f38a..06b79aec90be 100644
--- a/rust/ingestion-consumer/src/config.rs
+++ b/rust/ingestion-consumer/src/config.rs
@@ -373,8 +373,9 @@ pub struct Config {
/// Minimum aperture width for `INGESTION_ROUTING_STRATEGY=aperture`: how
/// many workers this dispatcher's ring slice spans. The effective width
- /// is floored at `ceil(pool / dispatchers)` so the fleet's slices always
- /// cover the whole pool. Small values maximize sub-batch consolidation.
+ /// is the smallest multiple of `pool / dispatchers` workers at or above
+ /// this value, so every worker gets the same share of the fleet's slices.
+ /// Small values maximize sub-batch consolidation.
#[envconfig(from = "INGESTION_MIN_APERTURE", default = "3")]
pub min_aperture: usize,
diff --git a/rust/ingestion-consumer/src/main.rs b/rust/ingestion-consumer/src/main.rs
index ca87894f81e0..e4e4cae02640 100644
--- a/rust/ingestion-consumer/src/main.rs
+++ b/rust/ingestion-consumer/src/main.rs
@@ -646,6 +646,7 @@ async fn build_debug_load(
let ring = aperture::sorted_ring(
snapshots
.iter()
+ .filter(|snap| !snap.draining)
.map(|snap| WorkerId::from(snap.url.as_str()))
.collect(),
);
@@ -659,7 +660,7 @@ async fn build_debug_load(
strategy: pool_source.strategy().as_str().to_string(),
min_aperture: pool_source.min_aperture(),
ring: ring.iter().map(|w| w.to_string()).collect(),
- slice: slice.map(|s| s.iter().map(|w| w.to_string()).collect()),
+ slice: slice.map(|s| s.iter().map(|c| c.worker.to_string()).collect()),
};
let workers = snapshots
.into_iter()
diff --git a/rust/ingestion-consumer/src/routing.rs b/rust/ingestion-consumer/src/routing.rs
index cbebce92db36..679ddb83ebad 100644
--- a/rust/ingestion-consumer/src/routing.rs
+++ b/rust/ingestion-consumer/src/routing.rs
@@ -8,6 +8,7 @@
//! subsetting) without touching the batcher's bookkeeping.
use std::collections::HashMap;
+use std::num::NonZeroU64;
use std::str::FromStr;
use rand::rngs::StdRng;
@@ -24,11 +25,11 @@ pub enum RoutingStrategy {
#[default]
P2c,
/// Deterministic aperture: each consumer places requests only within
- /// its slice of the worker ring (see [`crate::aperture`]), with P2C
- /// selection inside the slice. Consolidates small batches onto few workers
- /// while the fleet's slices still tile the whole pool. Requires peer
- /// awareness (`PEER_SERVICE_NAME`); behaves as plain P2C over the full
- /// pool while the peer set is unknown.
+ /// its weighted slice of the worker ring (see [`crate::aperture`]), with
+ /// weighted P2C selection inside the slice. Consolidates small batches onto
+ /// few workers while every worker gets the same share of the fleet's
+ /// slices. Requires peer awareness (`PEER_SERVICE_NAME`); behaves as plain
+ /// P2C over the full pool while the peer set is unknown.
Aperture,
}
@@ -56,6 +57,40 @@ impl RoutingStrategy {
}
}
+/// A worker that may take a request, and its share of this consumer's
+/// requests relative to the other candidates.
+#[derive(Clone, Debug, PartialEq, Eq)]
+pub struct Candidate {
+ pub worker: WorkerId,
+ pub weight: NonZeroU64,
+ /// The weight of a worker that the candidate set covers whole.
+ pub full_weight: NonZeroU64,
+}
+
+impl Candidate {
+ /// `weight` above `full_weight` is clamped to `full_weight`.
+ pub fn new(worker: WorkerId, weight: NonZeroU64, full_weight: NonZeroU64) -> Self {
+ Self {
+ worker,
+ weight: weight.min(full_weight),
+ full_weight,
+ }
+ }
+
+ pub fn uniform(worker: WorkerId) -> Self {
+ Self::new(worker, NonZeroU64::MIN, NonZeroU64::MIN)
+ }
+
+ /// The requests this consumer may keep in flight on the worker:
+ /// `max_per_worker` scaled by `weight / full_weight` and rounded up, so a
+ /// candidate keeps at least one slot when `max_per_worker` is not zero.
+ pub fn slots(&self, max_per_worker: usize) -> usize {
+ let full = u128::from(self.full_weight.get());
+ let scaled = (max_per_worker as u128 * u128::from(self.weight.get())).div_ceil(full);
+ scaled as usize
+ }
+}
+
/// Per-worker load for a single routing decision, keyed by worker id. A missing
/// entry counts as zero load.
pub type WorkerLoad = HashMap<WorkerId, usize>;
@@ -94,40 +129,52 @@ impl Router {
self.strategy
}
- /// Select a worker for one request. `healthy` lists candidate worker ids;
- /// `load` is the current per-worker load (outstanding messages plus what has
- /// been provisionally assigned earlier in this round). Returns `None` when
- /// no workers are healthy.
- pub fn select(&mut self, healthy: &[WorkerId], load: &WorkerLoad) -> Option<WorkerId> {
- // Aperture narrows the candidate slice upstream (in the worker pool);
- // within the slice it selects exactly like P2C.
- select_p2c(healthy, load, &mut self.rng)
+ /// Select a worker for one request. `load` is the current per-worker load
+ /// (outstanding messages plus what has been provisionally assigned earlier
+ /// in this round). Returns `None` when there are no candidates.
+ pub fn select(&mut self, candidates: &[Candidate], load: &WorkerLoad) -> Option<WorkerId> {
+ select_p2c(candidates, load, &mut self.rng)
}
}
-/// Pick the lighter of two distinct random healthy workers. Degenerates to the
-/// single worker when the pool has one, and to `None` when it is empty.
-fn select_p2c(healthy: &[WorkerId], load: &WorkerLoad, rng: &mut impl Rng) -> Option<WorkerId> {
- match healthy.len() {
+/// Weighted power-of-two-choices: sample two distinct candidates in proportion
+/// to weight, and pick the one with the lower load per unit of weight, so each
+/// worker's share of the requests follows its weight.
+fn select_p2c(candidates: &[Candidate], load: &WorkerLoad, rng: &mut impl Rng) -> Option<WorkerId> {
+ match candidates.len() {
0 => None,
- 1 => Some(healthy[0].clone()),
- len => {
- // Sample two distinct *positions* in one step (no redraw loop): pick
- // the first, then offset by 1..len to land on a different position.
- // This terminates regardless of the slice's contents, so it stays
- // correct even if a caller ever passes duplicate worker ids.
- let i = rng.gen_range(0..len);
- let j = (i + 1 + rng.gen_range(0..len - 1)) % len;
- let (a, b) = (&healthy[i], &healthy[j]);
- Some(if load_of(load, a) <= load_of(load, b) {
- a.clone()
- } else {
- b.clone()
- })
+ 1 => Some(candidates[0].worker.clone()),
+ _ => {
+ // Positions, not ids, so duplicate worker ids cannot make the
+ // second draw loop.
+ let total: u64 = candidates.iter().map(|c| c.weight.get()).sum();
+ let i = weighted_position(candidates, rng.gen_range(0..total), None);
+ let rest = total - candidates[i].weight.get();
+ let j = weighted_position(candidates, rng.gen_range(0..rest), Some(i));
+ let (a, b) = (&candidates[i], &candidates[j]);
+ // a.load / a.weight <= b.load / b.weight, without division.
+ let a_cost = load_of(load, &a.worker) as u128 * u128::from(b.weight.get());
+ let b_cost = load_of(load, &b.worker) as u128 * u128::from(a.weight.get());
+ Some(if a_cost <= b_cost { a } else { b }.worker.clone())
}
}
}
+/// `point` must be below the total weight of the positions other than `skip`.
+fn weighted_position(candidates: &[Candidate], mut point: u64, skip: Option<usize>) -> usize {
+ for (position, candidate) in candidates.iter().enumerate() {
+ if Some(position) == skip {
+ continue;
+ }
+ let weight = candidate.weight.get();
+ if point < weight {
+ return position;
+ }
+ point -= weight;
+ }
+ unreachable!("point is below the total weight of the positions not skipped")
+}
+
#[cfg(test)]
mod tests {
use super::*;
@@ -172,6 +219,75 @@ mod tests {
// ---- p2c ----
+ fn uniform(workers: &[&str]) -> Vec<Candidate> {
+ workers.iter().map(|w| Candidate::uniform(wid(w))).collect()
+ }
+
+ fn weighted(workers: &[(&str, u64)]) -> Vec<Candidate> {
+ workers
+ .iter()
+ .map(|(w, weight)| {
+ Candidate::new(wid(w), NonZeroU64::new(*weight).unwrap(), NonZeroU64::MAX)
+ })
+ .collect()
+ }
+
+ #[test]
+ fn test_slots_scale_the_per_worker_cap_by_the_covered_part_rounded_up() {
+ struct Case {
+ weight: u64,
+ full_weight: u64,
+ max_per_worker: usize,
+ slots: usize,
+ }
+ let cases = [
+ Case {
+ weight: 1,
+ full_weight: 1,
+ max_per_worker: 8,
+ slots: 8,
+ },
+ Case {
+ weight: 2,
+ full_weight: 4,
+ max_per_worker: 8,
+ slots: 4,
+ },
+ Case {
+ weight: 1,
+ full_weight: 3,
+ max_per_worker: 8,
+ slots: 3,
+ },
+ Case {
+ weight: 1,
+ full_weight: 100,
+ max_per_worker: 4,
+ slots: 1,
+ },
+ Case {
+ weight: 9,
+ full_weight: 4,
+ max_per_worker: 8,
+ slots: 8,
+ },
+ ];
+ for case in cases {
+ let candidate = Candidate::new(
+ wid(A),
+ NonZeroU64::new(case.weight).unwrap(),
+ NonZeroU64::new(case.full_weight).unwrap(),
+ );
+ assert_eq!(
+ candidate.slots(case.max_per_worker),
+ case.slots,
+ "{} of {}",
+ case.weight,
+ case.full_weight
+ );
+ }
+ }
+
#[test]
fn test_p2c_empty_pool_returns_none() {
let mut rng = StdRng::seed_from_u64(1);
@@ -182,21 +298,33 @@ mod tests {
fn test_p2c_single_worker_returns_it() {
let mut rng = StdRng::seed_from_u64(1);
assert_eq!(
- select_p2c(&[wid(C)], &load(&[(C, 9)]), &mut rng),
+ select_p2c(&uniform(&[C]), &load(&[(C, 9)]), &mut rng),
Some(wid(C))
);
}
#[test]
fn test_p2c_two_workers_always_picks_lighter() {
- // With exactly two candidates, sampling-without-replacement always draws
- // both, so the lighter one is chosen deterministically regardless of seed.
- let healthy = [wid(A), wid(B)];
+ let candidates = uniform(&[A, B]);
let l = load(&[(A, 3), (B, 0)]);
for seed in 0..16 {
let mut rng = StdRng::seed_from_u64(seed);
assert_eq!(
- select_p2c(&healthy, &l, &mut rng),
+ select_p2c(&candidates, &l, &mut rng),
+ Some(wid(B)),
+ "seed {seed}"
+ );
+ }
+ }
+
+ #[test]
+ fn test_p2c_two_workers_picks_the_lower_load_per_unit_of_weight() {
+ let candidates = weighted(&[(A, 1), (B, 4)]);
+ let l = load(&[(A, 2), (B, 4)]);
+ for seed in 0..16 {
+ let mut rng = StdRng::seed_from_u64(seed);
+ assert_eq!(
+ select_p2c(&candidates, &l, &mut rng),
Some(wid(B)),
"seed {seed}"
);
@@ -205,37 +333,63 @@ mod tests {
#[test]
fn test_p2c_terminates_with_duplicate_ids() {
- // Regression: selection must not loop even if the candidate slice
- // contains duplicate worker ids.
- let healthy = [wid(C), wid(C)];
+ let candidates = uniform(&[C, C]);
let mut rng = StdRng::seed_from_u64(1);
for _ in 0..50 {
assert_eq!(
- select_p2c(&healthy, &WorkerLoad::new(), &mut rng),
+ select_p2c(&candidates, &WorkerLoad::new(), &mut rng),
Some(wid(C))
);
}
}
#[test]
- fn test_p2c_only_ever_returns_a_healthy_worker() {
- let healthy = [wid(A), wid(B), wid(C)];
+ fn test_p2c_only_ever_returns_a_candidate() {
+ let candidates = weighted(&[(A, 1), (B, 5), (C, 2)]);
let l = load(&[(A, 2), (C, 5)]);
let mut rng = StdRng::seed_from_u64(7);
for _ in 0..200 {
- let pick = select_p2c(&healthy, &l, &mut rng).unwrap();
- assert!(healthy.contains(&pick));
+ let pick = select_p2c(&candidates, &l, &mut rng).unwrap();
+ assert!(candidates.iter().any(|c| c.worker == pick));
}
}
#[test]
fn test_p2c_spreads_across_pool() {
- let healthy = [wid(A), wid(B), wid(C)];
+ let candidates = uniform(&[A, B, C]);
let mut rng = StdRng::seed_from_u64(42);
let mut seen = std::collections::HashSet::new();
for _ in 0..200 {
- seen.insert(select_p2c(&healthy, &WorkerLoad::new(), &mut rng).unwrap());
+ seen.insert(select_p2c(&candidates, &WorkerLoad::new(), &mut rng).unwrap());
}
assert_eq!(seen.len(), 3, "every worker should be reachable");
}
+
+ #[test]
+ fn test_p2c_without_load_picks_in_proportion_to_weight() {
+ let candidates = weighted(&[(A, 1), (B, 3)]);
+ let mut rng = StdRng::seed_from_u64(3);
+ let picks_of_a = (0..4000)
+ .filter(|_| select_p2c(&candidates, &WorkerLoad::new(), &mut rng) == Some(wid(A)))
+ .count();
+ assert!((900..1100).contains(&picks_of_a), "{picks_of_a} of 4000");
+ }
+
+ #[test]
+ fn test_p2c_load_held_in_flight_follows_weight() {
+ let candidates = weighted(&[(A, 1), (B, 2), (C, 3)]);
+ let mut rng = StdRng::seed_from_u64(9);
+ let mut l = WorkerLoad::new();
+ for _ in 0..600 {
+ let pick = select_p2c(&candidates, &l, &mut rng).unwrap();
+ *l.entry(pick).or_insert(0) += 1;
+ }
+ for (worker, expected) in [(A, 100), (B, 200), (C, 300)] {
+ let held = load_of(&l, &wid(worker));
+ assert!(
+ held.abs_diff(expected) <= 10,
+ "{worker}: {held}, expected {expected}"
+ );
+ }
+ }
}
diff --git a/rust/ingestion-consumer/src/worker_registry.rs b/rust/ingestion-consumer/src/worker_registry.rs
index abbaf478d521..ccab556eccc5 100644
--- a/rust/ingestion-consumer/src/worker_registry.rs
+++ b/rust/ingestion-consumer/src/worker_registry.rs
@@ -318,6 +318,16 @@ impl WorkerRegistry {
self.workers.iter().map(|e| e.key().clone()).collect()
}
+ /// Snapshot of the workers that discovery lists, so not draining, in any
+ /// health state. Every consumer of the same Service sees the same set.
+ pub fn members(&self) -> Vec<WorkerId> {
+ self.workers
+ .iter()
+ .filter(|e| !e.value().is_draining())
+ .map(|e| e.key().clone())
+ .collect()
+ }
+
/// Snapshot of worker ids currently eligible for routing: Healthy or
/// Degraded, and not draining. Returned in arbitrary order.
pub fn healthy_workers(&self) -> Vec<WorkerId> {
@@ -809,6 +819,19 @@ mod tests {
assert!(!healthy.iter().any(|w| w.as_ref() == W2));
}
+ #[test]
+ fn test_members_keep_unhealthy_workers_and_drop_draining_ones() {
+ let registry = WorkerRegistry::new(&three_workers(), no_cooldown_config());
+ for _ in 0..4 {
+ registry.record_outcome(W2, true);
+ }
+ registry.start_draining(W3);
+
+ let mut members: Vec<String> = registry.members().iter().map(|w| w.to_string()).collect();
+ members.sort();
+ assert_eq!(members, vec![W1.to_string(), W2.to_string()]);
+ }
+
// --- Draining lifecycle ---
#[test]