:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(ingestion): give every worker the same share of aperture slices (#129711)
Paweł Ledwoń6a523ffeaa6b
diff --git a/rust/ingestion-consumer/src/aperture.rs b/rust/ingestion-consumer/src/aperture.rsindex 1162ea48746c..3d21dfacd538 100644--- a/rust/ingestion-consumer/src/aperture.rs+++ b/rust/ingestion-consumer/src/aperture.rs@@ -1,33 +1,45 @@-//! Deterministic aperture: ring-slice candidate selection.+//! Deterministic aperture: weighted ring-slice candidate selection.//!-//! Every consumer sorts the full worker set into the same ring, takes the-//! slice starting at `peer_index * ring_len / peer_count`, and places its-//! requests only within that slice (via the configured selection-//! strategy). Because peers agree on both the ring order (same EndpointSlices,-//! same sort) and their own indices ([`k8s_awareness::PeerTracker`]), the-//! slices tile the ring without any coordination, so each consumer talks to-//! a handful of workers instead of the whole pool, which is what keeps-//! sub-batches consolidated. Modeled on Finagle's deterministic aperture.+//! Every consumer sorts the workers into the same ring, takes an arc+//! of it, and places its requests only on the workers under that arc (via the+//! configured selection strategy). Because peers agree on both the ring order+//! (same EndpointSlices, same sort) and their own indices+//! ([`k8s_awareness::PeerTracker`]), the arcs need no coordination, so each+//! consumer talks to a handful of workers instead of the whole pool, which is+//! what keeps sub-batches consolidated. Modeled on Finagle's deterministic+//! aperture.//!-//! The configured width is a *minimum*: it's floored at-//! `ceil(ring_len / peer_count)` — the narrowest equal slices that can cover-//! the ring — so the union of the fleet's slices covers every worker even-//! when workers outnumber `width x dispatchers`. Every peer computes the same-//! floor from the same inputs, so the property stays coordination-free.+//! The ring is measured in units: each worker spans `peer_count` units. Peer+//! `i`'s arc starts at unit `i x ring_len`, so the arcs start at equal+//! distances, and its length is a multiple of `ring_len` units. So every unit+//! is under the same number of arcs. A worker's weight is the number of its+//! units under the arc, so every worker gets the same total weight from the+//! fleet. With 7 workers, 3 peers, and arcs of 7 units://!-//! Unroutable ring positions (unhealthy or draining workers) are skipped and-//! the slice extends further around the ring, so the *effective* width holds-//! under partial failure.+//! | peer | arc | weights |+//! |------|-------|------------------|+//! | 0 | 0-7 | w0 3, w1 3, w2 1 |+//! | 1 | 7-14 | w2 2, w3 3, w4 2 |+//! | 2 | 14-21 | w4 1, w5 3, w6 3 |+//!+//! The ring holds the workers that discovery lists, so a draining worker+//! leaves every peer's ring and the arcs move to share its units. A worker+//! that only this consumer finds unhealthy leaves only this slice, and the+//! other workers of the slice take its share. When the routable workers hold+//! less than half of the arc's weight, the consumer takes its arc of a ring of+//! the routable workers instead, so that a large failure spreads over the+//! whole routable pool and does not fall on the few workers left in a slice.use std::collections::HashSet;+use std::iter;+use std::num::NonZeroU64;+use crate::routing::Candidate;use crate::worker_registry::WorkerId;-/// The candidate workers for one assignment round: walk the ring from this-/// dispatcher's slice start, collecting healthy workers until the effective-/// width (`width` floored at `ceil(ring_len / peer_count)` for coverage) is-/// reached or the ring is exhausted. `ring` is the canonically sorted full-/// worker set; `healthy` the subset this dispatcher may route to.+/// The weighted candidate workers for one assignment round. `ring` is the+/// canonically sorted set of workers that discovery lists; `healthy` the+/// subset this dispatcher may route to.////// Returns `None` (caller falls back to the full healthy pool) when the/// slice can't be computed: no peers known yet, an out-of-range index, or an@@ -37,38 +49,125 @@ pub fn ring_slice(healthy: &[WorkerId],peer_index: Option<usize>,peer_count: usize,- width: usize,-) -> Option<Vec<WorkerId>> {+ min_width: usize,+) -> Option<Vec<Candidate>> {let index = peer_index?;- if ring.is_empty() || peer_count == 0 || index >= peer_count || width == 0 {+ if ring.is_empty() || peer_count == 0 || index >= peer_count || min_width == 0 {return None;}+ let healthy: HashSet<&WorkerId> = healthy.iter().collect();- // Coverage floor: consecutive slice starts are at most `ceil(len/peers)`- // apart, so any narrower width would leave ring positions no dispatcher- // ever routes fresh keys to. Flooring the width there guarantees the- // union of the fleet's slices covers the whole ring for any pool/peer- // ratio, at the cost of slight slice overlap when the division is uneven.- let width = width.max(ring.len().div_ceil(peer_count));+ let arc = arc_slice(ring, index, peer_count, min_width);+ let arc_weight = total_weight(&arc);+ let slice: Vec<Candidate> = arc+ .into_iter()+ .filter(|candidate| healthy.contains(&candidate.worker))+ .collect();+ if total_weight(&slice) * 2 >= arc_weight {+ return Some(slice);+ }+ let routable_ring: Vec<WorkerId> = ring+ .iter()+ .filter(|worker| healthy.contains(worker))+ .cloned()+ .collect();+ (!routable_ring.is_empty()).then(|| arc_slice(&routable_ring, index, peer_count, min_width))+}- let healthy_set: HashSet<&WorkerId> = healthy.iter().collect();- let start = index * ring.len() / peer_count;+fn total_weight(slice: &[Candidate]) -> u64 {+ slice.iter().map(|candidate| candidate.weight.get()).sum()+}- let mut candidates = Vec::with_capacity(width.min(ring.len()));- for offset in 0..ring.len() {- let worker = &ring[(start + offset) % ring.len()];- if healthy_set.contains(worker) {- candidates.push(worker.clone());- if candidates.len() == width {- break;- }+/// Every worker under peer `index`'s arc of `ring`, weighted by its units+/// under the arc. `ring` and `peer_count` must not be empty or zero.+fn arc_slice(+ ring: &[WorkerId],+ index: usize,+ peer_count: usize,+ min_width: usize,+) -> Vec<Candidate> {+ let arc = Arc::new(ring.len(), peer_count, index, min_width);+ let full_weight = NonZeroU64::new(arc.units_per_worker).expect("peer_count is above zero");+ let covered = take_units(arc.spans(ring.len()), arc.len);+ sum_by_position(covered, ring.len())+ .into_iter()+ .map(|(position, units)| {+ let weight = NonZeroU64::new(units).expect("take_units yields no empty span");+ Candidate::new(ring[position].clone(), weight, full_weight)+ })+ .collect()+}++/// The spans in order, cut so that their units add up to at most `units`.+fn take_units(+ spans: impl Iterator<Item = (usize, u64)>,+ mut units: u64,+) -> impl Iterator<Item = (usize, u64)> {+ spans.map_while(move |(position, span)| {+ let taken = span.min(units);+ units -= taken;+ (taken > 0).then_some((position, taken))+ })+}++/// The units of each position, in the order of each position's first span.+/// Every span must hold at least one unit.+fn sum_by_position(+ spans: impl Iterator<Item = (usize, u64)>,+ ring_len: usize,+) -> Vec<(usize, u64)> {+ let mut units_at = vec![0u64; ring_len];+ let mut order = Vec::new();+ for (position, units) in spans {+ if units_at[position] == 0 {+ order.push(position);}+ units_at[position] += units;}+ order+ .into_iter()+ .map(|position| (position, units_at[position]))+ .collect()+}- if candidates.is_empty() {- return None;+/// One peer's arc of the ring, in units.+struct Arc {+ units_per_worker: u64,+ start: u64,+ len: u64,+}++impl Arc {+ /// The arc is the smallest multiple of `ring_len` units that covers+ /// `min_width` workers. `spans` visits each unit of the ring once, so a+ /// longer arc would still cover the ring only once. The clamp keeps the+ /// numbers small.+ fn new(ring_len: usize, peer_count: usize, index: usize, min_width: usize) -> Self {+ let units_per_worker = peer_count as u64;+ let spacing = ring_len as u64;+ let multiple = (min_width as u64 * units_per_worker)+ .div_ceil(spacing)+ .clamp(1, units_per_worker);+ Self {+ units_per_worker,+ start: index as u64 * spacing,+ len: multiple * spacing,+ }+ }++ /// The ring positions in walk order from the worker that holds `start`,+ /// each with its units from the walk point on. That worker comes again+ /// last with its units before `start`. Only an arc as long as the ring+ /// gets to them.+ fn spans(&self, ring_len: usize) -> impl Iterator<Item = (usize, u64)> + '_ {+ let first = (self.start / self.units_per_worker) as usize;+ let before_start = self.start % self.units_per_worker;+ let whole =+ (1..ring_len).map(move |offset| ((first + offset) % ring_len, self.units_per_worker));+ iter::once((first, self.units_per_worker - before_start))+ .chain(whole)+ .chain(iter::once((first, before_start)))}- Some(candidates)}/// The canonical worker ring: the full registered set, sorted. All dispatchers@@ -80,6 +179,8 @@ pub fn sorted_ring(mut workers: Vec<WorkerId>) -> Vec<WorkerId> {#[cfg(test)]mod tests {+ use std::collections::HashMap;+use super::*;fn wid(n: usize) -> WorkerId {@@ -90,72 +191,206 @@ mod tests {sorted_ring((1..=n).map(wid).collect())}+ fn weights(slice: &[Candidate]) -> Vec<(WorkerId, u64)> {+ slice+ .iter()+ .map(|c| (c.worker.clone(), c.weight.get()))+ .collect()+ }++ fn fleet_weights(ring: &[WorkerId], peers: usize, min_width: usize) -> HashMap<WorkerId, u64> {+ let mut total = HashMap::new();+ for index in 0..peers {+ for candidate in ring_slice(ring, ring, Some(index), peers, min_width).unwrap() {+ *total.entry(candidate.worker).or_insert(0) += candidate.weight.get();+ }+ }+ total+ }+#[test]fn test_slices_tile_the_ring_across_peers() {- // 3 peers x width 2 over 6 workers: each peer gets a distinct pair and- // the union covers the whole ring — the coordination-free guarantee- // the whole design rests on.let ring = ring(6);let mut seen = Vec::new();for index in 0..3 {let slice = ring_slice(&ring, &ring, Some(index), 3, 2).unwrap();assert_eq!(slice.len(), 2, "peer {index}");- seen.extend(slice);+ seen.extend(slice.into_iter().map(|c| c.worker));}- let mut sorted = seen.clone();- sorted.sort_unstable();- sorted.dedup();- assert_eq!(sorted.len(), 6, "slices must tile the ring without overlap");+ seen.sort_unstable();+ seen.dedup();+ assert_eq!(seen.len(), 6, "slices must tile the ring without overlap");}#[test]- fn test_width_floored_so_slices_cover_an_oversized_ring() {- // 2 peers x configured width 3 over 8 workers: unfloored slices would- // be spaced 4 apart and leave positions 3 and 7 with no traffic. The ceil(8/2)=4 floor widens each slice just enough that- // the union covers the whole ring.- let ring = ring(8);- let mut seen = HashSet::new();- for index in 0..2 {- let slice = ring_slice(&ring, &ring, Some(index), 2, 3).unwrap();- assert_eq!(slice.len(), 4, "peer {index} floored to ceil(8/2)");- seen.extend(slice);+ fn test_every_worker_gets_the_same_fleet_weight_when_peers_do_not_divide_the_ring() {+ struct Case {+ workers: usize,+ peers: usize,+ min_width: usize,+ }+ let cases = [+ Case {+ workers: 7,+ peers: 3,+ min_width: 1,+ },+ Case {+ workers: 100,+ peers: 17,+ min_width: 3,+ },+ Case {+ workers: 250,+ peers: 30,+ min_width: 3,+ },+ Case {+ workers: 250,+ peers: 30,+ min_width: 32,+ },+ Case {+ workers: 50,+ peers: 8,+ min_width: 48,+ },+ Case {+ workers: 3,+ peers: 5,+ min_width: 1,+ },+ Case {+ workers: 5,+ peers: 3,+ min_width: 10,+ },+ ];+ for case in cases {+ let ring = ring(case.workers);+ let total = fleet_weights(&ring, case.peers, case.min_width);+ assert_eq!(total.len(), case.workers, "every worker is covered");+ let distinct: HashSet<u64> = total.values().copied().collect();+ assert_eq!(+ distinct.len(),+ 1,+ "{} workers, {} peers, min width {}: {distinct:?}",+ case.workers,+ case.peers,+ case.min_width+ );}- assert_eq!(seen.len(), 8, "every ring position must be covered");}#[test]- fn test_width_floor_overlaps_slices_on_uneven_division() {- // 3 peers over 7 workers: starts 0, 2, 4 with floor ceil(7/3)=3 give- // slices {0,1,2}, {2,3,4}, {4,5,6} — full coverage, with boundary- // positions shared by two peers rather than left uncovered.- let ring = ring(7);- let mut seen = HashSet::new();- for index in 0..3 {- seen.extend(ring_slice(&ring, &ring, Some(index), 3, 1).unwrap());+ fn test_every_worker_gets_the_same_fleet_slots_when_arcs_end_inside_workers() {+ let ring = ring(6);+ let mut slots: HashMap<WorkerId, usize> = HashMap::new();+ for index in 0..4 {+ for candidate in ring_slice(&ring, &ring, Some(index), 4, 3).unwrap() {+ *slots.entry(candidate.worker.clone()).or_insert(0) += candidate.slots(8);+ }}- assert_eq!(seen.len(), 7, "uneven division must still cover the ring");+ assert_eq!(slots.len(), 6);+ assert!(slots.values().all(|n| *n == 16), "{slots:?}");+ }++ #[test]+ fn test_arc_ends_get_the_covered_part_of_the_worker() {+ let ring = ring(7);+ let slice = ring_slice(&ring, &ring, Some(1), 3, 1).unwrap();+ assert_eq!(+ weights(&slice),+ vec![+ (ring[2].clone(), 2),+ (ring[3].clone(), 3),+ (ring[4].clone(), 2),+ ]+ );+ }++ #[test]+ fn test_width_is_at_least_the_minimum() {+ let ring = ring(8);+ let slice = ring_slice(&ring, &ring, Some(0), 2, 5).unwrap();+ assert_eq!(slice.len(), 8);}#[test]fn test_slice_wraps_around_the_ring() {- // Last peer's slice starts near the ring end and wraps to the front.let ring = ring(5);let slice = ring_slice(&ring, &ring, Some(2), 3, 3).unwrap();- // start = 2 * 5 / 3 = 3 → positions 3, 4, 0.assert_eq!(- slice,- vec![ring[3].clone(), ring[4].clone(), ring[0].clone()]+ weights(&slice),+ vec![+ (ring[3].clone(), 2),+ (ring[4].clone(), 3),+ (ring[0].clone(), 3),+ (ring[1].clone(), 2),+ ]);}#[test]- fn test_slice_extends_past_unhealthy_workers() {- // An unroutable worker inside the slice must not shrink the effective- // width — the slice walks on around the ring.+ fn test_an_arc_over_the_whole_ring_gives_every_worker_the_same_weight() {+ let ring = ring(4);+ let slice = ring_slice(&ring, &ring, Some(1), 3, 10).unwrap();+ assert_eq!(slice.len(), 4);+ assert!(slice.iter().all(|c| c.weight.get() == 3), "{slice:?}");+ }++ #[test]+ fn test_with_exactly_half_the_arc_healthy_the_unhealthy_worker_leaves_the_slice() {let ring = ring(6);let healthy: Vec<WorkerId> = ring.iter().filter(|w| **w != ring[1]).cloned().collect();let slice = ring_slice(&ring, &healthy, Some(0), 3, 2).unwrap();- assert_eq!(slice, vec![ring[0].clone(), ring[2].clone()]);+ assert_eq!(weights(&slice), vec![(ring[0].clone(), 3)]);+ }++ #[test]+ fn test_with_less_than_half_the_arc_healthy_the_peer_takes_its_arc_of_the_healthy_ring() {+ let ring = ring(7);+ let arc = ring_slice(&ring, &ring, Some(1), 3, 1).unwrap();+ assert_eq!(+ weights(&arc),+ vec![+ (ring[2].clone(), 2),+ (ring[3].clone(), 3),+ (ring[4].clone(), 2)+ ]+ );+ let healthy: Vec<WorkerId> = ring+ .iter()+ .filter(|w| **w != ring[2] && **w != ring[4])+ .cloned()+ .collect();++ let slice = ring_slice(&ring, &healthy, Some(1), 3, 1).unwrap();++ assert_eq!(+ slice,+ ring_slice(&healthy, &healthy, Some(1), 3, 1).unwrap()+ );+ assert_ne!(weights(&slice), vec![(ring[3].clone(), 3)]);+ }++ #[test]+ fn test_with_no_healthy_worker_under_the_arc_the_peer_takes_its_arc_of_the_healthy_ring() {+ let ring = ring(6);+ let healthy: Vec<WorkerId> = ring[2..].to_vec();+ let slice = ring_slice(&ring, &healthy, Some(0), 3, 2).unwrap();+ assert_eq!(+ slice,+ ring_slice(&healthy, &healthy, Some(0), 3, 2).unwrap()+ );+ assert_eq!(+ weights(&slice),+ vec![+ (ring[2].clone(), 3),+ (ring[3].clone(), 3),+ (ring[4].clone(), 2)+ ]+ );}#[test]diff --git a/rust/ingestion-consumer/src/batcher/driver.rs b/rust/ingestion-consumer/src/batcher/driver.rsindex f059a0903686..9dc315a1a03c 100644--- a/rust/ingestion-consumer/src/batcher/driver.rs+++ b/rust/ingestion-consumer/src/batcher/driver.rs@@ -24,6 +24,7 @@ use super::{make_batch_id, BatcherCompletion, BatcherOutputs, RevokeId};use crate::debug_recorder::{partition_offsets, record_if, DebugEventKind, DebugRecorder};use crate::grpc_transport::{GrpcTransport, SendOptions};use crate::order_sentinel::{KeyOrderSentinel, SendKind, SentinelBatch};+use crate::routing::Candidate;use crate::transport::{SendError, SendOutcome};use crate::types::{Accumulator, SerializedKafkaMessage};use crate::worker_registry::WorkerId;@@ -90,7 +91,7 @@ pub(super) trait RequestSender: Send + Sync + 'static {}pub(super) trait Workers: Send + 'static {- fn candidates(&self) -> Vec<WorkerId>;+ fn candidates(&self) -> Vec<Candidate>;fn record_outcome(&self, worker: &WorkerId, fault: bool);@@ -116,7 +117,7 @@ impl RequestSender for GrpcTransport {}impl Workers for WorkerPoolSource {- fn candidates(&self) -> Vec<WorkerId> {+ fn candidates(&self) -> Vec<Candidate> {WorkerPoolSource::candidates(self)}@@ -572,7 +573,7 @@ mod tests {#[derive(Default)]struct WorkersState {- candidates: Vec<WorkerId>,+ candidates: Vec<Candidate>,outcomes: Vec<(WorkerId, bool)>,idle: Vec<WorkerId>,}@@ -581,7 +582,7 @@ mod tests {struct FakeWorkers(Arc<Mutex<WorkersState>>);impl Workers for FakeWorkers {- fn candidates(&self) -> Vec<WorkerId> {+ fn candidates(&self) -> Vec<Candidate> {self.0.lock().unwrap().candidates.clone()}@@ -720,8 +721,10 @@ mod tests {impl FakeWorkers {fn set_candidates(&self, workers: &[&str]) {- self.0.lock().unwrap().candidates =- workers.iter().map(|w| WorkerId::from(*w)).collect();+ self.0.lock().unwrap().candidates = workers+ .iter()+ .map(|w| Candidate::uniform(WorkerId::from(*w)))+ .collect();}fn outcomes(&self) -> Vec<(WorkerId, bool)> {diff --git a/rust/ingestion-consumer/src/batcher/fleet_tests.rs b/rust/ingestion-consumer/src/batcher/fleet_tests.rsnew file mode 100644index 000000000000..d2409bd1608a--- /dev/null+++ b/rust/ingestion-consumer/src/batcher/fleet_tests.rs@@ -0,0 +1,263 @@+use std::collections::{HashMap, HashSet};+use std::num::NonZeroUsize;+use std::time::{Duration, Instant};++use super::governor::{Adaptation, GovernorSettings};+use super::key_queues::KeyRun;+use super::packer::{PackTargets, Packer};+use super::retry_policy::RetryPolicy;+use super::state_machine::BatcherStateMachine;+use super::test_support::message;+use super::worker_assigner::WorkerAssigner;+use crate::aperture::{ring_slice, sorted_ring};+use crate::routing::{Candidate, Router, RoutingStrategy};+use crate::worker_registry::WorkerId;++const LATENCY_BUDGET: Duration = Duration::from_secs(1);+const SLOW_RTT: Duration = Duration::from_secs(2);++#[derive(Debug)]+struct Fleet {+ workers: usize,+ peers: usize,+ min_width: usize,+ max_per_worker: usize,+ min_permits: usize,+ draining: Vec<usize>,+ unhealthy: Vec<usize>,+}++impl Fleet {+ fn new(workers: usize, peers: usize, min_width: usize, max: usize, min: usize) -> Self {+ Self {+ workers,+ peers,+ min_width,+ max_per_worker: max,+ min_permits: min,+ draining: Vec::new(),+ unhealthy: Vec::new(),+ }+ }++ fn draining(self, positions: impl IntoIterator<Item = usize>) -> Self {+ Self {+ draining: positions.into_iter().collect(),+ ..self+ }+ }++ fn unhealthy(self, positions: impl IntoIterator<Item = usize>) -> Self {+ Self {+ unhealthy: positions.into_iter().collect(),+ ..self+ }+ }++ fn worker(position: usize) -> WorkerId {+ WorkerId::from(format!("http://10.0.{}.{}:6738", position / 200, position % 200).as_str())+ }++ fn ring(&self) -> Vec<WorkerId> {+ sorted_ring(+ (0..self.workers)+ .filter(|p| !self.draining.contains(p))+ .map(Self::worker)+ .collect(),+ )+ }++ fn healthy(&self) -> Vec<WorkerId> {+ let unhealthy: HashSet<WorkerId> =+ self.unhealthy.iter().map(|p| Self::worker(*p)).collect();+ self.ring()+ .into_iter()+ .filter(|w| !unhealthy.contains(w))+ .collect()+ }++ fn slices(&self) -> Vec<Vec<Candidate>> {+ let (ring, healthy) = (self.ring(), self.healthy());+ (0..self.peers)+ .map(|index| {+ let slice = ring_slice(&ring, &healthy, Some(index), self.peers, self.min_width)+ .expect("every peer gets a slice");+ assert!(+ !slice.is_empty(),+ "{self:?}: peer {index} got an empty slice"+ );+ slice+ })+ .collect()+ }++ fn batcher(&self, now: Instant) -> BatcherStateMachine {+ let packer = Packer::new(PackTargets {+ events: NonZeroUsize::new(1),+ bytes: None,+ latency_budget: Duration::ZERO,+ });+ let assigner = WorkerAssigner::new(+ Router::with_seed(RoutingStrategy::Aperture, 0),+ self.max_per_worker,+ )+ .expect("valid request cap");+ let retry = RetryPolicy::new(+ Duration::from_millis(200),+ Duration::from_millis(20),+ Duration::from_millis(100),+ )+ .expect("valid retry policy");+ BatcherStateMachine::new(packer, assigner, retry, Duration::from_secs(60), now)+ .expect("valid batcher")+ .with_governor(GovernorSettings {+ adaptation: Adaptation::On,+ latency_budget: Some(LATENCY_BUDGET),+ min_permits_per_worker: self.min_permits,+ ..GovernorSettings::observing()+ })+ .expect("valid governor settings")+ }+}++fn backlog(requests: usize) -> Vec<KeyRun> {+ (0..requests)+ .map(|n| {+ let key = format!("k{n}");+ KeyRun {+ messages: vec![message(&key, 0, n as i64)],+ routing_key: key.into(),+ }+ })+ .collect()+}++fn requests_held_under_a_backlog(fleet: &Fleet) -> HashMap<WorkerId, usize> {+ let now = Instant::now();+ let mut held = HashMap::new();+ for slice in fleet.slices() {+ let requests = slice.len() * fleet.max_per_worker + 1;+ let (_, effects) = fleet+ .batcher(now)+ .on_groups(now, &slice, 0, backlog(requests));+ for send in effects.sends {+ *held.entry(send.worker).or_insert(0) += 1;+ }+ }+ held+}++fn traffic_shares(fleet: &Fleet) -> HashMap<WorkerId, f64> {+ let mut share = HashMap::new();+ for slice in fleet.slices() {+ let total: u64 = slice.iter().map(|c| c.weight.get()).sum();+ for candidate in slice {+ *share.entry(candidate.worker).or_insert(0.0) +=+ candidate.weight.get() as f64 / total as f64;+ }+ }+ share+}++fn fleets() -> Vec<Fleet> {+ vec![+ Fleet::new(6, 4, 3, 4, 3),+ Fleet::new(6, 4, 3, 1, 1),+ Fleet::new(7, 3, 1, 8, 7),+ Fleet::new(2, 4, 1, 4, 3),+ Fleet::new(3, 5, 1, 2, 1),+ Fleet::new(12, 30, 3, 4, 2),+ Fleet::new(60, 7, 3, 8, 4),+ Fleet::new(100, 17, 3, 8, 7).draining(25..50),+ Fleet::new(40, 40, 3, 4, 3).draining((0..40).step_by(5)),+ Fleet::new(100, 17, 3, 8, 4).unhealthy([10]),+ Fleet::new(60, 7, 3, 8, 4).unhealthy([3, 4, 5]),+ ]+}++#[test]+fn every_healthy_worker_is_in_some_slice() {+ for fleet in fleets() {+ let covered: HashSet<WorkerId> = fleet+ .slices()+ .into_iter()+ .flatten()+ .map(|c| c.worker)+ .collect();+ let healthy: HashSet<WorkerId> = fleet.healthy().into_iter().collect();+ assert_eq!(covered, healthy, "{fleet:?}");+ }+}++#[test]+fn without_unhealthy_workers_every_worker_gets_the_same_share_of_the_traffic() {+ for fleet in fleets().into_iter().filter(|f| f.unhealthy.is_empty()) {+ let expected = fleet.peers as f64 / fleet.ring().len() as f64;+ for (worker, got) in traffic_shares(&fleet) {+ assert!(+ (got - expected).abs() < 1e-9,+ "{fleet:?}: {worker} gets {got}"+ );+ }+ }+}++#[test]+fn with_a_large_unhealthy_block_no_worker_gets_more_than_two_and_a_half_times_the_mean_share() {+ let blocks = [+ Fleet::new(464, 77, 3, 8, 4).unhealthy(100..216),+ Fleet::new(334, 18, 3, 8, 4).unhealthy(0..84),+ Fleet::new(100, 17, 3, 8, 4).unhealthy(40..65),+ Fleet::new(60, 7, 3, 8, 4).unhealthy(10..25),+ ];+ for fleet in blocks {+ let mean = fleet.peers as f64 / fleet.healthy().len() as f64;+ let hottest = traffic_shares(&fleet).values().cloned().fold(0.0, f64::max);+ assert!(+ hottest <= 2.5 * mean,+ "{fleet:?}: {:.2} x the mean",+ hottest / mean+ );+ }+}++#[test]+fn under_a_backlog_no_worker_holds_more_than_one_extra_slot_in_max_per_worker() {+ for fleet in fleets() {+ let held = requests_held_under_a_backlog(&fleet);+ let most = *held.values().max().unwrap();+ let least = *held.values().min().unwrap();+ assert!(+ most * fleet.max_per_worker <= least * (fleet.max_per_worker + 1),+ "{fleet:?}: {least} to {most} requests per worker"+ );+ }+}++#[test]+fn a_slow_response_shrinks_every_peer_that_has_room_to_shrink() {+ let now = Instant::now();+ for fleet in fleets()+ .into_iter()+ .filter(|f| f.min_permits < f.max_per_worker)+ {+ for slice in fleet.slices() {+ let requests = slice.len() * fleet.max_per_worker;+ let (batcher, effects) =+ fleet+ .batcher(now)+ .on_groups(now, &slice, 0, backlog(requests));+ let ceiling = batcher.snapshot().governor.ceiling;+ if ceiling < 2 {+ continue;+ }+ let first = effects.sends[0].request;+ let (batcher, _) = batcher.on_request_succeeded(now + SLOW_RTT, &slice, first, 1);+ assert_eq!(+ batcher.snapshot().governor.permits,+ ceiling - 1,+ "{fleet:?}: {slice:?}"+ );+ }+ }+}diff --git a/rust/ingestion-consumer/src/batcher/governor.rs b/rust/ingestion-consumer/src/batcher/governor.rsindex 0d292c7001af..d2517d6186b1 100644--- a/rust/ingestion-consumer/src/batcher/governor.rs+++ b/rust/ingestion-consumer/src/batcher/governor.rs@@ -1,9 +1,8 @@//! Governs how many requests the batcher keeps in flight from request//! round-trip time.//!-//! The permit bounds follow the routable workers: the ceiling is the-//! per-worker cap times the pool, so at the ceiling the permits are the free-//! worker slots. Permits move by one at most once per step interval, because+//! The permit bounds follow the routable workers: the ceiling is the slots of+//! the pool, so at the ceiling the permits are the free worker slots. Permits move by one at most once per step interval, because//! a step per settlement walks the pool to the floor in one round trip and//! then sawtooths against the average. With adaptation off the pool is a//! shadow: it moves by the law and reports what it would have done, while the@@ -12,6 +11,8 @@use std::str::FromStr;use std::time::{Duration, Instant};+use crate::routing::Candidate;+const RTT_TIME_CONSTANT: Duration = Duration::from_secs(10);#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]@@ -212,11 +213,21 @@ impl Governor {}}- /// `in_flight` counts the requests on those workers only: a request on a+ /// The ceiling is the slots of `pool`. The floor is the min permits for+ /// each worker that `pool` covers, and stays at least one below the+ /// ceiling when the min is below the max, so the law can always shrink the+ /// pool. `in_flight` counts the requests on `pool` only: a request on a/// worker that left the pool holds none of the pool's permits.- pub fn set_pool(&mut self, workers: usize, in_flight: usize) {- self.ceiling = self.max_permits_per_worker * workers;- self.floor = self.settings.min_permits_per_worker * workers;+ pub fn set_pool(&mut self, pool: &[Candidate], in_flight: usize) {+ self.ceiling = pool+ .iter()+ .map(|candidate| candidate.slots(self.max_permits_per_worker))+ .sum();+ let mut floor = permits_for_covered_workers(pool, self.settings.min_permits_per_worker);+ if self.settings.min_permits_per_worker < self.max_permits_per_worker {+ floor = floor.min(self.ceiling.saturating_sub(1));+ }+ self.floor = floor.max(1).min(self.ceiling);self.in_flight = in_flight;}@@ -260,11 +271,24 @@ impl Governor {}}+/// `per_worker` permits for each worker that `pool` covers, rounded up once+/// for the whole pool. The candidates of one pool share one full weight.+fn permits_for_covered_workers(pool: &[Candidate], per_worker: usize) -> usize {+ let Some(first) = pool.first() else {+ return 0;+ };+ debug_assert!(pool.iter().all(|c| c.full_weight == first.full_weight));+ let weight: u128 = pool.iter().map(|c| u128::from(c.weight.get())).sum();+ (per_worker as u128 * weight).div_ceil(u128::from(first.full_weight.get())) as usize+}+#[cfg(test)]mod tests {use rstest::rstest;use super::*;+ use crate::aperture::{ring_slice, sorted_ring};+ use crate::worker_registry::WorkerId;const STEP: Duration = Duration::from_secs(1);const BUDGET: Duration = Duration::from_secs(1);@@ -281,10 +305,16 @@ mod tests {}}+ fn workers(count: usize) -> Vec<Candidate> {+ (0..count)+ .map(|n| Candidate::uniform(WorkerId::from(format!("w{n}").as_str())))+ .collect()+ }+/// Two workers: a ceiling of 8 and a floor of 2.fn governor(adaptation: Adaptation) -> Governor {let mut governor = Governor::new(settings(adaptation), MAX_PER_WORKER).expect("valid");- governor.set_pool(2, 0);+ governor.set_pool(&workers(2), 0);governor}@@ -388,11 +418,11 @@ mod tests {assert_eq!(governor.observe(SLOW, now), applied(Direction::Shrink));assert_eq!(governor.permits(), 7);- governor.set_pool(1, 0);+ governor.set_pool(&workers(1), 0);assert_eq!(governor.ceiling(), 4);assert_eq!(governor.permits(), 3, "the ceiling dropped by one worker");- governor.set_pool(3, 0);+ governor.set_pool(&workers(3), 0);assert_eq!(governor.ceiling(), 12);assert_eq!(governor.permits(),@@ -404,10 +434,56 @@ mod tests {governor.observe(SLOW, now + STEP * step);}assert_eq!(governor.permits(), 3, "at the floor of one per worker");- governor.set_pool(1, 0);+ governor.set_pool(&workers(1), 0);assert_eq!(governor.permits(), 1, "the floor dropped with the pool");}+ #[test]+ fn a_slice_with_partly_covered_workers_shrinks_to_its_weighted_floor() {+ let ring = sorted_ring(+ (1..=6)+ .map(|n| WorkerId::from(format!("w{n}").as_str()))+ .collect(),+ );+ let slice = ring_slice(&ring, &ring, Some(1), 4, 3).unwrap();+ assert_eq!(slice.len(), 4, "two whole workers and two halves");+ let settings = GovernorSettings {+ min_permits_per_worker: 3,+ ..settings(Adaptation::On)+ };+ let mut governor = Governor::new(settings, MAX_PER_WORKER).expect("valid");+ governor.set_pool(&slice, 0);+ assert_eq!(governor.ceiling(), 12);++ let now = Instant::now();+ for step in 0..3 {+ assert_eq!(+ governor.observe(SLOW, now + STEP * step),+ applied(Direction::Shrink)+ );+ }+ assert_eq!(governor.observe(SLOW, now + STEP * 3), None);+ assert_eq!(governor.permits(), 9, "3 permits x 3 covered workers");+ }++ #[test]+ fn a_slice_of_two_half_workers_can_still_shrink() {+ let ring = sorted_ring(vec![WorkerId::from("w1"), WorkerId::from("w2")]);+ let slice = ring_slice(&ring, &ring, Some(1), 4, 1).unwrap();+ assert_eq!(slice.len(), 2);+ let settings = GovernorSettings {+ min_permits_per_worker: 3,+ ..settings(Adaptation::On)+ };+ let mut governor = Governor::new(settings, MAX_PER_WORKER).expect("valid");+ governor.set_pool(&slice, 0);+ assert_eq!(governor.ceiling(), 4);++ let now = Instant::now();+ assert_eq!(governor.observe(SLOW, now), applied(Direction::Shrink));+ assert_eq!(governor.permits(), 3);+ }+#[test]fn an_empty_pool_keeps_the_shrink_until_workers_return() {let now = Instant::now();@@ -419,7 +495,7 @@ mod tests {);assert_eq!(governor.permits(), 6);- governor.set_pool(0, 0);+ governor.set_pool(&workers(0), 0);assert_eq!((governor.permits(), governor.free()), (0, 0));assert_eq!(governor.observe(SLOW, now + STEP * 2),@@ -427,7 +503,7 @@ mod tests {"nothing to govern");- governor.set_pool(2, 0);+ governor.set_pool(&workers(2), 0);assert_eq!(governor.permits(), 6);}@@ -435,12 +511,12 @@ mod tests {fn with_adaptation_off_steps_are_unapplied_and_the_shadow_pool_still_moves() {let now = Instant::now();let mut shadow = governor(Adaptation::Off);- shadow.set_pool(2, 8);+ shadow.set_pool(&workers(2), 8);assert_eq!(shadow.free(), 0);assert_eq!(shadow.observe(SLOW, now), unapplied(Direction::Shrink));assert_eq!(shadow.permits(), 7);- shadow.set_pool(2, 7);+ shadow.set_pool(&workers(2), 7);assert_eq!(shadow.free(),0,diff --git a/rust/ingestion-consumer/src/batcher/mod.rs b/rust/ingestion-consumer/src/batcher/mod.rsindex e7657de9f5bb..a71780cfa325 100644--- a/rust/ingestion-consumer/src/batcher/mod.rs+++ b/rust/ingestion-consumer/src/batcher/mod.rs@@ -17,6 +17,8 @@mod drains;mod driver;+#[cfg(test)]+mod fleet_tests;pub mod governor;pub mod in_flight;pub mod key_queues;diff --git a/rust/ingestion-consumer/src/batcher/state_machine.rs b/rust/ingestion-consumer/src/batcher/state_machine.rsindex 96eb1d068774..e69b8c9c5709 100644--- a/rust/ingestion-consumer/src/batcher/state_machine.rs+++ b/rust/ingestion-consumer/src/batcher/state_machine.rs@@ -31,6 +31,7 @@ use super::retry_policy::{RetryPolicy, RetryReason};use super::worker_assigner::{WorkerAssigner, WorkerInFlight};use super::RevokeId;use crate::order_sentinel::ack_mark;+use crate::routing::Candidate;use crate::types::SerializedKafkaMessage;use crate::worker_registry::WorkerId;@@ -180,7 +181,7 @@ impl BatcherStateMachine {pub fn on_groups(self,now: Instant,- workers: &[WorkerId],+ workers: &[Candidate],assignment_epoch: u64,runs: Vec<KeyRun>,) -> (Self, Effects) {@@ -199,7 +200,7 @@ impl BatcherStateMachine {pub fn on_request_succeeded(self,now: Instant,- workers: &[WorkerId],+ workers: &[Candidate],request: RequestId,accepted: u32,) -> (Self, Effects) {@@ -214,7 +215,7 @@ impl BatcherStateMachine {pub fn on_request_partial(self,now: Instant,- workers: &[WorkerId],+ workers: &[Candidate],request: RequestId,accepted: u32,unfinished: Vec<SerializedKafkaMessage>,@@ -229,7 +230,7 @@ impl BatcherStateMachine {pub fn on_request_failed(self,now: Instant,- workers: &[WorkerId],+ workers: &[Candidate],request: RequestId,cause: FailureCause,messages: Vec<SerializedKafkaMessage>,@@ -239,7 +240,7 @@ impl BatcherStateMachine {})}- pub fn on_wakeup(self, now: Instant, workers: &[WorkerId]) -> (Self, Effects) {+ pub fn on_wakeup(self, now: Instant, workers: &[Candidate]) -> (Self, Effects) {self.act(|active, draining| {let mut effects = Effects::default();active.advance(now, workers, draining, &mut effects)?;@@ -263,7 +264,7 @@ impl BatcherStateMachine {/// Takes no new groups from now on, and stops once nothing is pending or/// in flight.- pub fn on_shutdown(self, now: Instant, workers: &[WorkerId]) -> (Self, Effects) {+ pub fn on_shutdown(self, now: Instant, workers: &[Candidate]) -> (Self, Effects) {match self {BatcherStateMachine::Running(active) | BatcherStateMachine::Draining(active) => {BatcherStateMachine::Draining(active).on_wakeup(now, workers)@@ -394,7 +395,7 @@ impl ActiveState {fn on_groups(&mut self,now: Instant,- workers: &[WorkerId],+ workers: &[Candidate],assignment_epoch: u64,runs: Vec<KeyRun>,) -> Result<Effects, String> {@@ -411,7 +412,7 @@ impl ActiveState {fn on_request_succeeded(&mut self,now: Instant,- workers: &[WorkerId],+ workers: &[Candidate],draining: bool,request: RequestId,accepted: u32,@@ -454,7 +455,7 @@ impl ActiveState {fn on_request_partial(&mut self,now: Instant,- workers: &[WorkerId],+ workers: &[Candidate],draining: bool,request: RequestId,accepted: u32,@@ -505,7 +506,7 @@ impl ActiveState {fn on_request_failed(&mut self,now: Instant,- workers: &[WorkerId],+ workers: &[Candidate],draining: bool,request: RequestId,cause: FailureCause,@@ -602,14 +603,14 @@ impl ActiveState {/// A request on a worker that left the pool holds none of the pool's/// permits, so the governor counts only the requests on `workers`.- fn sync_governor(&mut self, workers: &[WorkerId]) {+ fn sync_governor(&mut self, workers: &[Candidate]) {let in_flight = workers.iter()- .map(|worker| self.assigner.requests_on(worker))+ .map(|candidate| self.assigner.requests_on(&candidate.worker)).sum();- self.governor.set_pool(workers.len(), in_flight);+ self.governor.set_pool(workers, in_flight);self.reconnecting- .retain(|worker, _| workers.contains(worker));+ .retain(|worker, _| workers.iter().any(|candidate| candidate.worker == *worker));}fn settle_key(@@ -635,7 +636,7 @@ impl ActiveState {fn advance(&mut self,now: Instant,- workers: &[WorkerId],+ workers: &[Candidate],draining: bool,effects: &mut Effects,) -> Result<(), String> {@@ -653,7 +654,7 @@ impl ActiveState {fn place(&mut self,now: Instant,- workers: &[WorkerId],+ workers: &[Candidate],draining: bool,effects: &mut Effects,) -> Result<(), String> {@@ -901,8 +902,11 @@ mod tests {.expect("valid governor settings")}- fn pool(workers: &[&str]) -> Vec<WorkerId> {- workers.iter().map(|w| WorkerId::from(*w)).collect()+ fn pool(workers: &[&str]) -> Vec<Candidate> {+ workers+ .iter()+ .map(|w| Candidate::uniform(WorkerId::from(*w)))+ .collect()}fn run(key: &str, offsets: &[i64]) -> KeyRun {@@ -1106,6 +1110,29 @@ mod tests {assert_eq!(shape(&effects.sends[0]), vec![("k9", vec![9])]);}+ #[test]+ fn a_backlog_fills_each_worker_to_its_weighted_slots() {+ let now = Instant::now();+ let quarters = |w: &str, n: u64| {+ Candidate::new(+ WorkerId::from(w),+ std::num::NonZeroU64::new(n).unwrap(),+ std::num::NonZeroU64::new(4).unwrap(),+ )+ };+ let workers = vec![quarters("whole", 4), quarters("half", 2)];+ let runs = (1..=10).map(|n| run(&format!("k{n}"), &[n])).collect();+ let (_, effects) = batcher(4, now).on_groups(now, &workers, 0, runs);+ let mut sent: HashMap<WorkerId, usize> = HashMap::new();+ for send in &effects.sends {+ *sent.entry(send.worker.clone()).or_insert(0) += 1;+ }+ assert_eq!(+ sent,+ HashMap::from([(WorkerId::from("whole"), 4), (WorkerId::from("half"), 2)])+ );+ }+#[test]fn each_key_run_is_its_own_request_and_its_next_run_waits_for_the_response() {let now = Instant::now();diff --git a/rust/ingestion-consumer/src/batcher/worker_assigner.rs b/rust/ingestion-consumer/src/batcher/worker_assigner.rsindex 415b5cc03181..455fbe9cd95c 100644--- a/rust/ingestion-consumer/src/batcher/worker_assigner.rs+++ b/rust/ingestion-consumer/src/batcher/worker_assigner.rs@@ -2,7 +2,7 @@use std::collections::HashMap;-use crate::routing::{Router, WorkerLoad};+use crate::routing::{Candidate, Router, WorkerLoad};use crate::worker_registry::WorkerId;pub struct WorkerAssigner {@@ -52,18 +52,24 @@ impl WorkerAssigner {.collect()}- pub fn free_slots(&self, pool: &[WorkerId]) -> usize {+ pub fn free_slots(&self, pool: &[Candidate]) -> usize {pool.iter()- .map(|worker| self.max_requests_per_worker - self.requests_on(worker))+ .map(|candidate| {+ candidate+ .slots(self.max_requests_per_worker)+ .saturating_sub(self.requests_on(&candidate.worker))+ }).sum()}- pub fn assign(&mut self, pool: &[WorkerId], message_count: usize) -> Option<WorkerId> {+ pub fn assign(&mut self, pool: &[Candidate], message_count: usize) -> Option<WorkerId> {// A worker at the cap is skipped, so one slow worker cannot hold// every send slot.- let open: Vec<WorkerId> = pool+ let open: Vec<Candidate> = pool.iter()- .filter(|worker| self.requests_on(worker) < self.max_requests_per_worker)+ .filter(|candidate| {+ self.requests_on(&candidate.worker) < candidate.slots(self.max_requests_per_worker)+ }).cloned().collect();let worker = self.router.select(&open, &self.message_load)?;@@ -98,6 +104,8 @@ impl WorkerAssigner {#[cfg(test)]mod tests {+ use std::num::NonZeroU64;+use super::*;use crate::routing::RoutingStrategy;@@ -105,6 +113,10 @@ mod tests {WorkerId::from(s)}+ fn pool(workers: &[&str]) -> Vec<Candidate> {+ workers.iter().map(|w| Candidate::uniform(wid(w))).collect()+ }+#[test]fn a_zero_request_cap_is_rejected() {assert!(WorkerAssigner::new(Router::new(RoutingStrategy::P2c), 0).is_err());@@ -112,7 +124,7 @@ mod tests {#[test]fn a_worker_at_the_request_cap_is_skipped_until_a_request_settles() {- let pool = vec![wid("a"), wid("b")];+ let pool = pool(&["a", "b"]);let mut assigner =WorkerAssigner::new(Router::with_seed(RoutingStrategy::P2c, 0), 1).unwrap();@@ -126,9 +138,31 @@ mod tests {assert_eq!(assigner.assign(&pool, 10), Some(first));}+ #[test]+ fn a_candidate_covering_part_of_a_worker_gets_that_part_of_its_slots() {+ let quarters = |w: &str, n: u64| {+ Candidate::new(+ wid(w),+ NonZeroU64::new(n).unwrap(),+ NonZeroU64::new(4).unwrap(),+ )+ };+ let pool = vec![quarters("whole", 4), quarters("half", 2)];+ let mut assigner =+ WorkerAssigner::new(Router::with_seed(RoutingStrategy::P2c, 0), 4).unwrap();+ assert_eq!(assigner.free_slots(&pool), 6);++ let mut sent = HashMap::new();+ while let Some(worker) = assigner.assign(&pool, 1) {+ *sent.entry(worker).or_insert(0) += 1;+ }+ assert_eq!(sent, HashMap::from([(wid("whole"), 4), (wid("half"), 2)]));+ assert_eq!(assigner.free_slots(&pool), 0);+ }+#[test]fn placement_follows_the_in_flight_message_load() {- let pool = vec![wid("a"), wid("b")];+ let pool = pool(&["a", "b"]);let mut assigner =WorkerAssigner::new(Router::with_seed(RoutingStrategy::P2c, 0), 10).unwrap();diff --git a/rust/ingestion-consumer/src/batcher/worker_pool.rs b/rust/ingestion-consumer/src/batcher/worker_pool.rsindex 01b12e4d389e..8522518d37a1 100644--- a/rust/ingestion-consumer/src/batcher/worker_pool.rs+++ b/rust/ingestion-consumer/src/batcher/worker_pool.rs@@ -3,16 +3,16 @@ use std::sync::Arc;use k8s_awareness::PeerTracker;use crate::aperture;-use crate::routing::RoutingStrategy;+use crate::routing::{Candidate, RoutingStrategy};use crate::worker_registry::{WorkerId, WorkerRegistry};/// The routable workers for one action.#[derive(Clone, Debug, Default)]pub struct WorkerPool {pub healthy: Vec<WorkerId>,- /// Candidates for fresh requests: the aperture ring slice when narrowing- /// applies, otherwise the healthy pool.- pub candidates: Vec<WorkerId>,+ /// Candidates for fresh requests: the weighted aperture ring slice when+ /// narrowing applies, otherwise the healthy pool with uniform weights.+ pub candidates: Vec<Candidate>,}/// Captures a [`WorkerPool`] from the registry and the peer set.@@ -33,7 +33,7 @@ impl WorkerPoolSource {}/// Narrow fresh candidates to this consumer's slice of the worker ring,- /// `min_aperture` wide. Only consulted under [`RoutingStrategy::Aperture`].+ /// at least `min_aperture` wide. Only consulted under [`RoutingStrategy::Aperture`].pub fn set_aperture(&mut self, tracker: Arc<PeerTracker>, min_aperture: usize) {self.aperture = Some((tracker, min_aperture.max(1)));}@@ -53,7 +53,7 @@ impl WorkerPoolSource {/// The ring slice of `healthy`, or `None` when narrowing does not apply./// The slice falls back to `None` while the peer set is unknown, at/// startup or with peer awareness disabled.- pub fn slice(&self, ring: &[WorkerId], healthy: &[WorkerId]) -> Option<Vec<WorkerId>> {+ pub fn slice(&self, ring: &[WorkerId], healthy: &[WorkerId]) -> Option<Vec<Candidate>> {let (tracker, width) = self.active_aperture()?;let peers = tracker.snapshot();aperture::ring_slice(ring, healthy, peers.self_index, peers.peer_count(), *width)@@ -61,16 +61,19 @@ impl WorkerPoolSource {pub fn pool(&self) -> WorkerPool {let healthy = self.registry.healthy_workers();- let candidates = self.narrow(&healthy).unwrap_or_else(|| healthy.clone());+ let candidates = self+ .narrow(&healthy)+ .unwrap_or_else(|| uniform(healthy.iter().cloned()));WorkerPool {healthy,candidates,}}- pub fn candidates(&self) -> Vec<WorkerId> {+ pub fn candidates(&self) -> Vec<Candidate> {let healthy = self.registry.healthy_workers();- self.narrow(&healthy).unwrap_or(healthy)+ self.narrow(&healthy)+ .unwrap_or_else(|| uniform(healthy.into_iter()))}fn active_aperture(&self) -> Option<&(Arc<PeerTracker>, usize)> {@@ -81,11 +84,13 @@ impl WorkerPoolSource {}}- fn narrow(&self, healthy: &[WorkerId]) -> Option<Vec<WorkerId>> {+ fn narrow(&self, healthy: &[WorkerId]) -> Option<Vec<Candidate>> {self.active_aperture()?;- let ring = aperture::sorted_ring(self.registry.workers());- // The fleet's slices tile the pool, so each consumer's requests- // consolidate onto few workers.+ let ring = aperture::sorted_ring(self.registry.members());self.slice(&ring, healthy)}}++fn uniform(workers: impl Iterator<Item = WorkerId>) -> Vec<Candidate> {+ workers.map(Candidate::uniform).collect()+}diff --git a/rust/ingestion-consumer/src/config.rs b/rust/ingestion-consumer/src/config.rsindex d5b8bd17f38a..06b79aec90be 100644--- a/rust/ingestion-consumer/src/config.rs+++ b/rust/ingestion-consumer/src/config.rs@@ -373,8 +373,9 @@ pub struct Config {/// Minimum aperture width for `INGESTION_ROUTING_STRATEGY=aperture`: how/// many workers this dispatcher's ring slice spans. The effective width- /// is floored at `ceil(pool / dispatchers)` so the fleet's slices always- /// cover the whole pool. Small values maximize sub-batch consolidation.+ /// is the smallest multiple of `pool / dispatchers` workers at or above+ /// this value, so every worker gets the same share of the fleet's slices.+ /// Small values maximize sub-batch consolidation.#[envconfig(from = "INGESTION_MIN_APERTURE", default = "3")]pub min_aperture: usize,diff --git a/rust/ingestion-consumer/src/main.rs b/rust/ingestion-consumer/src/main.rsindex ca87894f81e0..e4e4cae02640 100644--- a/rust/ingestion-consumer/src/main.rs+++ b/rust/ingestion-consumer/src/main.rs@@ -646,6 +646,7 @@ async fn build_debug_load(let ring = aperture::sorted_ring(snapshots.iter()+ .filter(|snap| !snap.draining).map(|snap| WorkerId::from(snap.url.as_str())).collect(),);@@ -659,7 +660,7 @@ async fn build_debug_load(strategy: pool_source.strategy().as_str().to_string(),min_aperture: pool_source.min_aperture(),ring: ring.iter().map(|w| w.to_string()).collect(),- slice: slice.map(|s| s.iter().map(|w| w.to_string()).collect()),+ slice: slice.map(|s| s.iter().map(|c| c.worker.to_string()).collect()),};let workers = snapshots.into_iter()diff --git a/rust/ingestion-consumer/src/routing.rs b/rust/ingestion-consumer/src/routing.rsindex cbebce92db36..679ddb83ebad 100644--- a/rust/ingestion-consumer/src/routing.rs+++ b/rust/ingestion-consumer/src/routing.rs@@ -8,6 +8,7 @@//! subsetting) without touching the batcher's bookkeeping.use std::collections::HashMap;+use std::num::NonZeroU64;use std::str::FromStr;use rand::rngs::StdRng;@@ -24,11 +25,11 @@ pub enum RoutingStrategy {#[default]P2c,/// Deterministic aperture: each consumer places requests only within- /// its slice of the worker ring (see [`crate::aperture`]), with P2C- /// selection inside the slice. Consolidates small batches onto few workers- /// while the fleet's slices still tile the whole pool. Requires peer- /// awareness (`PEER_SERVICE_NAME`); behaves as plain P2C over the full- /// pool while the peer set is unknown.+ /// its weighted slice of the worker ring (see [`crate::aperture`]), with+ /// weighted P2C selection inside the slice. Consolidates small batches onto+ /// few workers while every worker gets the same share of the fleet's+ /// slices. Requires peer awareness (`PEER_SERVICE_NAME`); behaves as plain+ /// P2C over the full pool while the peer set is unknown.Aperture,}@@ -56,6 +57,40 @@ impl RoutingStrategy {}}+/// A worker that may take a request, and its share of this consumer's+/// requests relative to the other candidates.+#[derive(Clone, Debug, PartialEq, Eq)]+pub struct Candidate {+ pub worker: WorkerId,+ pub weight: NonZeroU64,+ /// The weight of a worker that the candidate set covers whole.+ pub full_weight: NonZeroU64,+}++impl Candidate {+ /// `weight` above `full_weight` is clamped to `full_weight`.+ pub fn new(worker: WorkerId, weight: NonZeroU64, full_weight: NonZeroU64) -> Self {+ Self {+ worker,+ weight: weight.min(full_weight),+ full_weight,+ }+ }++ pub fn uniform(worker: WorkerId) -> Self {+ Self::new(worker, NonZeroU64::MIN, NonZeroU64::MIN)+ }++ /// The requests this consumer may keep in flight on the worker:+ /// `max_per_worker` scaled by `weight / full_weight` and rounded up, so a+ /// candidate keeps at least one slot when `max_per_worker` is not zero.+ pub fn slots(&self, max_per_worker: usize) -> usize {+ let full = u128::from(self.full_weight.get());+ let scaled = (max_per_worker as u128 * u128::from(self.weight.get())).div_ceil(full);+ scaled as usize+ }+}+/// Per-worker load for a single routing decision, keyed by worker id. A missing/// entry counts as zero load.pub type WorkerLoad = HashMap<WorkerId, usize>;@@ -94,40 +129,52 @@ impl Router {self.strategy}- /// Select a worker for one request. `healthy` lists candidate worker ids;- /// `load` is the current per-worker load (outstanding messages plus what has- /// been provisionally assigned earlier in this round). Returns `None` when- /// no workers are healthy.- pub fn select(&mut self, healthy: &[WorkerId], load: &WorkerLoad) -> Option<WorkerId> {- // Aperture narrows the candidate slice upstream (in the worker pool);- // within the slice it selects exactly like P2C.- select_p2c(healthy, load, &mut self.rng)+ /// Select a worker for one request. `load` is the current per-worker load+ /// (outstanding messages plus what has been provisionally assigned earlier+ /// in this round). Returns `None` when there are no candidates.+ pub fn select(&mut self, candidates: &[Candidate], load: &WorkerLoad) -> Option<WorkerId> {+ select_p2c(candidates, load, &mut self.rng)}}-/// Pick the lighter of two distinct random healthy workers. Degenerates to the-/// single worker when the pool has one, and to `None` when it is empty.-fn select_p2c(healthy: &[WorkerId], load: &WorkerLoad, rng: &mut impl Rng) -> Option<WorkerId> {- match healthy.len() {+/// Weighted power-of-two-choices: sample two distinct candidates in proportion+/// to weight, and pick the one with the lower load per unit of weight, so each+/// worker's share of the requests follows its weight.+fn select_p2c(candidates: &[Candidate], load: &WorkerLoad, rng: &mut impl Rng) -> Option<WorkerId> {+ match candidates.len() {0 => None,- 1 => Some(healthy[0].clone()),- len => {- // Sample two distinct *positions* in one step (no redraw loop): pick- // the first, then offset by 1..len to land on a different position.- // This terminates regardless of the slice's contents, so it stays- // correct even if a caller ever passes duplicate worker ids.- let i = rng.gen_range(0..len);- let j = (i + 1 + rng.gen_range(0..len - 1)) % len;- let (a, b) = (&healthy[i], &healthy[j]);- Some(if load_of(load, a) <= load_of(load, b) {- a.clone()- } else {- b.clone()- })+ 1 => Some(candidates[0].worker.clone()),+ _ => {+ // Positions, not ids, so duplicate worker ids cannot make the+ // second draw loop.+ let total: u64 = candidates.iter().map(|c| c.weight.get()).sum();+ let i = weighted_position(candidates, rng.gen_range(0..total), None);+ let rest = total - candidates[i].weight.get();+ let j = weighted_position(candidates, rng.gen_range(0..rest), Some(i));+ let (a, b) = (&candidates[i], &candidates[j]);+ // a.load / a.weight <= b.load / b.weight, without division.+ let a_cost = load_of(load, &a.worker) as u128 * u128::from(b.weight.get());+ let b_cost = load_of(load, &b.worker) as u128 * u128::from(a.weight.get());+ Some(if a_cost <= b_cost { a } else { b }.worker.clone())}}}+/// `point` must be below the total weight of the positions other than `skip`.+fn weighted_position(candidates: &[Candidate], mut point: u64, skip: Option<usize>) -> usize {+ for (position, candidate) in candidates.iter().enumerate() {+ if Some(position) == skip {+ continue;+ }+ let weight = candidate.weight.get();+ if point < weight {+ return position;+ }+ point -= weight;+ }+ unreachable!("point is below the total weight of the positions not skipped")+}+#[cfg(test)]mod tests {use super::*;@@ -172,6 +219,75 @@ mod tests {// ---- p2c ----+ fn uniform(workers: &[&str]) -> Vec<Candidate> {+ workers.iter().map(|w| Candidate::uniform(wid(w))).collect()+ }++ fn weighted(workers: &[(&str, u64)]) -> Vec<Candidate> {+ workers+ .iter()+ .map(|(w, weight)| {+ Candidate::new(wid(w), NonZeroU64::new(*weight).unwrap(), NonZeroU64::MAX)+ })+ .collect()+ }++ #[test]+ fn test_slots_scale_the_per_worker_cap_by_the_covered_part_rounded_up() {+ struct Case {+ weight: u64,+ full_weight: u64,+ max_per_worker: usize,+ slots: usize,+ }+ let cases = [+ Case {+ weight: 1,+ full_weight: 1,+ max_per_worker: 8,+ slots: 8,+ },+ Case {+ weight: 2,+ full_weight: 4,+ max_per_worker: 8,+ slots: 4,+ },+ Case {+ weight: 1,+ full_weight: 3,+ max_per_worker: 8,+ slots: 3,+ },+ Case {+ weight: 1,+ full_weight: 100,+ max_per_worker: 4,+ slots: 1,+ },+ Case {+ weight: 9,+ full_weight: 4,+ max_per_worker: 8,+ slots: 8,+ },+ ];+ for case in cases {+ let candidate = Candidate::new(+ wid(A),+ NonZeroU64::new(case.weight).unwrap(),+ NonZeroU64::new(case.full_weight).unwrap(),+ );+ assert_eq!(+ candidate.slots(case.max_per_worker),+ case.slots,+ "{} of {}",+ case.weight,+ case.full_weight+ );+ }+ }+#[test]fn test_p2c_empty_pool_returns_none() {let mut rng = StdRng::seed_from_u64(1);@@ -182,21 +298,33 @@ mod tests {fn test_p2c_single_worker_returns_it() {let mut rng = StdRng::seed_from_u64(1);assert_eq!(- select_p2c(&[wid(C)], &load(&[(C, 9)]), &mut rng),+ select_p2c(&uniform(&[C]), &load(&[(C, 9)]), &mut rng),Some(wid(C)));}#[test]fn test_p2c_two_workers_always_picks_lighter() {- // With exactly two candidates, sampling-without-replacement always draws- // both, so the lighter one is chosen deterministically regardless of seed.- let healthy = [wid(A), wid(B)];+ let candidates = uniform(&[A, B]);let l = load(&[(A, 3), (B, 0)]);for seed in 0..16 {let mut rng = StdRng::seed_from_u64(seed);assert_eq!(- select_p2c(&healthy, &l, &mut rng),+ select_p2c(&candidates, &l, &mut rng),+ Some(wid(B)),+ "seed {seed}"+ );+ }+ }++ #[test]+ fn test_p2c_two_workers_picks_the_lower_load_per_unit_of_weight() {+ let candidates = weighted(&[(A, 1), (B, 4)]);+ let l = load(&[(A, 2), (B, 4)]);+ for seed in 0..16 {+ let mut rng = StdRng::seed_from_u64(seed);+ assert_eq!(+ select_p2c(&candidates, &l, &mut rng),Some(wid(B)),"seed {seed}");@@ -205,37 +333,63 @@ mod tests {#[test]fn test_p2c_terminates_with_duplicate_ids() {- // Regression: selection must not loop even if the candidate slice- // contains duplicate worker ids.- let healthy = [wid(C), wid(C)];+ let candidates = uniform(&[C, C]);let mut rng = StdRng::seed_from_u64(1);for _ in 0..50 {assert_eq!(- select_p2c(&healthy, &WorkerLoad::new(), &mut rng),+ select_p2c(&candidates, &WorkerLoad::new(), &mut rng),Some(wid(C)));}}#[test]- fn test_p2c_only_ever_returns_a_healthy_worker() {- let healthy = [wid(A), wid(B), wid(C)];+ fn test_p2c_only_ever_returns_a_candidate() {+ let candidates = weighted(&[(A, 1), (B, 5), (C, 2)]);let l = load(&[(A, 2), (C, 5)]);let mut rng = StdRng::seed_from_u64(7);for _ in 0..200 {- let pick = select_p2c(&healthy, &l, &mut rng).unwrap();- assert!(healthy.contains(&pick));+ let pick = select_p2c(&candidates, &l, &mut rng).unwrap();+ assert!(candidates.iter().any(|c| c.worker == pick));}}#[test]fn test_p2c_spreads_across_pool() {- let healthy = [wid(A), wid(B), wid(C)];+ let candidates = uniform(&[A, B, C]);let mut rng = StdRng::seed_from_u64(42);let mut seen = std::collections::HashSet::new();for _ in 0..200 {- seen.insert(select_p2c(&healthy, &WorkerLoad::new(), &mut rng).unwrap());+ seen.insert(select_p2c(&candidates, &WorkerLoad::new(), &mut rng).unwrap());}assert_eq!(seen.len(), 3, "every worker should be reachable");}++ #[test]+ fn test_p2c_without_load_picks_in_proportion_to_weight() {+ let candidates = weighted(&[(A, 1), (B, 3)]);+ let mut rng = StdRng::seed_from_u64(3);+ let picks_of_a = (0..4000)+ .filter(|_| select_p2c(&candidates, &WorkerLoad::new(), &mut rng) == Some(wid(A)))+ .count();+ assert!((900..1100).contains(&picks_of_a), "{picks_of_a} of 4000");+ }++ #[test]+ fn test_p2c_load_held_in_flight_follows_weight() {+ let candidates = weighted(&[(A, 1), (B, 2), (C, 3)]);+ let mut rng = StdRng::seed_from_u64(9);+ let mut l = WorkerLoad::new();+ for _ in 0..600 {+ let pick = select_p2c(&candidates, &l, &mut rng).unwrap();+ *l.entry(pick).or_insert(0) += 1;+ }+ for (worker, expected) in [(A, 100), (B, 200), (C, 300)] {+ let held = load_of(&l, &wid(worker));+ assert!(+ held.abs_diff(expected) <= 10,+ "{worker}: {held}, expected {expected}"+ );+ }+ }}diff --git a/rust/ingestion-consumer/src/worker_registry.rs b/rust/ingestion-consumer/src/worker_registry.rsindex abbaf478d521..ccab556eccc5 100644--- a/rust/ingestion-consumer/src/worker_registry.rs+++ b/rust/ingestion-consumer/src/worker_registry.rs@@ -318,6 +318,16 @@ impl WorkerRegistry {self.workers.iter().map(|e| e.key().clone()).collect()}+ /// Snapshot of the workers that discovery lists, so not draining, in any+ /// health state. Every consumer of the same Service sees the same set.+ pub fn members(&self) -> Vec<WorkerId> {+ self.workers+ .iter()+ .filter(|e| !e.value().is_draining())+ .map(|e| e.key().clone())+ .collect()+ }+/// Snapshot of worker ids currently eligible for routing: Healthy or/// Degraded, and not draining. Returned in arbitrary order.pub fn healthy_workers(&self) -> Vec<WorkerId> {@@ -809,6 +819,19 @@ mod tests {assert!(!healthy.iter().any(|w| w.as_ref() == W2));}+ #[test]+ fn test_members_keep_unhealthy_workers_and_drop_draining_ones() {+ let registry = WorkerRegistry::new(&three_workers(), no_cooldown_config());+ for _ in 0..4 {+ registry.record_outcome(W2, true);+ }+ registry.start_draining(W3);++ let mut members: Vec<String> = registry.members().iter().map(|w| w.to_string()).collect();+ members.sort();+ assert_eq!(members, vec![W1.to_string(), W2.to_string()]);+ }+// --- Draining lifecycle ---#[test]