:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(ingestion-consumer): wire the governor into the batcher in observe mode (#121089)
José Sequeirae9ebf9b431b4
diff --git a/rust/ingestion-consumer/README.md b/rust/ingestion-consumer/README.mdindex 2b989da80774..200439867df6 100644--- a/rust/ingestion-consumer/README.md+++ b/rust/ingestion-consumer/README.md@@ -70,6 +70,7 @@ The batcher state machine in `src/batcher/` orders runs and places them on worke- Each key has a FIFO queue, and at most one request per key is in flight, which preserves per-key order.- The packer builds a request only for a free worker slot, from ready keys up to a target size. Keys stay unclaimed until then, so a key's run keeps growing while every slot is busy, up to the run cap. A key's backlog past the run cap leaves in consecutive requests, one in flight at a time. The run cap is the target unless `INGESTION_RUN_CAP_*` sets it apart. A request never mixes assignment epochs or fresh and replayed messages.- A request is placed on a worker when it is sent, against the load at that moment. Each worker takes at most `INGESTION_WORKER_CONCURRENT_BATCHES` requests at a time.+- A governor holds a permit pool between `INGESTION_GOVERNOR_MIN_PERMITS_PER_WORKER` and `INGESTION_WORKER_CONCURRENT_BATCHES` permits per routable worker, so at the ceiling the permits are the worker slots. A round-trip average over `INGESTION_GOVERNOR_LATENCY_BUDGET_MS` shrinks the pool by one. With the average within budget, the pool grows by one when the last pass was starved, which means it had ready work past the free permits. The pool moves at most once per `INGESTION_GOVERNOR_STEP_INTERVAL_MS`. Only accepted and partial acks feed the average. After a failed request, the requests sent to that worker until one of them settles feed nothing either, because they can wait out the stream's reconnect backoff. With `INGESTION_GOVERNOR_ADAPTATION=off` the pool is a shadow that reports what it would do while free slots size each pass.- A failed send returns its messages to the front of their keys' queues. They retry through the packer as replay after the retry delay.- A partial ack completes the messages the worker finished. Its unfinished messages return to the front of their keys' queues and go out again at once as replay. They need not be the last messages of their key.- A key resends without a budget once `INGESTION_REQUEST_BUDGET_ATTEMPTS` settles have left its messages unfinished since it last finished a run, or at once when its request finished nothing, because a budgeted retry would wait out the same budget again. So a message that never fits the budget still finishes. A failed request neither counts nor resets the count. The resend rides in a request that holds only such keys, ahead of budgeted replays.@@ -86,6 +87,10 @@ The batcher state machine in `src/batcher/` orders runs and places them on worke| `INGESTION_REQUEST_SOFT_BUDGET_MS` | `0` | How long a worker may spend on one request before it stops starting work and acks what finished as partial. The rest goes out again at once. `0` sends no budget. A request split into several frames goes without one. Startup fails if it is below `100` or above `2147483647`, if it times `INGESTION_WORKER_CONCURRENT_BATCHES` reaches `INGESTION_WORKER_STREAM_ACK_TIMEOUT_MS`, or if it plus `INGESTION_PACK_LATENCY_BUDGET_MS` reaches `CONSUMER_DEFERRED_FLUSH_TIMEOUT_MS`. || `INGESTION_REQUEST_BUDGET_ATTEMPTS` | `3` | How many budgeted attempts may leave a key's messages unfinished, since the key last finished a run, before it resends without a budget. At least 1, or startup fails. Only read when `INGESTION_REQUEST_SOFT_BUDGET_MS` is set. || `INGESTION_PARKED_RETRY_INTERVAL_MS` | `200` | The delay before a failed send retries, and how often a request with no routable worker tries again. |+| `INGESTION_GOVERNOR_ADAPTATION` | `off` | `off` keeps the governor observing: it measures request round-trip time and reports the permit pool it would run, while free worker slots size each pass. `on` sizes the pass by free permits too. |+| `INGESTION_GOVERNOR_LATENCY_BUDGET_MS` | `0` | A round-trip average above this shrinks the permit pool by one per step. `0` never shrinks. Startup fails when adaptation is on without it, when it reaches `INGESTION_REQUEST_SOFT_BUDGET_MS`, because a saturated worker answers partial at about the soft budget and a budget above it never shrinks on saturation, or when it reaches `INGESTION_WORKER_STREAM_ACK_TIMEOUT_MS`. |+| `INGESTION_GOVERNOR_MIN_PERMITS_PER_WORKER` | `1` | The permit pool never shrinks below this many permits per routable worker. Startup fails below `1`, above `INGESTION_WORKER_CONCURRENT_BATCHES`, or at it when adaptation is on, because the pool could not move. |+| `INGESTION_GOVERNOR_STEP_INTERVAL_MS` | `1000` | The least time between two permit steps. Startup fails below `100`. |Metrics:@@ -96,6 +101,10 @@ Metrics:- `ingestion_consumer_request_budget_partials_total` counts partial acks, and `ingestion_consumer_request_budget_unfinished_messages_total` counts the messages they left unfinished.- `ingestion_consumer_request_budget_escalations_total` counts keys that resend without a budget, after their budgeted attempts or after a partial ack that finished nothing. A steady rate means the budget is too tight for the lane, or that the workers are saturated: a frame that waits out its budget for a pipeline slot comes back as a partial ack with nothing finished, and the worker counts it in `ingestion_api_grpc_admission_expired_total`.- The `ingestion_consumer_batcher_*` gauges report keys, queued messages and bytes, claimed and waiting keys, unplaced work, and in-flight requests.+- `ingestion_consumer_governor_permits`, `ingestion_consumer_governor_ceiling` and `ingestion_consumer_governor_free_permits` report the permit pool, its ceiling, which is every slot on the routable workers, and the permits no request on a routable worker holds. With adaptation off the pool is the shadow pool.+- `ingestion_consumer_request_rtt_seconds` is the round-trip average the governor steps on.+- `ingestion_consumer_governor_adjustments_total{direction=grow|shrink,applied=true|false}` counts the governor's steps. `applied=false` means adaptation is off, so the step moved only the shadow pool. A step at a bound moves nothing and does not count.+- `ingestion_consumer_governor_starved_messages` reports the ready messages the last pass had past its free permits, which is demand the worker pool cannot take. With adaptation on they wait for a permit. With it off they went out on free slots, or wait for a slot.## Debug APIdiff --git a/rust/ingestion-consumer/src/batcher/driver.rs b/rust/ingestion-consumer/src/batcher/driver.rsindex d894caedd9a0..98ce4d4b2d87 100644--- a/rust/ingestion-consumer/src/batcher/driver.rs+++ b/rust/ingestion-consumer/src/batcher/driver.rs@@ -13,6 +13,7 @@ use tokio::sync::{mpsc, oneshot};use tokio::task::JoinHandle;use tracing::error;+use super::governor::Adaptation;use super::in_flight::RequestId;use super::key_queues::{payload_bytes, KeyRun};use super::state_machine::{@@ -55,6 +56,9 @@ pub(super) struct Load {pub pending_messages: AtomicUsize,pub in_flight_messages: AtomicUsize,pub busy_workers: Mutex<HashSet<WorkerId>>,+ /// Fixed at startup, so a snapshot that falls back to the totals still+ /// reports it.+ pub adaptation: Adaptation,}pub(super) struct RevokeSender(mpsc::UnboundedSender<Input>);@@ -161,7 +165,10 @@ impl StateMachineDriver {let (inputs_tx, inputs_rx) = mpsc::unbounded_channel();let (completions_tx, completions_rx) = mpsc::unbounded_channel();let (errors_tx, errors_rx) = mpsc::unbounded_channel();- let load = Arc::new(Load::default());+ let load = Arc::new(Load {+ adaptation: state.snapshot().governor.adaptation,+ ..Load::default()+ });let key_sentinel = Arc::new(KeyOrderSentinel::new());let assignment_epoch = sender.assignment_epoch();let task = BatcherTask {@@ -501,6 +508,8 @@ mod tests {use std::num::{NonZeroU32, NonZeroUsize};use std::time::Duration;+ use rstest::rstest;+use super::*;use crate::batcher::packer::{PackTargets, Packer};use crate::batcher::retry_policy::RetryPolicy;@@ -972,6 +981,30 @@ mod tests {);}+ #[rstest]+ #[case::accepted(SendOutcome::Accepted(1))]+ #[case::a_partial_that_finished_nothing(SendOutcome::Partial {+ accepted: 0,+ unfinished: vec![message("a", 0, 1)],+ })]+ #[tokio::test(start_paused = true)]+ async fn the_rtt_sample_is_the_time_between_dispatch_and_settlement(+ #[case] outcome: SendOutcome,+ ) {+ let h = Harness::new(&["w"], 4);+ let snapshots = h.driver.snapshot_sender();+ h.submit(vec![run("a", &[1])]);+ settle().await;+ let send = h.take_sent().pop().unwrap();++ let rtt = Duration::from_millis(750);+ tokio::time::advance(rtt).await;+ send.reply.send(Ok(outcome)).unwrap();+ settle().await;+ let snapshot = snapshots.snapshot().await.expect("the task answers");+ assert_eq!(snapshot.governor.rtt, Some(rtt));+ }+#[tokio::test(start_paused = true)]async fn request_events_follow_a_failed_send_and_its_replay() {let (h, recorder) = Harness::recorded(&["w"], 4);diff --git a/rust/ingestion-consumer/src/batcher/governor.rs b/rust/ingestion-consumer/src/batcher/governor.rsindex 49a33b4d72df..0d292c7001af 100644--- a/rust/ingestion-consumer/src/batcher/governor.rs+++ b/rust/ingestion-consumer/src/batcher/governor.rs@@ -9,6 +9,7 @@//! shadow: it moves by the law and reports what it would have done, while the//! batcher sizes its passes by free slots as before.+use std::str::FromStr;use std::time::{Duration, Instant};const RTT_TIME_CONSTANT: Duration = Duration::from_secs(10);@@ -20,12 +21,44 @@ pub enum Adaptation {On,}+impl Adaptation {+ pub fn as_str(self) -> &'static str {+ match self {+ Adaptation::Off => "off",+ Adaptation::On => "on",+ }+ }+}++impl FromStr for Adaptation {+ type Err = String;++ fn from_str(value: &str) -> Result<Self, Self::Err> {+ match value.trim().to_lowercase().as_str() {+ "off" => Ok(Adaptation::Off),+ "on" => Ok(Adaptation::On),+ other => Err(format!(+ "unknown governor adaptation '{other}' (expected 'on' or 'off')"+ )),+ }+ }+}+#[derive(Clone, Copy, Debug, PartialEq, Eq)]pub enum Direction {Grow,Shrink,}+impl Direction {+ pub fn as_str(self) -> &'static str {+ match self {+ Direction::Grow => "grow",+ Direction::Shrink => "shrink",+ }+ }+}+/// One step of the law that moved the pool. `applied` is false when/// adaptation is off, so the step moved only the shadow pool.#[derive(Clone, Copy, Debug, PartialEq, Eq)]diff --git a/rust/ingestion-consumer/src/batcher/in_flight.rs b/rust/ingestion-consumer/src/batcher/in_flight.rsindex 349772a6c7f9..e580a640dcc6 100644--- a/rust/ingestion-consumer/src/batcher/in_flight.rs+++ b/rust/ingestion-consumer/src/batcher/in_flight.rs@@ -3,6 +3,7 @@use std::collections::HashMap;use std::sync::Arc;+use std::time::Instant;use super::key_queues::KeyRun;use super::request::RequestClass;@@ -68,6 +69,7 @@ pub struct InFlightRequest {pub worker: WorkerId,pub class: RequestClass,pub message_count: usize,+ pub sent_at: Instant,runs: Vec<SentRun>,}@@ -94,6 +96,7 @@ impl InFlightRequests {&mut self,worker: WorkerId,class: RequestClass,+ sent_at: Instant,runs: &[KeyRun],) -> RequestId {let id = RequestId(self.next_id);@@ -112,6 +115,7 @@ impl InFlightRequests {worker,class,message_count,+ sent_at,runs,},);@@ -226,6 +230,7 @@ mod tests {let id = requests.register(WorkerId::from("w"),FRESH,+ Instant::now(),&[run("a", &[1, 2, 3]), run("b", &[10])],);requests.take(id).expect("registered")@@ -280,7 +285,12 @@ mod tests {#[test]fn a_partial_matches_a_message_the_request_carries_twice_by_occurrence() {let mut requests = InFlightRequests::new();- let id = requests.register(WorkerId::from("w"), FRESH, &[run("a", &[1, 2, 1])]);+ let id = requests.register(+ WorkerId::from("w"),+ FRESH,+ Instant::now(),+ &[run("a", &[1, 2, 1])],+ );let request = requests.take(id).expect("registered");let partial_runs = request.partially_accepted(2, vec![message("a", 0, 1)])diff --git a/rust/ingestion-consumer/src/batcher/mod.rs b/rust/ingestion-consumer/src/batcher/mod.rsindex fb552d784007..6b8405ff7e4c 100644--- a/rust/ingestion-consumer/src/batcher/mod.rs+++ b/rust/ingestion-consumer/src/batcher/mod.rs@@ -33,6 +33,7 @@ use common_kafka_consumer::GroupCompletion;use tokio::sync::mpsc;use self::driver::StateMachineDriver;+use self::governor::GovernorSnapshot;use self::state_machine::{BatcherSnapshot, BatcherStateMachine};use self::worker_pool::WorkerPoolSource;use crate::debug_recorder::DebugRecorder;@@ -133,6 +134,10 @@ impl BatcherObserver {None => BatcherSnapshot {pending_messages: self.pending_messages(),in_flight_messages: self.total_in_flight(),+ governor: GovernorSnapshot {+ adaptation: self.load.adaptation,+ ..GovernorSnapshot::default()+ },..BatcherSnapshot::default()},}diff --git a/rust/ingestion-consumer/src/batcher/state_machine.rs b/rust/ingestion-consumer/src/batcher/state_machine.rsindex 5aeb766d6764..073f320a50dd 100644--- a/rust/ingestion-consumer/src/batcher/state_machine.rs+++ b/rust/ingestion-consumer/src/batcher/state_machine.rs@@ -21,6 +21,7 @@ use std::time::{Duration, Instant};use common_kafka_consumer::{GroupCompletion, Offset, Partition};use metrics::{counter, gauge, histogram};+use super::governor::{Adaptation, Governor, GovernorSettings, GovernorSnapshot};use super::in_flight::{InFlightRequest, InFlightRequests, RequestId, SentMessage};use super::key_queues::{KeyQueues, KeyRun, Settlement};use super::packer::Packer;@@ -93,6 +94,7 @@ pub struct BatcherSnapshot {pub in_flight_requests: usize,pub in_flight_messages: usize,pub per_worker: Vec<WorkerInFlight>,+ pub governor: GovernorSnapshot,}pub enum BatcherStateMachine {@@ -126,6 +128,10 @@ impl BatcherStateMachine {if targets.events.is_none() && targets.bytes.is_none() {return Err("set a pack target in events or bytes".to_string());}+ let governor = Governor::new(+ GovernorSettings::observing(),+ assigner.max_requests_per_worker(),+ )?;Ok(BatcherStateMachine::Running(ActiveState {keys: KeyQueues::new(packer.run_cap()),packer,@@ -135,6 +141,8 @@ impl BatcherStateMachine {stall_timeout,last_progress: now,soft_budget: None,+ governor,+ reconnecting: HashMap::new(),}))}@@ -152,6 +160,19 @@ impl BatcherStateMachine {}}+ pub fn with_governor(self, settings: GovernorSettings) -> Result<Self, String> {+ match self {+ BatcherStateMachine::Running(active) => {+ let governor = Governor::new(settings, active.assigner.max_requests_per_worker())?;+ Ok(BatcherStateMachine::Running(ActiveState {+ governor,+ ..active+ }))+ }+ other => Ok(other),+ }+ }+pub fn on_groups(self,now: Instant,@@ -323,6 +344,12 @@ pub struct ActiveState {stall_timeout: Duration,last_progress: Instant,soft_budget: Option<Duration>,+ governor: Governor,+ /// Workers whose stream a failure fenced, with the time of that failure.+ /// The stream reconnects after a backoff, so a request sent since then+ /// may wait it out and is no RTT sample. The first such request to settle+ /// clears the mark.+ reconnecting: HashMap<WorkerId, Instant>,}impl ActiveState {@@ -340,6 +367,7 @@ impl ActiveState {in_flight_requests: self.in_flight.len(),in_flight_messages: self.assigner.in_flight_messages(),per_worker: self.assigner.in_flight_per_worker(),+ governor: self.governor.snapshot(),}}@@ -382,6 +410,7 @@ impl ActiveState {) -> Result<Effects, String> {let mut effects = Effects::default();let sent = self.take_request(request, &mut effects)?;+ self.observe_rtt(&sent, now);if accepted as usize != sent.message_count {return Err(format!("worker accepted {accepted} of {} messages",@@ -425,6 +454,7 @@ impl ActiveState {) -> Result<Effects, String> {let mut effects = Effects::default();let sent = self.take_request(request, &mut effects)?;+ self.observe_rtt(&sent, now);effects.worker_outcomes.push(WorkerOutcome {worker: sent.worker.clone(),fault: false,@@ -475,6 +505,7 @@ impl ActiveState {) -> Result<Effects, String> {let mut effects = Effects::default();let sent = self.take_request(request, &mut effects)?;+ self.reconnecting.entry(sent.worker.clone()).or_insert(now);effects.worker_outcomes.push(WorkerOutcome {worker: sent.worker.clone(),fault: cause == FailureCause::Fault,@@ -529,6 +560,45 @@ impl ActiveState {Ok(sent)}+ /// Only an accepted or partial settlement is a sample, because a fault or+ /// a busy answer measures the failure, not the worker. Saturation still+ /// reaches the average, because the workers send no busy answer: a+ /// request that finds no free slot waits for one, and at its soft budget+ /// it comes back as a partial with nothing finished. Faults and ack+ /// timeouts go to health routing instead, and a busy answer waits out the+ /// busy retry delay.+ fn observe_rtt(&mut self, sent: &InFlightRequest, now: Instant) {+ let sent_while_reconnecting = self+ .reconnecting+ .get(&sent.worker)+ .is_some_and(|failed_at| sent.sent_at >= *failed_at);+ if sent_while_reconnecting {+ self.reconnecting.remove(&sent.worker);+ return;+ }+ let rtt = now.saturating_duration_since(sent.sent_at);+ if let Some(decision) = self.governor.observe(rtt, now) {+ counter!(+ "ingestion_consumer_governor_adjustments_total",+ "direction" => decision.direction.as_str(),+ "applied" => if decision.applied { "true" } else { "false" }+ )+ .increment(1);+ }+ }++ /// A request on a worker that left the pool holds none of the pool's+ /// permits, so the governor counts only the requests on `workers`.+ fn sync_governor(&mut self, workers: &[WorkerId]) {+ let in_flight = workers+ .iter()+ .map(|worker| self.assigner.requests_on(worker))+ .sum();+ self.governor.set_pool(workers.len(), in_flight);+ self.reconnecting+ .retain(|worker, _| workers.contains(worker));+ }+fn settle_key(&mut self,routing_key: &Arc<str>,@@ -557,6 +627,7 @@ impl ActiveState {effects: &mut Effects,) -> Result<(), String> {self.restart_stall_clock_if_quiet(now);+ self.sync_governor(workers);// Past the stall deadline no new request starts, so overlapping// failures drain to nothing in flight and the watchdog can fire.let stalled = self.pending_messages() > 0 && now >= self.last_progress + self.stall_timeout;@@ -574,7 +645,19 @@ impl ActiveState {effects: &mut Effects,) -> Result<(), String> {let free_slots = self.assigner.free_slots(workers);- let batch = self.packer.pack(&mut self.keys, now, free_slots, draining);+ let free_permits = self.governor.free();+ let free = match self.governor.adaptation() {+ Adaptation::On => free_slots.min(free_permits),+ Adaptation::Off => free_slots,+ };+ let batch = self.packer.pack(&mut self.keys, now, free, draining);+ // With adaptation off free slots size the pass, so the requests past+ // the free permits are the ones the governor would have held back.+ let mut starved_messages: usize = batch+ .iter()+ .skip(free_permits)+ .map(|request| request.message_count)+ .sum();for request in batch {let worker = self.assigner@@ -585,6 +668,16 @@ impl ActiveState {}self.send(now, worker, request, effects);}+ self.sync_governor(workers);+ if self.governor.free() == 0 {+ starved_messages += self+ .keys+ .ready_sizes()+ .iter()+ .map(|(_, size)| size.messages)+ .sum::<usize>();+ }+ self.governor.set_starved_messages(starved_messages);Ok(())}@@ -602,7 +695,7 @@ impl ActiveState {);let id = self.in_flight- .register(worker.clone(), request.class, &request.runs);+ .register(worker.clone(), request.class, now, &request.runs);effects.sends.push(SendRequest {request: id,worker,@@ -649,6 +742,15 @@ impl ActiveState {gauge!("ingestion_consumer_batcher_claimed_keys").set(self.keys.claimed_keys() as f64);gauge!("ingestion_consumer_batcher_waiting_keys").set(self.keys.waiting_keys() as f64);gauge!("ingestion_consumer_batcher_in_flight_requests").set(self.in_flight.len() as f64);+ let governor = self.governor.snapshot();+ gauge!("ingestion_consumer_governor_permits").set(governor.permits as f64);+ gauge!("ingestion_consumer_governor_ceiling").set(governor.ceiling as f64);+ gauge!("ingestion_consumer_governor_free_permits").set(governor.free_permits as f64);+ gauge!("ingestion_consumer_governor_starved_messages")+ .set(governor.starved_messages as f64);+ if let Some(rtt) = governor.rtt {+ gauge!("ingestion_consumer_request_rtt_seconds").set(rtt.as_secs_f64());+ }}}@@ -726,6 +828,8 @@ mod tests {const NO_WORKER_DELAY: Duration = Duration::from_millis(100);const STALL: Duration = Duration::from_secs(60);const BUDGET: Duration = Duration::from_millis(250);+ const LATENCY_BUDGET: Duration = Duration::from_secs(1);+ const SLOW_RTT: Duration = Duration::from_secs(2);/// Distinct delays, so a wakeup time shows which retry reason applied.fn retry_policy() -> RetryPolicy {@@ -771,6 +875,18 @@ mod tests {}))}+ /// One worker with four slots, and a latency budget the slow settlements+ /// of these tests exceed.+ fn governed_batcher(adaptation: Adaptation, now: Instant) -> BatcherStateMachine {+ batcher(4, now)+ .with_governor(GovernorSettings {+ adaptation,+ latency_budget: Some(LATENCY_BUDGET),+ ..GovernorSettings::observing()+ })+ .expect("valid governor settings")+ }+fn pool(workers: &[&str]) -> Vec<WorkerId> {workers.iter().map(|w| WorkerId::from(*w)).collect()}@@ -823,6 +939,161 @@ mod tests {.collect()}+ fn saturate_then_settle_slowly(+ batcher: BatcherStateMachine,+ now: Instant,+ ) -> (BatcherStateMachine, Vec<String>) {+ let workers = pool(&["w"]);+ let runs = vec![+ run("a", &[1]),+ run("b", &[2]),+ run("c", &[3]),+ run("d", &[4]),+ ];+ let (batcher, effects) = batcher.on_groups(now, &workers, 0, runs);+ let first = effects.sends[0].request;+ let mut log = vec![format!("{effects:?}")];+ let settled_at = now + SLOW_RTT;+ let (batcher, effects) = batcher.on_request_succeeded(settled_at, &workers, first, 1);+ log.push(format!("{effects:?}"));+ let (batcher, effects) = batcher.on_groups(settled_at, &workers, 0, vec![run("e", &[5])]);+ log.push(format!("{effects:?}"));+ (batcher, log)+ }++ #[test]+ fn with_adaptation_off_the_governor_changes_no_effect_while_its_shadow_pool_moves() {+ let now = Instant::now();+ let (baseline, baseline_log) = saturate_then_settle_slowly(batcher(4, now), now);+ let (shadow, shadow_log) =+ saturate_then_settle_slowly(governed_batcher(Adaptation::Off, now), now);+ assert_eq!(baseline_log, shadow_log);++ assert_eq!(baseline.snapshot().governor.permits, 4);+ let governor = shadow.snapshot().governor;+ assert_eq!(governor.rtt, Some(SLOW_RTT));+ assert_eq!(+ (governor.permits, governor.ceiling),+ (3, 4),+ "the shadow pool shrank below the ceiling"+ );+ assert_eq!(+ governor.starved_messages, 1,+ "e went out on a free slot past the shadow pool's free permits"+ );+ }++ #[test]+ fn with_adaptation_on_free_permits_cap_the_pass_until_a_permit_returns() {+ let now = Instant::now();+ let workers = pool(&["w"]);+ let runs = vec![+ run("a", &[1]),+ run("b", &[2]),+ run("c", &[3]),+ run("d", &[4]),+ ];+ let (batcher, effects) =+ governed_batcher(Adaptation::On, now).on_groups(now, &workers, 0, runs);+ assert_eq!(effects.sends.len(), 4);+ let (a, b) = (effects.sends[0].request, effects.sends[1].request);++ let settled_at = now + SLOW_RTT;+ let (batcher, _) = batcher.on_request_succeeded(settled_at, &workers, a, 1);+ let (batcher, effects) = batcher.on_groups(settled_at, &workers, 0, vec![run("e", &[5])]);+ assert!(+ effects.sends.is_empty(),+ "a's slot is free but the pool shrank to three"+ );+ let governor = batcher.snapshot().governor;+ assert_eq!(+ (+ governor.permits,+ governor.free_permits,+ governor.starved_messages+ ),+ (3, 0, 1)+ );++ let (batcher, effects) = batcher.on_request_succeeded(settled_at, &workers, b, 1);+ assert_eq!(shape(&effects.sends[0]), vec![("e", vec![5])]);+ assert_eq!(+ batcher.snapshot().governor.starved_messages,+ 0,+ "the pass that sent e left nothing waiting"+ );+ }++ #[test]+ fn a_failure_and_the_requests_sent_until_its_worker_answers_again_give_no_rtt_sample() {+ let now = Instant::now();+ let workers = pool(&["w"]);+ let (batcher, effects) = batcher(4, now).on_groups(now, &workers, 0, vec![run("a", &[1])]);+ let request = effects.sends[0].request;+ assert_eq!(batcher.snapshot().governor.free_permits, 3);++ let failed_at = now + SLOW_RTT;+ let (batcher, _) = batcher.on_request_failed(+ failed_at,+ &workers,+ request,+ FailureCause::Busy,+ vec![message("a", 0, 1)],+ );+ let governor = batcher.snapshot().governor;+ assert_eq!(governor.free_permits, 4);+ assert_eq!(governor.rtt, None, "a busy answer is no round-trip sample");++ let (batcher, effects) = batcher.on_wakeup(failed_at + BUSY_DELAY, &workers);+ let retry = effects.sends[0].request;+ let answered_at = failed_at + BUSY_DELAY + SLOW_RTT;+ let (batcher, _) = batcher.on_request_succeeded(answered_at, &workers, retry, 1);+ assert_eq!(+ batcher.snapshot().governor.rtt,+ None,+ "the retry can wait out the stream's reconnect backoff"+ );++ let rtt = Duration::from_millis(100);+ let (batcher, effects) = batcher.on_groups(answered_at, &workers, 0, vec![run("b", &[2])]);+ let next = effects.sends[0].request;+ let (batcher, _) = batcher.on_request_succeeded(answered_at + rtt, &workers, next, 1);+ assert_eq!(batcher.snapshot().governor.rtt, Some(rtt));+ }++ #[test]+ fn a_draining_workers_requests_hold_none_of_the_routable_workers_permits() {+ let now = Instant::now();+ let both = pool(&["a", "b"]);+ let runs = (1..=8).map(|n| run(&format!("k{n}"), &[n])).collect();+ let (mut batcher, effects) =+ governed_batcher(Adaptation::On, now).on_groups(now, &both, 0, runs);+ assert_eq!(effects.sends.len(), 8);+ let on_a: Vec<_> = effects+ .sends+ .iter()+ .filter(|send| send.worker == WorkerId::from("a"))+ .map(|send| send.request)+ .collect();+ assert_eq!(on_a.len(), 4);++ let only_a = pool(&["a"]);+ for request in on_a {+ let effects;+ (batcher, effects) = batcher.on_request_succeeded(now, &only_a, request, 1);+ assert!(effects.sends.is_empty());+ }+ let governor = batcher.snapshot().governor;+ assert_eq!(+ (governor.ceiling, governor.free_permits),+ (4, 4),+ "b's four requests in flight hold none of a's permits"+ );++ let (_, effects) = batcher.on_groups(now, &only_a, 0, vec![run("k9", &[9])]);+ assert_eq!(shape(&effects.sends[0]), vec![("k9", vec![9])]);+ }+#[test]fn each_key_run_is_its_own_request_and_its_next_run_waits_for_the_response() {let now = Instant::now();diff --git a/rust/ingestion-consumer/src/batcher/worker_assigner.rs b/rust/ingestion-consumer/src/batcher/worker_assigner.rsindex 7091ea73eaef..415b5cc03181 100644--- a/rust/ingestion-consumer/src/batcher/worker_assigner.rs+++ b/rust/ingestion-consumer/src/batcher/worker_assigner.rs@@ -32,6 +32,10 @@ impl WorkerAssigner {})}+ pub fn max_requests_per_worker(&self) -> usize {+ self.max_requests_per_worker+ }+pub fn in_flight_messages(&self) -> usize {self.message_load.values().sum()}diff --git a/rust/ingestion-consumer/src/config.rs b/rust/ingestion-consumer/src/config.rsindex 5966cffd4ab0..5639e13547bb 100644--- a/rust/ingestion-consumer/src/config.rs+++ b/rust/ingestion-consumer/src/config.rs@@ -7,6 +7,7 @@ use envconfig::Envconfig;use rdkafka::ClientConfig;use tracing::info;+use crate::batcher::governor::{Adaptation, GovernorSettings};use crate::batcher::key_queues::RunCap;use crate::batcher::packer::{PackTargets, Packer};use crate::batcher::retry_policy::RetryPolicy;@@ -242,6 +243,28 @@ pub struct Config {#[envconfig(from = "INGESTION_REQUEST_BUDGET_ATTEMPTS", default = "3")]pub request_budget_attempts: u32,+ /// `off` (default) keeps the governor observing: it measures request+ /// round-trip time and reports the permit pool it would run, while free+ /// worker slots size each pass. `on` sizes the pass by free permits too.+ #[envconfig(from = "INGESTION_GOVERNOR_ADAPTATION", default = "off")]+ pub governor_adaptation: Adaptation,++ /// A round-trip average above this shrinks the permit pool by one per+ /// step (milliseconds). `0` (default) never shrinks. Required when+ /// adaptation is on. See `governor_settings` for the bounds against the+ /// soft budget and the ack timeout.+ #[envconfig(from = "INGESTION_GOVERNOR_LATENCY_BUDGET_MS", default = "0")]+ pub governor_latency_budget_ms: u64,++ /// The permit pool never shrinks below this many permits per routable+ /// worker. At least 1, at most `INGESTION_WORKER_CONCURRENT_BATCHES`.+ #[envconfig(from = "INGESTION_GOVERNOR_MIN_PERMITS_PER_WORKER", default = "1")]+ pub governor_min_permits_per_worker: usize,++ /// The least time between two permit steps (milliseconds). At least 100.+ #[envconfig(from = "INGESTION_GOVERNOR_STEP_INTERVAL_MS", default = "1000")]+ pub governor_step_interval_ms: u64,+/// Maximum Kafka batches to process concurrently. Matches the Node.js/// CONSUMER_MAX_BACKGROUND_TASKS setting used by the Kafka consumer wrapper.#[envconfig(from = "CONSUMER_MAX_BACKGROUND_TASKS", default = "1")]@@ -510,14 +533,73 @@ impl Config {self.ingestion_worker_concurrent_batches,)?;let retry = RetryPolicy::uniform(Duration::from_millis(self.parked_retry_interval_ms))?;- Ok(BatcherStateMachine::new(+ BatcherStateMachine::new(packer,assigner,retry,Duration::from_millis(self.consumer_deferred_flush_timeout_ms),Instant::now(),)?- .with_request_budget(self.request_budget()?))+ .with_request_budget(self.request_budget()?)+ .with_governor(self.governor_settings()?)+ }++ const MIN_GOVERNOR_STEP_INTERVAL_MS: u64 = 100;++ pub fn governor_settings(&self) -> Result<GovernorSettings, String> {+ let budget_ms = self.governor_latency_budget_ms;+ if self.governor_adaptation == Adaptation::On && budget_ms == 0 {+ return Err(+ "INGESTION_GOVERNOR_ADAPTATION=on needs INGESTION_GOVERNOR_LATENCY_BUDGET_MS"+ .to_string(),+ );+ }+ // A saturated worker answers partial at about the soft budget, so a+ // latency budget at or above it never shrinks on saturation.+ if budget_ms > 0+ && self.request_soft_budget_ms > 0+ && budget_ms >= self.request_soft_budget_ms+ {+ return Err("INGESTION_GOVERNOR_LATENCY_BUDGET_MS must be below \+ INGESTION_REQUEST_SOFT_BUDGET_MS"+ .to_string());+ }+ // The ack watchdog fails a request at the timeout, and a failure is+ // no sample.+ if budget_ms >= self.ingestion_worker_stream_ack_timeout_ms {+ return Err("INGESTION_GOVERNOR_LATENCY_BUDGET_MS must be below \+ INGESTION_WORKER_STREAM_ACK_TIMEOUT_MS"+ .to_string());+ }+ let min_permits = self.governor_min_permits_per_worker;+ if min_permits == 0 || min_permits > self.ingestion_worker_concurrent_batches {+ return Err(+ "INGESTION_GOVERNOR_MIN_PERMITS_PER_WORKER must be at least 1 and at most \+ INGESTION_WORKER_CONCURRENT_BATCHES"+ .to_string(),+ );+ }+ if self.governor_adaptation == Adaptation::On+ && min_permits == self.ingestion_worker_concurrent_batches+ {+ return Err(+ "INGESTION_GOVERNOR_ADAPTATION=on needs INGESTION_GOVERNOR_MIN_PERMITS_PER_WORKER \+ below INGESTION_WORKER_CONCURRENT_BATCHES, or the pool cannot move"+ .to_string(),+ );+ }+ if self.governor_step_interval_ms < Self::MIN_GOVERNOR_STEP_INTERVAL_MS {+ return Err(format!(+ "INGESTION_GOVERNOR_STEP_INTERVAL_MS must be at least {}",+ Self::MIN_GOVERNOR_STEP_INTERVAL_MS+ ));+ }+ Ok(GovernorSettings {+ adaptation: self.governor_adaptation,+ latency_budget: (budget_ms > 0).then(|| Duration::from_millis(budget_ms)),+ min_permits_per_worker: min_permits,+ step_interval: Duration::from_millis(self.governor_step_interval_ms),+ })}pub fn partition_budget(&self) -> Result<BudgetLimits, String> {@@ -740,6 +822,66 @@ mod tests {assert!(config(vars).partition_budget().is_err());}+ #[rstest]+ #[case::unset(&[], GovernorSettings::observing())]+ #[case::a_budget_with_adaptation_off(+ &[("INGESTION_GOVERNOR_LATENCY_BUDGET_MS", "3000")],+ GovernorSettings {+ latency_budget: Some(Duration::from_millis(3000)),+ ..GovernorSettings::observing()+ }+ )]+ #[case::adaptation_on(+ &[+ ("INGESTION_GOVERNOR_ADAPTATION", "on"),+ ("INGESTION_GOVERNOR_LATENCY_BUDGET_MS", "3000"),+ ("INGESTION_GOVERNOR_MIN_PERMITS_PER_WORKER", "2"),+ ("INGESTION_GOVERNOR_STEP_INTERVAL_MS", "100"),+ ("INGESTION_WORKER_CONCURRENT_BATCHES", "4"),+ ("INGESTION_REQUEST_SOFT_BUDGET_MS", "3001"),+ ],+ GovernorSettings {+ adaptation: Adaptation::On,+ latency_budget: Some(Duration::from_millis(3000)),+ min_permits_per_worker: 2,+ step_interval: Duration::from_millis(100),+ }+ )]+ fn valid_governor_settings_are_read(+ #[case] vars: &[(&str, &str)],+ #[case] expected: GovernorSettings,+ ) {+ assert_eq!(config(vars).governor_settings(), Ok(expected));+ }++ #[rstest]+ #[case::adaptation_on_without_a_budget(&[("INGESTION_GOVERNOR_ADAPTATION", "on")])]+ #[case::a_budget_at_the_soft_budget(&[+ ("INGESTION_GOVERNOR_LATENCY_BUDGET_MS", "3000"),+ ("INGESTION_REQUEST_SOFT_BUDGET_MS", "3000"),+ ])]+ #[case::a_budget_at_the_ack_timeout(&[("INGESTION_GOVERNOR_LATENCY_BUDGET_MS", "60000")])]+ #[case::no_floor(&[("INGESTION_GOVERNOR_MIN_PERMITS_PER_WORKER", "0")])]+ #[case::a_floor_above_the_per_worker_cap(&[+ ("INGESTION_GOVERNOR_MIN_PERMITS_PER_WORKER", "2"),+ ("INGESTION_WORKER_CONCURRENT_BATCHES", "1"),+ ])]+ #[case::a_step_interval_below_the_floor(&[("INGESTION_GOVERNOR_STEP_INTERVAL_MS", "99")])]+ #[case::adaptation_on_with_the_floor_at_the_cap(&[+ ("INGESTION_GOVERNOR_ADAPTATION", "on"),+ ("INGESTION_GOVERNOR_LATENCY_BUDGET_MS", "3000"),+ ])]+ fn invalid_governor_settings_are_rejected(#[case] vars: &[(&str, &str)]) {+ assert!(config(vars).governor_settings().is_err());+ }++ #[test]+ fn governor_adaptation_parses_on_and_off() {+ assert_eq!(Adaptation::from_str(" ON "), Ok(Adaptation::On));+ assert_eq!(Adaptation::from_str("off"), Ok(Adaptation::Off));+ assert!(Adaptation::from_str("true").is_err());+ }+#[test]fn completion_granularity_parses_known_values() {assert_eq!(diff --git a/rust/ingestion-consumer/src/debug_recorder.rs b/rust/ingestion-consumer/src/debug_recorder.rsindex ac9a5e814f76..1dbcfbf2f652 100644--- a/rust/ingestion-consumer/src/debug_recorder.rs+++ b/rust/ingestion-consumer/src/debug_recorder.rs@@ -201,6 +201,18 @@ pub struct BatcherLoad {pub waiting_keys: usize,/// Keys with a request in flight.pub claimed_keys: usize,+ pub governor: GovernorLoad,+}++#[derive(Serialize)]+pub struct GovernorLoad {+ pub adaptation: &'static str,+ pub permits: usize,+ pub ceiling: usize,+ pub free_permits: usize,+ pub starved_messages: usize,+ /// The round-trip average; `None` before the first sample.+ pub rtt_seconds: Option<f64>,}/// The consumer's routing configuration and, under aperture, its currentdiff --git a/rust/ingestion-consumer/src/main.rs b/rust/ingestion-consumer/src/main.rsindex 416489c7486a..48c7e3695e68 100644--- a/rust/ingestion-consumer/src/main.rs+++ b/rust/ingestion-consumer/src/main.rs@@ -25,7 +25,7 @@ use ingestion_consumer::batcher::{Batcher, BatcherObserver};use ingestion_consumer::config::Config;use ingestion_consumer::consumer::IngestionConsumer;use ingestion_consumer::debug_recorder::{- BatcherLoad, DebugLoad, DebugRecorder, DebugState, RoutingDebug, WorkerStatus,+ BatcherLoad, DebugLoad, DebugRecorder, DebugState, GovernorLoad, RoutingDebug, WorkerStatus,};use ingestion_consumer::discovery::{DiscoveryMode, EndpointSliceDiscovery, StaticDiscovery, WorkerDiscovery,@@ -681,6 +681,14 @@ async fn build_debug_load(queued_bytes: batcher.queued_bytes,waiting_keys: batcher.waiting_keys,claimed_keys: batcher.claimed_keys,+ governor: GovernorLoad {+ adaptation: batcher.governor.adaptation.as_str(),+ permits: batcher.governor.permits,+ ceiling: batcher.governor.ceiling,+ free_permits: batcher.governor.free_permits,+ starved_messages: batcher.governor.starved_messages,+ rtt_seconds: batcher.governor.rtt.map(|rtt| rtt.as_secs_f64()),+ },},peer_index,peer_count,