:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(personhog): keep a grow's parking floor until every born partition has a route (#121317)
Zach Brown808416fb2b39
diff --git a/rust/personhog-coordination/README.md b/rust/personhog-coordination/README.mdindex e8b37a406a57..a948781289e3 100644--- a/rust/personhog-coordination/README.md+++ b/rust/personhog-coordination/README.md@@ -135,7 +135,7 @@ The ordinary planner places only when no saga holds placement: no migration, orA leader serving a partition holds in its cache every acked write for that partition's keys that Postgres does not have yet, so a restart warms from every source that can hold one: for each draining generation, the old partitions congruent to the new one modulo `gcd(old, new)`, from the writer's committed offset to their sealed end and filtered to the keys that hash to the new partition, then the live partition itself. Marks from an old source keep their own stream (`topic`, source partition), so recovery fetches them from the old topic and the prune settles them against the writer's offsets there. A mark also names the cache partition it belongs to, and a release or prune of one partition only removes the marks and lifecycle fences homed there: after a flip, an old partition and a new one on the same pod share keys, and either may be released or pruned first. Nothing in the saga waits for the writer: it drains the oldest undrained generation first, flushes a sealed partition's tail as soon as it reaches the end, and re-subscribes in place once the group has drained the topic; its offsets carry their topic, since a resubscribe reuses the partition numbers. The control plane's retire op removes a generation from `draining` once the writer group has read each partition to its end; the config watch delivers a retire as it delivers a flip, and a leader settles the marks on a retired stream without the writer's offsets. Deleting the topic stays a Terraform change.-Routers park requests per key, so a key's requests keep one queue across the flip: a request goes straight to a leader only when its key has nothing parked or in flight and its partition is not stashing, and the live count decides only which partition's drain takes a key. A drain first reads the config from etcd, best effort, then takes the keys that map to its partition under the count the router has adopted; the leader bounces any key it took under an older one, and the reconcile pass drains every pending partition, with no owner when nothing names it; a write for a partition born in the flip parks until that partition's fresh handoff drains it. A leader answers a stale `x-partition` with `FAILED_PRECONDITION`, which the router treats as a bounce, so the write lands once the router's config watch delivers the flip; a router that has not adopted within its retries answers `UNAVAILABLE`, never an ack; a lifecycle batch with a member off the routed partition is refused with `INVALID_ARGUMENT`, which the identity service answers by retrying each member alone.+Routers park requests per key, so a key's requests keep one queue across the flip: a request goes straight to a leader only when its key has nothing parked or in flight and its partition is not stashing, and the live count decides only which partition's drain takes a key. A drain first reads the config from etcd, best effort, then takes the keys that map to its partition under the count the router has adopted; the leader bounces any key it took under an older one, and the reconcile pass drains every pending partition, with no owner when nothing names it; a write for a partition born in the flip parks until that partition's fresh handoff drains it. A router keeps a grow's floor past the retire until its own routing table has a route for every born partition, because its assignment and config watches are independent and the retire can arrive first. A leader answers a stale `x-partition` with `FAILED_PRECONDITION`, which the router treats as a bounce, so the write lands once the router's config watch delivers the flip; a router that has not adopted within its retries answers `UNAVAILABLE`, never an ack; a lifecycle batch with a member off the routed partition is refused with `INVALID_ARGUMENT`, which the identity service answers by retrying each member alone.## Modulesdiff --git a/rust/personhog-coordination/src/changelog.rs b/rust/personhog-coordination/src/changelog.rsindex 30134010cd79..8a288a0118a3 100644--- a/rust/personhog-coordination/src/changelog.rs+++ b/rust/personhog-coordination/src/changelog.rs@@ -1,7 +1,7 @@//! Boot-time load and the shared watch of `config/changelog`.use std::ops::ControlFlow;-use std::sync::atomic::{AtomicU32, Ordering};+use std::sync::atomic::{AtomicBool, AtomicU32, Ordering};use std::sync::Arc;use std::time::Duration;@@ -190,6 +190,10 @@ pub struct LiveChangelog {current: Arc<std::sync::RwLock<ChangelogConfig>>,partitions: Arc<AtomicU32>,draining_floor: Arc<AtomicU32>,+ /// The floor outlived the generation that set it, until `release_floor`.+ floor_held: Arc<AtomicBool>,+ /// Only a router routes by the floor, so only it holds one past a retire.+ holds_floor: bool,}/// The smallest partition count among the draining generations; `u32::MAX` when none drains.@@ -207,16 +211,25 @@ impl LiveChangelog {Self {partitions: Arc::new(AtomicU32::new(config.partitions)),draining_floor: Arc::new(AtomicU32::new(draining_floor_of(&config))),+ floor_held: Arc::new(AtomicBool::new(false)),+ holds_floor: false,current: Arc::new(std::sync::RwLock::new(config)),}}+ /// Keeps a grow's floor past its retire until `release_floor`, for a router.+ pub fn holding_floor(mut self) -> Self {+ self.holds_floor = true;+ metrics::gauge!("personhog_changelog_floor_held").set(0.0);+ self+ }+pub fn partitions(&self) -> u32 {self.partitions.load(Ordering::Acquire)}- /// Partitions at or above the floor were born in a flip some consumer- /// is still catching up on; `None` once nothing drains.+ /// Partitions at or above the floor were born in a flip; `None` once+ /// nothing drains and any held floor was released.pub fn draining_floor(&self) -> Option<u32> {let floor = self.draining_floor.load(Ordering::Acquire);(floor != u32::MAX).then_some(floor)@@ -244,13 +257,58 @@ impl LiveChangelog {if !config.supersedes(¤t) {return false;}+ // A retire can land before this process routes every partition the flip+ // created, so the floor of a grow stays until `release_floor`. A grow+ // adopted without its draining config holds one at the old count.+ let mut floor = draining_floor_of(config);+ let held = if self.holds_floor && floor == u32::MAX {+ let grown_from = if config.partitions > current.partitions {+ current.partitions+ } else {+ u32::MAX+ };+ let carried = self.draining_floor.load(Ordering::Acquire).min(grown_from);+ if carried < config.partitions {+ floor = carried;+ }+ carried < config.partitions+ } else {+ false+ };*current = config.clone();+ self.floor_held.store(held, Ordering::Release);// The floor first: a reader that sees the new count sees its floor too.- self.draining_floor- .store(draining_floor_of(config), Ordering::Release);+ self.draining_floor.store(floor, Ordering::Release);self.partitions.store(config.partitions, Ordering::Release);+ if self.holds_floor {+ metrics::gauge!("personhog_changelog_floor_held").set(if held { 1.0 } else { 0.0 });+ }true}++ /// Whether the floor outlived its draining generation.+ pub fn floor_held(&self) -> bool {+ self.floor_held.load(Ordering::Acquire)+ }++ /// Drops a held floor once the caller routes every partition in+ /// `floor..partitions`, if that is still the held range: an adoption+ /// between the caller's check and this call may have carried the floor+ /// over a larger count. A floor that a draining generation sets stays.+ pub fn release_floor(&self, floor: u32, partitions: u32) {+ let _current = self+ .current+ .write()+ .expect("changelog config lock poisoned");+ if self.floor_held.load(Ordering::Acquire)+ && self.draining_floor.load(Ordering::Acquire) == floor+ && self.partitions.load(Ordering::Acquire) == partitions+ {+ self.floor_held.store(false, Ordering::Release);+ self.draining_floor.store(u32::MAX, Ordering::Release);+ metrics::gauge!("personhog_changelog_floor_held").set(0.0);+ }+ }}pub async fn keep_changelog_live(@@ -322,6 +380,64 @@ mod tests {}}+ /// A grow's floor survives the retire until released; a shrink's never+ /// needs to; a grow adopted without its draining config holds one at the old count.+ #[test]+ fn a_grows_floor_is_held_past_the_retire_until_released() {+ let at_epoch = |epoch: u64, partitions: u32, draining: Vec<(u32, u64)>| ChangelogConfig {+ epoch,+ ..config(partitions, draining)+ };+ let cases = [+ (+ "grow, then retire",+ at_epoch(2, 8, vec![(4, 1)]),+ at_epoch(2, 8, vec![]),+ Some(4),+ ),+ (+ "shrink, then retire",+ at_epoch(2, 4, vec![(8, 1)]),+ at_epoch(2, 4, vec![]),+ None,+ ),+ (+ "grow seen only retired",+ at_epoch(1, 4, vec![]),+ at_epoch(2, 8, vec![]),+ Some(4),+ ),+ ];+ for (name, before, after, held) in cases {+ let live = LiveChangelog::new(before).holding_floor();+ assert!(live.adopt(&after), "{name}");+ assert_eq!(live.draining_floor(), held, "{name}");+ assert_eq!(live.floor_held(), held.is_some(), "{name}");+ live.release_floor(held.unwrap_or(u32::MAX), live.partitions());+ assert_eq!(live.draining_floor(), None, "{name}");+ }++ let draining = LiveChangelog::new(at_epoch(2, 8, vec![(4, 1)])).holding_floor();+ draining.release_floor(4, 8);+ assert_eq!(draining.draining_floor(), Some(4));++ // A second grow adopted between the check and the release carries the+ // floor over a larger count; the release of the checked range is stale.+ let regrown = LiveChangelog::new(at_epoch(2, 8, vec![(4, 1)])).holding_floor();+ assert!(regrown.adopt(&at_epoch(2, 8, vec![])));+ assert!(regrown.adopt(&at_epoch(3, 16, vec![])));+ regrown.release_floor(4, 8);+ assert_eq!(regrown.draining_floor(), Some(4));+ regrown.release_floor(4, 16);+ assert_eq!(regrown.draining_floor(), None);++ // A process that does not route never holds a floor past the retire.+ let leader = LiveChangelog::new(at_epoch(2, 8, vec![(4, 1)]));+ assert!(leader.adopt(&at_epoch(2, 8, vec![])));+ assert_eq!(leader.draining_floor(), None);+ assert!(!leader.floor_held());+ }+fn old_partitions(partitions: u32,draining: Vec<(u32, u64)>,diff --git a/rust/personhog-router/src/backend/leader.rs b/rust/personhog-router/src/backend/leader.rsindex 4b23e4fc8537..fb207b619e22 100644--- a/rust/personhog-router/src/backend/leader.rs+++ b/rust/personhog-router/src/backend/leader.rs@@ -437,15 +437,19 @@ impl LeaderBackend {let partition = self.partition_for_person(key.0, key.1);// A partition born in the flip parks its writes until its first handoff drains// them. The table's read lock ensures a route added later meets the stash.- if self- .config- .changelog+ let changelog = &self.config.changelog;+ if let Some(floor) = changelog.draining_floor()- .is_some_and(|floor| partition >= floor)+ .filter(|floor| partition >= *floor){let table = self.routing_table.read().await;if !table.contains_key(&partition) {self.stash.begin_stash(partition);+ } else if changelog.floor_held() {+ let partitions = changelog.partitions();+ if (floor..partitions).all(|p| table.contains_key(&p)) {+ changelog.release_floor(floor, partitions);+ }}}// The stash module emits its own enqueued/rejected countersdiff --git a/rust/personhog-router/src/main.rs b/rust/personhog-router/src/main.rsindex f8fe9c171072..c19739c3f5ab 100644--- a/rust/personhog-router/src/main.rs+++ b/rust/personhog-router/src/main.rs@@ -267,7 +267,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {epoch = loaded.epoch,"loaded changelog config");- let changelog = LiveChangelog::new(loaded);+ let changelog = LiveChangelog::new(loaded).holding_floor();// Build the routing table and leader backend, sharing the same// partition-to-pod mapping so both see consistent state.diff --git a/rust/personhog-router/tests/stash_integration.rs b/rust/personhog-router/tests/stash_integration.rsindex 12a0dd8aad8a..a520a580ac17 100644--- a/rust/personhog-router/tests/stash_integration.rs+++ b/rust/personhog-router/tests/stash_integration.rs@@ -212,6 +212,17 @@ async fn forward_read(/// a write for it parks and drains with that handoff instead of bouncing.#[tokio::test]async fn a_write_for_a_partition_born_in_the_flip_parks_until_its_handoff_drains() {+ born_partition_write_parks(false).await;+}++/// The retire can reach a router before the born partition's route does; the+/// write still parks rather than bouncing.+#[tokio::test]+async fn a_born_partition_write_parks_when_the_retire_lands_before_its_route() {+ born_partition_write_parks(true).await;+}++async fn born_partition_write_parks(retired_first: bool) {let old_count = NUM_PARTITIONS / 2;let team_id = 1;let person_id = (1..)@@ -242,7 +253,13 @@ async fn a_write_for_a_partition_born_in_the_flip_parks_until_its_handoff_drainspartitions: old_count,epoch: 1,}],- });+ })+ .holding_floor();+ if retired_first {+ let mut retired = changelog.current();+ retired.draining.clear();+ assert!(changelog.adopt(&retired));+ }let leader_url = format!("http://{leader_addr}");let resolver: personhog_router::backend::AddressResolver =Arc::new(move |_pod: &str| Some(leader_url.clone()));@@ -255,7 +272,7 @@ async fn a_write_for_a_partition_born_in_the_flip_parks_until_its_handoff_drainsnum_channels: 1,http2_windows: Http2Windows::default(),},- StashTable::with_bounds(usize::MAX, usize::MAX, changelog),+ StashTable::with_bounds(usize::MAX, usize::MAX, changelog.clone()),));let handler = new_test_handler(Arc::clone(&backend)).await;@@ -289,6 +306,23 @@ async fn a_write_for_a_partition_born_in_the_flip_parks_until_its_handoff_drains.await.expect("the write succeeds once its partition has an owner");assert_eq!(response.person.expect("a person").id, person_id);++ if retired_first {+ // Once every born partition has a route, the next write drops the held floor.+ for born in old_count..NUM_PARTITIONS {+ routing_table+ .write()+ .await+ .insert(born, "leader-0".to_string());+ }+ assert_eq!(changelog.draining_floor(), Some(old_count));+ forward(+ &backend,+ mk_request(team_id, person_id, "[email protected]"),+ )+ .await;+ assert_eq!(changelog.draining_floor(), None);+ }}/// A request that arrives while the stash is open must park on a oneshot,