:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(personhog): share the CDC parity rules in personhog-common (#119363)
José Sequeirafb9e55ff25b7
diff --git a/rust/personhog-common/README.md b/rust/personhog-common/README.mdindex 82ffae5b1449..92718a680d09 100644--- a/rust/personhog-common/README.md+++ b/rust/personhog-common/README.md@@ -65,3 +65,9 @@ graph TB- N -> M pods (deployment handoff)- crashed pods/corrupted disks- ensures handoffs do not introduce split brain invariant or cause service interruptions amongst the different actors in the distributed system++## CDC parity rules (`cdc_parity`)++`src/cdc_parity/` holds the rules for the two person CDC feeds, the `clickhouse_person_distinct_id` rows and the `person_merge_events` that both the Node Postgres path and `personhog-publisher` produce.+It covers strict decoding of the wire shapes, the fold to end state (the highest version per distinct id, one merge target per person, survivor chains), and the comparison between two feeds as the Kafka parity e2e (`nodejs/tests/service-e2e/personhog-kafka-parity.serial.test.ts`) states it: the same mapping per distinct id, the same survivor for every person both feeds named, and every person accounted for.+The test harness folds a published topic through it and the parity monitor compares the two feeds through it, so a rule lives in one place.diff --git a/rust/personhog-common/src/cdc_parity/compare.rs b/rust/personhog-common/src/cdc_parity/compare.rsnew file mode 100644index 000000000000..a769e5b0c4ca--- /dev/null+++ b/rust/personhog-common/src/cdc_parity/compare.rs@@ -0,0 +1,625 @@+//! The parity rules between the Node feed and the personhog feed: per key+//! for a streaming comparison, over whole states for a gate.++use std::collections::{BTreeSet, HashMap, HashSet};++use uuid::Uuid;++use super::feed::{DistinctIdKey, FeedState, Mapping, PersonKey};++#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]+pub enum Side {+ Node,+ Personhog,+}++impl Side {+ pub fn label(self) -> &'static str {+ match self {+ Side::Node => "node",+ Side::Personhog => "personhog",+ }+ }+}++/// A person difference outranks a tombstone difference.+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]+pub enum MappingOutcome {+ Agree,+ PersonMismatch,+ DeletedMismatch,+ NodeOnly,+ PersonhogOnly,+}++impl MappingOutcome {+ pub fn label(self) -> &'static str {+ match self {+ MappingOutcome::Agree => "agree",+ MappingOutcome::PersonMismatch => "person_mismatch",+ MappingOutcome::DeletedMismatch => "deleted_mismatch",+ MappingOutcome::NodeOnly => "node_only",+ MappingOutcome::PersonhogOnly => "personhog_only",+ }+ }+}++pub fn compare_mapping(node: Option<&Mapping>, personhog: Option<&Mapping>) -> MappingOutcome {+ match (node, personhog) {+ (None, None) => MappingOutcome::Agree,+ (Some(_), None) => MappingOutcome::NodeOnly,+ (None, Some(_)) => MappingOutcome::PersonhogOnly,+ (Some(node), Some(personhog)) if node.person_uuid != personhog.person_uuid => {+ MappingOutcome::PersonMismatch+ }+ (Some(node), Some(personhog)) if node.is_deleted != personhog.is_deleted => {+ MappingOutcome::DeletedMismatch+ }+ (Some(_), Some(_)) => MappingOutcome::Agree,+ }+}++/// A merge only one feed holds is that feed's stub when it also holds the+/// row that created the person and the other feed never named it; any+/// other silence from the other feed is one-sided.+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]+pub enum SurvivorOutcome {+ Agree,+ Mismatch,+ NodeOnly,+ PersonhogOnly,+ NodeStub,+ PersonhogStub,+ Loop(Side),+}++impl SurvivorOutcome {+ pub fn label(self) -> &'static str {+ match self {+ SurvivorOutcome::Agree => "agree",+ SurvivorOutcome::Mismatch => "survivor_mismatch",+ SurvivorOutcome::NodeOnly => "node_only",+ SurvivorOutcome::PersonhogOnly => "personhog_only",+ SurvivorOutcome::NodeStub => "node_stub",+ SurvivorOutcome::PersonhogStub => "personhog_stub",+ SurvivorOutcome::Loop(_) => "loop",+ }+ }+}++/// The same direct target is agreement on its own, so the verdict cannot+/// depend on whether the target's own later merge settled first. Different+/// direct targets are compared by survivor, because one feed may reach the+/// same person through a stub the other never made.+pub fn compare_survivor(+ node: &FeedState,+ personhog: &FeedState,+ old: PersonKey,+) -> SurvivorOutcome {+ match (node.merged_into(&old), personhog.merged_into(&old)) {+ (None, None) => SurvivorOutcome::Agree,+ (Some(_), None) => one_sided(+ node,+ personhog,+ &old,+ SurvivorOutcome::NodeOnly,+ SurvivorOutcome::NodeStub,+ ),+ (None, Some(_)) => one_sided(+ personhog,+ node,+ &old,+ SurvivorOutcome::PersonhogOnly,+ SurvivorOutcome::PersonhogStub,+ ),+ (Some(node_target), Some(personhog_target)) if node_target == personhog_target => {+ SurvivorOutcome::Agree+ }+ (Some(_), Some(_)) => match (node.survivor_of(old), personhog.survivor_of(old)) {+ (Err(_), _) => SurvivorOutcome::Loop(Side::Node),+ (_, Err(_)) => SurvivorOutcome::Loop(Side::Personhog),+ (Ok(node), Ok(personhog)) if node == personhog => SurvivorOutcome::Agree,+ _ => SurvivorOutcome::Mismatch,+ },+ }+}++/// A person the merging feed created in view and the other feed never saw+/// is the merging feed's stub. A person that predates the view, or that the+/// other feed named, makes the other feed's missing merge the finding.+fn one_sided(+ merging: &FeedState,+ other: &FeedState,+ old: &PersonKey,+ only: SurvivorOutcome,+ stub: SurvivorOutcome,+) -> SurvivorOutcome {+ if merging.has_row_for(old) && !other.names_person(old) {+ stub+ } else {+ only+ }+}++/// A feed that contradicts itself.+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]+pub enum AccountingViolation {+ Unaccounted {+ person: PersonKey,+ survivor: Uuid,+ },+ StaleMapping {+ key: DistinctIdKey,+ person: Uuid,+ survivor: Uuid,+ },+ Loop {+ person: PersonKey,+ },+}++/// Tombstoned rows count as holding too: a deleted person keeps its+/// tombstones under its own uuid, while a merged-away person's rows moved.+pub fn accounting_violations(feed: &FeedState) -> Vec<AccountingViolation> {+ let holders: HashSet<PersonKey> = feed+ .mappings()+ .map(|(key, mapping)| PersonKey::new(key.team_id, mapping.person_uuid))+ .collect();+ let deleted = deleted_persons(feed);+ let mut violations = Vec::new();+ for person in feed.named_persons() {+ match feed.survivor_of(person) {+ Err(_) => violations.push(AccountingViolation::Loop { person }),+ Ok(survivor) => {+ let survivor_key = PersonKey::new(person.team_id, survivor);+ if !holders.contains(&survivor_key) && !deleted.contains(&survivor_key) {+ violations.push(AccountingViolation::Unaccounted { person, survivor });+ }+ }+ }+ }+ for (key, mapping) in feed.mappings() {+ if mapping.is_deleted {+ continue;+ }+ if let Ok(survivor) = feed.survivor_of(PersonKey::new(key.team_id, mapping.person_uuid)) {+ if survivor != mapping.person_uuid {+ violations.push(AccountingViolation::StaleMapping {+ key: key.clone(),+ person: mapping.person_uuid,+ survivor,+ });+ }+ }+ }+ violations.sort();+ violations+}++/// Only a delete writes a tombstone: both feeds publish the rows a merge+/// moves as live rows under the target, so a missing merge event cannot+/// look like a delete.+fn deleted_persons(feed: &FeedState) -> HashSet<PersonKey> {+ let mut highest_rows: HashMap<(&DistinctIdKey, Uuid), &Mapping> = HashMap::new();+ for (key, row) in feed.history() {+ highest_rows+ .entry((key, row.person_uuid))+ .and_modify(|highest| {+ if row.version > highest.version {+ *highest = row;+ }+ })+ .or_insert(row);+ }+ let mut deleted = HashSet::new();+ let mut live = HashSet::new();+ for ((key, person_uuid), row) in highest_rows {+ let person = PersonKey::new(key.team_id, person_uuid);+ if row.is_deleted {+ deleted.insert(person);+ } else {+ live.insert(person);+ }+ }+ &deleted - &live+}++#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]+pub enum Divergence {+ Mapping {+ key: DistinctIdKey,+ outcome: MappingOutcome,+ node: Option<Mapping>,+ personhog: Option<Mapping>,+ },+ Survivor {+ person: PersonKey,+ node: Uuid,+ personhog: Uuid,+ },+ Accounting {+ side: Side,+ violation: AccountingViolation,+ },+}++/// A person only one feed named is ignored, as the e2e ignores it: a stub+/// the other backend never made and then merged away. Loops surface through+/// the accounting of the feed that holds them.+pub fn compare_states(node: &FeedState, personhog: &FeedState) -> Vec<Divergence> {+ let mut divergences = Vec::new();+ let keys: BTreeSet<&DistinctIdKey> = node+ .mappings()+ .chain(personhog.mappings())+ .map(|(key, _)| key)+ .collect();+ for key in keys {+ let (node_mapping, personhog_mapping) = (node.mapping(key), personhog.mapping(key));+ let outcome = compare_mapping(node_mapping, personhog_mapping);+ if outcome != MappingOutcome::Agree {+ divergences.push(Divergence::Mapping {+ key: key.clone(),+ outcome,+ node: node_mapping.cloned(),+ personhog: personhog_mapping.cloned(),+ });+ }+ }+ let node_persons = node.named_persons();+ let personhog_persons = personhog.named_persons();+ for person in node_persons.intersection(&personhog_persons) {+ if let (Ok(node_survivor), Ok(personhog_survivor)) =+ (node.survivor_of(*person), personhog.survivor_of(*person))+ {+ if node_survivor != personhog_survivor {+ divergences.push(Divergence::Survivor {+ person: *person,+ node: node_survivor,+ personhog: personhog_survivor,+ });+ }+ }+ }+ for (side, feed) in [(Side::Node, node), (Side::Personhog, personhog)] {+ for violation in accounting_violations(feed) {+ divergences.push(Divergence::Accounting { side, violation });+ }+ }+ divergences.sort();+ divergences+}++#[cfg(test)]+mod tests {+ use super::*;+ use crate::cdc_parity::decode::{DistinctIdRow, MergeEvent};++ const TEAM: i64 = 5;++ fn uuid(n: u128) -> Uuid {+ Uuid::from_u128(n)+ }++ fn mapping(person: u128, is_deleted: bool) -> Mapping {+ Mapping {+ person_uuid: uuid(person),+ is_deleted,+ version: 0,+ }+ }++ /// `rows` as (distinct id, person, version, deleted), `merges` as (old, new).+ fn feed(rows: &[(&str, u128, i64, bool)], merges: &[(u128, u128)]) -> FeedState {+ let mut feed = FeedState::new();+ for (distinct_id, person, version, is_deleted) in rows {+ feed.observe_mapping(&DistinctIdRow {+ team_id: TEAM,+ distinct_id: distinct_id.to_string(),+ person_uuid: uuid(*person),+ version: *version,+ is_deleted: *is_deleted,+ });+ }+ for (old, new) in merges {+ feed.observe_merge(&MergeEvent {+ team_id: TEAM,+ old_person_uuid: uuid(*old),+ new_person_uuid: uuid(*new),+ merged_at_ms: 10,+ });+ }+ feed+ }++ #[test]+ fn mapping_outcome_decision_table() {+ let live = |person| mapping(person, false);+ let gone = |person| mapping(person, true);+ assert_eq!(+ compare_mapping(Some(&live(1)), Some(&live(1))),+ MappingOutcome::Agree+ );+ assert_eq!(+ compare_mapping(Some(&gone(1)), Some(&gone(1))),+ MappingOutcome::Agree+ );+ assert_eq!(+ compare_mapping(Some(&live(1)), None),+ MappingOutcome::NodeOnly+ );+ assert_eq!(+ compare_mapping(None, Some(&live(1))),+ MappingOutcome::PersonhogOnly+ );+ assert_eq!(+ compare_mapping(Some(&live(1)), Some(&live(2))),+ MappingOutcome::PersonMismatch+ );+ assert_eq!(+ compare_mapping(Some(&live(1)), Some(&gone(1))),+ MappingOutcome::DeletedMismatch+ );+ assert_eq!(+ compare_mapping(Some(&live(1)), Some(&gone(2))),+ MappingOutcome::PersonMismatch+ );+ let newer = Mapping {+ version: 7,+ ..live(1)+ };+ assert_eq!(+ compare_mapping(Some(&live(1)), Some(&newer)),+ MappingOutcome::Agree+ );+ }++ #[test]+ fn survivor_outcome_decision_table() {+ let old = PersonKey::new(TEAM, uuid(1));+ let none = feed(&[], &[]);+ let knows_one = feed(&[("a", 1, 0, false)], &[]);+ let direct = feed(&[], &[(1, 2)]);+ let created_and_merged = feed(&[("a", 1, 0, false), ("a", 2, 1, false)], &[(1, 2)]);+ let chained = feed(&[], &[(1, 3), (3, 2)]);+ let downstream = feed(&[], &[(1, 2), (2, 5)]);+ let elsewhere = feed(&[], &[(1, 4)]);+ let looped_elsewhere = feed(&[], &[(1, 3), (3, 1)]);+ assert_eq!(compare_survivor(&none, &none, old), SurvivorOutcome::Agree);+ // The merging feed saw the person come to be; the other never named it.+ assert_eq!(+ compare_survivor(&created_and_merged, &none, old),+ SurvivorOutcome::NodeStub+ );+ assert_eq!(+ compare_survivor(&none, &created_and_merged, old),+ SurvivorOutcome::PersonhogStub+ );+ // The other feed named the person, or the person predates the view.+ assert_eq!(+ compare_survivor(&created_and_merged, &knows_one, old),+ SurvivorOutcome::NodeOnly+ );+ assert_eq!(+ compare_survivor(&direct, &none, old),+ SurvivorOutcome::NodeOnly+ );+ assert_eq!(+ compare_survivor(&none, &direct, old),+ SurvivorOutcome::PersonhogOnly+ );+ assert_eq!(+ compare_survivor(&direct, &chained, old),+ SurvivorOutcome::Agree+ );+ assert_eq!(+ compare_survivor(&direct, &downstream, old),+ SurvivorOutcome::Agree+ );+ assert_eq!(+ compare_survivor(&direct, &elsewhere, old),+ SurvivorOutcome::Mismatch+ );+ assert_eq!(+ compare_survivor(&looped_elsewhere, &direct, old),+ SurvivorOutcome::Loop(Side::Node)+ );+ assert_eq!(+ compare_survivor(&direct, &looped_elsewhere, old),+ SurvivorOutcome::Loop(Side::Personhog)+ );+ }++ // The shapes the Kafka parity e2e found: the same result reached by+ // different paths passes, and each way a feed can disagree is named.+ #[test]+ fn compare_states_decision_table() {+ // A birth on one backend, a create-then-merge on the other.+ let birth = feed(&[("anon", 2, 0, false), ("user", 2, 0, false)], &[]);+ let create_then_merge = feed(+ &[+ ("anon", 1, 0, false),+ ("anon", 2, 1, false),+ ("user", 2, 0, false),+ ],+ &[(1, 2)],+ );+ assert_eq!(compare_states(&birth, &create_then_merge), vec![]);+ assert_eq!(compare_states(&create_then_merge, &birth), vec![]);++ let deleted = feed(&[("d", 1, 1, true)], &[]);+ assert_eq!(compare_states(&deleted, &deleted), vec![]);+ let tombstone_left_behind = feed(&[("a", 1, 1, true), ("b", 2, 0, false)], &[(1, 2)]);+ assert_eq!(+ compare_states(&tombstone_left_behind, &tombstone_left_behind),+ vec![]+ );+ let reused_after_delete = feed(+ &[+ ("a", 1, 0, false),+ ("a", 2, 1, false),+ ("a", 2, 2, true),+ ("a", 3, 3, false),+ ],+ &[(1, 2)],+ );+ assert_eq!(+ compare_states(&reused_after_delete, &reused_after_delete),+ vec![]+ );+ let revived_then_moved = feed(+ &[+ ("a", 1, 0, false),+ ("a", 1, 1, true),+ ("a", 1, 2, false),+ ("a", 2, 3, false),+ ("b", 1, 1, true),+ ("b", 4, 2, false),+ ],+ &[],+ );+ assert_eq!(+ accounting_violations(&revived_then_moved),+ vec![AccountingViolation::Unaccounted {+ person: PersonKey::new(TEAM, uuid(1)),+ survivor: uuid(1),+ }]+ );++ let other_person = feed(&[("anon", 3, 0, false), ("user", 2, 0, false)], &[]);+ assert_eq!(+ compare_states(&birth, &other_person),+ vec![Divergence::Mapping {+ key: DistinctIdKey::new(TEAM, "anon"),+ outcome: MappingOutcome::PersonMismatch,+ node: Some(mapping(2, false)),+ personhog: Some(mapping(3, false)),+ }]+ );++ // A move without a merge event: the superseded row still names the+ // old person, which is what exposes the missing event.+ let silent_move = feed(+ &[+ ("anon", 1, 0, false),+ ("anon", 2, 1, false),+ ("user", 2, 0, false),+ ],+ &[],+ );+ assert_eq!(+ compare_states(&silent_move, &create_then_merge),+ vec![+ Divergence::Survivor {+ person: PersonKey::new(TEAM, uuid(1)),+ node: uuid(1),+ personhog: uuid(2),+ },+ Divergence::Accounting {+ side: Side::Node,+ violation: AccountingViolation::Unaccounted {+ person: PersonKey::new(TEAM, uuid(1)),+ survivor: uuid(1),+ },+ },+ ]+ );++ let into_two = feed(+ &[("a", 2, 1, false), ("b", 2, 0, false), ("c", 3, 0, false)],+ &[(1, 2)],+ );+ let into_three = feed(+ &[("a", 3, 1, false), ("b", 2, 0, false), ("c", 3, 0, false)],+ &[(1, 3)],+ );+ assert_eq!(+ compare_states(&into_two, &into_three),+ vec![+ Divergence::Mapping {+ key: DistinctIdKey::new(TEAM, "a"),+ outcome: MappingOutcome::PersonMismatch,+ node: Some(Mapping {+ version: 1,+ ..mapping(2, false)+ }),+ personhog: Some(Mapping {+ version: 1,+ ..mapping(3, false)+ }),+ },+ Divergence::Survivor {+ person: PersonKey::new(TEAM, uuid(1)),+ node: uuid(2),+ personhog: uuid(3),+ },+ ]+ );++ // Person 1 merged away but still mapped live, and survivor 2 holds nothing.+ let stale = feed(&[("a", 1, 0, false)], &[(1, 2)]);+ assert_eq!(+ compare_states(&stale, &stale),+ vec![+ Divergence::Accounting {+ side: Side::Node,+ violation: AccountingViolation::Unaccounted {+ person: PersonKey::new(TEAM, uuid(1)),+ survivor: uuid(2),+ },+ },+ Divergence::Accounting {+ side: Side::Node,+ violation: AccountingViolation::Unaccounted {+ person: PersonKey::new(TEAM, uuid(2)),+ survivor: uuid(2),+ },+ },+ Divergence::Accounting {+ side: Side::Node,+ violation: AccountingViolation::StaleMapping {+ key: DistinctIdKey::new(TEAM, "a"),+ person: uuid(1),+ survivor: uuid(2),+ },+ },+ Divergence::Accounting {+ side: Side::Personhog,+ violation: AccountingViolation::Unaccounted {+ person: PersonKey::new(TEAM, uuid(1)),+ survivor: uuid(2),+ },+ },+ Divergence::Accounting {+ side: Side::Personhog,+ violation: AccountingViolation::Unaccounted {+ person: PersonKey::new(TEAM, uuid(2)),+ survivor: uuid(2),+ },+ },+ Divergence::Accounting {+ side: Side::Personhog,+ violation: AccountingViolation::StaleMapping {+ key: DistinctIdKey::new(TEAM, "a"),+ person: uuid(1),+ survivor: uuid(2),+ },+ },+ ]+ );++ let looped = feed(&[("a", 2, 0, false)], &[(1, 2), (2, 1)]);+ assert_eq!(+ compare_states(&looped, &birth)+ .into_iter()+ .filter(|divergence| matches!(+ divergence,+ Divergence::Accounting {+ violation: AccountingViolation::Loop { .. },+ ..+ }+ ))+ .count(),+ 2+ );+ }+}diff --git a/rust/personhog-common/src/cdc_parity/decode.rs b/rust/personhog-common/src/cdc_parity/decode.rsnew file mode 100644index 000000000000..7f72544b207a--- /dev/null+++ b/rust/personhog-common/src/cdc_parity/decode.rs@@ -0,0 +1,299 @@+//! The wire shapes of the two topics, decoded as strictly as their+//! consumers read them: the ClickHouse Kafka table drops an unreadable row+//! without an error, so a payload it would lose is an error here.++use serde_json::Value;+use uuid::Uuid;++/// Mirrors `MERGE_EVENT_SCHEMA_VERSION` in the cohort stream processor.+pub const MERGE_EVENT_SCHEMA_VERSION: u64 = 1;++const DISTINCT_ID_ROW: &str = "distinct id row";+const MERGE_EVENT: &str = "merge event";++#[derive(Debug, thiserror::Error)]+pub enum DecodeError {+ #[error("{what} is not valid JSON: {source}")]+ Json {+ what: &'static str,+ #[source]+ source: serde_json::Error,+ },+ #[error("{what} without {field}")]+ MissingField {+ what: &'static str,+ field: &'static str,+ },+ #[error("distinct id row with is_deleted {value}, expected 0 or 1")]+ IsDeleted { value: Value },+ #[error("merge event with schema_version {value}, expected {MERGE_EVENT_SCHEMA_VERSION}")]+ SchemaVersion { value: Value },+ #[error("merge event with merged_at_ms {value}, expected a positive integer")]+ MergedAt { value: Value },+ #[error("merge event {old} -> {new} produced under key {actual:?}, expected {expected:?}")]+ Key {+ old: Uuid,+ new: Uuid,+ expected: String,+ actual: Option<String>,+ },+}++/// `person_id` on the wire is the person's uuid; `is_deleted` is the Kafka+/// table's `Int8`, 0 or 1.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct DistinctIdRow {+ pub team_id: i64,+ pub distinct_id: String,+ pub person_uuid: Uuid,+ pub version: i64,+ pub is_deleted: bool,+}++impl DistinctIdRow {+ pub fn decode(payload: &[u8]) -> Result<Self, DecodeError> {+ let value: Value = serde_json::from_slice(payload).map_err(|source| DecodeError::Json {+ what: DISTINCT_ID_ROW,+ source,+ })?;+ let missing = |field: &'static str| DecodeError::MissingField {+ what: DISTINCT_ID_ROW,+ field,+ };+ let team_id = value["team_id"]+ .as_i64()+ .ok_or_else(|| missing("an integer team_id"))?;+ let distinct_id = value["distinct_id"]+ .as_str()+ .ok_or_else(|| missing("a distinct_id"))?+ .to_string();+ let person_uuid =+ parse_uuid(&value["person_id"]).ok_or_else(|| missing("a person uuid"))?;+ let version = value["version"]+ .as_i64()+ .ok_or_else(|| missing("an integer version"))?;+ let is_deleted = match value["is_deleted"].as_u64() {+ Some(0) => false,+ Some(1) => true,+ _ => {+ return Err(DecodeError::IsDeleted {+ value: value["is_deleted"].clone(),+ })+ }+ };+ Ok(Self {+ team_id,+ distinct_id,+ person_uuid,+ version,+ is_deleted,+ })+ }+}++#[derive(Debug, Clone, PartialEq, Eq)]+pub struct MergeEvent {+ pub team_id: i64,+ pub old_person_uuid: Uuid,+ pub new_person_uuid: Uuid,+ pub merged_at_ms: i64,+}++impl MergeEvent {+ /// Checks the record key too: the cohort stream processor keeps the old+ /// person's state on the worker the key selects, so a wrong key lands+ /// the merge on the wrong worker without a decode error.+ pub fn decode(key: Option<&[u8]>, payload: &[u8]) -> Result<Self, DecodeError> {+ let value: Value = serde_json::from_slice(payload).map_err(|source| DecodeError::Json {+ what: MERGE_EVENT,+ source,+ })?;+ let missing = |field: &'static str| DecodeError::MissingField {+ what: MERGE_EVENT,+ field,+ };+ // The cohort stream processor reads team_id as i32.+ let team_id = value["team_id"]+ .as_i64()+ .filter(|team_id| i32::try_from(*team_id).is_ok())+ .ok_or_else(|| missing("a team_id in i32 range"))?;+ let old_person_uuid =+ parse_uuid(&value["old_person_uuid"]).ok_or_else(|| missing("an old_person_uuid"))?;+ let new_person_uuid =+ parse_uuid(&value["new_person_uuid"]).ok_or_else(|| missing("a new_person_uuid"))?;+ if value["schema_version"].as_u64() != Some(MERGE_EVENT_SCHEMA_VERSION) {+ return Err(DecodeError::SchemaVersion {+ value: value["schema_version"].clone(),+ });+ }+ let merged_at_ms = match value["merged_at_ms"].as_i64() {+ Some(ms) if ms > 0 => ms,+ _ => {+ return Err(DecodeError::MergedAt {+ value: value["merged_at_ms"].clone(),+ })+ }+ };+ let event = Self {+ team_id,+ old_person_uuid,+ new_person_uuid,+ merged_at_ms,+ };+ let expected = event.key();+ if key != Some(expected.as_bytes()) {+ return Err(DecodeError::Key {+ old: old_person_uuid,+ new: new_person_uuid,+ expected,+ actual: key.map(|key| String::from_utf8_lossy(key).into_owned()),+ });+ }+ Ok(event)+ }++ /// Node's key, `"{team_id}:{old_person_uuid}"`.+ pub fn key(&self) -> String {+ format!("{}:{}", self.team_id, self.old_person_uuid)+ }+}++fn parse_uuid(value: &Value) -> Option<Uuid> {+ value.as_str().and_then(|s| Uuid::parse_str(s).ok())+}++#[cfg(test)]+mod tests {+ use super::*;++ const UUID_1: &str = "00000000-0000-0000-0000-000000000001";+ const UUID_2: &str = "00000000-0000-0000-0000-000000000002";++ #[test]+ fn distinct_id_row_decode_decision_table() {+ let row = DistinctIdRow::decode(+ format!(r#"{{"person_id":"{UUID_1}","team_id":5,"distinct_id":"d","version":2,"is_deleted":1}}"#)+ .as_bytes(),+ )+ .expect("decodes");+ assert_eq!(+ row,+ DistinctIdRow {+ team_id: 5,+ distinct_id: "d".to_string(),+ person_uuid: Uuid::parse_str(UUID_1).unwrap(),+ version: 2,+ is_deleted: true,+ }+ );++ let error = |payload: &str| DistinctIdRow::decode(payload.as_bytes()).unwrap_err();+ assert!(matches!(error("not json"), DecodeError::Json { .. }));+ assert!(matches!(+ error(+ r#"{"person_id":"00000000-0000-0000-0000-000000000001","team_id":"5","distinct_id":"d","version":1,"is_deleted":0}"#+ ),+ DecodeError::MissingField {+ field: "an integer team_id",+ ..+ }+ ));+ assert!(matches!(+ error(r#"{"team_id":5,"distinct_id":"d","version":1,"is_deleted":0}"#),+ DecodeError::MissingField {+ field: "a person uuid",+ ..+ }+ ));+ assert!(matches!(+ error(+ r#"{"person_id":"not-a-uuid","team_id":5,"distinct_id":"d","version":1,"is_deleted":0}"#+ ),+ DecodeError::MissingField {+ field: "a person uuid",+ ..+ }+ ));+ assert!(matches!(+ error(+ r#"{"person_id":"00000000-0000-0000-0000-000000000001","team_id":5,"distinct_id":"d","version":1,"is_deleted":2}"#+ ),+ DecodeError::IsDeleted { .. }+ ));+ assert!(matches!(+ error(+ r#"{"person_id":"00000000-0000-0000-0000-000000000001","team_id":5,"distinct_id":"d","version":1,"is_deleted":true}"#+ ),+ DecodeError::IsDeleted { .. }+ ));+ }++ #[test]+ fn merge_event_decode_decision_table() {+ let payload = |schema_version: u64, merged_at_ms: i64| {+ format!(+ r#"{{"team_id":5,"old_person_uuid":"{UUID_1}","new_person_uuid":"{UUID_2}","merged_at_ms":{merged_at_ms},"schema_version":{schema_version}}}"#+ )+ };+ let key = format!("5:{UUID_1}");+ let event =+ MergeEvent::decode(Some(key.as_bytes()), payload(1, 10).as_bytes()).expect("decodes");+ assert_eq!(+ event,+ MergeEvent {+ team_id: 5,+ old_person_uuid: Uuid::parse_str(UUID_1).unwrap(),+ new_person_uuid: Uuid::parse_str(UUID_2).unwrap(),+ merged_at_ms: 10,+ }+ );+ assert_eq!(event.key(), key);++ let error = |key: Option<&[u8]>, payload: String| {+ MergeEvent::decode(key, payload.as_bytes()).unwrap_err()+ };+ assert!(matches!(+ error(Some(key.as_bytes()), payload(2, 10)),+ DecodeError::SchemaVersion { .. }+ ));+ assert!(matches!(+ error(Some(key.as_bytes()), payload(1, 0)),+ DecodeError::MergedAt { .. }+ ));+ assert!(matches!(+ error(None, payload(1, 10)),+ DecodeError::Key { actual: None, .. }+ ));+ assert!(matches!(+ error(Some(b"5:wrong".as_slice()), payload(1, 10)),+ DecodeError::Key {+ actual: Some(_),+ ..+ }+ ));+ assert!(matches!(+ error(+ Some(key.as_bytes()),+ format!(+ r#"{{"team_id":5,"old_person_uuid":"{UUID_1}","merged_at_ms":10,"schema_version":1}}"#+ )+ ),+ DecodeError::MissingField {+ field: "a new_person_uuid",+ ..+ }+ ));+ assert!(matches!(+ error(+ Some(format!("4294967301:{UUID_1}").as_bytes()),+ format!(+ r#"{{"team_id":4294967301,"old_person_uuid":"{UUID_1}","new_person_uuid":"{UUID_2}","merged_at_ms":10,"schema_version":1}}"#+ )+ ),+ DecodeError::MissingField {+ field: "a team_id in i32 range",+ ..+ }+ ));+ }+}diff --git a/rust/personhog-common/src/cdc_parity/feed.rs b/rust/personhog-common/src/cdc_parity/feed.rsnew file mode 100644index 000000000000..300cf880061f--- /dev/null+++ b/rust/personhog-common/src/cdc_parity/feed.rs@@ -0,0 +1,447 @@+//! The end state a reader of one feed is left with: the highest version per+//! distinct id, as a ReplacingMergeTree keeps it, and one merge target per+//! merged person, as the cohort stream processor applies it.++use std::collections::{HashMap, HashSet};++use uuid::Uuid;++use super::decode::{DistinctIdRow, MergeEvent};++/// The version only orders rows; it is never compared across feeds.+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]+pub struct Mapping {+ pub person_uuid: Uuid,+ pub is_deleted: bool,+ pub version: i64,+}++impl From<&DistinctIdRow> for Mapping {+ fn from(row: &DistinctIdRow) -> Self {+ Self {+ person_uuid: row.person_uuid,+ is_deleted: row.is_deleted,+ version: row.version,+ }+ }+}++#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]+pub struct DistinctIdKey {+ pub team_id: i64,+ pub distinct_id: String,+}++impl DistinctIdKey {+ pub fn new(team_id: i64, distinct_id: impl Into<String>) -> Self {+ Self {+ team_id,+ distinct_id: distinct_id.into(),+ }+ }+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]+pub struct PersonKey {+ pub team_id: i64,+ pub person_uuid: Uuid,+}++impl PersonKey {+ pub fn new(team_id: i64, person_uuid: Uuid) -> Self {+ Self {+ team_id,+ person_uuid,+ }+ }+}++/// Two different rows at one version, reported once per version.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct MappingConflict {+ pub key: DistinctIdKey,+ pub version: i64,+ pub first: Mapping,+ pub second: Mapping,+}++/// One person merged into two different persons.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct MergeConflict {+ pub old: PersonKey,+ pub first: Uuid,+ pub second: Uuid,+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, thiserror::Error)]+#[error("merge events of team {} loop through {}", .person.team_id, .person.person_uuid)]+pub struct MergeLoop {+ pub person: PersonKey,+}++/// The first row sits inline because most keys only ever see one; later+/// rows are indexed by version, so a busy key costs one lookup per arrival.+#[derive(Debug)]+struct MappingHistory {+ first: Mapping,+ more: HashMap<i64, Vec<Mapping>>,+ latest_version: i64,+}++impl MappingHistory {+ fn new(first: Mapping) -> Self {+ Self {+ latest_version: first.version,+ first,+ more: HashMap::new(),+ }+ }++ fn rows(&self) -> impl Iterator<Item = &Mapping> {+ std::iter::once(&self.first).chain(self.more.values().flatten())+ }++ /// The rows stored at one version, the first arrival first.+ fn at_version(&self, version: i64) -> impl Iterator<Item = &Mapping> {+ let first = (self.first.version == version).then_some(&self.first);+ first+ .into_iter()+ .chain(self.more.get(&version).into_iter().flatten())+ }++ /// The highest version; the first arrival among equal versions.+ fn latest(&self) -> &Mapping {+ if self.first.version == self.latest_version {+ return &self.first;+ }+ self.more+ .get(&self.latest_version)+ .and_then(|rows| rows.first())+ .unwrap_or(&self.first)+ }++ fn insert(&mut self, mapping: Mapping) {+ self.latest_version = self.latest_version.max(mapping.version);+ self.more.entry(mapping.version).or_default().push(mapping);+ }+}++/// How many stored rows and merge ends name one person.+#[derive(Debug, Default, Clone, Copy)]+struct Names {+ rows: u32,+ merges: u32,+}++#[derive(Debug, Default)]+pub struct FeedState {+ mappings: HashMap<DistinctIdKey, MappingHistory>,+ merged_into: HashMap<PersonKey, Uuid>,+ /// Kept exact through eviction, so naming is one lookup.+ named: HashMap<PersonKey, Names>,+ /// Rows retained across every key, for a memory bound.+ rows: usize,+}++impl FeedState {+ pub fn new() -> Self {+ Self::default()+ }++ pub fn observe_mapping(&mut self, row: &DistinctIdRow) -> Option<MappingConflict> {+ let key = DistinctIdKey::new(row.team_id, row.distinct_id.as_str());+ let mapping = Mapping::from(row);+ let person = PersonKey::new(row.team_id, row.person_uuid);+ let Some(history) = self.mappings.get_mut(&key) else {+ self.mappings.insert(key, MappingHistory::new(mapping));+ self.named.entry(person).or_default().rows += 1;+ self.rows += 1;+ return None;+ };+ if history+ .at_version(mapping.version)+ .any(|seen| *seen == mapping)+ {+ return None;+ }+ // A second row already stored at this version was reported then.+ let conflict = {+ let mut at_version = history.at_version(mapping.version);+ match (at_version.next(), at_version.next()) {+ (Some(first), None) => Some(MappingConflict {+ key: key.clone(),+ version: mapping.version,+ first: first.clone(),+ second: mapping.clone(),+ }),+ _ => None,+ }+ };+ history.insert(mapping);+ self.named.entry(person).or_default().rows += 1;+ self.rows += 1;+ conflict+ }++ /// Keeps the first target; a different second one is the conflict.+ pub fn observe_merge(&mut self, event: &MergeEvent) -> Option<MergeConflict> {+ let old = PersonKey::new(event.team_id, event.old_person_uuid);+ match self.merged_into.get(&old) {+ Some(&first) if first != event.new_person_uuid => Some(MergeConflict {+ old,+ first,+ second: event.new_person_uuid,+ }),+ Some(_) => None,+ None => {+ self.merged_into.insert(old, event.new_person_uuid);+ self.named.entry(old).or_default().merges += 1;+ self.named+ .entry(PersonKey::new(event.team_id, event.new_person_uuid))+ .or_default()+ .merges += 1;+ None+ }+ }+ }++ fn unname(&mut self, person: PersonKey, update: impl FnOnce(&mut Names)) {+ if let Some(names) = self.named.get_mut(&person) {+ update(names);+ if names.rows == 0 && names.merges == 0 {+ self.named.remove(&person);+ }+ }+ }++ pub fn mapping(&self, key: &DistinctIdKey) -> Option<&Mapping> {+ self.mappings.get(key).map(MappingHistory::latest)+ }++ pub fn merged_into(&self, old: &PersonKey) -> Option<Uuid> {+ self.merged_into.get(old).copied()+ }++ pub fn survivor_of(&self, person: PersonKey) -> Result<Uuid, MergeLoop> {+ let mut visited: Vec<Uuid> = Vec::new();+ let mut current = person;+ while let Some(&next) = self.merged_into.get(¤t) {+ if visited.contains(¤t.person_uuid) {+ return Err(MergeLoop { person: current });+ }+ visited.push(current.person_uuid);+ current = PersonKey::new(person.team_id, next);+ }+ Ok(current.person_uuid)+ }++ /// A person only one feed named is that feed's own path to the result.+ pub fn names_person(&self, person: &PersonKey) -> bool {+ self.named.contains_key(person)+ }++ /// Whether a stored row, at any version, maps a distinct id to the+ /// person: the feed saw the person come to be, not only a merge of it.+ pub fn has_row_for(&self, person: &PersonKey) -> bool {+ self.named.get(person).is_some_and(|names| names.rows > 0)+ }++ pub fn mappings(&self) -> impl Iterator<Item = (&DistinctIdKey, &Mapping)> {+ self.mappings+ .iter()+ .map(|(key, history)| (key, history.latest()))+ }++ pub fn history(&self) -> impl Iterator<Item = (&DistinctIdKey, &Mapping)> {+ self.mappings+ .iter()+ .flat_map(|(key, history)| history.rows().map(move |row| (key, row)))+ }++ pub fn merges(&self) -> impl Iterator<Item = (&PersonKey, Uuid)> {+ self.merged_into.iter().map(|(old, new)| (old, *new))+ }++ pub fn named_persons(&self) -> HashSet<PersonKey> {+ self.named.keys().copied().collect()+ }++ pub fn remove_mapping(&mut self, key: &DistinctIdKey) -> Option<Mapping> {+ let history = self.mappings.remove(key)?;+ let latest = history.latest().clone();+ for row in history.rows() {+ self.unname(PersonKey::new(key.team_id, row.person_uuid), |names| {+ names.rows -= 1+ });+ self.rows -= 1;+ }+ Some(latest)+ }++ pub fn remove_merge(&mut self, old: &PersonKey) -> Option<Uuid> {+ let new = self.merged_into.remove(old)?;+ self.unname(*old, |names| names.merges -= 1);+ self.unname(PersonKey::new(old.team_id, new), |names| names.merges -= 1);+ Some(new)+ }++ pub fn mapping_count(&self) -> usize {+ self.mappings.len()+ }++ /// Rows retained across every key, history included.+ pub fn row_count(&self) -> usize {+ self.rows+ }++ pub fn merge_count(&self) -> usize {+ self.merged_into.len()+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn uuid(n: u128) -> Uuid {+ Uuid::from_u128(n)+ }++ fn person(n: u128) -> PersonKey {+ PersonKey::new(5, uuid(n))+ }++ fn row(person: u128, version: i64, is_deleted: bool) -> DistinctIdRow {+ DistinctIdRow {+ team_id: 5,+ distinct_id: "d".to_string(),+ person_uuid: uuid(person),+ version,+ is_deleted,+ }+ }++ fn mapping(person: u128, version: i64, is_deleted: bool) -> Mapping {+ Mapping {+ person_uuid: uuid(person),+ is_deleted,+ version,+ }+ }++ fn merge(old: u128, new: u128) -> MergeEvent {+ MergeEvent {+ team_id: 5,+ old_person_uuid: uuid(old),+ new_person_uuid: uuid(new),+ merged_at_ms: 10,+ }+ }++ // A replay after a crash repeats rows in any order; only a different+ // row at an already seen version is a finding, and only once.+ #[test]+ fn rows_fold_to_the_highest_version_and_report_a_tie_once() {+ let key = DistinctIdKey::new(5, "d");+ let mut feed = FeedState::new();+ for entry in [+ row(1, 1, false),+ row(1, 2, false),+ row(1, 2, false),+ row(1, 1, false),+ ] {+ assert_eq!(feed.observe_mapping(&entry), None);+ }+ assert_eq!(feed.mapping(&key), Some(&mapping(1, 2, false)));++ assert_eq!(+ feed.observe_mapping(&row(2, 2, false)),+ Some(MappingConflict {+ key: key.clone(),+ version: 2,+ first: mapping(1, 2, false),+ second: mapping(2, 2, false),+ })+ );+ assert_eq!(feed.observe_mapping(&row(2, 2, false)), None);+ assert_eq!(feed.observe_mapping(&row(2, 2, true)), None);+ assert_eq!(feed.observe_mapping(&row(1, 3, false)), None);+ assert_eq!(feed.mapping(&key), Some(&mapping(1, 3, false)));+ // A tie at the first row's own version is a conflict too.+ assert_eq!(+ feed.observe_mapping(&row(2, 1, false)),+ Some(MappingConflict {+ key: key.clone(),+ version: 1,+ first: mapping(1, 1, false),+ second: mapping(2, 1, false),+ })+ );+ assert_eq!(feed.observe_mapping(&row(2, 1, false)), None);+ assert_eq!(feed.mapping(&key), Some(&mapping(1, 3, false)));+ assert_eq!(feed.row_count(), 6);++ // The same disagreement read after a higher version is still one.+ let mut later = FeedState::new();+ assert_eq!(later.observe_mapping(&row(1, 3, false)), None);+ assert_eq!(later.observe_mapping(&row(1, 2, false)), None);+ assert!(later.observe_mapping(&row(2, 2, false)).is_some());+ assert_eq!(later.mapping(&key), Some(&mapping(1, 3, false)));+ assert!(later.names_person(&person(2)));+ assert!(later.has_row_for(&person(2)));++ // Eviction forgets the history and the persons only it named.+ assert_eq!(later.row_count(), 3);+ assert_eq!(later.remove_mapping(&key), Some(mapping(1, 3, false)));+ assert_eq!(later.row_count(), 0);+ assert!(!later.names_person(&person(1)));+ assert!(!later.names_person(&person(2)));+ assert_eq!(later.observe_mapping(&row(2, 2, false)), None);+ assert_eq!(later.mapping(&key), Some(&mapping(2, 2, false)));+ assert_eq!(later.mapping_count(), 1);+ assert_eq!(later.row_count(), 1);+ }++ #[test]+ fn merges_keep_the_first_target_and_follow_chains() {+ let mut feed = FeedState::new();+ assert_eq!(feed.observe_merge(&merge(1, 2)), None);+ assert_eq!(feed.observe_merge(&merge(1, 2)), None);+ assert_eq!(feed.observe_merge(&merge(2, 3)), None);+ assert_eq!(feed.survivor_of(person(1)), Ok(uuid(3)));+ assert_eq!(feed.survivor_of(person(3)), Ok(uuid(3)));+ assert_eq!(feed.survivor_of(person(9)), Ok(uuid(9)));++ assert_eq!(+ feed.observe_merge(&merge(1, 4)),+ Some(MergeConflict {+ old: person(1),+ first: uuid(2),+ second: uuid(4),+ })+ );+ assert_eq!(feed.merged_into(&person(1)), Some(uuid(2)));++ assert_eq!(feed.observe_merge(&merge(3, 1)), None);+ assert_eq!(+ feed.survivor_of(person(1)),+ Err(MergeLoop { person: person(1) })+ );+ assert_eq!(+ feed.named_persons(),+ HashSet::from([person(1), person(2), person(3)])+ );+ // A merge end names the person without a row for it.+ assert!(feed.names_person(&person(1)));+ assert!(!feed.has_row_for(&person(1)));++ // A person stays named while any merge still ends in it.+ assert_eq!(feed.remove_merge(&person(3)), Some(uuid(1)));+ assert_eq!(feed.survivor_of(person(1)), Ok(uuid(3)));+ assert!(feed.names_person(&person(3)));+ assert_eq!(feed.remove_merge(&person(2)), Some(uuid(3)));+ assert!(!feed.names_person(&person(3)));+ assert!(feed.names_person(&person(1)));+ assert_eq!(feed.merge_count(), 1);+ }+}diff --git a/rust/personhog-common/src/cdc_parity/mod.rs b/rust/personhog-common/src/cdc_parity/mod.rsnew file mode 100644index 000000000000..619f1e8d1b44--- /dev/null+++ b/rust/personhog-common/src/cdc_parity/mod.rs@@ -0,0 +1,18 @@+//! The parity rules for the two person CDC feeds: the+//! `clickhouse_person_distinct_id` rows and `person_merge_events` that the+//! Node Postgres path and `personhog-publisher` both produce. Feeds are+//! compared by end state, not message by message, because the two backends+//! take different paths to the same place.++pub mod compare;+pub mod decode;+pub mod feed;++pub use compare::{+ accounting_violations, compare_mapping, compare_states, compare_survivor, AccountingViolation,+ Divergence, MappingOutcome, Side, SurvivorOutcome,+};+pub use decode::{DecodeError, DistinctIdRow, MergeEvent, MERGE_EVENT_SCHEMA_VERSION};+pub use feed::{+ DistinctIdKey, FeedState, Mapping, MappingConflict, MergeConflict, MergeLoop, PersonKey,+};diff --git a/rust/personhog-common/src/lib.rs b/rust/personhog-common/src/lib.rsindex 3298ea96a87a..1cada8f94ac4 100644--- a/rust/personhog-common/src/lib.rs+++ b/rust/personhog-common/src/lib.rs@@ -1,4 +1,5 @@pub mod async_gzip;+pub mod cdc_parity;pub mod client;pub mod grpc;pub mod h2_window;diff --git a/rust/personhog-common/src/partitioning.rs b/rust/personhog-common/src/partitioning.rsindex 64c280e44d3d..f116990c56fd 100644--- a/rust/personhog-common/src/partitioning.rs+++ b/rust/personhog-common/src/partitioning.rs@@ -16,9 +16,15 @@ pub fn partition_for_person(team_id: i64, person_id: i64, num_partitions: u32) -write!(cursor, "{team_id}:{person_id}").unwrap();cursor.position() as usize};- let hash = kafka_murmur2(&buf[..len]);+ partition_for_key(&buf[..len], num_partitions)+}++/// Kafka's default partitioner for a keyed record, which Node's+/// `murmur2Partition` and librdkafka's `murmur2_random` both reproduce.+pub fn partition_for_key(key: &[u8], num_partitions: u32) -> u32 {+ assert!(num_partitions > 0, "num_partitions must be > 0");// Kafka's toPositive: hash & 0x7fffffff- let positive = (hash & 0x7fffffff) as u32;+ let positive = (kafka_murmur2(key) & 0x7fffffff) as u32;positive % num_partitions}@@ -105,6 +111,17 @@ mod tests {assert_eq!(kafka_murmur2(b"test-key"), -1341026247);}+ // The fixture `nodejs/src/common/kafka/murmur2.test.ts` pins.+ #[test]+ fn partition_for_key_matches_nodes_partitioner() {+ let key = b"2:01928aaa-bbbb-cccc-dddd-eeeeeeeeeeee";+ assert_eq!(kafka_murmur2(key), 989609914);+ assert_eq!(partition_for_key(key, 64), 58);+ // A negative hash: the mask, not the absolute value, picks the partition.+ assert_eq!(kafka_murmur2(b"21"), -973932308);+ assert_eq!(partition_for_key(b"21", 10), 0);+ }+#[test]fn partition_for_person_deterministic() {let p1 = partition_for_person(1, 42, 16);diff --git a/rust/personhog-test-harness/README.md b/rust/personhog-test-harness/README.mdindex 7333467156d1..ad4eb454f0ed 100644--- a/rust/personhog-test-harness/README.md+++ b/rust/personhog-test-harness/README.md@@ -120,6 +120,7 @@ It implies `--create-via-identity`. The publishers claim every team's rows, so t- Distinct ids: after the Postgres quiesce, the highest-version row per distinct id on the topic must equal the Postgres mapping (person uuid, `is_deleted`, version), and no distinct id may be on the topic that Postgres does not hold. Rows repeated at the same version are fine: a replay after a crash is the publisher's contract, and the consumers absorb it. Two different rows at one version are a violation, because a ReplacingMergeTree would keep either.- Merge events: the set of (old, new) person uuid pairs on the topic must equal the merged sources the merge lane journaled, each event carrying `schema_version` 1, a positive `merged_at_ms` and the key `<team_id>:<old uuid>` the cohort processor places it by. A source whose merge op never settled is excused either way; those are violations already. The event count is printed next to the pair count, so a replay shows in the output without failing the gate.- Deletes: after the delete leg, Postgres must hold every mapping as a tombstone under the same uuid, and the topic must match those rows exactly, version included.+- The fold itself (strict decoding, highest version per distinct id, one merge target per person, the key check) is `personhog_common::cdc_parity`, shared with the Kafka feed parity checks; the gate only maps its findings to violations and adds the Postgres side.- The outbox must drain within 60 s after the Postgres quiesce, and again after the delete leg. A row the publisher cannot read stays pending with its `last_error` and is listed when the drain misses its deadline; a row parked by hand (`given_up_at`) is a violation, because only an operator sets it.`--publish-rows-per-pass` (default 100) keeps `LIFECYCLE_PUBLISH_MAX_ROWS_PER_PASS` small, so a wide merge drains across several passes and the pointer cursor is exercised.diff --git a/rust/personhog-test-harness/src/publish.rs b/rust/personhog-test-harness/src/publish.rsindex eb5dbec5b3a6..47172ae0b3be 100644--- a/rust/personhog-test-harness/src/publish.rs+++ b/rust/personhog-test-harness/src/publish.rs@@ -9,6 +9,10 @@ use std::collections::{HashMap, HashSet};use std::time::{Duration, Instant};use anyhow::{bail, Context, Result};+pub use personhog_common::cdc_parity::Mapping as PublishedMapping;+use personhog_common::cdc_parity::{+ DistinctIdRow, FeedState, MappingConflict, MergeConflict, MergeEvent,+};use serde_json::{json, Value};use sqlx::postgres::PgPool;use sqlx::Row;@@ -24,13 +28,6 @@ use crate::state::MergedSource;pub const DRAIN_DEADLINE: Duration = Duration::from_secs(60);const READ_DEADLINE: Duration = Duration::from_secs(30);-#[derive(Debug, Clone, PartialEq, Eq)]-pub struct PublishedMapping {- pub person_uuid: Uuid,- pub is_deleted: bool,- pub version: i64,-}-#[derive(Debug, Clone, PartialEq, Eq)]pub struct ExpectedMapping {pub person_id: i64,@@ -51,8 +48,10 @@ pub struct PublishedMappings {pub struct PublishedMerges {pub pairs: HashSet<(Uuid, Uuid)>,+ /// Resolved to violations where the person ids are known.+ pub conflicts: Vec<MergeConflict>,/// Events read, duplicates included. More events than pairs is a- /// replay, which only a crash or a produce timeout should cause.+ /// replay or a conflict.pub events: usize,pub foreign_events: usize,}@@ -272,73 +271,28 @@ pub async fn read_published_mappings(fold_published_mappings(records.iter().map(|r| r.payload.as_slice()), team_id)}-/// Fold distinct id rows into the state a ClickHouse reader would see: the-/// highest version per distinct id. Two rows that differ at one version-/// are a conflict whatever the version, because partition order decides-/// which one a reader sees. Rows of other teams are counted and skipped; a-/// row that does not decode is a publisher bug and fails the read outright.+/// A row that does not decode is a publisher bug and fails the read.pub fn fold_published_mappings<'a>(payloads: impl IntoIterator<Item = &'a [u8]>,team_id: i64,) -> Result<PublishedMappings> {- let mut rows: HashMap<String, PublishedMapping> = HashMap::new();- let mut seen: HashMap<(String, i64), PublishedMapping> = HashMap::new();- let mut conflicted: HashSet<(String, i64)> = HashSet::new();+ let mut feed = FeedState::new();let mut conflicts = Vec::new();let mut foreign_rows = 0;for payload in payloads {- let value: Value = serde_json::from_slice(payload).context("decoding a distinct id row")?;- let Some(row_team) = value["team_id"].as_i64() else {- bail!("distinct id row without an integer team_id");- };- if row_team != team_id {+ let row = DistinctIdRow::decode(payload)?;+ if row.team_id != team_id {foreign_rows += 1;continue;}- let distinct_id = value["distinct_id"]- .as_str()- .context("distinct id row without a distinct_id")?- .to_string();- let person_uuid = value["person_id"]- .as_str()- .and_then(|s| Uuid::parse_str(s).ok())- .context("distinct id row without a person uuid")?;- let version = value["version"]- .as_i64()- .context("distinct id row without an integer version")?;- let is_deleted = match value["is_deleted"].as_u64() {- Some(0) => false,- Some(1) => true,- other => bail!("distinct id row with is_deleted {other:?}, expected 0 or 1"),- };- let row = PublishedMapping {- person_uuid,- is_deleted,- version,- };- let versioned = (distinct_id.clone(), version);- match seen.get(&versioned) {- Some(first) if *first != row && !conflicted.contains(&versioned) => {- conflicts.push(ConsistencyViolation {- person_id: 0,- key: format!("__published_mapping_conflict:{distinct_id}"),- expected: json!(format!("one row at version {version}")),- actual: json!([mapping_json(first), mapping_json(&row)]),- });- conflicted.insert(versioned);- }- Some(_) => {}- None => {- seen.insert(versioned, row.clone());- }- }- match rows.get(&distinct_id) {- Some(current) if current.version >= version => {}- _ => {- rows.insert(distinct_id, row);- }+ if let Some(conflict) = feed.observe_mapping(&row) {+ conflicts.push(mapping_conflict_violation(conflict));}}+ let rows = feed+ .mappings()+ .map(|(key, mapping)| (key.distinct_id.clone(), mapping.clone()))+ .collect();Ok(PublishedMappings {rows,conflicts,@@ -346,6 +300,18 @@ pub fn fold_published_mappings<'a>(})}+fn mapping_conflict_violation(conflict: MappingConflict) -> ConsistencyViolation {+ ConsistencyViolation {+ person_id: 0,+ key: format!("__published_mapping_conflict:{}", conflict.key.distinct_id),+ expected: json!(format!("one row at version {}", conflict.version)),+ actual: json!([+ mapping_json(&conflict.first),+ mapping_json(&conflict.second)+ ]),+ }+}+fn mapping_json(row: &PublishedMapping) -> Value {json!({"person_uuid": row.person_uuid,@@ -479,59 +445,34 @@ pub async fn read_published_merges()}-/// The (old, new) pairs on the merge events topic. Every event must carry-/// the wire shape and the key the cohort stream processor depends on; one-/// that does not is a publisher bug and fails the read outright.+/// An event without the wire shape or the key the cohort stream processor+/// depends on is a publisher bug and fails the read.pub fn fold_published_merges<'a>(records: impl IntoIterator<Item = (Option<&'a [u8]>, &'a [u8])>,team_id: i64,) -> Result<PublishedMerges> {- let mut pairs = HashSet::new();+ let mut feed = FeedState::new();+ let mut conflicts = Vec::new();let mut events = 0;let mut foreign_events = 0;for (key, payload) in records {- let value: Value = serde_json::from_slice(payload).context("decoding a merge event")?;- let Some(event_team) = value["team_id"].as_i64() else {- bail!("merge event without an integer team_id");- };- if event_team != team_id {+ let event = MergeEvent::decode(key, payload)?;+ if event.team_id != team_id {foreign_events += 1;continue;}- let uuid = |field: &str| {- value[field]- .as_str()- .and_then(|s| Uuid::parse_str(s).ok())- .with_context(|| format!("merge event without a {field} uuid"))- };- let old = uuid("old_person_uuid")?;- let new = uuid("new_person_uuid")?;- if value["schema_version"].as_u64() != Some(1) {- bail!(- "merge event with schema_version {}, expected 1",- value["schema_version"]- );- }- if value["merged_at_ms"].as_i64().is_none_or(|ms| ms <= 0) {- bail!(- "merge event with merged_at_ms {}, expected a positive integer",- value["merged_at_ms"]- );- }- // The processor keeps the old person's state on the worker this- // key selects, so a different key misplaces the merge silently.- let expected_key = format!("{team_id}:{old}");- if key != Some(expected_key.as_bytes()) {- bail!(- "merge event {old} -> {new} produced under key {:?}, expected {expected_key:?}",- key.map(String::from_utf8_lossy)- );- }events += 1;- pairs.insert((old, new));+ if let Some(conflict) = feed.observe_merge(&event) {+ conflicts.push(conflict);+ }}+ let pairs = feed+ .merges()+ .map(|(old, new)| (old.person_uuid, new))+ .collect();Ok(PublishedMerges {pairs,+ conflicts,events,foreign_events,})@@ -539,9 +480,22 @@ pub fn fold_published_merges<'a>(pub fn verify_published_merges(expected: &ExpectedMerges,- published: &HashSet<(Uuid, Uuid)>,+ published: &PublishedMerges,) -> Vec<ConsistencyViolation> {let mut violations = Vec::new();+ for conflict in &published.conflicts {+ violations.push(ConsistencyViolation {+ person_id: expected+ .ids+ .get(&conflict.old.person_uuid)+ .copied()+ .unwrap_or_default(),+ key: "__published_merge_conflict".to_string(),+ expected: json!(format!("one merge target for {}", conflict.old.person_uuid)),+ actual: json!([conflict.first, conflict.second]),+ });+ }+ let published = &published.pairs;let mut pairs: Vec<(&Uuid, &Uuid)> = expected.pairs.iter().collect();pairs.sort();for (old, new) in pairs {@@ -575,6 +529,8 @@ pub fn verify_published_merges(#[cfg(test)]mod tests {+ use personhog_common::cdc_parity::PersonKey;+use super::*;fn keys(violations: Vec<ConsistencyViolation>) -> Vec<String> {@@ -668,61 +624,60 @@ mod tests {);}- // A crash between the acks and the commit republishes rows; the fold- // must land on the highest version whatever the order, identical- // duplicates must not count as anything, and two different rows at one- // version must, because a ReplacingMergeTree would keep either.+ // The fold rules live in personhog-common; this covers the gate's own+ // mapping of its findings to violations and the foreign-row count.#[test]- fn duplicate_and_replayed_rows_fold_to_the_highest_version() {- let row = |uuid: u128, version: i64, is_deleted: u8| {+ fn fold_findings_become_violations() {+ let row = |uuid: u128, version: i64| {format!(- r#"{{"person_id":"{}","team_id":5,"distinct_id":"d","version":{version},"is_deleted":{is_deleted}}}"#,+ r#"{{"person_id":"{}","team_id":5,"distinct_id":"d","version":{version},"is_deleted":0}}"#,Uuid::from_u128(uuid))};let other_team = r#"{"person_id":"00000000-0000-0000-0000-000000000002","team_id":6,"distinct_id":"d","version":9,"is_deleted":0}"#.to_string();- let payloads = [- row(1, 1, 0),- row(1, 2, 0),- row(1, 2, 0),- row(1, 1, 0),- other_team,- ];+ let payloads = [row(1, 2), row(2, 2), row(1, 3), other_team];let folded =fold_published_mappings(payloads.iter().map(|p| p.as_bytes()), 5).expect("decodes");- assert_eq!(- folded.rows,- HashMap::from([("d".to_string(), published(uuid(1), false, 2))])- );- assert!(folded.conflicts.is_empty());- assert_eq!(folded.foreign_rows, 1);-- let conflicting = [row(1, 2, 0), row(2, 2, 0), row(1, 3, 0)];- let folded =- fold_published_mappings(conflicting.iter().map(|p| p.as_bytes()), 5).expect("decodes");assert_eq!(keys(folded.conflicts),vec!["__published_mapping_conflict:d"]);- assert_eq!(folded.rows["d"].version, 3);- // The same disagreement read after a higher version is still one.- let later = [row(1, 3, 0), row(1, 2, 0), row(2, 2, 0), row(2, 2, 0)];- let folded =- fold_published_mappings(later.iter().map(|p| p.as_bytes()), 5).expect("decodes");assert_eq!(- keys(folded.conflicts),- vec!["__published_mapping_conflict:d"]+ folded.rows,+ HashMap::from([("d".to_string(), published(uuid(1), false, 3))]));- assert!(fold_published_mappings(- [br#"{"person_id":"00000000-0000-0000-0000-000000000001","team_id":"5","distinct_id":"d","version":1,"is_deleted":0}"#.as_slice()],- 5+ assert_eq!(folded.foreign_rows, 1);++ let event = |new: u128| {+ format!(+ r#"{{"team_id":5,"old_person_uuid":"{}","new_person_uuid":"{}","merged_at_ms":10,"schema_version":1}}"#,+ uuid(1),+ uuid(new)+ )+ };+ let key = format!("5:{}", uuid(1)).into_bytes();+ let records = [event(2), event(2), event(3)];+ let folded = fold_published_merges(+ records.iter().map(|p| (Some(key.as_slice()), p.as_bytes())),+ 5,)- .is_err());+ .expect("decodes");+ assert_eq!(+ folded.conflicts,+ vec![MergeConflict {+ old: PersonKey::new(5, uuid(1)),+ first: uuid(2),+ second: uuid(3),+ }]+ );+ assert_eq!(folded.pairs, HashSet::from([(uuid(1), uuid(2))]));+ assert_eq!(folded.events, 3);+ // Skipping an unreadable payload would pass a gate over a publisher that corrupts rows.assert!(fold_published_mappings([br#"{"team_id":5,"distinct_id":"d"}"#.as_slice()], 5).is_err());- assert!(fold_published_mappings([row(1, 3, 2).as_bytes()], 5).is_err());+ assert!(fold_published_merges([(None, event(2).as_bytes())], 5).is_err());}#[test]@@ -732,13 +687,30 @@ mod tests {uncertain: HashSet::from([uuid(3)]),ids: HashMap::from([(uuid(1), 1), (uuid(2), 2), (uuid(3), 3)]),};+ let read = |pairs: &[(Uuid, Uuid)], conflicts: Vec<MergeConflict>| PublishedMerges {+ pairs: pairs.iter().copied().collect(),+ conflicts,+ events: pairs.len(),+ foreign_events: 0,+ };let check = |pairs: &[(Uuid, Uuid)]| {- keys(verify_published_merges(- &expected,- &pairs.iter().copied().collect(),- ))+ keys(verify_published_merges(&expected, &read(pairs, Vec::new())))};assert!(check(&[(uuid(1), uuid(2))]).is_empty());+ // A second target is reported against the journal's id of the person.+ let conflicted = verify_published_merges(+ &expected,+ &read(+ &[(uuid(1), uuid(2))],+ vec![MergeConflict {+ old: PersonKey::new(5, uuid(1)),+ first: uuid(2),+ second: uuid(5),+ }],+ ),+ );+ assert_eq!(conflicted[0].person_id, 1);+ assert_eq!(keys(conflicted), vec!["__published_merge_conflict"]);assert_eq!(check(&[]), vec!["__published_merge_missing"]);assert_eq!(check(&[(uuid(1), uuid(2)), (uuid(4), uuid(5))]),@@ -751,35 +723,4 @@ mod tests {// An uncertain source may or may not have merged.assert!(check(&[(uuid(1), uuid(2)), (uuid(3), uuid(2))]).is_empty());}-- // The processor places a merge by its key; an event under another key- // decodes fine and still lands on the wrong worker.- #[test]- fn merge_events_fold_to_pairs_and_keep_their_key() {- let event = |schema_version: u32, merged_at_ms: i64| {- format!(- r#"{{"team_id":5,"old_person_uuid":"{}","new_person_uuid":"{}","merged_at_ms":{merged_at_ms},"schema_version":{schema_version}}}"#,- uuid(1),- uuid(2)- )- };- let key = format!("5:{}", uuid(1));- let keyed = |payload: &str| -> (Option<Vec<u8>>, Vec<u8>) {- (Some(key.clone().into_bytes()), payload.as_bytes().to_vec())- };- let twice = [keyed(&event(1, 10)), keyed(&event(1, 10))];- let folded =- fold_published_merges(twice.iter().map(|(k, p)| (k.as_deref(), p.as_slice())), 5)- .expect("decodes");- assert_eq!(folded.pairs, HashSet::from([(uuid(1), uuid(2))]));- assert_eq!(folded.events, 2);-- let bad = |key: Option<&[u8]>, payload: String| {- fold_published_merges([(key, payload.as_bytes())], 5).is_err()- };- assert!(bad(Some(key.as_bytes()), event(2, 10)));- assert!(bad(Some(key.as_bytes()), event(1, 0)));- assert!(bad(None, event(1, 10)));- assert!(bad(Some(b"5:wrong".as_slice()), event(1, 10)));- }}diff --git a/rust/personhog-test-harness/src/scenarios/gate.rs b/rust/personhog-test-harness/src/scenarios/gate.rsindex 8839b80532fb..ea483e271c67 100644--- a/rust/personhog-test-harness/src/scenarios/gate.rs+++ b/rust/personhog-test-harness/src/scenarios/gate.rs@@ -829,7 +829,7 @@ async fn run_gate(.await?;violations.extend(publish::verify_published_merges(&expected_merges,- &published_merges.pairs,+ &published_merges,));println!(" {} mappings and {} merge events for {} pairs read back from Kafka ({foreign_rows} rows and {} events of other teams skipped)",