:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(personhog): add the CDC parity monitor (#119364)
José Sequeira4f991386927a
diff --git a/.github/rust-images.yml b/.github/rust-images.ymlindex 35ffcdb6eabc..de7ce57a5d9b 100644--- a/.github/rust-images.yml+++ b/.github/rust-images.yml@@ -153,6 +153,10 @@dockerfile: ./rust/Dockerfileproject: 6tl5lmb8l8+- image: personhog-parity-monitor+ dockerfile: ./rust/Dockerfile+ project: jfz76fgvqd+- image: kafka-assignerdockerfile: ./rust/Dockerfileproject: 6kjc85bwm7diff --git a/rust/Cargo.lock b/rust/Cargo.lockindex 37689993e62b..a8a37c494b60 100644--- a/rust/Cargo.lock+++ b/rust/Cargo.lock@@ -9013,6 +9013,26 @@ dependencies = ["zstd 0.13.3",]+[[package]]+name = "personhog-parity-monitor"+version = "0.1.0"+dependencies = [+ "anyhow",+ "axum 0.7.9",+ "common-alloc",+ "common-metrics",+ "envconfig",+ "health",+ "metrics",+ "personhog-common",+ "rdkafka",+ "serde_json",+ "tokio",+ "tracing",+ "tracing-subscriber",+ "uuid",+]+[[package]]name = "personhog-proto"version = "0.1.0"diff --git a/rust/Cargo.toml b/rust/Cargo.tomlindex a30b97a47bf1..e269164773d9 100644--- a/rust/Cargo.toml+++ b/rust/Cargo.toml@@ -66,6 +66,7 @@ members = ["personhog-coordination","personhog-identity","personhog-leader",+ "personhog-parity-monitor","personhog-writer","personhog-proto","prometheus-rw-proto",diff --git a/rust/owners.yaml b/rust/owners.yamlindex 0ff7bace8a84..db0987225df1 100644--- a/rust/owners.yaml+++ b/rust/owners.yaml@@ -23,6 +23,7 @@ rules:- '/personhog-common/'- '/personhog-coordination/'- '/personhog-leader/'+ - '/personhog-parity-monitor/'- '/personhog-proto/'- '/personhog-replica/'- '/personhog-router/'diff --git a/rust/personhog-parity-monitor/Cargo.toml b/rust/personhog-parity-monitor/Cargo.tomlnew file mode 100644index 000000000000..3cf6761f479c--- /dev/null+++ b/rust/personhog-parity-monitor/Cargo.toml@@ -0,0 +1,27 @@+[package]+name = "personhog-parity-monitor"+version = "0.1.0"+edition = "2021"+publish = false++[dependencies]+personhog-common = { path = "../personhog-common" }+common-alloc = { path = "../common/alloc" }+common-metrics = { path = "../common/metrics" }+health = { path = "../common/health" }++anyhow = { workspace = true }+axum = { workspace = true }+envconfig = { workspace = true }+metrics = { workspace = true }+rdkafka = { workspace = true }+serde_json = { workspace = true }+tokio = { workspace = true }+tracing = { workspace = true }+tracing-subscriber = { workspace = true }++[dev-dependencies]+uuid = { workspace = true }++[lints]+workspace = truediff --git a/rust/personhog-parity-monitor/README.md b/rust/personhog-parity-monitor/README.mdnew file mode 100644index 000000000000..06300cc2822f--- /dev/null+++ b/rust/personhog-parity-monitor/README.md@@ -0,0 +1,76 @@+# personhog-parity-monitor++A read-only service that follows the two person CDC feeds and reports, as Prometheus metrics, whether they end in the same state.+The Node Postgres path produces `clickhouse_person_distinct_id` rows and `person_merge_events`; `personhog-publisher` produces the same two topics from the identity service's outbox.+When a lane runs in `shadow` store mode both feeds describe the same events, so every distinct id and every merge should settle identically on both.++The rules are `personhog_common::cdc_parity`, the ones the Kafka parity e2e (`nodejs/tests/service-e2e/personhog-kafka-parity.serial.test.ts`) and the test harness gate apply: end state, not messages.+Per distinct id only the highest-version row counts, and of it only the person uuid and the deleted flag.+Per merged person only the survivor at the end of its merge chain counts.++## How it settles++- Every partition of the four topics is assigned at the end of the log. There is no consumer group and nothing is committed: a group would split the partitions between replicas and break the cross-feed join, and the output is a rate over a window, so a restart has nothing to resume. Run one replica.+- Each distinct id topic tracks the newest message timestamp per partition. A feed's watermark is the slowest of its partitions that delivered within `PARITY_IDLE_PARTITION_SECONDS`. When both feeds are quiet the watermarks advance by the wall clock; when one feed is quiet next to a live one it has stalled, so its watermark holds, its `watermark_age_seconds` grows and nothing settles against its absence. The merge topics do not count: they are sparse and would hold the watermark an idle window behind. The global watermark is the slower feed's.+- An entry settles once its newest row is older than the global watermark minus `PARITY_GRACE_SECONDS`. The grace has to exceed the publisher's drain lag, or pairs settle one-sided before the personhog copy arrives.+- Entries older than the first watermark plus `PARITY_WARMUP_SECONDS` (the grace by default) are dropped unjudged and counted as `warmup_evictions_total`: the readers start at the end of the log at different moments, so a row's counterpart may have been produced before its reader started. A fixture whose readers start before any traffic sets it to `0`.+- Merges settle first and mappings `PARITY_MAPPING_LAG_MS` later, so a settling merge still sees the rows that named its persons and can tell a stub from a missing merge event.+- A merge agrees when both feeds name the same direct target; what became of that target is the target's own verdict. Different direct targets are followed to their survivors, because one feed may reach the same person through a stub the other never made.+- Settled entries are evicted from both feeds. `PARITY_MAX_ENTRIES` bounds the rows and merges retained across both feeds, history included, so a key that keeps receiving versions cannot grow past it behind one entry. Above the cap the oldest entries are evicted down to nine tenths of it without a verdict and counted as overflow. A row or merge costs about half a kilobyte.+- Distinct id rows carry null keys on both sides, so their partitions are random and only the partition counts are compared. Merge events are keyed `"{team_id}:{old_person_uuid}"`; each one is checked against `murmur2(key) % partitions` of its own topic, and a merge present on both feeds is checked for landing on the same partition number while both merge topics have the same count.++## Metrics++All names start with `personhog_parity_`. Labels carry outcomes only, never a team or an id; `/debug/mismatches` holds the last `PARITY_SAMPLE_RING_SIZE` disagreements with both sides' rows.++| Metric | Labels | Meaning |+| --- | --- | --- |+| `settled_mappings_total` | `outcome`: `agree`, `person_mismatch`, `deleted_mismatch`, `node_only`, `personhog_only` | One per settled distinct id |+| `settled_merges_total` | `outcome`: `agree`, `survivor_mismatch`, `node_only`, `personhog_only`, `node_stub`, `personhog_stub`, `loop` | One per settled merged person. `*_stub`: the merging feed created the person inside the window and the other feed never named it, its own path to the result. `*_only`: the other feed named the person, or the person predates the window, and no merge for it arrived |+| `merge_partition_total` | `outcome`: `same`, `different` | Merges present on both feeds, by partition number; only counted while both merge topics have the same partition count |+| `merge_placement_total` | `feed`, `outcome`: `expected`, `unexpected` | Arrival partition against the key's murmur2 partition |+| `version_ties_total`, `merge_conflicts_total` | `feed` | A feed contradicting itself |+| `messages_total`, `decode_failures_total`, `read_errors_total`, `filtered_total` | `feed`, `topic` | Reader counters |+| `topic_partitions` | `feed`, `topic` | Partition counts from metadata, read once at startup |+| `topic_pair_partitions_equal` | `topic` | Whether the two feeds' topics of that kind have the same count |+| `pending_entries` | `kind` | Entries waiting to settle |+| `watermark_age_seconds` | `feed` | Wall clock minus the feed's watermark; a stalled feed shows here |+| `overflow_evictions_total` | | Entries dropped by the cap, so the outcome counters are degraded while this moves |+| `warmup_evictions_total` | | Entries dropped during the first grace after start |++## Configuration++| Variable | Default | Meaning |+| --- | --- | --- |+| `NODE_KAFKA_HOSTS`, `NODE_KAFKA_TLS` | `localhost:9092`, `false` | The Node feed's cluster |+| `NODE_DISTINCT_ID_TOPIC`, `NODE_MERGE_EVENTS_TOPIC` | `clickhouse_person_distinct_id`, `person_merge_events` | The Node feed's topics |+| `PERSONHOG_KAFKA_HOSTS`, `PERSONHOG_KAFKA_TLS` | `localhost:9092`, `false` | The personhog feed's cluster |+| `PERSONHOG_DISTINCT_ID_TOPIC`, `PERSONHOG_MERGE_EVENTS_TOPIC` | required | The publisher's topics |+| `PARITY_GRACE_SECONDS` | `60` | How long an entry waits for the other feed |+| `PARITY_IDLE_PARTITION_SECONDS` | `30` | When a quiet partition stops holding the watermark |+| `PARITY_MAPPING_LAG_MS` | `2000` | How much longer than merges the mappings wait |+| `PARITY_WARMUP_SECONDS` | the grace | How long after the first watermark entries are dropped unjudged; `0` for a fixture whose readers start before any traffic |+| `PARITY_MAX_ENTRIES` | `1000000` | The cap on retained rows and merges across both feeds, about half a gigabyte |+| `PARITY_TEAMS` | empty, every team | Comma separated team ids; set it to the merge events allowlist when that is a subset |+| `PARITY_SAMPLE_RING_SIZE` | `500` | Disagreements kept for `/debug/mismatches` |+| `PARITY_SWEEP_INTERVAL_MS` | `1000` | How often entries are settled |+| `DEBUG_API_SECRET` | unset, endpoint off | Required in the `x-debug-api-secret` header to read `/debug/mismatches`, which carries distinct ids |+| `BIND_HOST`, `BIND_PORT` | `0.0.0.0`, `3301` | `/metrics`, `/_readiness`, `/_liveness`, `/debug/mismatches` |++## Running it locally++The harness `stack --publish` brings up identity and the publishers with per-run topics, and a Node ingestion API worker in `PERSONS_STORE_MODE=shadow` fed by the Rust ingestion consumer writes the Node side; the Kafka parity e2e shows the worker's environment.++```bash+target/debug/personhog-test-harness stack --publish --ready-file /tmp/personhog-stack.json++NODE_DISTINCT_ID_TOPIC=<the worker's INGESTION_OUTPUT_PERSON_DISTINCT_IDS_TOPIC> \+NODE_MERGE_EVENTS_TOPIC=<the worker's INGESTION_OUTPUT_PERSON_MERGE_EVENTS_TOPIC> \+PERSONHOG_DISTINCT_ID_TOPIC=$(jq -r .distinct_ids_topic /tmp/personhog-stack.json) \+PERSONHOG_MERGE_EVENTS_TOPIC=$(jq -r .merge_events_topic /tmp/personhog-stack.json) \+PARITY_GRACE_SECONDS=10 DEBUG_API_SECRET=local \+target/debug/personhog-parity-monitor++curl -s localhost:3301/metrics | grep personhog_parity_settled+curl -s -H 'x-debug-api-secret: local' localhost:3301/debug/mismatches+```diff --git a/rust/personhog-parity-monitor/src/config.rs b/rust/personhog-parity-monitor/src/config.rsnew file mode 100644index 000000000000..2bc43c98faff--- /dev/null+++ b/rust/personhog-parity-monitor/src/config.rs@@ -0,0 +1,185 @@+use std::collections::HashSet;+use std::time::Duration;++use envconfig::Envconfig;+use personhog_common::cdc_parity::Side;++#[derive(Envconfig, Clone, Debug)]+pub struct Config {+ #[envconfig(default = "localhost:9092")]+ pub node_kafka_hosts: String,+ #[envconfig(default = "false")]+ pub node_kafka_tls: bool,+ #[envconfig(default = "clickhouse_person_distinct_id")]+ pub node_distinct_id_topic: String,+ #[envconfig(default = "person_merge_events")]+ pub node_merge_events_topic: String,++ #[envconfig(default = "localhost:9092")]+ pub personhog_kafka_hosts: String,+ #[envconfig(default = "false")]+ pub personhog_kafka_tls: bool,+ pub personhog_distinct_id_topic: String,+ pub personhog_merge_events_topic: String,++ /// Has to exceed the publisher's drain lag, or every pair settles+ /// one-sided before the personhog copy arrives.+ #[envconfig(default = "60")]+ pub parity_grace_seconds: u64,+ /// A quiet partition stops holding the watermark back after this long.+ #[envconfig(default = "30")]+ pub parity_idle_partition_seconds: u64,+ /// Covers the timestamp skew between a merge event and the rows it moved.+ #[envconfig(default = "2000")]+ pub parity_mapping_lag_ms: u64,+ /// Defaults to the grace. Zero turns the warmup off, for a fixture+ /// whose readers start before any traffic.+ pub parity_warmup_seconds: Option<u64>,+ /// Retained rows and merges across both feeds; each costs about half a+ /// kilobyte.+ #[envconfig(default = "1000000")]+ pub parity_max_entries: usize,+ /// Empty means every team. Set it to the merge events allowlist when+ /// that is a subset, or every other team's merges settle one-sided.+ #[envconfig(default = "")]+ pub parity_teams: String,+ #[envconfig(default = "500")]+ pub parity_sample_ring_size: usize,+ #[envconfig(default = "1000")]+ pub parity_sweep_interval_ms: u64,++ /// Unset leaves `/debug/mismatches` off; its samples carry distinct ids.+ pub debug_api_secret: Option<String>,++ #[envconfig(default = "0.0.0.0")]+ pub bind_host: String,+ #[envconfig(default = "3301")]+ pub bind_port: u16,+}++#[derive(Clone, Debug)]+pub struct FeedConfig {+ pub side: Side,+ pub kafka_hosts: String,+ pub kafka_tls: bool,+ pub distinct_id_topic: String,+ pub merge_events_topic: String,+}++impl Config {+ pub fn feeds(&self) -> [FeedConfig; 2] {+ [+ FeedConfig {+ side: Side::Node,+ kafka_hosts: self.node_kafka_hosts.clone(),+ kafka_tls: self.node_kafka_tls,+ distinct_id_topic: self.node_distinct_id_topic.clone(),+ merge_events_topic: self.node_merge_events_topic.clone(),+ },+ FeedConfig {+ side: Side::Personhog,+ kafka_hosts: self.personhog_kafka_hosts.clone(),+ kafka_tls: self.personhog_kafka_tls,+ distinct_id_topic: self.personhog_distinct_id_topic.clone(),+ merge_events_topic: self.personhog_merge_events_topic.clone(),+ },+ ]+ }++ pub fn teams(&self) -> Result<Option<HashSet<i64>>, String> {+ let teams = self+ .parity_teams+ .split(',')+ .map(str::trim)+ .filter(|team| !team.is_empty())+ .map(|team| {+ team.parse::<i64>()+ .map_err(|_| format!("PARITY_TEAMS holds {team:?}, expected integer team ids"))+ })+ .collect::<Result<HashSet<i64>, String>>()?;+ Ok((!teams.is_empty()).then_some(teams))+ }++ pub fn grace(&self) -> Duration {+ Duration::from_secs(self.parity_grace_seconds)+ }++ pub fn idle_partition(&self) -> Duration {+ Duration::from_secs(self.parity_idle_partition_seconds)+ }++ pub fn mapping_lag(&self) -> Duration {+ Duration::from_millis(self.parity_mapping_lag_ms)+ }++ pub fn warmup(&self) -> Duration {+ Duration::from_secs(+ self.parity_warmup_seconds+ .unwrap_or(self.parity_grace_seconds),+ )+ }++ pub fn sweep_interval(&self) -> Duration {+ Duration::from_millis(self.parity_sweep_interval_ms.max(100))+ }++ pub fn bind(&self) -> String {+ format!("{}:{}", self.bind_host, self.bind_port)+ }++ /// An empty value is unset: an empty header must not open the endpoint.+ pub fn debug_api_secret(&self) -> Option<&str> {+ self.debug_api_secret+ .as_deref()+ .filter(|secret| !secret.is_empty())+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ fn config(vars: &[(&str, &str)]) -> Config {+ let mut env = std::collections::HashMap::from([+ ("PERSONHOG_DISTINCT_ID_TOPIC".to_string(), "a".to_string()),+ ("PERSONHOG_MERGE_EVENTS_TOPIC".to_string(), "b".to_string()),+ ]);+ env.extend(vars.iter().map(|(k, v)| (k.to_string(), v.to_string())));+ Config::init_from_hashmap(&env).expect("config")+ }++ #[test]+ fn team_filter_parses_or_refuses() {+ assert_eq!(config(&[("PARITY_TEAMS", "")]).teams(), Ok(None));+ assert_eq!(+ config(&[("PARITY_TEAMS", " 2, 15 ")]).teams(),+ Ok(Some(HashSet::from([2, 15])))+ );+ assert!(config(&[("PARITY_TEAMS", "2,all")]).teams().is_err());+ }++ #[test]+ fn the_warmup_follows_the_grace_unless_set() {+ assert_eq!(+ config(&[("PARITY_GRACE_SECONDS", "7")]).warmup(),+ Duration::from_secs(7)+ );+ assert_eq!(+ config(&[+ ("PARITY_GRACE_SECONDS", "7"),+ ("PARITY_WARMUP_SECONDS", "0")+ ])+ .warmup(),+ Duration::ZERO+ );+ }++ #[test]+ fn an_empty_debug_secret_leaves_the_endpoint_off() {+ assert_eq!(config(&[("DEBUG_API_SECRET", "")]).debug_api_secret(), None);+ assert_eq!(+ config(&[("DEBUG_API_SECRET", "s")]).debug_api_secret(),+ Some("s")+ );+ }+}diff --git a/rust/personhog-parity-monitor/src/feed.rs b/rust/personhog-parity-monitor/src/feed.rsnew file mode 100644index 000000000000..4bddf1005a41--- /dev/null+++ b/rust/personhog-parity-monitor/src/feed.rs@@ -0,0 +1,178 @@+//! One reader per topic, every partition assigned at the end of the log,+//! nothing committed: a consumer group would split the partitions between+//! replicas and break the cross-feed join, and a restart resumes nothing.++use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};++use anyhow::{bail, Context, Result};+use health::HealthHandle;+use personhog_common::cdc_parity::{DecodeError, DistinctIdRow, MergeEvent};+use rdkafka::consumer::{Consumer, StreamConsumer};+use rdkafka::error::KafkaError;+use rdkafka::message::BorrowedMessage;+use rdkafka::{ClientConfig, Message, Offset, TopicPartitionList};+use tokio::sync::mpsc;+use tracing::{debug, error, warn};++use crate::config::FeedConfig;+use crate::join::{Event, Kind, Record, Topic};++const METADATA_TIMEOUT: Duration = Duration::from_secs(10);+/// Bounds a `recv` on a quiet topic so the reader still reports liveness.+const RECV_TIMEOUT: Duration = Duration::from_secs(5);+const HEALTH_REPORT_INTERVAL: Duration = Duration::from_secs(1);+const ERROR_BACKOFF: Duration = Duration::from_secs(1);+/// librdkafka's default prefetch is 64 MiB per consumer.+const PREFETCH_KBYTES: &str = "16384";++pub struct TopicReader {+ topic: Topic,+ name: String,+ consumer: StreamConsumer,+}++impl TopicReader {+ pub fn connect(feed: &FeedConfig, kind: Kind) -> Result<(Self, u32)> {+ let name = match kind {+ Kind::DistinctIds => feed.distinct_id_topic.clone(),+ Kind::MergeEvents => feed.merge_events_topic.clone(),+ };+ let topic = Topic {+ side: feed.side,+ kind,+ };+ let mut client = ClientConfig::new();+ client+ .set("bootstrap.servers", &feed.kafka_hosts)+ .set("group.id", "personhog-parity-monitor")+ .set("enable.auto.commit", "false")+ .set("enable.auto.offset.store", "false")+ .set("auto.offset.reset", "latest")+ .set("queued.max.messages.kbytes", PREFETCH_KBYTES)+ .set(+ "client.id",+ format!("personhog-parity-monitor-{}", feed.side.label()),+ );+ if feed.kafka_tls {+ client+ .set("security.protocol", "ssl")+ .set("enable.ssl.certificate.verification", "false");+ }+ let consumer: StreamConsumer = client+ .create()+ .with_context(|| format!("creating the {} reader for {name}", feed.side.label()))?;+ let metadata = consumer+ .fetch_metadata(Some(&name), METADATA_TIMEOUT)+ .with_context(|| format!("fetching metadata of {name}"))?;+ let partitions: Vec<i32> = metadata+ .topics()+ .iter()+ .find(|entry| entry.name() == name)+ .map(|entry| entry.partitions().iter().map(|p| p.id()).collect())+ .unwrap_or_default();+ if partitions.is_empty() {+ bail!("topic {name} has no partitions on {}", feed.kafka_hosts);+ }+ let mut assignment = TopicPartitionList::new();+ for partition in &partitions {+ // The end offset is read here, before anyone learns the reader is+ // up; `Offset::End` would resolve later, in the fetcher, and a row+ // produced in between would be skipped.+ let (_, end) = consumer+ .fetch_watermarks(&name, *partition, METADATA_TIMEOUT)+ .with_context(|| format!("fetching the end offset of {name}/{partition}"))?;+ assignment.add_partition_offset(&name, *partition, Offset::Offset(end))?;+ }+ consumer+ .assign(&assignment)+ .with_context(|| format!("assigning {name}"))?;+ Ok((+ Self {+ topic,+ name,+ consumer,+ },+ partitions.len() as u32,+ ))+ }++ pub fn topic(&self) -> Topic {+ self.topic+ }++ /// The first decode failure per topic is logged in full; the counter is the signal.+ pub async fn run(self, events: mpsc::Sender<Event>, health: HealthHandle) {+ let feed = self.topic.side.label();+ let kind = self.topic.kind.label();+ let messages =+ metrics::counter!("personhog_parity_messages_total", "feed" => feed, "topic" => kind);+ let read_errors = metrics::counter!("personhog_parity_read_errors_total", "feed" => feed, "topic" => kind);+ let decode_failures = metrics::counter!(+ "personhog_parity_decode_failures_total",+ "feed" => feed,+ "topic" => kind+ );+ let mut last_health_report = Instant::now() - HEALTH_REPORT_INTERVAL;+ let mut decode_failure_logged = false;+ loop {+ if last_health_report.elapsed() >= HEALTH_REPORT_INTERVAL {+ health.report_healthy().await;+ last_health_report = Instant::now();+ }+ let message = match tokio::time::timeout(RECV_TIMEOUT, self.consumer.recv()).await {+ Err(_) => continue,+ // The consumer cannot recover; the reader stops reporting, so+ // liveness fails and the pod restarts into a fresh window.+ Ok(Err(KafkaError::MessageConsumptionFatal(code))) => {+ error!(topic = %self.name, %code, "the consumer failed for good; stopping the reader");+ read_errors.increment(1);+ return;+ }+ Ok(Err(error)) => {+ warn!(topic = %self.name, %error, "reading the topic failed");+ read_errors.increment(1);+ tokio::time::sleep(ERROR_BACKOFF).await;+ continue;+ }+ Ok(Ok(message)) => message,+ };+ messages.increment(1);+ let record = match self.decode(&message) {+ Ok(record) => record,+ Err(error) => {+ decode_failures.increment(1);+ if decode_failure_logged {+ debug!(topic = %self.name, %error, "undecodable message");+ } else {+ warn!(topic = %self.name, %error, "undecodable message; further ones are counted only");+ decode_failure_logged = true;+ }+ continue;+ }+ };+ let event = Event::Arrival {+ topic: self.topic,+ partition: message.partition(),+ timestamp_ms: message.timestamp().to_millis().unwrap_or_else(now_ms),+ record,+ };+ if events.send(event).await.is_err() {+ return;+ }+ }+ }++ fn decode(&self, message: &BorrowedMessage<'_>) -> Result<Record, DecodeError> {+ let payload = message.payload().unwrap_or_default();+ match self.topic.kind {+ Kind::DistinctIds => DistinctIdRow::decode(payload).map(Record::Mapping),+ Kind::MergeEvents => MergeEvent::decode(message.key(), payload).map(Record::Merge),+ }+ }+}++pub fn now_ms() -> i64 {+ SystemTime::now()+ .duration_since(UNIX_EPOCH)+ .map_or(0, |elapsed| elapsed.as_millis() as i64)+}diff --git a/rust/personhog-parity-monitor/src/join.rs b/rust/personhog-parity-monitor/src/join.rsnew file mode 100644index 000000000000..43f01c67935a--- /dev/null+++ b/rust/personhog-parity-monitor/src/join.rs@@ -0,0 +1,985 @@+//! The two feeds folded side by side and settled by watermark: an entry+//! waits until the slower feed has delivered past it by the grace, so a+//! lagging feed never reads as one-sided. Pure; the pipeline emits metrics.++use std::collections::{BTreeMap, HashMap, HashSet, VecDeque};+use std::sync::{Arc, Mutex};+use std::time::{Duration, Instant};++use personhog_common::cdc_parity::{+ compare_mapping, compare_survivor, DistinctIdKey, DistinctIdRow, FeedState, Mapping,+ MappingOutcome, MergeEvent, PersonKey, Side, SurvivorOutcome,+};+use personhog_common::partitioning::partition_for_key;+use serde_json::{json, Value};++#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]+pub enum Kind {+ DistinctIds,+ MergeEvents,+}++impl Kind {+ pub fn label(self) -> &'static str {+ match self {+ Kind::DistinctIds => "distinct_ids",+ Kind::MergeEvents => "merge_events",+ }+ }+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]+pub struct Topic {+ pub side: Side,+ pub kind: Kind,+}++#[derive(Debug, Clone)]+pub enum Record {+ Mapping(DistinctIdRow),+ Merge(MergeEvent),+}++#[derive(Debug, Clone)]+pub enum Event {+ /// The count places merge keys.+ Assigned { topic: Topic, partitions: u32 },+ Arrival {+ topic: Topic,+ partition: i32,+ timestamp_ms: i64,+ record: Record,+ },+}++impl Event {+ pub fn topic(&self) -> Topic {+ match self {+ Event::Assigned { topic, .. } | Event::Arrival { topic, .. } => *topic,+ }+ }+}++#[derive(Debug, Clone)]+pub struct JoinConfig {+ pub grace: Duration,+ pub idle_partition: Duration,+ pub max_entries: usize,+ pub teams: Option<HashSet<i64>>,+ /// Covers the timestamp skew between a merge event and the rows it+ /// moved, so a settling merge still finds the rows naming its persons.+ pub mapping_lag: Duration,+ /// Entries older than the first watermark plus this are dropped+ /// unjudged, because a counterpart may predate the readers. Zero+ /// turns the warmup off.+ pub warmup: Duration,+}++#[derive(Debug, Default, Clone, PartialEq, Eq)]+pub struct ObserveReport {+ pub filtered: bool,+ pub version_tie: Option<Side>,+ pub merge_conflict: Option<Side>,+ /// Whether the merge event sits on the partition its key selects.+ pub placement: Option<(Side, bool)>,+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]+pub enum PartitionAgreement {+ Same,+ Different,+}++impl PartitionAgreement {+ pub fn label(self) -> &'static str {+ match self {+ PartitionAgreement::Same => "same",+ PartitionAgreement::Different => "different",+ }+ }+}++#[derive(Debug, Default, Clone, PartialEq, Eq)]+pub struct SweepReport {+ pub mapping_outcomes: BTreeMap<MappingOutcome, u64>,+ pub merge_outcomes: BTreeMap<SurvivorOutcome, u64>,+ pub merge_partitions: BTreeMap<PartitionAgreement, u64>,+ pub overflow_evictions: u64,+ pub warmup_evictions: u64,+ pub pending_mappings: usize,+ pub pending_merges: usize,+ /// The global watermark the sweep settled against.+ pub watermark_ms: Option<i64>,+ /// Each feed's own watermark, indexed by [`side_index`].+ pub feed_watermarks_ms: [Option<i64>; 2],+}++pub fn side_index(side: Side) -> usize {+ match side {+ Side::Node => 0,+ Side::Personhog => 1,+ }+}++/// Metrics carry only outcome labels; this is where an operator sees an example.+#[derive(Clone)]+pub struct SampleRing {+ inner: Arc<Mutex<VecDeque<Value>>>,+ capacity: usize,+}++impl SampleRing {+ pub fn new(capacity: usize) -> Self {+ Self {+ inner: Arc::new(Mutex::new(VecDeque::with_capacity(capacity))),+ capacity,+ }+ }++ fn push(&self, sample: Value) {+ if self.capacity == 0 {+ return;+ }+ let mut ring = self+ .inner+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ if ring.len() == self.capacity {+ ring.pop_front();+ }+ ring.push_back(sample);+ }++ pub fn snapshot(&self) -> Vec<Value> {+ self.inner+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner())+ .iter()+ .cloned()+ .collect()+ }+}++struct PartitionProgress {+ max_timestamp_ms: i64,+ last_seen: Instant,+}++#[derive(Default)]+struct TopicProgress {+ partitions: HashMap<i32, PartitionProgress>,+ partition_count: Option<u32>,+}++impl TopicProgress {+ /// The slowest of the partitions that delivered within the idle window.+ fn active_min(&self, now: Instant, idle: Duration) -> Option<i64> {+ self.partitions+ .values()+ .filter(|progress| now.duration_since(progress.last_seen) <= idle)+ .map(|progress| progress.max_timestamp_ms)+ .min()+ }++ fn newest(&self) -> Option<i64> {+ self.partitions+ .values()+ .map(|progress| progress.max_timestamp_ms)+ .max()+ }++ /// The newest message, moved forward by the time since it arrived.+ fn newest_advanced(&self, now: Instant) -> Option<i64> {+ self.partitions+ .values()+ .map(|progress| {+ progress.max_timestamp_ms+ + now.duration_since(progress.last_seen).as_millis() as i64+ })+ .max()+ }+}++struct MappingEntry {+ last_update_ms: i64,+}++struct MergeEntry {+ last_update_ms: i64,+ partitions: [Option<i32>; 2],+ /// Each feed's recorded target, released when the entry is evicted.+ targets: [Option<PersonKey>; 2],+ /// Judged already; kept while a pending merge still points at it.+ settled: bool,+}++pub struct Join {+ config: JoinConfig,+ node: FeedState,+ personhog: FeedState,+ mappings: HashMap<DistinctIdKey, MappingEntry>,+ merges: HashMap<PersonKey, MergeEntry>,+ /// How many merge entries point at each person on either feed.+ target_refs: HashMap<PersonKey, usize>,+ progress: HashMap<Topic, TopicProgress>,+ warmup_until_ms: Option<i64>,+ samples: SampleRing,+}++impl Join {+ pub fn new(config: JoinConfig, samples: SampleRing) -> Self {+ Self {+ config,+ node: FeedState::new(),+ personhog: FeedState::new(),+ mappings: HashMap::new(),+ merges: HashMap::new(),+ target_refs: HashMap::new(),+ progress: HashMap::new(),+ warmup_until_ms: None,+ samples,+ }+ }++ fn feed(&self, side: Side) -> &FeedState {+ match side {+ Side::Node => &self.node,+ Side::Personhog => &self.personhog,+ }+ }++ fn feed_mut(&mut self, side: Side) -> &mut FeedState {+ match side {+ Side::Node => &mut self.node,+ Side::Personhog => &mut self.personhog,+ }+ }++ pub fn observe(&mut self, event: Event, now: Instant) -> ObserveReport {+ let mut report = ObserveReport::default();+ let (topic, partition, timestamp_ms, record) = match event {+ Event::Assigned { topic, partitions } => {+ self.progress.entry(topic).or_default().partition_count = Some(partitions);+ return report;+ }+ Event::Arrival {+ topic,+ partition,+ timestamp_ms,+ record,+ } => (topic, partition, timestamp_ms, record),+ };++ let progress = self.progress.entry(topic).or_default();+ let partition_count = progress.partition_count;+ let seen = progress+ .partitions+ .entry(partition)+ .or_insert(PartitionProgress {+ max_timestamp_ms: timestamp_ms,+ last_seen: now,+ });+ seen.max_timestamp_ms = seen.max_timestamp_ms.max(timestamp_ms);+ seen.last_seen = now;++ let team_id = match &record {+ Record::Mapping(row) => row.team_id,+ Record::Merge(event) => event.team_id,+ };+ if let Some(teams) = &self.config.teams {+ if !teams.contains(&team_id) {+ report.filtered = true;+ return report;+ }+ }++ match record {+ Record::Mapping(row) => {+ if self.feed_mut(topic.side).observe_mapping(&row).is_some() {+ report.version_tie = Some(topic.side);+ }+ let key = DistinctIdKey::new(row.team_id, row.distinct_id);+ let entry = self.mappings.entry(key).or_insert(MappingEntry {+ last_update_ms: timestamp_ms,+ });+ entry.last_update_ms = entry.last_update_ms.max(timestamp_ms);+ }+ Record::Merge(event) => {+ if let Some(count) = partition_count {+ let expected = partition_for_key(event.key().as_bytes(), count) as i32;+ report.placement = Some((topic.side, expected == partition));+ }+ let side = side_index(topic.side);+ if self.feed_mut(topic.side).observe_merge(&event).is_some() {+ report.merge_conflict = Some(topic.side);+ }+ let old = PersonKey::new(event.team_id, event.old_person_uuid);+ let target = self+ .feed(topic.side)+ .merged_into(&old)+ .map(|new| PersonKey::new(event.team_id, new));+ let entry = self.merges.entry(old).or_insert(MergeEntry {+ last_update_ms: timestamp_ms,+ partitions: [None, None],+ targets: [None, None],+ settled: false,+ });+ entry.last_update_ms = entry.last_update_ms.max(timestamp_ms);+ entry.partitions[side] = Some(partition);+ if let (None, Some(target)) = (entry.targets[side], target) {+ entry.targets[side] = Some(target);+ *self.target_refs.entry(target).or_default() += 1;+ }+ }+ }+ report+ }++ /// Merges settle first and mappings a little later, so a merge's+ /// verdict can still ask which persons each feed named. Entries the+ /// cap evicts are not counted: they never got their grace.+ pub fn settle(&mut self, now: Instant) -> SweepReport {+ let mut report = SweepReport {+ feed_watermarks_ms: self.feed_watermarks(now),+ ..Default::default()+ };+ if let [Some(node), Some(personhog)] = report.feed_watermarks_ms {+ let watermark = node.min(personhog);+ report.watermark_ms = Some(watermark);+ let warmup_until = if self.config.warmup.is_zero() {+ i64::MIN+ } else {+ *self+ .warmup_until_ms+ .get_or_insert(watermark + self.config.warmup.as_millis() as i64)+ };+ let merge_cutoff = watermark - self.config.grace.as_millis() as i64;+ let mapping_cutoff = merge_cutoff - self.config.mapping_lag.as_millis() as i64;+ let due: Vec<(PersonKey, i64)> = self+ .merges+ .iter()+ .filter(|(_, entry)| !entry.settled && entry.last_update_ms <= merge_cutoff)+ .map(|(old, entry)| (*old, entry.last_update_ms))+ .collect();+ for (old, last_update_ms) in due {+ if last_update_ms < warmup_until {+ self.evict_merge(old);+ report.warmup_evictions += 1;+ } else {+ self.settle_merge(old, &mut report);+ self.release_merge(old);+ }+ }+ let due: Vec<(DistinctIdKey, i64)> = self+ .mappings+ .iter()+ .filter(|(_, entry)| entry.last_update_ms <= mapping_cutoff)+ .map(|(key, entry)| (key.clone(), entry.last_update_ms))+ .collect();+ for (key, last_update_ms) in due {+ if last_update_ms < warmup_until {+ self.evict_mapping(&key);+ report.warmup_evictions += 1;+ } else {+ self.settle_mapping(&key, &mut report);+ }+ }+ }+ self.enforce_cap(&mut report);+ report.pending_mappings = self.mappings.len();+ report.pending_merges = self.merges.values().filter(|entry| !entry.settled).count();+ report+ }++ /// Only the distinct id topic counts: the sparse merge topic would hold+ /// the watermark an idle window behind. A quiet feed advances by the+ /// wall clock only while the other feed is quiet too; next to a live+ /// feed it has stalled, so its watermark holds and nothing settles.+ fn feed_watermarks(&self, now: Instant) -> [Option<i64>; 2] {+ let idle = self.config.idle_partition;+ let progress = [Side::Node, Side::Personhog].map(|side| {+ self.progress.get(&Topic {+ side,+ kind: Kind::DistinctIds,+ })+ });+ let active = progress.map(|topic| topic.and_then(|topic| topic.active_min(now, idle)));+ let both_quiet = active.iter().all(Option::is_none);+ let mut watermarks = [None, None];+ for index in 0..2 {+ watermarks[index] = match (active[index], progress[index]) {+ (Some(active), _) => Some(active),+ (None, Some(topic)) if both_quiet => topic.newest_advanced(now),+ (None, Some(topic)) => topic.newest(),+ (None, None) => None,+ };+ }+ watermarks+ }++ fn settle_mapping(&mut self, key: &DistinctIdKey, report: &mut SweepReport) {+ let (node, personhog) = (self.node.mapping(key), self.personhog.mapping(key));+ let outcome = compare_mapping(node, personhog);+ if outcome != MappingOutcome::Agree {+ self.samples.push(json!({+ "kind": "mapping",+ "team_id": key.team_id,+ "distinct_id": key.distinct_id,+ "outcome": outcome.label(),+ "node": node.map(mapping_json),+ "personhog": personhog.map(mapping_json),+ }));+ }+ *report.mapping_outcomes.entry(outcome).or_default() += 1;+ self.evict_mapping(key);+ }++ fn settle_merge(&mut self, old: PersonKey, report: &mut SweepReport) {+ let outcome = compare_survivor(&self.node, &self.personhog, old);+ let partitions = self+ .merges+ .get(&old)+ .map_or([None, None], |entry| entry.partitions);+ if let [Some(node), Some(personhog)] = partitions {+ if self.merge_topics_comparable() {+ let agreement = if node == personhog {+ PartitionAgreement::Same+ } else {+ PartitionAgreement::Different+ };+ *report.merge_partitions.entry(agreement).or_default() += 1;+ }+ }+ if outcome != SurvivorOutcome::Agree {+ self.samples.push(json!({+ "kind": "merge",+ "team_id": old.team_id,+ "old_person_uuid": old.person_uuid.to_string(),+ "outcome": outcome.label(),+ "node": self.node.merged_into(&old).map(|new| new.to_string()),+ "personhog": self.personhog.merged_into(&old).map(|new| new.to_string()),+ "partitions": partitions,+ }));+ }+ *report.merge_outcomes.entry(outcome).or_default() += 1;+ if let Some(entry) = self.merges.get_mut(&old) {+ entry.settled = true;+ }+ }++ /// A settled merge keeps its edge while a pending merge still points at+ /// it: that merge's verdict follows the chain through it, and the sweep+ /// order must not decide the verdict.+ fn release_merge(&mut self, old: PersonKey) {+ if self.target_refs.get(&old).copied().unwrap_or(0) == 0 {+ self.evict_merge(old);+ }+ }++ /// Partition numbers only compare across topics with the same count.+ fn merge_topics_comparable(&self) -> bool {+ let counts = [Side::Node, Side::Personhog].map(|side| {+ self.progress+ .get(&Topic {+ side,+ kind: Kind::MergeEvents,+ })+ .and_then(|topic| topic.partition_count)+ });+ matches!(counts, [Some(node), Some(personhog)] if node == personhog)+ }++ fn evict_mapping(&mut self, key: &DistinctIdKey) {+ self.node.remove_mapping(key);+ self.personhog.remove_mapping(key);+ self.mappings.remove(key);+ }++ /// Releases the evicted entry's targets; a settled target nothing points+ /// at any more goes with it.+ fn evict_merge(&mut self, old: PersonKey) {+ let mut stack = vec![old];+ while let Some(old) = stack.pop() {+ let Some(entry) = self.merges.remove(&old) else {+ continue;+ };+ self.node.remove_merge(&old);+ self.personhog.remove_merge(&old);+ for target in entry.targets.into_iter().flatten() {+ let Some(refs) = self.target_refs.get_mut(&target) else {+ continue;+ };+ *refs -= 1;+ if *refs == 0 {+ self.target_refs.remove(&target);+ if self.merges.get(&target).is_some_and(|entry| entry.settled) {+ stack.push(target);+ }+ }+ }+ }+ }++ /// Bounds retained rows and merges, not keys: a key that keeps receiving+ /// versions grows behind one entry. Trims to nine tenths of the cap, one+ /// pass per tenth rather than one per sweep, by key age; every key holds+ /// at least one row, so evicting as many keys as the excess suffices.+ fn enforce_cap(&mut self, report: &mut SweepReport) {+ let total = self.node.row_count() + self.personhog.row_count() + self.merges.len();+ if total <= self.config.max_entries {+ return;+ }+ let target = self.config.max_entries - self.config.max_entries / 10;+ let keys = self.mappings.len() + self.merges.len();+ let excess = (total - target).min(keys);+ let mut ages: Vec<i64> = self+ .mappings+ .values()+ .map(|entry| entry.last_update_ms)+ .chain(self.merges.values().map(|entry| entry.last_update_ms))+ .collect();+ let (_, threshold, _) = ages.select_nth_unstable(excess - 1);+ let threshold = *threshold;+ let old_merges: Vec<PersonKey> = self+ .merges+ .iter()+ .filter(|(_, entry)| entry.last_update_ms <= threshold)+ .map(|(old, _)| *old)+ .collect();+ for old in old_merges {+ self.evict_merge(old);+ report.overflow_evictions += 1;+ }+ let old_mappings: Vec<DistinctIdKey> = self+ .mappings+ .iter()+ .filter(|(_, entry)| entry.last_update_ms <= threshold)+ .map(|(key, _)| key.clone())+ .collect();+ for key in old_mappings {+ self.evict_mapping(&key);+ report.overflow_evictions += 1;+ }+ }+}++fn mapping_json(mapping: &Mapping) -> Value {+ json!({+ "person_uuid": mapping.person_uuid.to_string(),+ "is_deleted": mapping.is_deleted,+ "version": mapping.version,+ })+}++#[cfg(test)]+mod tests {+ use super::*;+ use uuid::Uuid;++ const TEAM: i64 = 5;+ const GRACE: Duration = Duration::from_secs(60);+ const IDLE: Duration = Duration::from_secs(30);+ const MAPPING_LAG: Duration = Duration::from_secs(2);++ fn uuid(n: u128) -> Uuid {+ Uuid::from_u128(n)+ }++ fn join_with(max_entries: usize, teams: Option<HashSet<i64>>, warmup: Duration) -> Join {+ Join::new(+ JoinConfig {+ grace: GRACE,+ idle_partition: IDLE,+ max_entries,+ teams,+ mapping_lag: MAPPING_LAG,+ warmup,+ },+ SampleRing::new(10),+ )+ }++ fn join(max_entries: usize, teams: Option<HashSet<i64>>) -> Join {+ join_with(max_entries, teams, Duration::ZERO)+ }++ fn topic(side: Side, kind: Kind) -> Topic {+ Topic { side, kind }+ }++ fn row_of_team(+ team_id: i64,+ side: Side,+ partition: i32,+ timestamp_ms: i64,+ distinct_id: &str,+ person: u128,+ version: i64,+ ) -> Event {+ Event::Arrival {+ topic: topic(side, Kind::DistinctIds),+ partition,+ timestamp_ms,+ record: Record::Mapping(DistinctIdRow {+ team_id,+ distinct_id: distinct_id.to_string(),+ person_uuid: uuid(person),+ version,+ is_deleted: false,+ }),+ }+ }++ fn row(+ side: Side,+ partition: i32,+ timestamp_ms: i64,+ distinct_id: &str,+ person: u128,+ ) -> Event {+ row_of_team(TEAM, side, partition, timestamp_ms, distinct_id, person, 0)+ }++ fn assigned(join: &mut Join, kind: Kind, counts: [u32; 2], at: Instant) {+ for (side, partitions) in [Side::Node, Side::Personhog].into_iter().zip(counts) {+ join.observe(+ Event::Assigned {+ topic: topic(side, kind),+ partitions,+ },+ at,+ );+ }+ }++ fn merge(side: Side, partition: i32, timestamp_ms: i64, old: u128, new: u128) -> Event {+ Event::Arrival {+ topic: topic(side, Kind::MergeEvents),+ partition,+ timestamp_ms,+ record: Record::Merge(MergeEvent {+ team_id: TEAM,+ old_person_uuid: uuid(old),+ new_person_uuid: uuid(new),+ merged_at_ms: timestamp_ms,+ }),+ }+ }++ fn mapping_outcomes(report: &SweepReport) -> Vec<(MappingOutcome, u64)> {+ report+ .mapping_outcomes+ .iter()+ .map(|(k, v)| (*k, *v))+ .collect()+ }++ fn both(+ join: &mut Join,+ partition: i32,+ timestamp_ms: i64,+ distinct_id: &str,+ person: u128,+ at: Instant,+ ) {+ for side in [Side::Node, Side::Personhog] {+ join.observe(row(side, partition, timestamp_ms, distinct_id, person), at);+ }+ }++ // Nothing settles until the slower feed passes the grace; the merge+ // topics never count towards the watermark.+ #[test]+ fn entries_settle_behind_the_slower_feed_after_the_grace() {+ let t0 = Instant::now();+ let mut join = join(usize::MAX, None);+ assigned(&mut join, Kind::MergeEvents, [8, 8], t0);+ join.observe(row(Side::Node, 0, 1_000, "d", 1), t0);+ join.observe(row(Side::Node, 0, 2_000, "only", 2), t0);+ join.observe(row(Side::Node, 0, 70_000, "ahead", 3), t0);+ join.observe(merge(Side::Node, 3, 1_000, 7, 8), t0);+ join.observe(row(Side::Personhog, 0, 1_500, "d", 1), t0);+ join.observe(merge(Side::Personhog, 4, 1_200, 7, 8), t0);++ let report = join.settle(t0);+ assert_eq!(report.watermark_ms, Some(1_500));+ assert_eq!(report.mapping_outcomes, BTreeMap::new());+ assert_eq!(report.pending_mappings, 3);+ assert_eq!(report.pending_merges, 1);++ join.observe(row(Side::Personhog, 0, 70_000, "ahead", 3), t0);+ let report = join.settle(t0);+ assert_eq!(report.watermark_ms, Some(70_000));+ assert_eq!(+ mapping_outcomes(&report),+ vec![(MappingOutcome::Agree, 1), (MappingOutcome::NodeOnly, 1)]+ );+ assert_eq!(+ report.merge_outcomes,+ BTreeMap::from([(SurvivorOutcome::Agree, 1)])+ );+ assert_eq!(+ report.merge_partitions,+ BTreeMap::from([(PartitionAgreement::Different, 1)])+ );+ assert_eq!(report.pending_mappings, 1);+ assert_eq!(report.pending_merges, 0);+ assert_eq!(join.node.mapping_count(), 1);+ assert_eq!(join.personhog.mapping_count(), 1);+ assert_eq!(join.node.merge_count() + join.personhog.merge_count(), 0);+ let samples = join.samples.snapshot();+ assert_eq!(samples.len(), 1);+ assert_eq!(samples[0]["distinct_id"], "only");+ assert_eq!(samples[0]["outcome"], "node_only");+ }++ // Topics with different counts place the same key on different+ // partition numbers; only the per-feed placement check applies there.+ #[test]+ fn merge_partitions_compare_only_across_equal_counts() {+ let t0 = Instant::now();+ let mut join = join(usize::MAX, None);+ assigned(&mut join, Kind::MergeEvents, [1, 4], t0);+ join.observe(merge(Side::Node, 0, 1_000, 7, 8), t0);+ join.observe(merge(Side::Personhog, 3, 1_000, 7, 8), t0);+ both(&mut join, 0, 70_000, "w", 9, t0);++ let report = join.settle(t0);+ assert_eq!(+ report.merge_outcomes,+ BTreeMap::from([(SurvivorOutcome::Agree, 1)])+ );+ assert_eq!(report.merge_partitions, BTreeMap::new());+ }++ // A lull on both feeds walks the watermark forward.+ #[test]+ fn idle_partitions_release_the_watermark() {+ let t0 = Instant::now();+ let mut join = join(usize::MAX, None);+ both(&mut join, 0, 1_000, "a", 1, t0);+ let later = t0 + Duration::from_secs(40);+ both(&mut join, 1, 70_000, "b", 2, later);++ let report = join.settle(later);+ assert_eq!(report.watermark_ms, Some(70_000));+ assert_eq!(mapping_outcomes(&report), vec![(MappingOutcome::Agree, 1)]);++ let report = join.settle(later + Duration::from_secs(62));+ assert_eq!(report.watermark_ms, Some(132_000));+ assert_eq!(mapping_outcomes(&report), vec![(MappingOutcome::Agree, 1)]);+ }++ // A feed quiet next to a live one has stalled: its watermark holds, so+ // nothing settles against its absence.+ #[test]+ fn a_stalled_feed_holds_the_watermark() {+ let t0 = Instant::now();+ let mut join = join(usize::MAX, None);+ both(&mut join, 0, 1_000, "a", 1, t0);+ let stalled = t0 + Duration::from_secs(100);+ join.observe(+ row(Side::Node, 0, 70_000, "b", 2),+ t0 + Duration::from_secs(40),+ );+ join.observe(row(Side::Node, 0, 130_000, "c", 3), stalled);++ let report = join.settle(stalled);+ assert_eq!(report.feed_watermarks_ms, [Some(130_000), Some(1_000)]);+ assert_eq!(report.watermark_ms, Some(1_000));+ assert_eq!(report.mapping_outcomes, BTreeMap::new());+ assert_eq!(report.pending_mappings, 3);++ let resumed = stalled + Duration::from_secs(1);+ join.observe(row(Side::Personhog, 0, 70_000, "b", 2), resumed);+ join.observe(row(Side::Personhog, 0, 130_000, "c", 3), resumed);+ let report = join.settle(resumed);+ assert_eq!(report.watermark_ms, Some(130_000));+ assert_eq!(mapping_outcomes(&report), vec![(MappingOutcome::Agree, 1)]);+ assert_eq!(report.pending_mappings, 2);+ }++ // A counterpart may predate the readers; the first grace is not judged.+ #[test]+ fn the_warmup_drops_entries_whose_counterparts_may_predate_the_readers() {+ let t0 = Instant::now();+ let mut join = join_with(usize::MAX, None, GRACE);+ join.observe(row(Side::Node, 0, 1_000, "x", 1), t0);+ join.observe(row(Side::Personhog, 0, 1_000, "y", 2), t0);+ let report = join.settle(t0);+ assert_eq!(report.watermark_ms, Some(1_000));++ both(&mut join, 0, 130_000, "z", 3, t0);+ let report = join.settle(t0);+ assert_eq!(report.warmup_evictions, 2);+ assert_eq!(report.mapping_outcomes, BTreeMap::new());+ assert_eq!(report.pending_mappings, 1);+ assert_eq!(join.node.mapping_count(), 1);+ assert_eq!(join.personhog.mapping_count(), 1);+ }++ // Evicted entries never had their grace, so they get no verdict.+ #[test]+ fn the_cap_evicts_the_oldest_entries_without_settling_them() {+ let t0 = Instant::now();+ let mut join = join(3, None);+ join.observe(row(Side::Node, 0, 3_000, "newest", 1), t0);+ join.observe(row(Side::Node, 0, 1_000, "oldest", 2), t0);+ join.observe(merge(Side::Node, 0, 2_000, 7, 8), t0);+ join.observe(row(Side::Personhog, 0, 3_000, "newest", 1), t0);++ let report = join.settle(t0);+ assert_eq!(report.mapping_outcomes, BTreeMap::new());+ assert_eq!(report.overflow_evictions, 1);+ assert_eq!(report.pending_mappings, 1);+ assert_eq!(report.pending_merges, 1);+ assert!(join+ .mappings+ .contains_key(&DistinctIdKey::new(TEAM, "newest")));+ assert_eq!(join.node.mapping_count(), 1);++ // A key's history counts: three versions of one key weigh three rows.+ let mut hot = Join::new(+ JoinConfig {+ grace: GRACE,+ idle_partition: IDLE,+ max_entries: 4,+ teams: None,+ mapping_lag: MAPPING_LAG,+ warmup: Duration::ZERO,+ },+ SampleRing::new(10),+ );+ for (version, person) in [(0, 1), (1, 2), (2, 3)] {+ let at = 1_000 + version * 100;+ hot.observe(+ row_of_team(TEAM, Side::Node, 0, at, "a", person, version),+ t0,+ );+ }+ hot.observe(row(Side::Personhog, 0, 1_000, "a", 3), t0);+ hot.observe(row(Side::Node, 0, 5_000, "b", 4), t0);+ let report = hot.settle(t0);+ assert_eq!(report.overflow_evictions, 1);+ assert!(hot.mappings.contains_key(&DistinctIdKey::new(TEAM, "b")));+ assert_eq!(hot.node.row_count(), 1);+ }++ // The lag keeps the rows a merge's verdict reads when they are older+ // than the merge; the order keeps them when both fall due in one sweep.+ #[test]+ fn mappings_outlive_the_merges_they_explain() {+ let t0 = Instant::now();+ let mut join = join(usize::MAX, None);+ join.observe(+ row_of_team(TEAM, Side::Personhog, 0, 1_000, "anon", 1, 0),+ t0,+ );+ join.observe(+ row_of_team(TEAM, Side::Personhog, 0, 1_000, "anon", 2, 1),+ t0,+ );+ join.observe(merge(Side::Personhog, 0, 1_500, 1, 2), t0);+ join.observe(row(Side::Node, 0, 1_000, "anon", 2), t0);+ join.observe(row_of_team(TEAM, Side::Personhog, 0, 1_000, "b", 3, 0), t0);+ join.observe(row_of_team(TEAM, Side::Personhog, 0, 1_000, "b", 4, 1), t0);+ join.observe(merge(Side::Personhog, 0, 1_500, 3, 4), t0);+ join.observe(row(Side::Node, 0, 1_000, "b", 4), t0);++ // The rows of `anon` are due for a lag of zero, the merge is not yet.+ both(&mut join, 0, 61_200, "w1", 9, t0);+ let report = join.settle(t0);+ assert_eq!(report.merge_outcomes, BTreeMap::new());+ assert_eq!(report.mapping_outcomes, BTreeMap::new());++ both(&mut join, 0, 61_600, "w2", 9, t0);+ let report = join.settle(t0);+ assert_eq!(+ report.merge_outcomes,+ BTreeMap::from([(SurvivorOutcome::PersonhogStub, 2)])+ );+ assert_eq!(report.mapping_outcomes, BTreeMap::new());++ both(&mut join, 0, 70_000, "w3", 9, t0);+ let report = join.settle(t0);+ assert_eq!(mapping_outcomes(&report), vec![(MappingOutcome::Agree, 2)]);+ }++ // Node merges A into C directly; personhog goes through its own B, whose+ // merge is older and settles first. A's verdict still reaches C.+ #[test]+ fn merge_edges_outlive_the_merges_that_depend_on_them() {+ let t0 = Instant::now();+ let mut join = join(usize::MAX, None);+ join.observe(merge(Side::Node, 0, 1_000, 1, 3), t0);+ join.observe(merge(Side::Personhog, 0, 500, 2, 3), t0);+ join.observe(merge(Side::Personhog, 0, 1_000, 1, 2), t0);++ both(&mut join, 0, 60_600, "w1", 9, t0);+ let report = join.settle(t0);+ assert_eq!(report.merge_outcomes.values().sum::<u64>(), 1);+ assert!(!report+ .merge_outcomes+ .contains_key(&SurvivorOutcome::Mismatch));+ assert_eq!(report.pending_merges, 1);+ assert_eq!(+ join.personhog.merged_into(&PersonKey::new(TEAM, uuid(2))),+ Some(uuid(3))+ );++ both(&mut join, 0, 61_100, "w2", 9, t0);+ let report = join.settle(t0);+ assert_eq!(+ report.merge_outcomes,+ BTreeMap::from([(SurvivorOutcome::Agree, 1)])+ );+ assert_eq!(join.node.merge_count() + join.personhog.merge_count(), 0);+ assert!(join.merges.is_empty());+ assert!(join.target_refs.is_empty());+ }++ #[test]+ fn observe_reports_filters_ties_conflicts_and_placement() {+ let t0 = Instant::now();+ let mut join = join(usize::MAX, Some(HashSet::from([TEAM])));+ assert!(+ join.observe(row_of_team(TEAM + 1, Side::Node, 0, 1_000, "f", 1, 0), t0)+ .filtered+ );+ assert_eq!(join.mappings.len(), 0);++ assert_eq!(+ join.observe(row(Side::Node, 0, 1_000, "d", 1), t0),+ ObserveReport::default()+ );+ assert_eq!(+ join.observe(row(Side::Node, 0, 1_000, "d", 2), t0)+ .version_tie,+ Some(Side::Node)+ );++ // Placement is only judged once the topic's partition count is known.+ let key = format!("{TEAM}:{}", uuid(7));+ let expected = partition_for_key(key.as_bytes(), 64) as i32;+ assert_eq!(+ join.observe(merge(Side::Node, expected, 1_000, 7, 8), t0)+ .placement,+ None+ );+ join.observe(+ Event::Assigned {+ topic: topic(Side::Personhog, Kind::MergeEvents),+ partitions: 64,+ },+ t0,+ );+ assert_eq!(+ join.observe(merge(Side::Personhog, expected, 1_000, 7, 8), t0)+ .placement,+ Some((Side::Personhog, true))+ );+ let report = join.observe(merge(Side::Personhog, expected + 1, 1_000, 7, 9), t0);+ assert_eq!(report.placement, Some((Side::Personhog, false)));+ assert_eq!(report.merge_conflict, Some(Side::Personhog));+ }+}diff --git a/rust/personhog-parity-monitor/src/lib.rs b/rust/personhog-parity-monitor/src/lib.rsnew file mode 100644index 000000000000..cedd5bce3395--- /dev/null+++ b/rust/personhog-parity-monitor/src/lib.rs@@ -0,0 +1,9 @@+//! Follows the Node Postgres path's and `personhog-publisher`'s person CDC+//! feeds and reports as metrics whether they end in the same state. State+//! is in memory under a settle window; the rules are `cdc_parity`.++pub mod config;+pub mod feed;+pub mod join;+pub mod pipeline;+pub mod server;diff --git a/rust/personhog-parity-monitor/src/main.rs b/rust/personhog-parity-monitor/src/main.rsnew file mode 100644index 000000000000..cf5417be33c4--- /dev/null+++ b/rust/personhog-parity-monitor/src/main.rs@@ -0,0 +1,109 @@+use std::time::Duration;++use anyhow::{Context, Result};+use envconfig::Envconfig;+use health::HealthRegistry;+use tokio::sync::mpsc;+use tracing::info;+use tracing::level_filters::LevelFilter;+use tracing_subscriber::layer::SubscriberExt;+use tracing_subscriber::util::SubscriberInitExt;+use tracing_subscriber::{fmt, EnvFilter};++use personhog_parity_monitor::config::Config;+use personhog_parity_monitor::feed::TopicReader;+use personhog_parity_monitor::join::{Event, Join, JoinConfig, Kind, SampleRing};+use personhog_parity_monitor::{pipeline, server};++common_alloc::used!();++/// A reader or the join that reports nothing for this long is dead.+const LIVENESS_DEADLINE: Duration = Duration::from_secs(60);+const EVENT_QUEUE: usize = 10_000;++#[tokio::main]+async fn main() -> Result<()> {+ let config = Config::init_from_env().context("reading the configuration")?;+ tracing_subscriber::registry()+ .with(fmt::layer().with_target(true).with_level(true))+ .with(+ EnvFilter::builder()+ .with_default_directive(LevelFilter::INFO.into())+ .from_env_lossy(),+ )+ .init();++ let join_config = JoinConfig {+ grace: config.grace(),+ idle_partition: config.idle_partition(),+ max_entries: config.parity_max_entries,+ teams: config.teams().map_err(anyhow::Error::msg)?,+ mapping_lag: config.mapping_lag(),+ warmup: config.warmup(),+ };+ info!(+ grace_seconds = config.parity_grace_seconds,+ max_entries = config.parity_max_entries,+ teams = ?join_config.teams,+ debug_api = config.debug_api_secret().is_some(),+ "starting personhog-parity-monitor"+ );++ let health = HealthRegistry::new("personhog-parity-monitor");+ let samples = SampleRing::new(config.parity_sample_ring_size);+ // Building the router installs the metrics recorder; counter handles the+ // tasks create before that would bind to the no-op recorder for good.+ let router = server::router(+ health.clone(),+ samples.clone(),+ config.debug_api_secret().map(str::to_string),+ );+ let (events, arrivals) = mpsc::channel(EVENT_QUEUE);++ // The join drains the queue, so it runs before the first reader fills it.+ let join_health = health.register("join".to_string(), LIVENESS_DEADLINE).await;+ tokio::spawn(pipeline::run(+ Join::new(join_config, samples),+ arrivals,+ config.sweep_interval(),+ join_health,+ ));++ for feed in config.feeds() {+ for kind in [Kind::DistinctIds, Kind::MergeEvents] {+ let feed_for_reader = feed.clone();+ // rdkafka's metadata fetch and assignment block the thread.+ let (reader, partitions) =+ tokio::task::spawn_blocking(move || TopicReader::connect(&feed_for_reader, kind))+ .await+ .context("the reader setup task panicked")??;+ let topic = reader.topic();+ info!(+ feed = topic.side.label(),+ topic = topic.kind.label(),+ partitions,+ hosts = %feed.kafka_hosts,+ "assigned every partition at the end of the log"+ );+ events+ .send(Event::Assigned { topic, partitions })+ .await+ .context("the join is gone")?;+ let handle = health+ .register(+ format!("{}-{}", topic.side.label(), topic.kind.label()),+ LIVENESS_DEADLINE,+ )+ .await;+ tokio::spawn(reader.run(events.clone(), handle));+ }+ }+ drop(events);++ let bind = config.bind();+ info!(%bind, "serving metrics");+ common_metrics::serve(router, &bind)+ .await+ .context("serving metrics")?;+ Ok(())+}diff --git a/rust/personhog-parity-monitor/src/pipeline.rs b/rust/personhog-parity-monitor/src/pipeline.rsnew file mode 100644index 000000000000..50ccff9f0ad3--- /dev/null+++ b/rust/personhog-parity-monitor/src/pipeline.rs@@ -0,0 +1,124 @@+//! Owns the join and turns its reports into metrics, which carry outcome+//! labels only, never a team or an id.++use std::collections::HashMap;+use std::time::{Duration, Instant};++use health::HealthHandle;+use personhog_common::cdc_parity::Side;+use tokio::sync::mpsc;+use tokio::time::MissedTickBehavior;++use crate::feed::now_ms;+use crate::join::{side_index, Event, Join, Kind, ObserveReport, SweepReport, Topic};++pub async fn run(+ mut join: Join,+ mut events: mpsc::Receiver<Event>,+ sweep_interval: Duration,+ health: HealthHandle,+) {+ let mut ticker = tokio::time::interval(sweep_interval);+ ticker.set_missed_tick_behavior(MissedTickBehavior::Delay);+ let mut partition_counts: HashMap<Topic, u32> = HashMap::new();+ loop {+ tokio::select! {+ event = events.recv() => {+ let Some(event) = event else {+ return;+ };+ if let Event::Assigned { topic, partitions } = &event {+ partition_counts.insert(*topic, *partitions);+ record_partition_gauges(&partition_counts);+ }+ let topic = event.topic();+ let report = join.observe(event, Instant::now());+ record_observation(topic, &report);+ }+ _ = ticker.tick() => {+ let report = join.settle(Instant::now());+ record_sweep(&report);+ health.report_healthy().await;+ }+ }+ }+}++fn record_partition_gauges(counts: &HashMap<Topic, u32>) {+ for (topic, count) in counts {+ metrics::gauge!(+ "personhog_parity_topic_partitions",+ "feed" => topic.side.label(),+ "topic" => topic.kind.label()+ )+ .set(*count as f64);+ }+ for kind in [Kind::DistinctIds, Kind::MergeEvents] {+ let pair = [Side::Node, Side::Personhog].map(|side| counts.get(&Topic { side, kind }));+ if let [Some(node), Some(personhog)] = pair {+ metrics::gauge!(+ "personhog_parity_topic_pair_partitions_equal",+ "topic" => kind.label()+ )+ .set(if node == personhog { 1.0 } else { 0.0 });+ }+ }+}++fn record_observation(topic: Topic, report: &ObserveReport) {+ let feed = topic.side.label();+ if report.filtered {+ metrics::counter!("personhog_parity_filtered_total", "feed" => feed, "topic" => topic.kind.label())+ .increment(1);+ }+ if let Some(side) = report.version_tie {+ metrics::counter!("personhog_parity_version_ties_total", "feed" => side.label())+ .increment(1);+ }+ if let Some(side) = report.merge_conflict {+ metrics::counter!("personhog_parity_merge_conflicts_total", "feed" => side.label())+ .increment(1);+ }+ if let Some((side, expected)) = report.placement {+ metrics::counter!(+ "personhog_parity_merge_placement_total",+ "feed" => side.label(),+ "outcome" => if expected { "expected" } else { "unexpected" }+ )+ .increment(1);+ }+}++fn record_sweep(report: &SweepReport) {+ for (outcome, count) in &report.mapping_outcomes {+ metrics::counter!("personhog_parity_settled_mappings_total", "outcome" => outcome.label())+ .increment(*count);+ }+ for (outcome, count) in &report.merge_outcomes {+ metrics::counter!("personhog_parity_settled_merges_total", "outcome" => outcome.label())+ .increment(*count);+ }+ for (agreement, count) in &report.merge_partitions {+ metrics::counter!("personhog_parity_merge_partition_total", "outcome" => agreement.label())+ .increment(*count);+ }+ if report.overflow_evictions > 0 {+ metrics::counter!("personhog_parity_overflow_evictions_total")+ .increment(report.overflow_evictions);+ }+ if report.warmup_evictions > 0 {+ metrics::counter!("personhog_parity_warmup_evictions_total")+ .increment(report.warmup_evictions);+ }+ metrics::gauge!("personhog_parity_pending_entries", "kind" => "mappings")+ .set(report.pending_mappings as f64);+ metrics::gauge!("personhog_parity_pending_entries", "kind" => "merges")+ .set(report.pending_merges as f64);+ let now = now_ms();+ for side in [Side::Node, Side::Personhog] {+ if let Some(watermark) = report.feed_watermarks_ms[side_index(side)] {+ metrics::gauge!("personhog_parity_watermark_age_seconds", "feed" => side.label())+ .set((now - watermark).max(0) as f64 / 1000.0);+ }+ }+}diff --git a/rust/personhog-parity-monitor/src/server.rs b/rust/personhog-parity-monitor/src/server.rsnew file mode 100644index 000000000000..1706fe458b16--- /dev/null+++ b/rust/personhog-parity-monitor/src/server.rs@@ -0,0 +1,56 @@+use axum::http::{HeaderMap, StatusCode};+use axum::response::{IntoResponse, Response};+use axum::routing::get;+use axum::{Json, Router};+use common_metrics::setup_metrics_routes;+use health::HealthRegistry;++use crate::join::SampleRing;++const DEBUG_API_SECRET_HEADER: &str = "x-debug-api-secret";++/// The mismatch ring carries distinct ids, so it answers only with the shared secret.+pub fn router(+ health: HealthRegistry,+ samples: SampleRing,+ debug_api_secret: Option<String>,+) -> Router {+ let readiness = health.clone();+ let router = Router::new()+ .route(+ "/_readiness",+ get(move || {+ let health = readiness.clone();+ async move { health.get_status() }+ }),+ )+ .route(+ "/_liveness",+ get(move || {+ let health = health.clone();+ async move { health.get_status() }+ }),+ )+ .route(+ "/debug/mismatches",+ get(move |headers: HeaderMap| {+ let samples = samples.clone();+ let secret = debug_api_secret.clone();+ async move { mismatches(&headers, secret.as_deref(), &samples) }+ }),+ );+ setup_metrics_routes(router)+}++fn mismatches(headers: &HeaderMap, secret: Option<&str>, samples: &SampleRing) -> Response {+ let Some(secret) = secret else {+ return StatusCode::NOT_FOUND.into_response();+ };+ let presented = headers+ .get(DEBUG_API_SECRET_HEADER)+ .and_then(|value| value.to_str().ok());+ if presented != Some(secret) {+ return StatusCode::UNAUTHORIZED.into_response();+ }+ Json(samples.snapshot()).into_response()+}