Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(personhog): add the CDC parity monitor (#119364)

José Sequeira4f991386927a
diff --git a/.github/rust-images.yml b/.github/rust-images.yml
index 35ffcdb6eabc..de7ce57a5d9b 100644
--- a/.github/rust-images.yml
+++ b/.github/rust-images.yml
@@ -153,6 +153,10 @@
dockerfile: ./rust/Dockerfile
project: 6tl5lmb8l8
+- image: personhog-parity-monitor
+ dockerfile: ./rust/Dockerfile
+ project: jfz76fgvqd
+
- image: kafka-assigner
dockerfile: ./rust/Dockerfile
project: 6kjc85bwm7
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index 37689993e62b..a8a37c494b60 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -9013,6 +9013,26 @@ dependencies = [
"zstd 0.13.3",
]
+[[package]]
+name = "personhog-parity-monitor"
+version = "0.1.0"
+dependencies = [
+ "anyhow",
+ "axum 0.7.9",
+ "common-alloc",
+ "common-metrics",
+ "envconfig",
+ "health",
+ "metrics",
+ "personhog-common",
+ "rdkafka",
+ "serde_json",
+ "tokio",
+ "tracing",
+ "tracing-subscriber",
+ "uuid",
+]
+
[[package]]
name = "personhog-proto"
version = "0.1.0"
diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index a30b97a47bf1..e269164773d9 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -66,6 +66,7 @@ members = [
"personhog-coordination",
"personhog-identity",
"personhog-leader",
+ "personhog-parity-monitor",
"personhog-writer",
"personhog-proto",
"prometheus-rw-proto",
diff --git a/rust/owners.yaml b/rust/owners.yaml
index 0ff7bace8a84..db0987225df1 100644
--- a/rust/owners.yaml
+++ b/rust/owners.yaml
@@ -23,6 +23,7 @@ rules:
- '/personhog-common/'
- '/personhog-coordination/'
- '/personhog-leader/'
+ - '/personhog-parity-monitor/'
- '/personhog-proto/'
- '/personhog-replica/'
- '/personhog-router/'
diff --git a/rust/personhog-parity-monitor/Cargo.toml b/rust/personhog-parity-monitor/Cargo.toml
new file mode 100644
index 000000000000..3cf6761f479c
--- /dev/null
+++ b/rust/personhog-parity-monitor/Cargo.toml
@@ -0,0 +1,27 @@
+[package]
+name = "personhog-parity-monitor"
+version = "0.1.0"
+edition = "2021"
+publish = false
+
+[dependencies]
+personhog-common = { path = "../personhog-common" }
+common-alloc = { path = "../common/alloc" }
+common-metrics = { path = "../common/metrics" }
+health = { path = "../common/health" }
+
+anyhow = { workspace = true }
+axum = { workspace = true }
+envconfig = { workspace = true }
+metrics = { workspace = true }
+rdkafka = { workspace = true }
+serde_json = { workspace = true }
+tokio = { workspace = true }
+tracing = { workspace = true }
+tracing-subscriber = { workspace = true }
+
+[dev-dependencies]
+uuid = { workspace = true }
+
+[lints]
+workspace = true
diff --git a/rust/personhog-parity-monitor/README.md b/rust/personhog-parity-monitor/README.md
new file mode 100644
index 000000000000..06300cc2822f
--- /dev/null
+++ b/rust/personhog-parity-monitor/README.md
@@ -0,0 +1,76 @@
+# personhog-parity-monitor
+
+A read-only service that follows the two person CDC feeds and reports, as Prometheus metrics, whether they end in the same state.
+The Node Postgres path produces `clickhouse_person_distinct_id` rows and `person_merge_events`; `personhog-publisher` produces the same two topics from the identity service's outbox.
+When a lane runs in `shadow` store mode both feeds describe the same events, so every distinct id and every merge should settle identically on both.
+
+The rules are `personhog_common::cdc_parity`, the ones the Kafka parity e2e (`nodejs/tests/service-e2e/personhog-kafka-parity.serial.test.ts`) and the test harness gate apply: end state, not messages.
+Per distinct id only the highest-version row counts, and of it only the person uuid and the deleted flag.
+Per merged person only the survivor at the end of its merge chain counts.
+
+## How it settles
+
+- Every partition of the four topics is assigned at the end of the log. There is no consumer group and nothing is committed: a group would split the partitions between replicas and break the cross-feed join, and the output is a rate over a window, so a restart has nothing to resume. Run one replica.
+- Each distinct id topic tracks the newest message timestamp per partition. A feed's watermark is the slowest of its partitions that delivered within `PARITY_IDLE_PARTITION_SECONDS`. When both feeds are quiet the watermarks advance by the wall clock; when one feed is quiet next to a live one it has stalled, so its watermark holds, its `watermark_age_seconds` grows and nothing settles against its absence. The merge topics do not count: they are sparse and would hold the watermark an idle window behind. The global watermark is the slower feed's.
+- An entry settles once its newest row is older than the global watermark minus `PARITY_GRACE_SECONDS`. The grace has to exceed the publisher's drain lag, or pairs settle one-sided before the personhog copy arrives.
+- Entries older than the first watermark plus `PARITY_WARMUP_SECONDS` (the grace by default) are dropped unjudged and counted as `warmup_evictions_total`: the readers start at the end of the log at different moments, so a row's counterpart may have been produced before its reader started. A fixture whose readers start before any traffic sets it to `0`.
+- Merges settle first and mappings `PARITY_MAPPING_LAG_MS` later, so a settling merge still sees the rows that named its persons and can tell a stub from a missing merge event.
+- A merge agrees when both feeds name the same direct target; what became of that target is the target's own verdict. Different direct targets are followed to their survivors, because one feed may reach the same person through a stub the other never made.
+- Settled entries are evicted from both feeds. `PARITY_MAX_ENTRIES` bounds the rows and merges retained across both feeds, history included, so a key that keeps receiving versions cannot grow past it behind one entry. Above the cap the oldest entries are evicted down to nine tenths of it without a verdict and counted as overflow. A row or merge costs about half a kilobyte.
+- Distinct id rows carry null keys on both sides, so their partitions are random and only the partition counts are compared. Merge events are keyed `"{team_id}:{old_person_uuid}"`; each one is checked against `murmur2(key) % partitions` of its own topic, and a merge present on both feeds is checked for landing on the same partition number while both merge topics have the same count.
+
+## Metrics
+
+All names start with `personhog_parity_`. Labels carry outcomes only, never a team or an id; `/debug/mismatches` holds the last `PARITY_SAMPLE_RING_SIZE` disagreements with both sides' rows.
+
+| Metric | Labels | Meaning |
+| --- | --- | --- |
+| `settled_mappings_total` | `outcome`: `agree`, `person_mismatch`, `deleted_mismatch`, `node_only`, `personhog_only` | One per settled distinct id |
+| `settled_merges_total` | `outcome`: `agree`, `survivor_mismatch`, `node_only`, `personhog_only`, `node_stub`, `personhog_stub`, `loop` | One per settled merged person. `*_stub`: the merging feed created the person inside the window and the other feed never named it, its own path to the result. `*_only`: the other feed named the person, or the person predates the window, and no merge for it arrived |
+| `merge_partition_total` | `outcome`: `same`, `different` | Merges present on both feeds, by partition number; only counted while both merge topics have the same partition count |
+| `merge_placement_total` | `feed`, `outcome`: `expected`, `unexpected` | Arrival partition against the key's murmur2 partition |
+| `version_ties_total`, `merge_conflicts_total` | `feed` | A feed contradicting itself |
+| `messages_total`, `decode_failures_total`, `read_errors_total`, `filtered_total` | `feed`, `topic` | Reader counters |
+| `topic_partitions` | `feed`, `topic` | Partition counts from metadata, read once at startup |
+| `topic_pair_partitions_equal` | `topic` | Whether the two feeds' topics of that kind have the same count |
+| `pending_entries` | `kind` | Entries waiting to settle |
+| `watermark_age_seconds` | `feed` | Wall clock minus the feed's watermark; a stalled feed shows here |
+| `overflow_evictions_total` | | Entries dropped by the cap, so the outcome counters are degraded while this moves |
+| `warmup_evictions_total` | | Entries dropped during the first grace after start |
+
+## Configuration
+
+| Variable | Default | Meaning |
+| --- | --- | --- |
+| `NODE_KAFKA_HOSTS`, `NODE_KAFKA_TLS` | `localhost:9092`, `false` | The Node feed's cluster |
+| `NODE_DISTINCT_ID_TOPIC`, `NODE_MERGE_EVENTS_TOPIC` | `clickhouse_person_distinct_id`, `person_merge_events` | The Node feed's topics |
+| `PERSONHOG_KAFKA_HOSTS`, `PERSONHOG_KAFKA_TLS` | `localhost:9092`, `false` | The personhog feed's cluster |
+| `PERSONHOG_DISTINCT_ID_TOPIC`, `PERSONHOG_MERGE_EVENTS_TOPIC` | required | The publisher's topics |
+| `PARITY_GRACE_SECONDS` | `60` | How long an entry waits for the other feed |
+| `PARITY_IDLE_PARTITION_SECONDS` | `30` | When a quiet partition stops holding the watermark |
+| `PARITY_MAPPING_LAG_MS` | `2000` | How much longer than merges the mappings wait |
+| `PARITY_WARMUP_SECONDS` | the grace | How long after the first watermark entries are dropped unjudged; `0` for a fixture whose readers start before any traffic |
+| `PARITY_MAX_ENTRIES` | `1000000` | The cap on retained rows and merges across both feeds, about half a gigabyte |
+| `PARITY_TEAMS` | empty, every team | Comma separated team ids; set it to the merge events allowlist when that is a subset |
+| `PARITY_SAMPLE_RING_SIZE` | `500` | Disagreements kept for `/debug/mismatches` |
+| `PARITY_SWEEP_INTERVAL_MS` | `1000` | How often entries are settled |
+| `DEBUG_API_SECRET` | unset, endpoint off | Required in the `x-debug-api-secret` header to read `/debug/mismatches`, which carries distinct ids |
+| `BIND_HOST`, `BIND_PORT` | `0.0.0.0`, `3301` | `/metrics`, `/_readiness`, `/_liveness`, `/debug/mismatches` |
+
+## Running it locally
+
+The harness `stack --publish` brings up identity and the publishers with per-run topics, and a Node ingestion API worker in `PERSONS_STORE_MODE=shadow` fed by the Rust ingestion consumer writes the Node side; the Kafka parity e2e shows the worker's environment.
+
+```bash
+target/debug/personhog-test-harness stack --publish --ready-file /tmp/personhog-stack.json
+
+NODE_DISTINCT_ID_TOPIC=<the worker's INGESTION_OUTPUT_PERSON_DISTINCT_IDS_TOPIC> \
+NODE_MERGE_EVENTS_TOPIC=<the worker's INGESTION_OUTPUT_PERSON_MERGE_EVENTS_TOPIC> \
+PERSONHOG_DISTINCT_ID_TOPIC=$(jq -r .distinct_ids_topic /tmp/personhog-stack.json) \
+PERSONHOG_MERGE_EVENTS_TOPIC=$(jq -r .merge_events_topic /tmp/personhog-stack.json) \
+PARITY_GRACE_SECONDS=10 DEBUG_API_SECRET=local \
+target/debug/personhog-parity-monitor
+
+curl -s localhost:3301/metrics | grep personhog_parity_settled
+curl -s -H 'x-debug-api-secret: local' localhost:3301/debug/mismatches
+```
diff --git a/rust/personhog-parity-monitor/src/config.rs b/rust/personhog-parity-monitor/src/config.rs
new file mode 100644
index 000000000000..2bc43c98faff
--- /dev/null
+++ b/rust/personhog-parity-monitor/src/config.rs
@@ -0,0 +1,185 @@
+use std::collections::HashSet;
+use std::time::Duration;
+
+use envconfig::Envconfig;
+use personhog_common::cdc_parity::Side;
+
+#[derive(Envconfig, Clone, Debug)]
+pub struct Config {
+ #[envconfig(default = "localhost:9092")]
+ pub node_kafka_hosts: String,
+ #[envconfig(default = "false")]
+ pub node_kafka_tls: bool,
+ #[envconfig(default = "clickhouse_person_distinct_id")]
+ pub node_distinct_id_topic: String,
+ #[envconfig(default = "person_merge_events")]
+ pub node_merge_events_topic: String,
+
+ #[envconfig(default = "localhost:9092")]
+ pub personhog_kafka_hosts: String,
+ #[envconfig(default = "false")]
+ pub personhog_kafka_tls: bool,
+ pub personhog_distinct_id_topic: String,
+ pub personhog_merge_events_topic: String,
+
+ /// Has to exceed the publisher's drain lag, or every pair settles
+ /// one-sided before the personhog copy arrives.
+ #[envconfig(default = "60")]
+ pub parity_grace_seconds: u64,
+ /// A quiet partition stops holding the watermark back after this long.
+ #[envconfig(default = "30")]
+ pub parity_idle_partition_seconds: u64,
+ /// Covers the timestamp skew between a merge event and the rows it moved.
+ #[envconfig(default = "2000")]
+ pub parity_mapping_lag_ms: u64,
+ /// Defaults to the grace. Zero turns the warmup off, for a fixture
+ /// whose readers start before any traffic.
+ pub parity_warmup_seconds: Option<u64>,
+ /// Retained rows and merges across both feeds; each costs about half a
+ /// kilobyte.
+ #[envconfig(default = "1000000")]
+ pub parity_max_entries: usize,
+ /// Empty means every team. Set it to the merge events allowlist when
+ /// that is a subset, or every other team's merges settle one-sided.
+ #[envconfig(default = "")]
+ pub parity_teams: String,
+ #[envconfig(default = "500")]
+ pub parity_sample_ring_size: usize,
+ #[envconfig(default = "1000")]
+ pub parity_sweep_interval_ms: u64,
+
+ /// Unset leaves `/debug/mismatches` off; its samples carry distinct ids.
+ pub debug_api_secret: Option<String>,
+
+ #[envconfig(default = "0.0.0.0")]
+ pub bind_host: String,
+ #[envconfig(default = "3301")]
+ pub bind_port: u16,
+}
+
+#[derive(Clone, Debug)]
+pub struct FeedConfig {
+ pub side: Side,
+ pub kafka_hosts: String,
+ pub kafka_tls: bool,
+ pub distinct_id_topic: String,
+ pub merge_events_topic: String,
+}
+
+impl Config {
+ pub fn feeds(&self) -> [FeedConfig; 2] {
+ [
+ FeedConfig {
+ side: Side::Node,
+ kafka_hosts: self.node_kafka_hosts.clone(),
+ kafka_tls: self.node_kafka_tls,
+ distinct_id_topic: self.node_distinct_id_topic.clone(),
+ merge_events_topic: self.node_merge_events_topic.clone(),
+ },
+ FeedConfig {
+ side: Side::Personhog,
+ kafka_hosts: self.personhog_kafka_hosts.clone(),
+ kafka_tls: self.personhog_kafka_tls,
+ distinct_id_topic: self.personhog_distinct_id_topic.clone(),
+ merge_events_topic: self.personhog_merge_events_topic.clone(),
+ },
+ ]
+ }
+
+ pub fn teams(&self) -> Result<Option<HashSet<i64>>, String> {
+ let teams = self
+ .parity_teams
+ .split(',')
+ .map(str::trim)
+ .filter(|team| !team.is_empty())
+ .map(|team| {
+ team.parse::<i64>()
+ .map_err(|_| format!("PARITY_TEAMS holds {team:?}, expected integer team ids"))
+ })
+ .collect::<Result<HashSet<i64>, String>>()?;
+ Ok((!teams.is_empty()).then_some(teams))
+ }
+
+ pub fn grace(&self) -> Duration {
+ Duration::from_secs(self.parity_grace_seconds)
+ }
+
+ pub fn idle_partition(&self) -> Duration {
+ Duration::from_secs(self.parity_idle_partition_seconds)
+ }
+
+ pub fn mapping_lag(&self) -> Duration {
+ Duration::from_millis(self.parity_mapping_lag_ms)
+ }
+
+ pub fn warmup(&self) -> Duration {
+ Duration::from_secs(
+ self.parity_warmup_seconds
+ .unwrap_or(self.parity_grace_seconds),
+ )
+ }
+
+ pub fn sweep_interval(&self) -> Duration {
+ Duration::from_millis(self.parity_sweep_interval_ms.max(100))
+ }
+
+ pub fn bind(&self) -> String {
+ format!("{}:{}", self.bind_host, self.bind_port)
+ }
+
+ /// An empty value is unset: an empty header must not open the endpoint.
+ pub fn debug_api_secret(&self) -> Option<&str> {
+ self.debug_api_secret
+ .as_deref()
+ .filter(|secret| !secret.is_empty())
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn config(vars: &[(&str, &str)]) -> Config {
+ let mut env = std::collections::HashMap::from([
+ ("PERSONHOG_DISTINCT_ID_TOPIC".to_string(), "a".to_string()),
+ ("PERSONHOG_MERGE_EVENTS_TOPIC".to_string(), "b".to_string()),
+ ]);
+ env.extend(vars.iter().map(|(k, v)| (k.to_string(), v.to_string())));
+ Config::init_from_hashmap(&env).expect("config")
+ }
+
+ #[test]
+ fn team_filter_parses_or_refuses() {
+ assert_eq!(config(&[("PARITY_TEAMS", "")]).teams(), Ok(None));
+ assert_eq!(
+ config(&[("PARITY_TEAMS", " 2, 15 ")]).teams(),
+ Ok(Some(HashSet::from([2, 15])))
+ );
+ assert!(config(&[("PARITY_TEAMS", "2,all")]).teams().is_err());
+ }
+
+ #[test]
+ fn the_warmup_follows_the_grace_unless_set() {
+ assert_eq!(
+ config(&[("PARITY_GRACE_SECONDS", "7")]).warmup(),
+ Duration::from_secs(7)
+ );
+ assert_eq!(
+ config(&[
+ ("PARITY_GRACE_SECONDS", "7"),
+ ("PARITY_WARMUP_SECONDS", "0")
+ ])
+ .warmup(),
+ Duration::ZERO
+ );
+ }
+
+ #[test]
+ fn an_empty_debug_secret_leaves_the_endpoint_off() {
+ assert_eq!(config(&[("DEBUG_API_SECRET", "")]).debug_api_secret(), None);
+ assert_eq!(
+ config(&[("DEBUG_API_SECRET", "s")]).debug_api_secret(),
+ Some("s")
+ );
+ }
+}
diff --git a/rust/personhog-parity-monitor/src/feed.rs b/rust/personhog-parity-monitor/src/feed.rs
new file mode 100644
index 000000000000..4bddf1005a41
--- /dev/null
+++ b/rust/personhog-parity-monitor/src/feed.rs
@@ -0,0 +1,178 @@
+//! One reader per topic, every partition assigned at the end of the log,
+//! nothing committed: a consumer group would split the partitions between
+//! replicas and break the cross-feed join, and a restart resumes nothing.
+
+use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
+
+use anyhow::{bail, Context, Result};
+use health::HealthHandle;
+use personhog_common::cdc_parity::{DecodeError, DistinctIdRow, MergeEvent};
+use rdkafka::consumer::{Consumer, StreamConsumer};
+use rdkafka::error::KafkaError;
+use rdkafka::message::BorrowedMessage;
+use rdkafka::{ClientConfig, Message, Offset, TopicPartitionList};
+use tokio::sync::mpsc;
+use tracing::{debug, error, warn};
+
+use crate::config::FeedConfig;
+use crate::join::{Event, Kind, Record, Topic};
+
+const METADATA_TIMEOUT: Duration = Duration::from_secs(10);
+/// Bounds a `recv` on a quiet topic so the reader still reports liveness.
+const RECV_TIMEOUT: Duration = Duration::from_secs(5);
+const HEALTH_REPORT_INTERVAL: Duration = Duration::from_secs(1);
+const ERROR_BACKOFF: Duration = Duration::from_secs(1);
+/// librdkafka's default prefetch is 64 MiB per consumer.
+const PREFETCH_KBYTES: &str = "16384";
+
+pub struct TopicReader {
+ topic: Topic,
+ name: String,
+ consumer: StreamConsumer,
+}
+
+impl TopicReader {
+ pub fn connect(feed: &FeedConfig, kind: Kind) -> Result<(Self, u32)> {
+ let name = match kind {
+ Kind::DistinctIds => feed.distinct_id_topic.clone(),
+ Kind::MergeEvents => feed.merge_events_topic.clone(),
+ };
+ let topic = Topic {
+ side: feed.side,
+ kind,
+ };
+ let mut client = ClientConfig::new();
+ client
+ .set("bootstrap.servers", &feed.kafka_hosts)
+ .set("group.id", "personhog-parity-monitor")
+ .set("enable.auto.commit", "false")
+ .set("enable.auto.offset.store", "false")
+ .set("auto.offset.reset", "latest")
+ .set("queued.max.messages.kbytes", PREFETCH_KBYTES)
+ .set(
+ "client.id",
+ format!("personhog-parity-monitor-{}", feed.side.label()),
+ );
+ if feed.kafka_tls {
+ client
+ .set("security.protocol", "ssl")
+ .set("enable.ssl.certificate.verification", "false");
+ }
+ let consumer: StreamConsumer = client
+ .create()
+ .with_context(|| format!("creating the {} reader for {name}", feed.side.label()))?;
+ let metadata = consumer
+ .fetch_metadata(Some(&name), METADATA_TIMEOUT)
+ .with_context(|| format!("fetching metadata of {name}"))?;
+ let partitions: Vec<i32> = metadata
+ .topics()
+ .iter()
+ .find(|entry| entry.name() == name)
+ .map(|entry| entry.partitions().iter().map(|p| p.id()).collect())
+ .unwrap_or_default();
+ if partitions.is_empty() {
+ bail!("topic {name} has no partitions on {}", feed.kafka_hosts);
+ }
+ let mut assignment = TopicPartitionList::new();
+ for partition in &partitions {
+ // The end offset is read here, before anyone learns the reader is
+ // up; `Offset::End` would resolve later, in the fetcher, and a row
+ // produced in between would be skipped.
+ let (_, end) = consumer
+ .fetch_watermarks(&name, *partition, METADATA_TIMEOUT)
+ .with_context(|| format!("fetching the end offset of {name}/{partition}"))?;
+ assignment.add_partition_offset(&name, *partition, Offset::Offset(end))?;
+ }
+ consumer
+ .assign(&assignment)
+ .with_context(|| format!("assigning {name}"))?;
+ Ok((
+ Self {
+ topic,
+ name,
+ consumer,
+ },
+ partitions.len() as u32,
+ ))
+ }
+
+ pub fn topic(&self) -> Topic {
+ self.topic
+ }
+
+ /// The first decode failure per topic is logged in full; the counter is the signal.
+ pub async fn run(self, events: mpsc::Sender<Event>, health: HealthHandle) {
+ let feed = self.topic.side.label();
+ let kind = self.topic.kind.label();
+ let messages =
+ metrics::counter!("personhog_parity_messages_total", "feed" => feed, "topic" => kind);
+ let read_errors = metrics::counter!("personhog_parity_read_errors_total", "feed" => feed, "topic" => kind);
+ let decode_failures = metrics::counter!(
+ "personhog_parity_decode_failures_total",
+ "feed" => feed,
+ "topic" => kind
+ );
+ let mut last_health_report = Instant::now() - HEALTH_REPORT_INTERVAL;
+ let mut decode_failure_logged = false;
+ loop {
+ if last_health_report.elapsed() >= HEALTH_REPORT_INTERVAL {
+ health.report_healthy().await;
+ last_health_report = Instant::now();
+ }
+ let message = match tokio::time::timeout(RECV_TIMEOUT, self.consumer.recv()).await {
+ Err(_) => continue,
+ // The consumer cannot recover; the reader stops reporting, so
+ // liveness fails and the pod restarts into a fresh window.
+ Ok(Err(KafkaError::MessageConsumptionFatal(code))) => {
+ error!(topic = %self.name, %code, "the consumer failed for good; stopping the reader");
+ read_errors.increment(1);
+ return;
+ }
+ Ok(Err(error)) => {
+ warn!(topic = %self.name, %error, "reading the topic failed");
+ read_errors.increment(1);
+ tokio::time::sleep(ERROR_BACKOFF).await;
+ continue;
+ }
+ Ok(Ok(message)) => message,
+ };
+ messages.increment(1);
+ let record = match self.decode(&message) {
+ Ok(record) => record,
+ Err(error) => {
+ decode_failures.increment(1);
+ if decode_failure_logged {
+ debug!(topic = %self.name, %error, "undecodable message");
+ } else {
+ warn!(topic = %self.name, %error, "undecodable message; further ones are counted only");
+ decode_failure_logged = true;
+ }
+ continue;
+ }
+ };
+ let event = Event::Arrival {
+ topic: self.topic,
+ partition: message.partition(),
+ timestamp_ms: message.timestamp().to_millis().unwrap_or_else(now_ms),
+ record,
+ };
+ if events.send(event).await.is_err() {
+ return;
+ }
+ }
+ }
+
+ fn decode(&self, message: &BorrowedMessage<'_>) -> Result<Record, DecodeError> {
+ let payload = message.payload().unwrap_or_default();
+ match self.topic.kind {
+ Kind::DistinctIds => DistinctIdRow::decode(payload).map(Record::Mapping),
+ Kind::MergeEvents => MergeEvent::decode(message.key(), payload).map(Record::Merge),
+ }
+ }
+}
+
+pub fn now_ms() -> i64 {
+ SystemTime::now()
+ .duration_since(UNIX_EPOCH)
+ .map_or(0, |elapsed| elapsed.as_millis() as i64)
+}
diff --git a/rust/personhog-parity-monitor/src/join.rs b/rust/personhog-parity-monitor/src/join.rs
new file mode 100644
index 000000000000..43f01c67935a
--- /dev/null
+++ b/rust/personhog-parity-monitor/src/join.rs
@@ -0,0 +1,985 @@
+//! The two feeds folded side by side and settled by watermark: an entry
+//! waits until the slower feed has delivered past it by the grace, so a
+//! lagging feed never reads as one-sided. Pure; the pipeline emits metrics.
+
+use std::collections::{BTreeMap, HashMap, HashSet, VecDeque};
+use std::sync::{Arc, Mutex};
+use std::time::{Duration, Instant};
+
+use personhog_common::cdc_parity::{
+ compare_mapping, compare_survivor, DistinctIdKey, DistinctIdRow, FeedState, Mapping,
+ MappingOutcome, MergeEvent, PersonKey, Side, SurvivorOutcome,
+};
+use personhog_common::partitioning::partition_for_key;
+use serde_json::{json, Value};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub enum Kind {
+ DistinctIds,
+ MergeEvents,
+}
+
+impl Kind {
+ pub fn label(self) -> &'static str {
+ match self {
+ Kind::DistinctIds => "distinct_ids",
+ Kind::MergeEvents => "merge_events",
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord)]
+pub struct Topic {
+ pub side: Side,
+ pub kind: Kind,
+}
+
+#[derive(Debug, Clone)]
+pub enum Record {
+ Mapping(DistinctIdRow),
+ Merge(MergeEvent),
+}
+
+#[derive(Debug, Clone)]
+pub enum Event {
+ /// The count places merge keys.
+ Assigned { topic: Topic, partitions: u32 },
+ Arrival {
+ topic: Topic,
+ partition: i32,
+ timestamp_ms: i64,
+ record: Record,
+ },
+}
+
+impl Event {
+ pub fn topic(&self) -> Topic {
+ match self {
+ Event::Assigned { topic, .. } | Event::Arrival { topic, .. } => *topic,
+ }
+ }
+}
+
+#[derive(Debug, Clone)]
+pub struct JoinConfig {
+ pub grace: Duration,
+ pub idle_partition: Duration,
+ pub max_entries: usize,
+ pub teams: Option<HashSet<i64>>,
+ /// Covers the timestamp skew between a merge event and the rows it
+ /// moved, so a settling merge still finds the rows naming its persons.
+ pub mapping_lag: Duration,
+ /// Entries older than the first watermark plus this are dropped
+ /// unjudged, because a counterpart may predate the readers. Zero
+ /// turns the warmup off.
+ pub warmup: Duration,
+}
+
+#[derive(Debug, Default, Clone, PartialEq, Eq)]
+pub struct ObserveReport {
+ pub filtered: bool,
+ pub version_tie: Option<Side>,
+ pub merge_conflict: Option<Side>,
+ /// Whether the merge event sits on the partition its key selects.
+ pub placement: Option<(Side, bool)>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
+pub enum PartitionAgreement {
+ Same,
+ Different,
+}
+
+impl PartitionAgreement {
+ pub fn label(self) -> &'static str {
+ match self {
+ PartitionAgreement::Same => "same",
+ PartitionAgreement::Different => "different",
+ }
+ }
+}
+
+#[derive(Debug, Default, Clone, PartialEq, Eq)]
+pub struct SweepReport {
+ pub mapping_outcomes: BTreeMap<MappingOutcome, u64>,
+ pub merge_outcomes: BTreeMap<SurvivorOutcome, u64>,
+ pub merge_partitions: BTreeMap<PartitionAgreement, u64>,
+ pub overflow_evictions: u64,
+ pub warmup_evictions: u64,
+ pub pending_mappings: usize,
+ pub pending_merges: usize,
+ /// The global watermark the sweep settled against.
+ pub watermark_ms: Option<i64>,
+ /// Each feed's own watermark, indexed by [`side_index`].
+ pub feed_watermarks_ms: [Option<i64>; 2],
+}
+
+pub fn side_index(side: Side) -> usize {
+ match side {
+ Side::Node => 0,
+ Side::Personhog => 1,
+ }
+}
+
+/// Metrics carry only outcome labels; this is where an operator sees an example.
+#[derive(Clone)]
+pub struct SampleRing {
+ inner: Arc<Mutex<VecDeque<Value>>>,
+ capacity: usize,
+}
+
+impl SampleRing {
+ pub fn new(capacity: usize) -> Self {
+ Self {
+ inner: Arc::new(Mutex::new(VecDeque::with_capacity(capacity))),
+ capacity,
+ }
+ }
+
+ fn push(&self, sample: Value) {
+ if self.capacity == 0 {
+ return;
+ }
+ let mut ring = self
+ .inner
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner());
+ if ring.len() == self.capacity {
+ ring.pop_front();
+ }
+ ring.push_back(sample);
+ }
+
+ pub fn snapshot(&self) -> Vec<Value> {
+ self.inner
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner())
+ .iter()
+ .cloned()
+ .collect()
+ }
+}
+
+struct PartitionProgress {
+ max_timestamp_ms: i64,
+ last_seen: Instant,
+}
+
+#[derive(Default)]
+struct TopicProgress {
+ partitions: HashMap<i32, PartitionProgress>,
+ partition_count: Option<u32>,
+}
+
+impl TopicProgress {
+ /// The slowest of the partitions that delivered within the idle window.
+ fn active_min(&self, now: Instant, idle: Duration) -> Option<i64> {
+ self.partitions
+ .values()
+ .filter(|progress| now.duration_since(progress.last_seen) <= idle)
+ .map(|progress| progress.max_timestamp_ms)
+ .min()
+ }
+
+ fn newest(&self) -> Option<i64> {
+ self.partitions
+ .values()
+ .map(|progress| progress.max_timestamp_ms)
+ .max()
+ }
+
+ /// The newest message, moved forward by the time since it arrived.
+ fn newest_advanced(&self, now: Instant) -> Option<i64> {
+ self.partitions
+ .values()
+ .map(|progress| {
+ progress.max_timestamp_ms
+ + now.duration_since(progress.last_seen).as_millis() as i64
+ })
+ .max()
+ }
+}
+
+struct MappingEntry {
+ last_update_ms: i64,
+}
+
+struct MergeEntry {
+ last_update_ms: i64,
+ partitions: [Option<i32>; 2],
+ /// Each feed's recorded target, released when the entry is evicted.
+ targets: [Option<PersonKey>; 2],
+ /// Judged already; kept while a pending merge still points at it.
+ settled: bool,
+}
+
+pub struct Join {
+ config: JoinConfig,
+ node: FeedState,
+ personhog: FeedState,
+ mappings: HashMap<DistinctIdKey, MappingEntry>,
+ merges: HashMap<PersonKey, MergeEntry>,
+ /// How many merge entries point at each person on either feed.
+ target_refs: HashMap<PersonKey, usize>,
+ progress: HashMap<Topic, TopicProgress>,
+ warmup_until_ms: Option<i64>,
+ samples: SampleRing,
+}
+
+impl Join {
+ pub fn new(config: JoinConfig, samples: SampleRing) -> Self {
+ Self {
+ config,
+ node: FeedState::new(),
+ personhog: FeedState::new(),
+ mappings: HashMap::new(),
+ merges: HashMap::new(),
+ target_refs: HashMap::new(),
+ progress: HashMap::new(),
+ warmup_until_ms: None,
+ samples,
+ }
+ }
+
+ fn feed(&self, side: Side) -> &FeedState {
+ match side {
+ Side::Node => &self.node,
+ Side::Personhog => &self.personhog,
+ }
+ }
+
+ fn feed_mut(&mut self, side: Side) -> &mut FeedState {
+ match side {
+ Side::Node => &mut self.node,
+ Side::Personhog => &mut self.personhog,
+ }
+ }
+
+ pub fn observe(&mut self, event: Event, now: Instant) -> ObserveReport {
+ let mut report = ObserveReport::default();
+ let (topic, partition, timestamp_ms, record) = match event {
+ Event::Assigned { topic, partitions } => {
+ self.progress.entry(topic).or_default().partition_count = Some(partitions);
+ return report;
+ }
+ Event::Arrival {
+ topic,
+ partition,
+ timestamp_ms,
+ record,
+ } => (topic, partition, timestamp_ms, record),
+ };
+
+ let progress = self.progress.entry(topic).or_default();
+ let partition_count = progress.partition_count;
+ let seen = progress
+ .partitions
+ .entry(partition)
+ .or_insert(PartitionProgress {
+ max_timestamp_ms: timestamp_ms,
+ last_seen: now,
+ });
+ seen.max_timestamp_ms = seen.max_timestamp_ms.max(timestamp_ms);
+ seen.last_seen = now;
+
+ let team_id = match &record {
+ Record::Mapping(row) => row.team_id,
+ Record::Merge(event) => event.team_id,
+ };
+ if let Some(teams) = &self.config.teams {
+ if !teams.contains(&team_id) {
+ report.filtered = true;
+ return report;
+ }
+ }
+
+ match record {
+ Record::Mapping(row) => {
+ if self.feed_mut(topic.side).observe_mapping(&row).is_some() {
+ report.version_tie = Some(topic.side);
+ }
+ let key = DistinctIdKey::new(row.team_id, row.distinct_id);
+ let entry = self.mappings.entry(key).or_insert(MappingEntry {
+ last_update_ms: timestamp_ms,
+ });
+ entry.last_update_ms = entry.last_update_ms.max(timestamp_ms);
+ }
+ Record::Merge(event) => {
+ if let Some(count) = partition_count {
+ let expected = partition_for_key(event.key().as_bytes(), count) as i32;
+ report.placement = Some((topic.side, expected == partition));
+ }
+ let side = side_index(topic.side);
+ if self.feed_mut(topic.side).observe_merge(&event).is_some() {
+ report.merge_conflict = Some(topic.side);
+ }
+ let old = PersonKey::new(event.team_id, event.old_person_uuid);
+ let target = self
+ .feed(topic.side)
+ .merged_into(&old)
+ .map(|new| PersonKey::new(event.team_id, new));
+ let entry = self.merges.entry(old).or_insert(MergeEntry {
+ last_update_ms: timestamp_ms,
+ partitions: [None, None],
+ targets: [None, None],
+ settled: false,
+ });
+ entry.last_update_ms = entry.last_update_ms.max(timestamp_ms);
+ entry.partitions[side] = Some(partition);
+ if let (None, Some(target)) = (entry.targets[side], target) {
+ entry.targets[side] = Some(target);
+ *self.target_refs.entry(target).or_default() += 1;
+ }
+ }
+ }
+ report
+ }
+
+ /// Merges settle first and mappings a little later, so a merge's
+ /// verdict can still ask which persons each feed named. Entries the
+ /// cap evicts are not counted: they never got their grace.
+ pub fn settle(&mut self, now: Instant) -> SweepReport {
+ let mut report = SweepReport {
+ feed_watermarks_ms: self.feed_watermarks(now),
+ ..Default::default()
+ };
+ if let [Some(node), Some(personhog)] = report.feed_watermarks_ms {
+ let watermark = node.min(personhog);
+ report.watermark_ms = Some(watermark);
+ let warmup_until = if self.config.warmup.is_zero() {
+ i64::MIN
+ } else {
+ *self
+ .warmup_until_ms
+ .get_or_insert(watermark + self.config.warmup.as_millis() as i64)
+ };
+ let merge_cutoff = watermark - self.config.grace.as_millis() as i64;
+ let mapping_cutoff = merge_cutoff - self.config.mapping_lag.as_millis() as i64;
+ let due: Vec<(PersonKey, i64)> = self
+ .merges
+ .iter()
+ .filter(|(_, entry)| !entry.settled && entry.last_update_ms <= merge_cutoff)
+ .map(|(old, entry)| (*old, entry.last_update_ms))
+ .collect();
+ for (old, last_update_ms) in due {
+ if last_update_ms < warmup_until {
+ self.evict_merge(old);
+ report.warmup_evictions += 1;
+ } else {
+ self.settle_merge(old, &mut report);
+ self.release_merge(old);
+ }
+ }
+ let due: Vec<(DistinctIdKey, i64)> = self
+ .mappings
+ .iter()
+ .filter(|(_, entry)| entry.last_update_ms <= mapping_cutoff)
+ .map(|(key, entry)| (key.clone(), entry.last_update_ms))
+ .collect();
+ for (key, last_update_ms) in due {
+ if last_update_ms < warmup_until {
+ self.evict_mapping(&key);
+ report.warmup_evictions += 1;
+ } else {
+ self.settle_mapping(&key, &mut report);
+ }
+ }
+ }
+ self.enforce_cap(&mut report);
+ report.pending_mappings = self.mappings.len();
+ report.pending_merges = self.merges.values().filter(|entry| !entry.settled).count();
+ report
+ }
+
+ /// Only the distinct id topic counts: the sparse merge topic would hold
+ /// the watermark an idle window behind. A quiet feed advances by the
+ /// wall clock only while the other feed is quiet too; next to a live
+ /// feed it has stalled, so its watermark holds and nothing settles.
+ fn feed_watermarks(&self, now: Instant) -> [Option<i64>; 2] {
+ let idle = self.config.idle_partition;
+ let progress = [Side::Node, Side::Personhog].map(|side| {
+ self.progress.get(&Topic {
+ side,
+ kind: Kind::DistinctIds,
+ })
+ });
+ let active = progress.map(|topic| topic.and_then(|topic| topic.active_min(now, idle)));
+ let both_quiet = active.iter().all(Option::is_none);
+ let mut watermarks = [None, None];
+ for index in 0..2 {
+ watermarks[index] = match (active[index], progress[index]) {
+ (Some(active), _) => Some(active),
+ (None, Some(topic)) if both_quiet => topic.newest_advanced(now),
+ (None, Some(topic)) => topic.newest(),
+ (None, None) => None,
+ };
+ }
+ watermarks
+ }
+
+ fn settle_mapping(&mut self, key: &DistinctIdKey, report: &mut SweepReport) {
+ let (node, personhog) = (self.node.mapping(key), self.personhog.mapping(key));
+ let outcome = compare_mapping(node, personhog);
+ if outcome != MappingOutcome::Agree {
+ self.samples.push(json!({
+ "kind": "mapping",
+ "team_id": key.team_id,
+ "distinct_id": key.distinct_id,
+ "outcome": outcome.label(),
+ "node": node.map(mapping_json),
+ "personhog": personhog.map(mapping_json),
+ }));
+ }
+ *report.mapping_outcomes.entry(outcome).or_default() += 1;
+ self.evict_mapping(key);
+ }
+
+ fn settle_merge(&mut self, old: PersonKey, report: &mut SweepReport) {
+ let outcome = compare_survivor(&self.node, &self.personhog, old);
+ let partitions = self
+ .merges
+ .get(&old)
+ .map_or([None, None], |entry| entry.partitions);
+ if let [Some(node), Some(personhog)] = partitions {
+ if self.merge_topics_comparable() {
+ let agreement = if node == personhog {
+ PartitionAgreement::Same
+ } else {
+ PartitionAgreement::Different
+ };
+ *report.merge_partitions.entry(agreement).or_default() += 1;
+ }
+ }
+ if outcome != SurvivorOutcome::Agree {
+ self.samples.push(json!({
+ "kind": "merge",
+ "team_id": old.team_id,
+ "old_person_uuid": old.person_uuid.to_string(),
+ "outcome": outcome.label(),
+ "node": self.node.merged_into(&old).map(|new| new.to_string()),
+ "personhog": self.personhog.merged_into(&old).map(|new| new.to_string()),
+ "partitions": partitions,
+ }));
+ }
+ *report.merge_outcomes.entry(outcome).or_default() += 1;
+ if let Some(entry) = self.merges.get_mut(&old) {
+ entry.settled = true;
+ }
+ }
+
+ /// A settled merge keeps its edge while a pending merge still points at
+ /// it: that merge's verdict follows the chain through it, and the sweep
+ /// order must not decide the verdict.
+ fn release_merge(&mut self, old: PersonKey) {
+ if self.target_refs.get(&old).copied().unwrap_or(0) == 0 {
+ self.evict_merge(old);
+ }
+ }
+
+ /// Partition numbers only compare across topics with the same count.
+ fn merge_topics_comparable(&self) -> bool {
+ let counts = [Side::Node, Side::Personhog].map(|side| {
+ self.progress
+ .get(&Topic {
+ side,
+ kind: Kind::MergeEvents,
+ })
+ .and_then(|topic| topic.partition_count)
+ });
+ matches!(counts, [Some(node), Some(personhog)] if node == personhog)
+ }
+
+ fn evict_mapping(&mut self, key: &DistinctIdKey) {
+ self.node.remove_mapping(key);
+ self.personhog.remove_mapping(key);
+ self.mappings.remove(key);
+ }
+
+ /// Releases the evicted entry's targets; a settled target nothing points
+ /// at any more goes with it.
+ fn evict_merge(&mut self, old: PersonKey) {
+ let mut stack = vec![old];
+ while let Some(old) = stack.pop() {
+ let Some(entry) = self.merges.remove(&old) else {
+ continue;
+ };
+ self.node.remove_merge(&old);
+ self.personhog.remove_merge(&old);
+ for target in entry.targets.into_iter().flatten() {
+ let Some(refs) = self.target_refs.get_mut(&target) else {
+ continue;
+ };
+ *refs -= 1;
+ if *refs == 0 {
+ self.target_refs.remove(&target);
+ if self.merges.get(&target).is_some_and(|entry| entry.settled) {
+ stack.push(target);
+ }
+ }
+ }
+ }
+ }
+
+ /// Bounds retained rows and merges, not keys: a key that keeps receiving
+ /// versions grows behind one entry. Trims to nine tenths of the cap, one
+ /// pass per tenth rather than one per sweep, by key age; every key holds
+ /// at least one row, so evicting as many keys as the excess suffices.
+ fn enforce_cap(&mut self, report: &mut SweepReport) {
+ let total = self.node.row_count() + self.personhog.row_count() + self.merges.len();
+ if total <= self.config.max_entries {
+ return;
+ }
+ let target = self.config.max_entries - self.config.max_entries / 10;
+ let keys = self.mappings.len() + self.merges.len();
+ let excess = (total - target).min(keys);
+ let mut ages: Vec<i64> = self
+ .mappings
+ .values()
+ .map(|entry| entry.last_update_ms)
+ .chain(self.merges.values().map(|entry| entry.last_update_ms))
+ .collect();
+ let (_, threshold, _) = ages.select_nth_unstable(excess - 1);
+ let threshold = *threshold;
+ let old_merges: Vec<PersonKey> = self
+ .merges
+ .iter()
+ .filter(|(_, entry)| entry.last_update_ms <= threshold)
+ .map(|(old, _)| *old)
+ .collect();
+ for old in old_merges {
+ self.evict_merge(old);
+ report.overflow_evictions += 1;
+ }
+ let old_mappings: Vec<DistinctIdKey> = self
+ .mappings
+ .iter()
+ .filter(|(_, entry)| entry.last_update_ms <= threshold)
+ .map(|(key, _)| key.clone())
+ .collect();
+ for key in old_mappings {
+ self.evict_mapping(&key);
+ report.overflow_evictions += 1;
+ }
+ }
+}
+
+fn mapping_json(mapping: &Mapping) -> Value {
+ json!({
+ "person_uuid": mapping.person_uuid.to_string(),
+ "is_deleted": mapping.is_deleted,
+ "version": mapping.version,
+ })
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use uuid::Uuid;
+
+ const TEAM: i64 = 5;
+ const GRACE: Duration = Duration::from_secs(60);
+ const IDLE: Duration = Duration::from_secs(30);
+ const MAPPING_LAG: Duration = Duration::from_secs(2);
+
+ fn uuid(n: u128) -> Uuid {
+ Uuid::from_u128(n)
+ }
+
+ fn join_with(max_entries: usize, teams: Option<HashSet<i64>>, warmup: Duration) -> Join {
+ Join::new(
+ JoinConfig {
+ grace: GRACE,
+ idle_partition: IDLE,
+ max_entries,
+ teams,
+ mapping_lag: MAPPING_LAG,
+ warmup,
+ },
+ SampleRing::new(10),
+ )
+ }
+
+ fn join(max_entries: usize, teams: Option<HashSet<i64>>) -> Join {
+ join_with(max_entries, teams, Duration::ZERO)
+ }
+
+ fn topic(side: Side, kind: Kind) -> Topic {
+ Topic { side, kind }
+ }
+
+ fn row_of_team(
+ team_id: i64,
+ side: Side,
+ partition: i32,
+ timestamp_ms: i64,
+ distinct_id: &str,
+ person: u128,
+ version: i64,
+ ) -> Event {
+ Event::Arrival {
+ topic: topic(side, Kind::DistinctIds),
+ partition,
+ timestamp_ms,
+ record: Record::Mapping(DistinctIdRow {
+ team_id,
+ distinct_id: distinct_id.to_string(),
+ person_uuid: uuid(person),
+ version,
+ is_deleted: false,
+ }),
+ }
+ }
+
+ fn row(
+ side: Side,
+ partition: i32,
+ timestamp_ms: i64,
+ distinct_id: &str,
+ person: u128,
+ ) -> Event {
+ row_of_team(TEAM, side, partition, timestamp_ms, distinct_id, person, 0)
+ }
+
+ fn assigned(join: &mut Join, kind: Kind, counts: [u32; 2], at: Instant) {
+ for (side, partitions) in [Side::Node, Side::Personhog].into_iter().zip(counts) {
+ join.observe(
+ Event::Assigned {
+ topic: topic(side, kind),
+ partitions,
+ },
+ at,
+ );
+ }
+ }
+
+ fn merge(side: Side, partition: i32, timestamp_ms: i64, old: u128, new: u128) -> Event {
+ Event::Arrival {
+ topic: topic(side, Kind::MergeEvents),
+ partition,
+ timestamp_ms,
+ record: Record::Merge(MergeEvent {
+ team_id: TEAM,
+ old_person_uuid: uuid(old),
+ new_person_uuid: uuid(new),
+ merged_at_ms: timestamp_ms,
+ }),
+ }
+ }
+
+ fn mapping_outcomes(report: &SweepReport) -> Vec<(MappingOutcome, u64)> {
+ report
+ .mapping_outcomes
+ .iter()
+ .map(|(k, v)| (*k, *v))
+ .collect()
+ }
+
+ fn both(
+ join: &mut Join,
+ partition: i32,
+ timestamp_ms: i64,
+ distinct_id: &str,
+ person: u128,
+ at: Instant,
+ ) {
+ for side in [Side::Node, Side::Personhog] {
+ join.observe(row(side, partition, timestamp_ms, distinct_id, person), at);
+ }
+ }
+
+ // Nothing settles until the slower feed passes the grace; the merge
+ // topics never count towards the watermark.
+ #[test]
+ fn entries_settle_behind_the_slower_feed_after_the_grace() {
+ let t0 = Instant::now();
+ let mut join = join(usize::MAX, None);
+ assigned(&mut join, Kind::MergeEvents, [8, 8], t0);
+ join.observe(row(Side::Node, 0, 1_000, "d", 1), t0);
+ join.observe(row(Side::Node, 0, 2_000, "only", 2), t0);
+ join.observe(row(Side::Node, 0, 70_000, "ahead", 3), t0);
+ join.observe(merge(Side::Node, 3, 1_000, 7, 8), t0);
+ join.observe(row(Side::Personhog, 0, 1_500, "d", 1), t0);
+ join.observe(merge(Side::Personhog, 4, 1_200, 7, 8), t0);
+
+ let report = join.settle(t0);
+ assert_eq!(report.watermark_ms, Some(1_500));
+ assert_eq!(report.mapping_outcomes, BTreeMap::new());
+ assert_eq!(report.pending_mappings, 3);
+ assert_eq!(report.pending_merges, 1);
+
+ join.observe(row(Side::Personhog, 0, 70_000, "ahead", 3), t0);
+ let report = join.settle(t0);
+ assert_eq!(report.watermark_ms, Some(70_000));
+ assert_eq!(
+ mapping_outcomes(&report),
+ vec![(MappingOutcome::Agree, 1), (MappingOutcome::NodeOnly, 1)]
+ );
+ assert_eq!(
+ report.merge_outcomes,
+ BTreeMap::from([(SurvivorOutcome::Agree, 1)])
+ );
+ assert_eq!(
+ report.merge_partitions,
+ BTreeMap::from([(PartitionAgreement::Different, 1)])
+ );
+ assert_eq!(report.pending_mappings, 1);
+ assert_eq!(report.pending_merges, 0);
+ assert_eq!(join.node.mapping_count(), 1);
+ assert_eq!(join.personhog.mapping_count(), 1);
+ assert_eq!(join.node.merge_count() + join.personhog.merge_count(), 0);
+ let samples = join.samples.snapshot();
+ assert_eq!(samples.len(), 1);
+ assert_eq!(samples[0]["distinct_id"], "only");
+ assert_eq!(samples[0]["outcome"], "node_only");
+ }
+
+ // Topics with different counts place the same key on different
+ // partition numbers; only the per-feed placement check applies there.
+ #[test]
+ fn merge_partitions_compare_only_across_equal_counts() {
+ let t0 = Instant::now();
+ let mut join = join(usize::MAX, None);
+ assigned(&mut join, Kind::MergeEvents, [1, 4], t0);
+ join.observe(merge(Side::Node, 0, 1_000, 7, 8), t0);
+ join.observe(merge(Side::Personhog, 3, 1_000, 7, 8), t0);
+ both(&mut join, 0, 70_000, "w", 9, t0);
+
+ let report = join.settle(t0);
+ assert_eq!(
+ report.merge_outcomes,
+ BTreeMap::from([(SurvivorOutcome::Agree, 1)])
+ );
+ assert_eq!(report.merge_partitions, BTreeMap::new());
+ }
+
+ // A lull on both feeds walks the watermark forward.
+ #[test]
+ fn idle_partitions_release_the_watermark() {
+ let t0 = Instant::now();
+ let mut join = join(usize::MAX, None);
+ both(&mut join, 0, 1_000, "a", 1, t0);
+ let later = t0 + Duration::from_secs(40);
+ both(&mut join, 1, 70_000, "b", 2, later);
+
+ let report = join.settle(later);
+ assert_eq!(report.watermark_ms, Some(70_000));
+ assert_eq!(mapping_outcomes(&report), vec![(MappingOutcome::Agree, 1)]);
+
+ let report = join.settle(later + Duration::from_secs(62));
+ assert_eq!(report.watermark_ms, Some(132_000));
+ assert_eq!(mapping_outcomes(&report), vec![(MappingOutcome::Agree, 1)]);
+ }
+
+ // A feed quiet next to a live one has stalled: its watermark holds, so
+ // nothing settles against its absence.
+ #[test]
+ fn a_stalled_feed_holds_the_watermark() {
+ let t0 = Instant::now();
+ let mut join = join(usize::MAX, None);
+ both(&mut join, 0, 1_000, "a", 1, t0);
+ let stalled = t0 + Duration::from_secs(100);
+ join.observe(
+ row(Side::Node, 0, 70_000, "b", 2),
+ t0 + Duration::from_secs(40),
+ );
+ join.observe(row(Side::Node, 0, 130_000, "c", 3), stalled);
+
+ let report = join.settle(stalled);
+ assert_eq!(report.feed_watermarks_ms, [Some(130_000), Some(1_000)]);
+ assert_eq!(report.watermark_ms, Some(1_000));
+ assert_eq!(report.mapping_outcomes, BTreeMap::new());
+ assert_eq!(report.pending_mappings, 3);
+
+ let resumed = stalled + Duration::from_secs(1);
+ join.observe(row(Side::Personhog, 0, 70_000, "b", 2), resumed);
+ join.observe(row(Side::Personhog, 0, 130_000, "c", 3), resumed);
+ let report = join.settle(resumed);
+ assert_eq!(report.watermark_ms, Some(130_000));
+ assert_eq!(mapping_outcomes(&report), vec![(MappingOutcome::Agree, 1)]);
+ assert_eq!(report.pending_mappings, 2);
+ }
+
+ // A counterpart may predate the readers; the first grace is not judged.
+ #[test]
+ fn the_warmup_drops_entries_whose_counterparts_may_predate_the_readers() {
+ let t0 = Instant::now();
+ let mut join = join_with(usize::MAX, None, GRACE);
+ join.observe(row(Side::Node, 0, 1_000, "x", 1), t0);
+ join.observe(row(Side::Personhog, 0, 1_000, "y", 2), t0);
+ let report = join.settle(t0);
+ assert_eq!(report.watermark_ms, Some(1_000));
+
+ both(&mut join, 0, 130_000, "z", 3, t0);
+ let report = join.settle(t0);
+ assert_eq!(report.warmup_evictions, 2);
+ assert_eq!(report.mapping_outcomes, BTreeMap::new());
+ assert_eq!(report.pending_mappings, 1);
+ assert_eq!(join.node.mapping_count(), 1);
+ assert_eq!(join.personhog.mapping_count(), 1);
+ }
+
+ // Evicted entries never had their grace, so they get no verdict.
+ #[test]
+ fn the_cap_evicts_the_oldest_entries_without_settling_them() {
+ let t0 = Instant::now();
+ let mut join = join(3, None);
+ join.observe(row(Side::Node, 0, 3_000, "newest", 1), t0);
+ join.observe(row(Side::Node, 0, 1_000, "oldest", 2), t0);
+ join.observe(merge(Side::Node, 0, 2_000, 7, 8), t0);
+ join.observe(row(Side::Personhog, 0, 3_000, "newest", 1), t0);
+
+ let report = join.settle(t0);
+ assert_eq!(report.mapping_outcomes, BTreeMap::new());
+ assert_eq!(report.overflow_evictions, 1);
+ assert_eq!(report.pending_mappings, 1);
+ assert_eq!(report.pending_merges, 1);
+ assert!(join
+ .mappings
+ .contains_key(&DistinctIdKey::new(TEAM, "newest")));
+ assert_eq!(join.node.mapping_count(), 1);
+
+ // A key's history counts: three versions of one key weigh three rows.
+ let mut hot = Join::new(
+ JoinConfig {
+ grace: GRACE,
+ idle_partition: IDLE,
+ max_entries: 4,
+ teams: None,
+ mapping_lag: MAPPING_LAG,
+ warmup: Duration::ZERO,
+ },
+ SampleRing::new(10),
+ );
+ for (version, person) in [(0, 1), (1, 2), (2, 3)] {
+ let at = 1_000 + version * 100;
+ hot.observe(
+ row_of_team(TEAM, Side::Node, 0, at, "a", person, version),
+ t0,
+ );
+ }
+ hot.observe(row(Side::Personhog, 0, 1_000, "a", 3), t0);
+ hot.observe(row(Side::Node, 0, 5_000, "b", 4), t0);
+ let report = hot.settle(t0);
+ assert_eq!(report.overflow_evictions, 1);
+ assert!(hot.mappings.contains_key(&DistinctIdKey::new(TEAM, "b")));
+ assert_eq!(hot.node.row_count(), 1);
+ }
+
+ // The lag keeps the rows a merge's verdict reads when they are older
+ // than the merge; the order keeps them when both fall due in one sweep.
+ #[test]
+ fn mappings_outlive_the_merges_they_explain() {
+ let t0 = Instant::now();
+ let mut join = join(usize::MAX, None);
+ join.observe(
+ row_of_team(TEAM, Side::Personhog, 0, 1_000, "anon", 1, 0),
+ t0,
+ );
+ join.observe(
+ row_of_team(TEAM, Side::Personhog, 0, 1_000, "anon", 2, 1),
+ t0,
+ );
+ join.observe(merge(Side::Personhog, 0, 1_500, 1, 2), t0);
+ join.observe(row(Side::Node, 0, 1_000, "anon", 2), t0);
+ join.observe(row_of_team(TEAM, Side::Personhog, 0, 1_000, "b", 3, 0), t0);
+ join.observe(row_of_team(TEAM, Side::Personhog, 0, 1_000, "b", 4, 1), t0);
+ join.observe(merge(Side::Personhog, 0, 1_500, 3, 4), t0);
+ join.observe(row(Side::Node, 0, 1_000, "b", 4), t0);
+
+ // The rows of `anon` are due for a lag of zero, the merge is not yet.
+ both(&mut join, 0, 61_200, "w1", 9, t0);
+ let report = join.settle(t0);
+ assert_eq!(report.merge_outcomes, BTreeMap::new());
+ assert_eq!(report.mapping_outcomes, BTreeMap::new());
+
+ both(&mut join, 0, 61_600, "w2", 9, t0);
+ let report = join.settle(t0);
+ assert_eq!(
+ report.merge_outcomes,
+ BTreeMap::from([(SurvivorOutcome::PersonhogStub, 2)])
+ );
+ assert_eq!(report.mapping_outcomes, BTreeMap::new());
+
+ both(&mut join, 0, 70_000, "w3", 9, t0);
+ let report = join.settle(t0);
+ assert_eq!(mapping_outcomes(&report), vec![(MappingOutcome::Agree, 2)]);
+ }
+
+ // Node merges A into C directly; personhog goes through its own B, whose
+ // merge is older and settles first. A's verdict still reaches C.
+ #[test]
+ fn merge_edges_outlive_the_merges_that_depend_on_them() {
+ let t0 = Instant::now();
+ let mut join = join(usize::MAX, None);
+ join.observe(merge(Side::Node, 0, 1_000, 1, 3), t0);
+ join.observe(merge(Side::Personhog, 0, 500, 2, 3), t0);
+ join.observe(merge(Side::Personhog, 0, 1_000, 1, 2), t0);
+
+ both(&mut join, 0, 60_600, "w1", 9, t0);
+ let report = join.settle(t0);
+ assert_eq!(report.merge_outcomes.values().sum::<u64>(), 1);
+ assert!(!report
+ .merge_outcomes
+ .contains_key(&SurvivorOutcome::Mismatch));
+ assert_eq!(report.pending_merges, 1);
+ assert_eq!(
+ join.personhog.merged_into(&PersonKey::new(TEAM, uuid(2))),
+ Some(uuid(3))
+ );
+
+ both(&mut join, 0, 61_100, "w2", 9, t0);
+ let report = join.settle(t0);
+ assert_eq!(
+ report.merge_outcomes,
+ BTreeMap::from([(SurvivorOutcome::Agree, 1)])
+ );
+ assert_eq!(join.node.merge_count() + join.personhog.merge_count(), 0);
+ assert!(join.merges.is_empty());
+ assert!(join.target_refs.is_empty());
+ }
+
+ #[test]
+ fn observe_reports_filters_ties_conflicts_and_placement() {
+ let t0 = Instant::now();
+ let mut join = join(usize::MAX, Some(HashSet::from([TEAM])));
+ assert!(
+ join.observe(row_of_team(TEAM + 1, Side::Node, 0, 1_000, "f", 1, 0), t0)
+ .filtered
+ );
+ assert_eq!(join.mappings.len(), 0);
+
+ assert_eq!(
+ join.observe(row(Side::Node, 0, 1_000, "d", 1), t0),
+ ObserveReport::default()
+ );
+ assert_eq!(
+ join.observe(row(Side::Node, 0, 1_000, "d", 2), t0)
+ .version_tie,
+ Some(Side::Node)
+ );
+
+ // Placement is only judged once the topic's partition count is known.
+ let key = format!("{TEAM}:{}", uuid(7));
+ let expected = partition_for_key(key.as_bytes(), 64) as i32;
+ assert_eq!(
+ join.observe(merge(Side::Node, expected, 1_000, 7, 8), t0)
+ .placement,
+ None
+ );
+ join.observe(
+ Event::Assigned {
+ topic: topic(Side::Personhog, Kind::MergeEvents),
+ partitions: 64,
+ },
+ t0,
+ );
+ assert_eq!(
+ join.observe(merge(Side::Personhog, expected, 1_000, 7, 8), t0)
+ .placement,
+ Some((Side::Personhog, true))
+ );
+ let report = join.observe(merge(Side::Personhog, expected + 1, 1_000, 7, 9), t0);
+ assert_eq!(report.placement, Some((Side::Personhog, false)));
+ assert_eq!(report.merge_conflict, Some(Side::Personhog));
+ }
+}
diff --git a/rust/personhog-parity-monitor/src/lib.rs b/rust/personhog-parity-monitor/src/lib.rs
new file mode 100644
index 000000000000..cedd5bce3395
--- /dev/null
+++ b/rust/personhog-parity-monitor/src/lib.rs
@@ -0,0 +1,9 @@
+//! Follows the Node Postgres path's and `personhog-publisher`'s person CDC
+//! feeds and reports as metrics whether they end in the same state. State
+//! is in memory under a settle window; the rules are `cdc_parity`.
+
+pub mod config;
+pub mod feed;
+pub mod join;
+pub mod pipeline;
+pub mod server;
diff --git a/rust/personhog-parity-monitor/src/main.rs b/rust/personhog-parity-monitor/src/main.rs
new file mode 100644
index 000000000000..cf5417be33c4
--- /dev/null
+++ b/rust/personhog-parity-monitor/src/main.rs
@@ -0,0 +1,109 @@
+use std::time::Duration;
+
+use anyhow::{Context, Result};
+use envconfig::Envconfig;
+use health::HealthRegistry;
+use tokio::sync::mpsc;
+use tracing::info;
+use tracing::level_filters::LevelFilter;
+use tracing_subscriber::layer::SubscriberExt;
+use tracing_subscriber::util::SubscriberInitExt;
+use tracing_subscriber::{fmt, EnvFilter};
+
+use personhog_parity_monitor::config::Config;
+use personhog_parity_monitor::feed::TopicReader;
+use personhog_parity_monitor::join::{Event, Join, JoinConfig, Kind, SampleRing};
+use personhog_parity_monitor::{pipeline, server};
+
+common_alloc::used!();
+
+/// A reader or the join that reports nothing for this long is dead.
+const LIVENESS_DEADLINE: Duration = Duration::from_secs(60);
+const EVENT_QUEUE: usize = 10_000;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+ let config = Config::init_from_env().context("reading the configuration")?;
+ tracing_subscriber::registry()
+ .with(fmt::layer().with_target(true).with_level(true))
+ .with(
+ EnvFilter::builder()
+ .with_default_directive(LevelFilter::INFO.into())
+ .from_env_lossy(),
+ )
+ .init();
+
+ let join_config = JoinConfig {
+ grace: config.grace(),
+ idle_partition: config.idle_partition(),
+ max_entries: config.parity_max_entries,
+ teams: config.teams().map_err(anyhow::Error::msg)?,
+ mapping_lag: config.mapping_lag(),
+ warmup: config.warmup(),
+ };
+ info!(
+ grace_seconds = config.parity_grace_seconds,
+ max_entries = config.parity_max_entries,
+ teams = ?join_config.teams,
+ debug_api = config.debug_api_secret().is_some(),
+ "starting personhog-parity-monitor"
+ );
+
+ let health = HealthRegistry::new("personhog-parity-monitor");
+ let samples = SampleRing::new(config.parity_sample_ring_size);
+ // Building the router installs the metrics recorder; counter handles the
+ // tasks create before that would bind to the no-op recorder for good.
+ let router = server::router(
+ health.clone(),
+ samples.clone(),
+ config.debug_api_secret().map(str::to_string),
+ );
+ let (events, arrivals) = mpsc::channel(EVENT_QUEUE);
+
+ // The join drains the queue, so it runs before the first reader fills it.
+ let join_health = health.register("join".to_string(), LIVENESS_DEADLINE).await;
+ tokio::spawn(pipeline::run(
+ Join::new(join_config, samples),
+ arrivals,
+ config.sweep_interval(),
+ join_health,
+ ));
+
+ for feed in config.feeds() {
+ for kind in [Kind::DistinctIds, Kind::MergeEvents] {
+ let feed_for_reader = feed.clone();
+ // rdkafka's metadata fetch and assignment block the thread.
+ let (reader, partitions) =
+ tokio::task::spawn_blocking(move || TopicReader::connect(&feed_for_reader, kind))
+ .await
+ .context("the reader setup task panicked")??;
+ let topic = reader.topic();
+ info!(
+ feed = topic.side.label(),
+ topic = topic.kind.label(),
+ partitions,
+ hosts = %feed.kafka_hosts,
+ "assigned every partition at the end of the log"
+ );
+ events
+ .send(Event::Assigned { topic, partitions })
+ .await
+ .context("the join is gone")?;
+ let handle = health
+ .register(
+ format!("{}-{}", topic.side.label(), topic.kind.label()),
+ LIVENESS_DEADLINE,
+ )
+ .await;
+ tokio::spawn(reader.run(events.clone(), handle));
+ }
+ }
+ drop(events);
+
+ let bind = config.bind();
+ info!(%bind, "serving metrics");
+ common_metrics::serve(router, &bind)
+ .await
+ .context("serving metrics")?;
+ Ok(())
+}
diff --git a/rust/personhog-parity-monitor/src/pipeline.rs b/rust/personhog-parity-monitor/src/pipeline.rs
new file mode 100644
index 000000000000..50ccff9f0ad3
--- /dev/null
+++ b/rust/personhog-parity-monitor/src/pipeline.rs
@@ -0,0 +1,124 @@
+//! Owns the join and turns its reports into metrics, which carry outcome
+//! labels only, never a team or an id.
+
+use std::collections::HashMap;
+use std::time::{Duration, Instant};
+
+use health::HealthHandle;
+use personhog_common::cdc_parity::Side;
+use tokio::sync::mpsc;
+use tokio::time::MissedTickBehavior;
+
+use crate::feed::now_ms;
+use crate::join::{side_index, Event, Join, Kind, ObserveReport, SweepReport, Topic};
+
+pub async fn run(
+ mut join: Join,
+ mut events: mpsc::Receiver<Event>,
+ sweep_interval: Duration,
+ health: HealthHandle,
+) {
+ let mut ticker = tokio::time::interval(sweep_interval);
+ ticker.set_missed_tick_behavior(MissedTickBehavior::Delay);
+ let mut partition_counts: HashMap<Topic, u32> = HashMap::new();
+ loop {
+ tokio::select! {
+ event = events.recv() => {
+ let Some(event) = event else {
+ return;
+ };
+ if let Event::Assigned { topic, partitions } = &event {
+ partition_counts.insert(*topic, *partitions);
+ record_partition_gauges(&partition_counts);
+ }
+ let topic = event.topic();
+ let report = join.observe(event, Instant::now());
+ record_observation(topic, &report);
+ }
+ _ = ticker.tick() => {
+ let report = join.settle(Instant::now());
+ record_sweep(&report);
+ health.report_healthy().await;
+ }
+ }
+ }
+}
+
+fn record_partition_gauges(counts: &HashMap<Topic, u32>) {
+ for (topic, count) in counts {
+ metrics::gauge!(
+ "personhog_parity_topic_partitions",
+ "feed" => topic.side.label(),
+ "topic" => topic.kind.label()
+ )
+ .set(*count as f64);
+ }
+ for kind in [Kind::DistinctIds, Kind::MergeEvents] {
+ let pair = [Side::Node, Side::Personhog].map(|side| counts.get(&Topic { side, kind }));
+ if let [Some(node), Some(personhog)] = pair {
+ metrics::gauge!(
+ "personhog_parity_topic_pair_partitions_equal",
+ "topic" => kind.label()
+ )
+ .set(if node == personhog { 1.0 } else { 0.0 });
+ }
+ }
+}
+
+fn record_observation(topic: Topic, report: &ObserveReport) {
+ let feed = topic.side.label();
+ if report.filtered {
+ metrics::counter!("personhog_parity_filtered_total", "feed" => feed, "topic" => topic.kind.label())
+ .increment(1);
+ }
+ if let Some(side) = report.version_tie {
+ metrics::counter!("personhog_parity_version_ties_total", "feed" => side.label())
+ .increment(1);
+ }
+ if let Some(side) = report.merge_conflict {
+ metrics::counter!("personhog_parity_merge_conflicts_total", "feed" => side.label())
+ .increment(1);
+ }
+ if let Some((side, expected)) = report.placement {
+ metrics::counter!(
+ "personhog_parity_merge_placement_total",
+ "feed" => side.label(),
+ "outcome" => if expected { "expected" } else { "unexpected" }
+ )
+ .increment(1);
+ }
+}
+
+fn record_sweep(report: &SweepReport) {
+ for (outcome, count) in &report.mapping_outcomes {
+ metrics::counter!("personhog_parity_settled_mappings_total", "outcome" => outcome.label())
+ .increment(*count);
+ }
+ for (outcome, count) in &report.merge_outcomes {
+ metrics::counter!("personhog_parity_settled_merges_total", "outcome" => outcome.label())
+ .increment(*count);
+ }
+ for (agreement, count) in &report.merge_partitions {
+ metrics::counter!("personhog_parity_merge_partition_total", "outcome" => agreement.label())
+ .increment(*count);
+ }
+ if report.overflow_evictions > 0 {
+ metrics::counter!("personhog_parity_overflow_evictions_total")
+ .increment(report.overflow_evictions);
+ }
+ if report.warmup_evictions > 0 {
+ metrics::counter!("personhog_parity_warmup_evictions_total")
+ .increment(report.warmup_evictions);
+ }
+ metrics::gauge!("personhog_parity_pending_entries", "kind" => "mappings")
+ .set(report.pending_mappings as f64);
+ metrics::gauge!("personhog_parity_pending_entries", "kind" => "merges")
+ .set(report.pending_merges as f64);
+ let now = now_ms();
+ for side in [Side::Node, Side::Personhog] {
+ if let Some(watermark) = report.feed_watermarks_ms[side_index(side)] {
+ metrics::gauge!("personhog_parity_watermark_age_seconds", "feed" => side.label())
+ .set((now - watermark).max(0) as f64 / 1000.0);
+ }
+ }
+}
diff --git a/rust/personhog-parity-monitor/src/server.rs b/rust/personhog-parity-monitor/src/server.rs
new file mode 100644
index 000000000000..1706fe458b16
--- /dev/null
+++ b/rust/personhog-parity-monitor/src/server.rs
@@ -0,0 +1,56 @@
+use axum::http::{HeaderMap, StatusCode};
+use axum::response::{IntoResponse, Response};
+use axum::routing::get;
+use axum::{Json, Router};
+use common_metrics::setup_metrics_routes;
+use health::HealthRegistry;
+
+use crate::join::SampleRing;
+
+const DEBUG_API_SECRET_HEADER: &str = "x-debug-api-secret";
+
+/// The mismatch ring carries distinct ids, so it answers only with the shared secret.
+pub fn router(
+ health: HealthRegistry,
+ samples: SampleRing,
+ debug_api_secret: Option<String>,
+) -> Router {
+ let readiness = health.clone();
+ let router = Router::new()
+ .route(
+ "/_readiness",
+ get(move || {
+ let health = readiness.clone();
+ async move { health.get_status() }
+ }),
+ )
+ .route(
+ "/_liveness",
+ get(move || {
+ let health = health.clone();
+ async move { health.get_status() }
+ }),
+ )
+ .route(
+ "/debug/mismatches",
+ get(move |headers: HeaderMap| {
+ let samples = samples.clone();
+ let secret = debug_api_secret.clone();
+ async move { mismatches(&headers, secret.as_deref(), &samples) }
+ }),
+ );
+ setup_metrics_routes(router)
+}
+
+fn mismatches(headers: &HeaderMap, secret: Option<&str>, samples: &SampleRing) -> Response {
+ let Some(secret) = secret else {
+ return StatusCode::NOT_FOUND.into_response();
+ };
+ let presented = headers
+ .get(DEBUG_API_SECRET_HEADER)
+ .and_then(|value| value.to_str().ok());
+ if presented != Some(secret) {
+ return StatusCode::UNAUTHORIZED.into_response();
+ }
+ Json(samples.snapshot()).into_response()
+}