Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(ingestion): show parity monitor disagreements in the control plane (#120360)

José Sequeira17d3b5bc256f
diff --git a/rust/ingestion-control-plane/README.md b/rust/ingestion-control-plane/README.md
index b6cfff3a58c1..3e1ba3300402 100644
--- a/rust/ingestion-control-plane/README.md
+++ b/rust/ingestion-control-plane/README.md
@@ -25,6 +25,10 @@ Lists live ingestion-consumer pods and serves the consumer's routing debug UI pe
Protocol-aware view of personhog's etcd coordination state (`PERSONHOG_ETCD_PREFIX`, default `/personhog/`): the whole prefix is read in one consistent snapshot and interpreted with the `personhog-coordination` types and the coordinator's own pure predicates, so the derived diagnostics cannot drift from the protocol. Shows the elected coordinator, registered pods and routers (with remaining lease TTLs), per-partition ownership, and in-flight handoffs with their phase age and what each is waiting on — distinguishing a stuck participant ("waiting on freeze acks from router-2") from a stuck coordinator ("quorum met; waiting on the coordinator to advance"). Derived issues cover stuck handoffs past their per-phase deadline, assignments owned by unregistered pods with no replanning handoff, lingering `Complete` records the coordinator hasn't cleaned up, dead new owners awaiting cancellation, unassigned partitions, stale acks from previous handoff attempts, and unparseable records. The deadline knobs (`PERSONHOG_HANDOFF_DEADLINE_SECS`, `PERSONHOG_WARMING_DEADLINE_SECS`) must match the coordinator's to agree with its cancellation decisions. Read-only: the tool never writes coordination state, and a force-reassignment must go through a handoff record, never a raw assignment write.
+### Personhog parity
+
+Shows the personhog parity monitor's recent disagreements between the Node and personhog CDC feeds, newest first. A distinct id row shows the mapping each feed ended with, and a merge row shows each feed's survivor and partition, with the differing fields highlighted. The server reads the monitor's `/debug/mismatches` sample with `DEBUG_API_SECRET`, so the monitor must run with the same secret. The monitor keeps the sample in memory, so older disagreements drop out and a monitor restart clears it. The view is disabled (its API returns 503) unless `PARITY_MONITOR_URL` is set.
+
### Etcd explorer
Generic key browser for the same etcd cluster: keys under a prefix render as a collapsible tree (with per-key version, lease, and size; the top level and single-child chains expand automatically), and a key's detail shows its value, revisions, and remaining lease TTL, with edit, create, and delete. Operations are bounded to `ETCD_ALLOWED_PREFIXES`. Writes bypass every coordination protocol, so each one confirms first; writing a lease-backed key warns that it detaches the key from its lease, and writing a personhog `assignments/` key gets an extra warning because routers only observe ownership through handoff `Complete` events — a raw assignment write is invisible to them and leaves an unfenced old owner serving. Deletes are single-key only; there is deliberately no prefix delete.
@@ -55,6 +59,8 @@ Both etcd tools are disabled (their APIs return 503) unless `ETCD_ENDPOINTS` is
| `POD_LABEL_SELECTORS` | `ingestion-analytics-main/app=ingestion-analytics-main,ingestion-analytics-async/app=ingestion-analytics-async` | One `namespace/key=value` per entry (each lane runs in its own namespace); bare `key=value` uses `K8S_NAMESPACE` |
| `K8S_NAMESPACE` | `posthog` | Default namespace for unqualified selector entries |
| `DEBUG_PORT` | `3301` | Consumer debug API port (kubernetes mode) |
+| `DEBUG_API_SECRET` | empty | Sent as `X-Debug-Api-Secret` to the consumers' debug APIs and to the parity monitor; must match theirs |
+| `PARITY_MONITOR_URL` | empty | Parity monitor base URL, for example `http://personhog-parity-monitor.personhog-parity-monitor.svc.cluster.local:3301`; empty disables the parity view |
| `ETCD_ENDPOINTS` | empty | Comma-separated etcd endpoints; empty disables the etcd explorer and personhog topology tools |
| `ETCD_ALLOWED_PREFIXES` | `/` | Comma-separated key prefixes the etcd explorer may read and write |
| `PERSONHOG_ETCD_PREFIX` | `/personhog/` | Prefix the personhog coordination state lives under; must match the personhog services' `ETCD_PREFIX` |
@@ -83,4 +89,4 @@ Then open `http://localhost:3305/`.
## Deployment notes
- Image `ingestion-control-plane` is built via `.github/rust-images.yml`; a deploy needs a matrix entry in `rust-docker-build.yml` plus a charts-repo app.
-- Kubernetes mode needs RBAC: `get`/`list` on `pods` in the ingestion namespace, and network egress to pod IPs on the debug port, Kafka, the Postgres replica, and (when the etcd tools are enabled) the personhog etcd cluster.
+- Kubernetes mode needs RBAC: `get`/`list` on `pods` in the ingestion namespace, and network egress to pod IPs on the debug port, Kafka, the Postgres replica, (when the etcd tools are enabled) the personhog etcd cluster, and (when the parity view is enabled) the parity monitor's Service.
diff --git a/rust/ingestion-control-plane/src/api.rs b/rust/ingestion-control-plane/src/api.rs
index 16dfda8f3a25..cff55de0d597 100644
--- a/rust/ingestion-control-plane/src/api.rs
+++ b/rust/ingestion-control-plane/src/api.rs
@@ -19,6 +19,7 @@ use crate::jobs::{AnalysisRequest, JobView};
use crate::kafka::browse::{self, BrowseParams, BrowseStop, MessageFilter, MessageRecord};
use crate::kafka::client;
use crate::kafka::lag::{self, ConsumerTarget, GroupLag, LagOverview};
+use crate::parity;
use crate::personhog;
use crate::proxy;
use crate::state::AppState;
@@ -462,6 +463,7 @@ pub fn router(state: AppState) -> Router {
get(etcd_get_key).put(etcd_put_key).delete(etcd_delete_key),
)
.route("/api/personhog/topology", get(personhog_topology))
+ .route("/api/personhog/parity/mismatches", get(parity::mismatches))
// The consumer debug UI is served per pod; its relative `debug/...`
// fetches resolve to the proxy route below. Pods are addressed by
// namespace + name so identical names across lanes cannot collide.
diff --git a/rust/ingestion-control-plane/src/config.rs b/rust/ingestion-control-plane/src/config.rs
index 88ba808de8b1..74916b38dc87 100644
--- a/rust/ingestion-control-plane/src/config.rs
+++ b/rust/ingestion-control-plane/src/config.rs
@@ -141,6 +141,9 @@ pub struct Config {
#[envconfig(from = "DEBUG_API_SECRET")]
pub debug_api_secret: Option<String>,
+ #[envconfig(from = "PARITY_MONITOR_URL")]
+ pub parity_monitor_url: Option<String>,
+
#[envconfig(default = "10000")]
pub kafka_metadata_timeout_ms: u64,
diff --git a/rust/ingestion-control-plane/src/lib.rs b/rust/ingestion-control-plane/src/lib.rs
index 5d0ef05aeafa..8473acde5312 100644
--- a/rust/ingestion-control-plane/src/lib.rs
+++ b/rust/ingestion-control-plane/src/lib.rs
@@ -5,6 +5,7 @@ pub mod etcd;
pub mod jobs;
pub mod k8s;
pub mod kafka;
+pub mod parity;
pub mod personhog;
pub mod proxy;
pub mod state;
diff --git a/rust/ingestion-control-plane/src/parity.rs b/rust/ingestion-control-plane/src/parity.rs
new file mode 100644
index 000000000000..5c4a54846bc2
--- /dev/null
+++ b/rust/ingestion-control-plane/src/parity.rs
@@ -0,0 +1,90 @@
+use std::time::Duration;
+
+use axum::extract::State;
+use axum::Json;
+use chrono::{DateTime, Utc};
+use reqwest::StatusCode;
+use serde::Serialize;
+
+use crate::api::ApiError;
+use crate::state::AppState;
+
+/// The shared client has no timeout because the pod proxy streams SSE.
+const REQUEST_TIMEOUT: Duration = Duration::from_secs(10);
+
+#[derive(Serialize)]
+pub struct ParityMismatches {
+ pub fetched_at: DateTime<Utc>,
+ pub samples: Vec<serde_json::Value>,
+}
+
+fn mismatches_url(base: &str) -> String {
+ format!("{}/debug/mismatches", base.trim_end_matches('/'))
+}
+
+pub async fn mismatches(State(state): State<AppState>) -> Result<Json<ParityMismatches>, ApiError> {
+ let base = state
+ .config
+ .parity_monitor_url
+ .as_deref()
+ .filter(|url| !url.is_empty())
+ .ok_or_else(|| {
+ ApiError::unavailable("the parity view is disabled (PARITY_MONITOR_URL not set)")
+ })?;
+
+ let mut request = state
+ .http
+ .get(mismatches_url(base))
+ .timeout(REQUEST_TIMEOUT);
+ if let Some(secret) = state
+ .config
+ .debug_api_secret
+ .as_deref()
+ .filter(|s| !s.is_empty())
+ {
+ request = request.header("X-Debug-Api-Secret", secret);
+ }
+ let response = request
+ .send()
+ .await
+ .map_err(|e| ApiError::upstream(format!("parity monitor request failed: {e}")))?;
+
+ match response.status() {
+ StatusCode::NOT_FOUND => {
+ return Err(ApiError::unavailable(
+ "the parity monitor answered 404: its debug API is off (DEBUG_API_SECRET not set on the monitor) or PARITY_MONITOR_URL points elsewhere",
+ ))
+ }
+ StatusCode::UNAUTHORIZED => {
+ return Err(ApiError::upstream(
+ "the parity monitor rejected the secret: DEBUG_API_SECRET differs between the control plane and the monitor",
+ ))
+ }
+ status if !status.is_success() => {
+ return Err(ApiError::upstream(format!(
+ "parity monitor answered {status}"
+ )))
+ }
+ _ => {}
+ }
+ let samples = response
+ .json()
+ .await
+ .map_err(|e| ApiError::upstream(format!("parity monitor sent an unreadable body: {e}")))?;
+ Ok(Json(ParityMismatches {
+ fetched_at: Utc::now(),
+ samples,
+ }))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn mismatches_url_tolerates_a_trailing_slash() {
+ for base in ["http://monitor:3301", "http://monitor:3301/"] {
+ assert_eq!(mismatches_url(base), "http://monitor:3301/debug/mismatches");
+ }
+ }
+}
diff --git a/rust/ingestion-control-plane/src/ui/index.html b/rust/ingestion-control-plane/src/ui/index.html
index 7c7a9f8d6405..4d4e4cb1827c 100644
--- a/rust/ingestion-control-plane/src/ui/index.html
+++ b/rust/ingestion-control-plane/src/ui/index.html
@@ -57,6 +57,10 @@
th, td { text-align: left; padding: 6px 14px; border-bottom: 1px solid var(--border); font-size: 12px; }
th { color: var(--muted); font-weight: 500; font-size: 11px; text-transform: uppercase; letter-spacing: 0.04em; }
td.mono, th.mono { font-family: var(--mono); }
+ td.diff { color: var(--error); }
+ td.nowrap { white-space: nowrap; }
+ .scroll-x { overflow-x: auto; }
+ .clip { display: inline-block; max-width: 260px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; vertical-align: bottom; }
tr.clickable:hover td { background: rgba(79,124,255,0.06); cursor: pointer; }
tr.selected td { background: rgba(79,124,255,0.1); }
td .bar { height: 6px; border-radius: 3px; background: var(--accent); min-width: 2px; display: inline-block; vertical-align: middle; }
@@ -1472,8 +1476,153 @@
},
};
+ // ---------- section: personhog parity ----------
+ const paritySection = {
+ id: "parity",
+ title: "Personhog parity",
+ description: "Recent disagreements between the Node and personhog CDC feeds, from the parity monitor's sample. Newest first. Refreshes every 30 seconds.",
+ timers: [],
+ refreshSeq: 0,
+ unmount() {
+ this.timers.forEach(clearInterval);
+ this.timers = [];
+ },
+ mount(root) {
+ this.loaded = false;
+ this.ui = {
+ summary: el("div", { class: "row", style: "padding: 0 0 10px" }),
+ mappingsBody: el("div", {}, el("div", { class: "empty" }, "Loading…")),
+ mergesBody: el("div", {}, el("div", { class: "empty" }, "Loading…")),
+ };
+ const matchesAny = (values, q) =>
+ values.some((v) => v !== null && v !== undefined && String(v).toLowerCase().includes(q));
+ this.mappingsView = listView({
+ placeholder: "Filter by team, id, or outcome…",
+ pageSize: 100,
+ matches: (s, q) => matchesAny(
+ [s.team_id, s.distinct_id, s.outcome, s.node && s.node.person_uuid, s.personhog && s.personhog.person_uuid], q),
+ renderTable: (rows) => this.renderMappings(rows),
+ emptyText: "No distinct id disagreements in the sample.",
+ });
+ this.mergesView = listView({
+ placeholder: "Filter by team, person, or outcome…",
+ pageSize: 100,
+ matches: (s, q) => matchesAny([s.team_id, s.old_person_uuid, s.outcome, s.node, s.personhog], q),
+ renderTable: (rows) => this.renderMerges(rows),
+ emptyText: "No merge disagreements in the sample.",
+ });
+ root.append(
+ this.ui.summary,
+ el("div", { class: "panel" }, el("h2", {}, "Distinct ids"), this.ui.mappingsBody),
+ el("div", { class: "panel" }, el("h2", {}, "Merges"), this.ui.mergesBody)
+ );
+ this.refresh();
+ this.timers.push(setInterval(() => this.refresh(), 30000));
+ },
+
+ async refresh() {
+ const seq = ++this.refreshSeq;
+ let data;
+ try {
+ data = await fetchJSON("api/personhog/parity/mismatches");
+ } catch (e) {
+ if (seq !== this.refreshSeq) return;
+ this.ui.summary.replaceChildren(el("div", { class: "error-box", style: "flex: 1 1 100%" }, String(e)));
+ if (!this.loaded) {
+ this.ui.mappingsBody.replaceChildren(el("div", { class: "empty" }, "No data."));
+ this.ui.mergesBody.replaceChildren(el("div", { class: "empty" }, "No data."));
+ }
+ return;
+ }
+ if (seq !== this.refreshSeq) return;
+ this.loaded = true;
+ this.render(data);
+ },
+
+ render(data) {
+ // One sweep pushes its verdicts in hash map order, so the ring alone is not time ordered.
+ const at = (s) => (typeof s.last_update_ms === "number" ? s.last_update_ms : Number.MIN_SAFE_INTEGER);
+ const samples = data.samples.slice().reverse().sort((a, b) => at(b) - at(a));
+ const counts = new Map();
+ for (const s of samples) {
+ const label = s.kind === "merge" ? `merge ${s.outcome}` : s.outcome;
+ counts.set(label, (counts.get(label) || 0) + 1);
+ }
+ this.ui.summary.replaceChildren(
+ el("span", { class: "pill muted" }, `${fmtInt(samples.length)} in sample`),
+ ...[...counts.entries()].map(([label, n]) => el("span", { class: "pill warn" }, `${label} ${fmtInt(n)}`)),
+ el("span", { class: "muted" }, "The monitor keeps recent disagreements in memory, so older ones drop out and a restart clears them."),
+ el("span", { class: "muted mono", style: "margin-left: auto" }, `as of ${new Date(data.fetched_at).toLocaleTimeString()}`),
+ el("button", { class: "small", onclick: () => this.refresh() }, "Refresh")
+ );
+ this.mappingsView.setRows(samples.filter((s) => s.kind === "mapping"));
+ this.ui.mappingsBody.replaceChildren(this.mappingsView.body);
+ this.mergesView.setRows(samples.filter((s) => s.kind === "merge"));
+ this.ui.mergesBody.replaceChildren(this.mergesView.body);
+ },
+
+ when(sample) {
+ if (sample.last_update_ms === null || sample.last_update_ms === undefined) return "–";
+ const at = new Date(sample.last_update_ms);
+ return el("span", { title: at.toISOString() }, `${at.toLocaleTimeString()} · ${fmtAgo(sample.last_update_ms)}`);
+ },
+
+ shortId(uuid) {
+ if (!uuid) return el("span", { class: "pill bad" }, "missing");
+ return el("span", { class: "mono", title: `${uuid} (click to copy)`, onclick: (e) => { e.stopPropagation(); copy(uuid); } },
+ `${uuid.slice(0, 8)}…`);
+ },
+
+ renderMappings(rows) {
+ const person = (row) => this.shortId(row ? row.person_uuid : null);
+ return el("div", { class: "scroll-x" }, el("table", {},
+ el("thead", {}, el("tr", {},
+ el("th", {}, "time"), el("th", {}, "team"), el("th", {}, "distinct id"), el("th", {}, "outcome"),
+ el("th", {}, "node person"), el("th", {}, "personhog person"), el("th", {}, "deleted"),
+ el("th", { title: "Versions order rows within one feed and are not compared across feeds" }, "versions"))),
+ el("tbody", {}, rows.map((s) => {
+ const node = s.node, personhog = s.personhog;
+ const both = node && personhog;
+ const personClass = both && node.person_uuid !== personhog.person_uuid ? "mono diff" : "mono";
+ const deletedClass = both && node.is_deleted !== personhog.is_deleted ? "mono diff" : "mono";
+ const pair = (f) => `${node ? f(node) : "–"} / ${personhog ? f(personhog) : "–"}`;
+ return el("tr", {},
+ el("td", { class: "muted mono nowrap" }, this.when(s)),
+ el("td", { class: "mono" }, String(s.team_id)),
+ el("td", { class: "mono" }, copyable(s.distinct_id, "clip")),
+ el("td", { class: "nowrap" }, el("span", { class: "pill warn" }, s.outcome)),
+ el("td", { class: personClass }, person(node)),
+ el("td", { class: personClass }, person(personhog)),
+ el("td", { class: `${deletedClass} nowrap` }, pair((r) => (r.is_deleted ? "yes" : "no"))),
+ el("td", { class: "muted mono nowrap" }, pair((r) => String(r.version))));
+ }))
+ ));
+ },
+
+ renderMerges(rows) {
+ const orDash = (v) => (v === null || v === undefined ? "–" : String(v));
+ return el("div", { class: "scroll-x" }, el("table", {},
+ el("thead", {}, el("tr", {},
+ el("th", {}, "time"), el("th", {}, "team"), el("th", {}, "merged person"), el("th", {}, "outcome"),
+ el("th", {}, "node survivor"), el("th", {}, "personhog survivor"), el("th", {}, "partitions"))),
+ el("tbody", {}, rows.map((s) => {
+ const survivorClass = s.node && s.personhog && s.node !== s.personhog ? "mono diff" : "mono";
+ const [nodePartition, personhogPartition] = s.partitions || [null, null];
+ return el("tr", {},
+ el("td", { class: "muted mono nowrap" }, this.when(s)),
+ el("td", { class: "mono" }, String(s.team_id)),
+ el("td", { class: "mono" }, this.shortId(s.old_person_uuid)),
+ el("td", { class: "nowrap" }, el("span", { class: "pill warn" }, s.outcome)),
+ el("td", { class: survivorClass }, this.shortId(s.node)),
+ el("td", { class: survivorClass }, this.shortId(s.personhog)),
+ el("td", { class: "muted mono nowrap" }, `${orDash(nodePartition)} / ${orDash(personhogPartition)}`));
+ }))
+ ));
+ },
+ };
+
// ---------- boot ----------
- SECTIONS.push(lagSection, debugSection, personhogSection, etcdSection);
+ SECTIONS.push(lagSection, debugSection, personhogSection, paritySection, etcdSection);
const nav = document.getElementById("nav");
for (const section of SECTIONS) {
nav.append(el("span", { class: "nav-item", "data-id": section.id, onclick: () => navigate(section.id) }, section.title));
diff --git a/rust/personhog-parity-monitor/README.md b/rust/personhog-parity-monitor/README.md
index 1861d0d096f2..e74ed2f3e3a0 100644
--- a/rust/personhog-parity-monitor/README.md
+++ b/rust/personhog-parity-monitor/README.md
@@ -21,7 +21,7 @@ Per merged person only the survivor at the end of its merge chain counts.
## Metrics
-All names start with `personhog_parity_`. Labels carry outcomes only, never a team or an id; `/debug/mismatches` holds the last `PARITY_SAMPLE_RING_SIZE` disagreements with both sides' rows.
+All names start with `personhog_parity_`. Labels carry outcomes only, never a team or an id; `/debug/mismatches` holds the last `PARITY_SAMPLE_RING_SIZE` disagreements with both sides' rows and `last_update_ms`, the newest message timestamp behind each verdict.
| Metric | Labels | Meaning |
| --- | --- | --- |
diff --git a/rust/personhog-parity-monitor/src/join.rs b/rust/personhog-parity-monitor/src/join.rs
index 7f207a8167a0..a83c8a367457 100644
--- a/rust/personhog-parity-monitor/src/join.rs
+++ b/rust/personhog-parity-monitor/src/join.rs
@@ -366,7 +366,7 @@ impl Join {
self.evict_merge(old);
report.warmup_evictions += 1;
} else {
- self.settle_merge(old, &mut report);
+ self.settle_merge(old, last_update_ms, &mut report);
self.release_merge(old);
}
}
@@ -381,7 +381,7 @@ impl Join {
self.evict_mapping(&key);
report.warmup_evictions += 1;
} else {
- self.settle_mapping(&key, &mut report);
+ self.settle_mapping(&key, last_update_ms, &mut report);
}
}
}
@@ -417,7 +417,12 @@ impl Join {
watermarks
}
- fn settle_mapping(&mut self, key: &DistinctIdKey, report: &mut SweepReport) {
+ fn settle_mapping(
+ &mut self,
+ key: &DistinctIdKey,
+ last_update_ms: i64,
+ report: &mut SweepReport,
+ ) {
let (node, personhog) = (self.node.mapping(key), self.personhog.mapping(key));
let outcome = compare_mapping(node, personhog);
if outcome != MappingOutcome::Agree {
@@ -426,6 +431,7 @@ impl Join {
"team_id": key.team_id,
"distinct_id": key.distinct_id,
"outcome": outcome.label(),
+ "last_update_ms": last_update_ms,
"node": node.map(mapping_json),
"personhog": personhog.map(mapping_json),
}));
@@ -434,7 +440,7 @@ impl Join {
self.evict_mapping(key);
}
- fn settle_merge(&mut self, old: PersonKey, report: &mut SweepReport) {
+ fn settle_merge(&mut self, old: PersonKey, last_update_ms: i64, report: &mut SweepReport) {
let outcome = compare_survivor(&self.node, &self.personhog, old);
let partitions = self
.merges
@@ -456,6 +462,7 @@ impl Join {
"team_id": old.team_id,
"old_person_uuid": old.person_uuid.to_string(),
"outcome": outcome.label(),
+ "last_update_ms": last_update_ms,
"node": self.node.merged_into(&old).map(|new| new.to_string()),
"personhog": self.personhog.merged_into(&old).map(|new| new.to_string()),
"partitions": partitions,
@@ -731,6 +738,7 @@ mod tests {
assert_eq!(samples.len(), 1);
assert_eq!(samples[0]["distinct_id"], "only");
assert_eq!(samples[0]["outcome"], "node_only");
+ assert_eq!(samples[0]["last_update_ms"], 2_000);
}
// Topics with different counts place the same key on different