:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
personhog-leader cluster
Requirements
- provides API contract allowing strong consistent reads/writes to person state
- enables faster/more efficient writes to person properties than writing directly to Postgres
- durably stores any write that a personhog client has received a status OK for
- new service versions can be deployed without service disruptions or inconsistent writes
- pods can scale up and down without service disruptions or inconsistent writes
- crashed pods can be recovered from with minimal downtime
API contract: every RPC must be safe under at-least-once delivery
A leader-path request can be delivered more than once: clients retry
UNAVAILABLE after ambiguous failures, and the router internally
replays fence- and transport-bounced requests. Neither layer can know
whether an ambiguous first delivery applied, so the safety burden sits
here, on the RPC's semantics — not in any retry policy.
Be precise about what "safe" means today. Reads are pure.
UpdatePersonProperties is a merge command with no request-level
identity: re-applying the same merge converges to the same state
unless a different caller wrote the same field between the original
apply and the replay, in which case the replay clobbers the newer
write. That interleaved case is the accepted at-least-once residual —
the exposure is identical under client-driven retries, which the
platform has always had — and its planned closure is operation
identity: the router stamps each logical request with an op id, the
leader keeps a short ring of recently applied ids on the cached person
entry, and the id rides the produced record so warming rebuilds the
ring on a new owner (the stamp and the ring land together).
Future RPCs must clear the same bar: convergent under redelivery
(tombstone-style deletes, max-merge version floors, re-appliable
merges), or carrying explicit operation identity. An RPC that is
neither (an unguarded increment, an append) is broken in this
architecture regardless of router behavior and must not be added; this
is a review requirement for new leader RPCs (see the
adding-personhog-rpc skill).
To Implement
Known Implementation Details
---
title: PersonHog Leader Cluster
---
graph TB
subgraph P[Leader Pod]
C1[vNode Cache]
end
P -->|produce merged state| K[(Kafka person_state topic)]
K -.->|"consume from O_pg_writer (cache warm)"| P
K -->|consume + batch| PGW[Postgres Writer Service]
PGW -->|idempotent upsert| PG[(PostgreSQL)]
Efficient Writes
- stateful API that caches person data on pods
- API can receive a list of property updates and only update/writes the changed property fields, doesn't replace the entire property field
TBD:
- what technology to use for the cache? how does a pod recover from a crash/restore its cache? how long does that take? does every pod crash result in service disruption? for how long?
Durability
- with writes going to the cache, the head of application state now lives in the cache (single point of failure), not in Postgres (durable store). we need durability
- after each person write to the cache and before we ack to client, we emit a message to kafka (acting as a distributed log)
- the head of our application state can always be materialized through replaying Kafka messages onto the outdated PG state
- a separate PG writer service consumes messages from the kafka topic and batch write to PG
- maintains a committed offset per partition, i.e. O_pg_writer(P)
- this offset is the boundary:
- below the offset: state is durably in Postgres
- at or above the offset: state is PG + the changes in our distributed log (the kafka topic)
- death documents (tombstoned persons) stay cached only while their record awaits the writer: once the committed offset passes it, the prune drops the document and reads fall through to Postgres, which then holds the tombstone or a revival that superseded it
- a revival therefore becomes visible at most one prune tick after the writer applies its tombstone
- warming leaves no residue for an applied death record — neither the document (unmarked, it would answer not-found forever) nor its earlier records for the same key
Admission
Every record the leader produces to the changelog must be applyable by the writer's upsert verbatim: the changelog is the source of truth, Postgres is a downstream consumer of it, and a record the writer cannot apply would leave acked state that never lands (the writer halts rather than skip, since every later snapshot for a person builds on the same state). Admission enforces this before the ack, under the per-person lock, on the merged result of each update:
- Sanitize (
personhog_common::properties::sanitize_for_jsonb): NUL (\u0000) becomes\u{FFFD}in every string, matching the Node pipeline'ssanitizeJsonbValue— Postgres jsonb refuses NUL. Floats beyond ±1e307 are clamped: Postgres renders jsonb numerics in expanded decimal, and serde_json cannot parse expansions of ~1e308+, so larger values would apply but never load back. - Measure (
jsonb_column_size): an exact reimplementation ofpg_column_sizefor JSONB, welded byte-equal to a live Postgres bypersonhog-common/tests/jsonb_size_pg.rs. A merged result above the threshold (thecheck_properties_sizeceiling) follows the Node pipeline's policy (handleOversizedPersonProperties): if the stored row was already over the threshold, it is remediated — non-protected properties trimmed alphabetically to the target, the triggering update's property changes discarded; if the row was within limits, the update is rejected withINVALID_ARGUMENT. Errors carry sizes, never values, and remediation that cannot fit (protected properties alone exceed the target) also rejects.PropertySizeLimits::newrefuses inverted configuration at startup. - Assert (
assert_writeable): identity fields that originate from earlier state (uuid parses, team_id fits the column's integer, created_at within sane bounds) — corrupt state must never reach the changelog.
The merge saga's fold (FoldPersonDocument) cannot reject for size — the
saga would re-drive it forever — so its oversized path always completes:
trim candidates are the fold's own contribution (a within-limit target
never loses a key it held), the target's own keys join only when its
stored document already exceeded the limit, and a trim that cannot reach
the hysteresis target retries against the hard ceiling, since any document
at or under the threshold is still applyable. The residual — a stored
document whose protected keys alone exceed the ceiling, so no applyable
fold document exists — completes without producing: the person keeps its
pre-fold state, the fold's property and scalar effects are skipped, and
the case is surfaced via personhog_leader_folds_total{outcome= "unapplyable"}, an error log, and a size-violation warning. This is an
accepted gap: the alternatives are wedging the saga (reject) or halting
the writer (produce an unapplyable record).
Trims and rejections emit person_properties_size_violation ingestion
warnings (src/warnings.rs), throttled per (team, type) to match the Node
pipeline's limiter. They go through a dedicated fire-and-forget producer,
not the changelog producer, because the warnings topic can live on a
different cluster from the changelog. INGESTION_WARNINGS_ENABLED
(default false) turns it on, and INGESTION_WARNINGS_KAFKA_HOSTS,
INGESTION_WARNINGS_KAFKA_TLS and INGESTION_WARNINGS_TOPIC (default
clickhouse_ingestion_warnings) name the destination. Enabled with no
hosts set, the leader logs a warning and emits nothing. Health reads from
ingestion_warnings_emitter_enabled{reason} and
ingestion_warnings_total{source="personhog-leader",outcome}. The writer-side weld
(personhog-writer/tests/admission_weld.rs) runs hostile property fixtures
through these exact functions and the writer's real statement against live
Postgres, asserting byte-exact round-trips.
Cache warming on partition handoff
When a partition moves between leader pods, the new owner repopulates
its cache by replaying the slice of personhog_updates that the
writer has not yet persisted to PG. The warming pipeline lives in
src/warming.rs and is invoked by LeaderHandoffHandler::warm_partition
when the handoff reaches the Warming phase (see
personhog-coordination's README for the full
Freezing → Draining → Warming → Complete protocol).
Pre-conditions established by the protocol:
Freezingcollected freeze quorum from every router → no router forwards to the old owner anymore.Drainingwaited for the old owner's in-flight handlers to complete and producedPodDrainedAck→ no producer can append to this partition's Kafka log.
By the time Warming runs, the partition's HWM is therefore stable
and warming can consume to a known endpoint without racing producers.
The pipeline:
- Query the writer's consumer group's committed offset
(
O_pg_writer(P)) via a short-lived OffsetFetch consumer. Anything at or after this offset still needs to be in cache; anything below is already durable in PG. - Resolve the start offset: the committed offset, clamped to the partition's earliest available offset. Every record from there is marked dirty as it warms.
assign()(notsubscribe()) the warming consumer to the partition at the resolved start offset and consume until HWM.- Buffer decoded records locally; only commit them to the cache
after the entire range warms successfully via
PartitionedCache::install_warmed_partition, which builds the populatedPersonCachefirst and publishes it via a singleDashMap::insert. Any decode/IO failure mid-range aborts warming with no observable cache mutation, preventing a partial cache from silently masking PG fallback reads.
Configurable knobs (env vars, see src/config.rs):
WRITER_CONSUMER_GROUP— group whose committed offset bounds the warming range.WARM_COMMITTED_OFFSETS_TIMEOUT_SECS,WARM_FETCH_WATERMARKS_TIMEOUT_SECS,WARM_RECV_TIMEOUT_SECS— Kafka call timeouts.WARM_RETRY_MAX_ATTEMPTS,WARM_RETRY_INITIAL_BACKOFF_MS,WARM_RETRY_MAX_BACKOFF_MS— retry policy for transient Kafka metadata failures.
The synchronous rdkafka calls (committed_offsets, fetch_watermarks)
run on the blocking pool via tokio::task::spawn_blocking so a slow
broker can't park the runtime thread.
vNode ownership
The pod participates in personhog-coordination's handoff protocol
via LeaderHandoffHandler (see src/coordination/mod.rs):
drain_partition_inflight(Draining): fences the partition against new writes (InflightTracker::fence— fenced writes are refused withFAILED_PRECONDITION), then waits for the per-partition counter to drop to zero, then writesPodDrainedAck. Fencing before waiting is what makes "counter reached zero" final: without the fence a write admitted after the wait could advance the Kafka HWM past the point warming snapshots. Combined with the produce path's sync-await on Kafka delivery, this gives the protocol the guarantee that "no in-flight" implies "every acked write durable in Kafka." Reads are never fenced — until cutover the frozen cache state is still the latest.warm_partition(Warming): see the section above. Takes the partition's changelog epoch before the warm read, which fences every predecessor at the broker, and lifts any write fence left over from a previous ownership of the same partition.release_partition(Complete): lifts the write fence, gives the changelog epoch back, and drops the partition's cache slot viaPartitionedCache::drop_partitiononce the routing table has flipped to the new owner.resume_partition(handoff cancelled): a handoff deleted beforeCompletewhile this pod still owns the partition is a cancellation — re-takes the changelog epoch, which the cancelled handoff's target may have taken, then lifts the write fence so the partition serves writes again.
Request Path
graph TB
C[Client] -->|"UPDATE /persons/<id>"| R
subgraph R[Router]
direction TB
PARSE[Parse request] --> DECIDE{Consistent Read/Write?}
DECIDE -->|"Yes"| HASH["Hash person_id → vnode"]
HASH --> WATCH["Lookup vnode → pod(from metadata store cache)"]
end
WATCH -->|"handoff"| MS[(Metadata Store)]
MS -->|"vnode assignments"| LOOKUP
LOOKUP --> POD
subgraph POD[PersonHog Leader BE]
direction TB
VALIDATE[Validate ownership] --> COMPUTE[Compute write]
COMPUTE --> ADMIT[Admission: sanitize, measure, reject/remediate]
ADMIT --> CACHE[Update in-memory cache]
CACHE --> KAFKA[Durably store to Kafka]
end
KAFKA --> ACK[Ack to client]