:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Ingestion control plane
Internal web tool for the ingestion team. Single binary serving an embedded UI, with a sidebar of tools that is meant to grow over time.
⚠️ Purely internal service — never expose publicly. This tool has no authentication of its own and its analysis results contain cross-customer data (team tokens, distinct_ids, event names). It must only be deployed behind the internal ingress (VPN/SSO), reachable by PostHog employees exclusively. It is an operational tool, not a product surface.
Tools
Lagging partitions
Discovers ingestion topics and consumer groups from the cluster by prefix (a group maps to the topics it has committed offsets on — no per-environment target config), scans committed offsets vs watermarks, and sorts by total outstanding messages. Drilling into a group shows per-partition lag; from there an analysis job reads a slice of the partition and aggregates message headers only (payloads are dropped, only their size is recorded):
headmode starts at the group's committed offset — what the consumer is stuck on.tailmode samples the newest messages before the high watermark — what's arriving now.
Results are broken down per token (resolved to team_id via Postgres when DATABASE_URL is set), with events and distinct_ids nested under each token, plus a message-size distribution and header-flag counts. Analyses use a dedicated consumer group (ingestion-control-plane-inspector) and never touch the real group's offsets.
Clicking a team, event, or distinct_id in an analysis result opens the message browser: a filtered scan over the same partition (/api/messages) that lists individual matching messages. Filters are equality matches on the token, event, and distinct_id headers; a facets sidebar aggregates the loaded messages so results can be narrowed further. Each request returns up to 100 matches and a cursor, and "Load more" resumes from it; per-request scanning is bounded by a message budget, a deadline, and a byte budget so a rarely-matching filter still returns promptly. Matched messages carry their payload (truncated to 64 KB per message), shown by expanding a row, so this surface exposes raw event payloads — another reason the tool must stay behind the internal ingress.
Consumer debug
Lists live ingestion-consumer pods and serves the consumer's routing debug UI per pod at /pods/<namespace>/<name>/ (static pods use the static pseudo-namespace), backed by a proxy to the pod's debug API (/debug/state, /debug/load, SSE /debug/events). The UI lives here; the consumer only exposes the JSON/SSE API (gated behind DEBUG_UI_ENABLED on the consumer side).
Personhog topology
Protocol-aware view of personhog's etcd coordination state (PERSONHOG_ETCD_PREFIX, default /personhog/): the whole prefix is read in one consistent snapshot and interpreted with the personhog-coordination types and the coordinator's own pure predicates, so the derived diagnostics cannot drift from the protocol. Shows the elected coordinator, registered pods and routers (with remaining lease TTLs), per-partition ownership, and in-flight handoffs with their phase age and what each is waiting on — distinguishing a stuck participant ("waiting on freeze acks from router-2") from a stuck coordinator ("quorum met; waiting on the coordinator to advance"). Derived issues cover stuck handoffs past their per-phase deadline, assignments owned by unregistered pods with no replanning handoff, lingering Complete records the coordinator hasn't cleaned up, dead new owners awaiting cancellation, unassigned partitions, stale acks from previous handoff attempts, and unparseable records. The deadline knobs (PERSONHOG_HANDOFF_DEADLINE_SECS, PERSONHOG_WARMING_DEADLINE_SECS) must match the coordinator's to agree with its cancellation decisions. The tool writes no coordination state except from the changelog panel (seeding config/changelog, adding or removing a consumer group that retire waits on, requesting or cancelling a migration, retiring a drained generation, and restoring config/changelog from a backup), and a force-reassignment must go through a handoff record, never a raw assignment write.
Changelog backups
With CHANGELOG_BACKUP_BUCKET set, the service keeps a copy of config/changelog in that versioned bucket at <prefix>/config-changelog.json, as {"nonce", "config"} with the stored bytes in config. It copies the config once at startup and again after every change, from whichever writer made it, and skips a write when the current copy already holds it. A deletion is never written, so a wiped etcd still has its last good config in the bucket. The bucket's versions are the history.
Every replica runs the copy. Each one reads the backup, then etcd, and uploads only if the backup it read is still current. When it is not, S3 refuses the upload and the replica tries again after a backoff, from a fresh read. So an upload never replaces a copy of a newer config. The nonce gives each upload its own ETag, because the config alone can return to bytes it held before. Each replica reports ingestion_control_plane_changelog_backup_in_sync: 1 after a sync succeeds, 0 from startup until then and after a failed attempt. The backup is at risk when no replica reports 1.
When etcd holds no config, the changelog panel offers a restore from the current backup or an earlier version. A restore writes the stored bytes back unchanged, only while the key is absent, and only after checking the topic's live partition count matches the backup. Running services ignore a config whose epoch is older than the one they adopted, so a rebuilt etcd is restored with personhog scaled to zero.
Personhog parity
Shows the personhog parity monitor's recent disagreements between the Node and personhog CDC feeds, newest first. A distinct id row shows the mapping each feed ended with, and a merge row shows each feed's survivor and partition, with the differing fields highlighted. The server reads the monitor's /debug/mismatches sample with DEBUG_API_SECRET, so the monitor must run with the same secret. The monitor keeps the sample in memory, so older disagreements drop out and a monitor restart clears it. The view is disabled (its API returns 503) unless PARITY_MONITOR_URL is set.
Etcd explorer
Generic key browser for the same etcd cluster: keys under a prefix render as a collapsible tree (with per-key version, lease, and size; the top level and single-child chains expand automatically), and a key's detail shows its value, revisions, and remaining lease TTL, with edit, create, and delete. Operations are bounded to ETCD_ALLOWED_PREFIXES. Writes bypass every coordination protocol, so each one confirms first; writing a lease-backed key warns that it detaches the key from its lease, and writing a personhog assignments/ key gets an extra warning because routers only observe ownership through handoff Complete events — a raw assignment write is invisible to them and leaves an unfenced old owner serving. Deletes are single-key only; there is deliberately no prefix delete.
Both etcd tools are disabled (their APIs return 503) unless ETCD_ENDPOINTS is set.
Configuration
| Env var | Default | Notes |
|---|---|---|
BIND_HOST / BIND_PORT |
0.0.0.0 / 3305 |
Single listener: UI, API, /_liveness, /_readiness, /metrics |
KAFKA_HOSTS |
localhost:19092 |
|
KAFKA_TLS |
false |
|
TOPIC_PREFIX |
ingestion- |
Topics are discovered from cluster metadata by prefix |
GROUP_PREFIX |
ingestion- |
Consumer groups are discovered by prefix; a group maps to the topics it has committed offsets on |
DISCOVERY_CACHE_TTL_SECS |
300 |
Discovered targets are cached; topology changes rarely |
OVERVIEW_CACHE_TTL_SECS |
15 |
Overview scans are cached with single-flight refresh, bounding broker load from repeated requests |
DATABASE_URL |
empty | Read replica; empty disables token → team resolution |
ANALYSIS_MESSAGE_COUNT |
10000 |
Messages per analysis |
ANALYSIS_DEADLINE_SECS |
120 |
|
ANALYSIS_MAX_FETCH_BYTES |
536870912 |
Kafka transfers full records even for header-only analysis |
BROWSE_SCAN_MESSAGE_COUNT |
25000 |
Messages scanned per message-browser request |
BROWSE_DEADLINE_SECS |
10 |
|
BROWSE_MAX_FETCH_BYTES |
268435456 |
Byte budget per message-browser request |
BROWSE_MAX_CONCURRENT |
3 |
Concurrent browse requests; more get 429 |
POD_DISCOVERY_MODE |
kubernetes |
static for local testing |
STATIC_PODS |
local=127.0.0.1:3301 |
name=host:port pairs for static mode |
POD_LABEL_SELECTORS |
ingestion-analytics-main/app=ingestion-analytics-main,ingestion-analytics-async/app=ingestion-analytics-async |
One namespace/key=value per entry (each lane runs in its own namespace); bare key=value uses K8S_NAMESPACE |
K8S_NAMESPACE |
posthog |
Default namespace for unqualified selector entries |
DEBUG_PORT |
3301 |
Consumer debug API port (kubernetes mode) |
DEBUG_API_SECRET |
empty | Sent as X-Debug-Api-Secret to the consumers' debug APIs and to the parity monitor; must match theirs |
PARITY_MONITOR_URL |
empty | Parity monitor base URL, for example http://personhog-parity-monitor.personhog-parity-monitor.svc.cluster.local:3301; empty disables the parity view |
ETCD_ENDPOINTS |
empty | Comma-separated etcd endpoints; empty disables the etcd explorer and personhog topology tools |
ETCD_ALLOWED_PREFIXES |
/ |
Comma-separated key prefixes the etcd explorer may read and write |
PERSONHOG_ETCD_PREFIX |
/personhog/ |
Prefix the personhog coordination state lives under; must match the personhog services' ETCD_PREFIX |
PERSONHOG_KAFKA_HOSTS |
KAFKA_HOSTS |
Cluster the personhog changelog lives on, for the changelog panel |
CHANGELOG_BACKUP_BUCKET |
empty | Versioned S3 bucket for config/changelog backups; empty disables backups and restore. Credentials and region come from the standard AWS environment |
PERSONHOG_KAFKA_TLS |
KAFKA_TLS |
TLS for PERSONHOG_KAFKA_HOSTS |
CHANGELOG_KAFKA_CLUSTERS |
empty | Clusters the changelog can live on, by name, as JSON {"name": {"hosts": "...", "tls": bool}}; seeding and a migration write the named endpoint into the changelog config, and the changelog panel offers the names as a migration's target |
PERSONHOG_CROSS_CLUSTER_MIGRATIONS |
false |
Allows a migration onto another cluster; enable only once every personhog leader, writer, relay and router runs a build that reads the cluster, because an older router's coordinator drops the cluster at the flip and an older leader, writer or relay uses its own KAFKA_HOSTS for the new topic |
PERSONHOG_HANDOFF_DEADLINE_SECS |
120 |
Per-phase deadline for flagging stuck handoffs; must match COORDINATOR_HANDOFF_DEADLINE_SECS |
PERSONHOG_WARMING_DEADLINE_SECS |
1800 |
Warming-phase deadline; must match COORDINATOR_WARMING_DEADLINE_SECS |
Local development
# Seed a lagging-partition scenario (one dominant team on partition 0):
DATABASE_URL=postgres://posthog:posthog@db:5432/posthog \
KAFKA_HOSTS=localhost:9092 \
cargo run -p ingestion-control-plane --example seed_lag
# Run the service against it, with a locally running ingestion-consumer on :3401.
# The seeded ingestion-lag-demo topic/group are picked up by the default
# `ingestion-` discovery prefixes:
BIND_PORT=3305 KAFKA_HOSTS=localhost:9092 \
DATABASE_URL=postgres://posthog:posthog@db:5432/posthog \
POD_DISCOVERY_MODE=static STATIC_PODS="local-consumer=127.0.0.1:3401" \
cargo run -p ingestion-control-plane
Then open http://localhost:3305/.
Deployment notes
- Image
ingestion-control-planeis built via.github/rust-images.yml; a deploy needs a matrix entry inrust-docker-build.ymlplus a charts-repo app. - Kubernetes mode needs RBAC:
get/listonpodsin the ingestion namespace, and network egress to pod IPs on the debug port, Kafka, the Postgres replica, (when the etcd tools are enabled) the personhog etcd cluster, and (when the parity view is enabled) the parity monitor's Service.