:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
personhog-writer
Consumes person state updates from the personhog_updates Kafka topic and batch-upserts them to Postgres, closing the durability loop between the personhog-leader's in-memory cache and persistent storage.
Architecture
personhog_updates (Kafka, compacted)
│
▼
┌──────────────────────────────────────────────────────┐
│ personhog-writer │
│ │
│ Consumer Task Writer Task │
│ ┌────────────────┐ ┌───────────────────┐ │
│ │ Kafka recv │ │ PG batch upsert │ │
│ │ Proto decode │────────│ Per-row fallback │ │
│ │ Buffer dedup │ channel│ Offset commit │ │
│ │ Flush triggers │ └───────────────────┘ │
│ └────────────────┘ │
└──────────────────────────────────────────────────────┘
│
▼
personhog_person_tmp (PG)
The service runs two concurrent tokio tasks connected by a bounded channel:
- Consumer task: reads from Kafka, decodes Person protobuf messages, deduplicates in an in-memory buffer keyed by (team_id, person_id), and sends batches to the writer task on flush triggers.
- Writer task: receives batches, upserts to Postgres via
INSERT ... ON CONFLICT, isolates failures with per-row fallback, and commits Kafka offsets only after every row in the flush has been applied.
The writer is a pure applier: the leader admits every record against this service's exact rejection surface (the pg_column_size measure, jsonb content rules, bindability of every column) before acking, so everything on the topic applies verbatim. The writer never corrects, trims, or skips — any writer-side edit or drop would permanently diverge Postgres from the leader's cache and changelog, because every later snapshot for the person builds on the same state.
Code organization
| Module | Responsibility |
|---|---|
kafka.rs |
Kafka consumer (PersonConsumer). Owns all Kafka config construction. |
consumer.rs |
Consumer loop: recv, decode, buffer, flush triggers, backpressure. |
writer.rs |
Writer loop: batch dispatch, retry orchestration, offset commit. Generic over PersonStore for testability. |
store.rs |
Batch orchestration: parallel chunk execution, outcome partitioning, per-row violation isolation. Defines the PersonDb trait; holds Arc<D: PersonDb>. |
pg.rs |
PG implementation of PersonDb: UNNEST-based chunk upsert, single-row upsert, type conversion, sqlx error classification. |
buffer.rs |
In-memory dedup buffer keyed by (team_id, person_id), keeps highest version. |
config.rs |
Envconfig-based configuration. |
Flush triggers
The consumer flushes the buffer when any of these conditions are met:
- Timer: every
FLUSH_INTERVAL_MS(default 30s) - Size threshold: buffer reaches
FLUSH_BUFFER_SIZEentries (default 10k) - Backpressure: buffer reaches
BUFFER_CAPACITY(default 100k), flush immediately - Shutdown: graceful shutdown flushes remaining buffer
Every flush drains at most FLUSH_BUFFER_SIZE rows (rounded up to whole partitions), so a backlogged buffer empties as a sequence of bounded batches paced by the writer instead of one giant batch. Whole partitions only: a partition's max offset is safe to commit exactly when every buffered person from it has flushed, so a drained batch's offsets never skip past a person still in the buffer.
Idempotency
The upsert uses WHERE EXCLUDED.version > COALESCE(table.version, -1), so out-of-order or replayed messages from Kafka are safely ignored. The COALESCE handles nullable version columns (existing rows with NULL version will accept any update).
A person whose fields cannot be bound losslessly (malformed uuid, team_id beyond the column's integer range, non-UTF-8 JSON bytes, out-of-range timestamp) fails its chunk as a Data error rather than being silently dropped or repaired; the per-row pass then isolates it as an invariant violation (see below).
Error handling
Failures are handled at chunk granularity: the store splits each batch into chunks (sized by upsert_batch_size), runs them in parallel against PgStore::execute_chunk, and reports successful, transient-failed, and data-failed chunks separately so only the affected rows need retry.
- Transient (connection loss, deadlock): retry just the failed chunks with exponential backoff (1s, 2s, 4s). Successful chunks are not re-executed. After 3 consecutive failures, signal unhealthy and shut down.
- Saturation (local pool acquire timeout): the writer's own in-flight statements exceeded pool capacity — self-inflicted, so it retries with the same backoff but never counts toward the consecutive-failure shutdown. Escalating on it would let a slow-PG burst crash-loop the pod, and each restart drops the buffers and redelivers, worsening the load that caused it.
- Data (constraint violation, invalid input): fall back to per-row inserts for the failed chunks' rows, isolating the bad records. Per-row upserts run with bounded concurrency (
ROW_FALLBACK_CONCURRENCY). Successful chunks are not re-executed. - Non-transient row failure (constraint violation, unbindable field): an invariant violation — the leader admitted a record Postgres cannot apply, so admission has a gap. The flush halts via
signal_failurewithout committing; Kafka redelivers after restart, and the alarm stands until the gap is fixed. Skipping is never an option: it would permanently diverge PG from the cache and changelog. - Chunk task panic: a spawned chunk task that panics cannot hand its persons back — the task's stack is unwound. The writer treats this as a fatal error, signals failure, and exits. Because Kafka offsets are committed only on full batch success, redelivery after restart recovers the records; the panic payload is captured in the error message for diagnosis.
- Slow flush: the lane keeps its liveness heartbeat ticking while a flush runs, for about five minutes, so a slow database does not restart the pod. A longer flush reports the lane unhealthy, and the pod restarts unless the flush completes before the next health check.
User-facing size warnings are emitted by the leader at admission time, where the client also gets synchronous feedback; the writer emits none.
JSON handling
The leader serializes a parsed serde_json::Value into proto bytes, so they are valid JSON by construction. The writer passes them through as text[] in the UNNEST query and lets PostgreSQL cast ::jsonb — it never parses JSON itself. The tests/admission_weld.rs suite welds the leader's admission functions to this exact statement against a live Postgres.
Backpressure
Total in-flight statement concurrency is capped by a pod-wide permit budget (UPSERT_CONCURRENCY) shared across all writer lanes and both the chunk and per-row paths. A burst of chunks queues at the semaphore — a cheap, unbounded wait — instead of oversubscribing the pool and converting its own fan-out into acquire timeouts.
When the writer is slow (PG latency, pool saturation), size-triggered flushes stop being accepted, the lane buffer grows, and at the hard cap the consumer blocks on channel send, which stops Kafka consumption. Capped drains keep the resulting batches bounded, so the backlog releases at the writer's pace. This naturally limits memory usage and prevents unbounded buffering.
Kafka consumer configuration
- Cooperative-sticky assignment: during autoscaling, only partitions that need to move are revoked
- Static group membership: when
KAFKA_CLIENT_IDis set, the broker holds partition assignments during pod restarts (requires StatefulSet for stable pod names) - Manual offset commits: offsets are committed only after a successful Postgres write, and only for partitions still assigned to this pod
- Revocation: when the broker takes partitions away, their buffered rows are dropped, and a batch already queued for the writer is filtered the same way when the writer picks it up. Their offsets were never committed, so the new owner reads them from the last commit instead of both pods writing the same rows. A batch already executing against Postgres, including its retries, still completes
Metrics
| Metric | Type | Description |
|---|---|---|
personhog_writer_messages_consumed_total |
counter | Messages decoded from Kafka |
personhog_writer_decode_errors_total |
counter | Proto decode failures |
personhog_writer_kafka_errors_total |
counter | Kafka recv errors |
personhog_writer_flushes_total |
counter | Total flush events |
personhog_writer_flushes_by_trigger_total{trigger} |
counter | Flush trigger breakdown (timer, size, backpressure, shutdown) |
personhog_writer_rows_upserted_total{mode} |
counter | Rows PG reported as affected (mode: chunk, row) |
personhog_writer_rows_version_skipped_total{mode} |
counter | Rows skipped by version guard (mode: chunk, row) |
personhog_writer_upsert_errors_total{mode} |
counter | PG write failures (mode: chunk, row) |
personhog_writer_batch_fallback_total |
counter | Batches that fell back to per-row |
personhog_writer_unbindable_field_total{field} |
counter | Rows whose field could not be bound losslessly (invariant violation) |
personhog_writer_unapplyable_rows_total{kind} |
counter | Rows PG refused non-transiently (invariant violation, halts the flush) |
personhog_writer_chunk_fallback_rows_total |
counter | Rows from data-failed chunks sent to per-row fallback |
personhog_writer_chunk_retry_rows_total |
counter | Rows from transient-failed chunks retried as a batch |
personhog_writer_chunk_saturated_rows_total |
counter | Rows from chunks that hit local pool saturation |
personhog_writer_saturation_retries_total |
counter | Retry rounds caused only by pool saturation (never escalate) |
personhog_writer_chunk_fatal_total |
counter | Chunk tasks that panicked or were cancelled |
personhog_writer_row_fallback_duration_seconds |
histogram | Duration of the per-row fallback for a batch |
personhog_writer_row_fallback_in_flight |
gauge | Concurrent per-row upserts in flight during fallback |
personhog_writer_pg_pool_size |
gauge | Total sqlx pool connections (sampled every 5s) |
personhog_writer_pg_pool_idle |
gauge | Idle sqlx pool connections (sampled every 5s) |
personhog_writer_offset_commits_total |
counter | Successful offset commits |
personhog_writer_offset_commit_errors_total |
counter | Failed offset commits |
personhog_writer_offset_commits_skipped_total{reason} |
counter | Partition offsets left uncommitted because the partition is no longer assigned |
personhog_writer_partitions_revoked_total |
counter | Partitions the broker revoked from this pod |
personhog_writer_revoked_rows_dropped_total{stage} |
counter | Rows dropped because their partition was revoked (stage: buffer, queued) |
personhog_writer_flush_duration_seconds |
histogram | PG write latency per flush |
personhog_writer_flush_rows |
histogram | Rows per flush |
personhog_writer_channel_send_duration_seconds |
histogram | Time waiting on the writer channel (backpressure indicator) |
personhog_writer_e2e_latency_seconds |
histogram | End-to-end latency from Kafka message creation to PG commit |
personhog_writer_buffer_size |
gauge | Current buffer entries |
personhog_writer_partition_offset{partition} |
gauge | Current offset per partition |
Consumer lag per partition is monitored externally via KMinion (kminion_kafka_consumer_group_topic_lag).
Configuration
| Variable | Default | Description |
|---|---|---|
KAFKA_HOSTS |
localhost:9092 |
Kafka bootstrap servers |
KAFKA_TLS |
false |
Enable TLS for Kafka |
KAFKA_CLIENT_ID |
(empty) | Pod name for static group membership |
KAFKA_CLIENT_RACK |
(empty) | Rack ID for rack-aware consumption |
KAFKA_CONSUMER_GROUP |
personhog-writer |
Consumer group ID |
ETCD_ENDPOINTS |
http://localhost:2379 |
etcd endpoints; the topic comes from config/changelog |
ETCD_PREFIX |
/personhog/ |
etcd key prefix |
DATABASE_URL |
(required) | Postgres connection string |
PG_MAX_CONNECTIONS |
20 |
Connection pool size |
PG_TARGET_TABLE |
personhog_person_tmp |
Target table (posthog_person for production cutover) |
FLUSH_INTERVAL_MS |
30000 |
Timer-based flush interval (longer = better dedup, higher latency) |
FLUSH_BUFFER_SIZE |
10000 |
Size-based flush trigger and per-flush drain cap |
BUFFER_CAPACITY |
100000 |
Hard cap for backpressure |
UPSERT_BATCH_SIZE |
5000 |
Max rows per INSERT statement (chunks execute in parallel) |
UPSERT_CONCURRENCY |
8 |
Pod-wide cap on concurrent upsert statements, shared across lanes |
ROW_FALLBACK_CONCURRENCY |
16 |
Max concurrent per-row upserts during per-row fallback |
FLUSH_CHANNEL_CAPACITY |
8 |
Channel capacity between tasks |
METRICS_PORT |
9103 |
Prometheus metrics HTTP port |