Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

13.3 KB
Raw Blame

personhog-writer

Consumes person state updates from the personhog_updates Kafka topic and batch-upserts them to Postgres, closing the durability loop between the personhog-leader's in-memory cache and persistent storage.

Architecture

personhog_updates (Kafka, compacted)
    │
    ▼
┌──────────────────────────────────────────────────────┐
│  personhog-writer                                    │
│                                                      │
│  Consumer Task              Writer Task              │
│  ┌────────────────┐        ┌───────────────────┐    │
│  │ Kafka recv     │        │ PG batch upsert   │    │
│  │ Proto decode   │────────│ Per-row fallback   │    │
│  │ Buffer dedup   │ channel│ Offset commit      │    │
│  │ Flush triggers │        └───────────────────┘    │
│  └────────────────┘                                  │
└──────────────────────────────────────────────────────┘
    │
    ▼
personhog_person_tmp (PG)

The service runs two concurrent tokio tasks connected by a bounded channel:

  • Consumer task: reads from Kafka, decodes Person protobuf messages, deduplicates in an in-memory buffer keyed by (team_id, person_id), and sends batches to the writer task on flush triggers.
  • Writer task: receives batches, upserts to Postgres via INSERT ... ON CONFLICT, isolates failures with per-row fallback, and commits Kafka offsets only after every row in the flush has been applied.

The writer is a pure applier: the leader admits every record against this service's exact rejection surface (the pg_column_size measure, jsonb content rules, bindability of every column) before acking, so everything on the topic applies verbatim. The writer never corrects, trims, or skips — any writer-side edit or drop would permanently diverge Postgres from the leader's cache and changelog, because every later snapshot for the person builds on the same state.

Code organization

Module Responsibility
kafka.rs Kafka consumer (PersonConsumer). Owns all Kafka config construction.
consumer.rs Consumer loop: recv, decode, buffer, flush triggers, backpressure.
writer.rs Writer loop: batch dispatch, retry orchestration, offset commit. Generic over PersonStore for testability.
store.rs Batch orchestration: parallel chunk execution, outcome partitioning, per-row violation isolation. Defines the PersonDb trait; holds Arc<D: PersonDb>.
pg.rs PG implementation of PersonDb: UNNEST-based chunk upsert, single-row upsert, type conversion, sqlx error classification.
buffer.rs In-memory dedup buffer keyed by (team_id, person_id), keeps highest version.
config.rs Envconfig-based configuration.

Flush triggers

The consumer flushes the buffer when any of these conditions are met:

  • Timer: every FLUSH_INTERVAL_MS (default 30s)
  • Size threshold: buffer reaches FLUSH_BUFFER_SIZE entries (default 10k)
  • Backpressure: buffer reaches BUFFER_CAPACITY (default 100k), flush immediately
  • Shutdown: graceful shutdown flushes remaining buffer

Every flush drains at most FLUSH_BUFFER_SIZE rows (rounded up to whole partitions), so a backlogged buffer empties as a sequence of bounded batches paced by the writer instead of one giant batch. Whole partitions only: a partition's max offset is safe to commit exactly when every buffered person from it has flushed, so a drained batch's offsets never skip past a person still in the buffer.

Idempotency

The upsert uses WHERE EXCLUDED.version > COALESCE(table.version, -1), so out-of-order or replayed messages from Kafka are safely ignored. The COALESCE handles nullable version columns (existing rows with NULL version will accept any update).

A person whose fields cannot be bound losslessly (malformed uuid, team_id beyond the column's integer range, non-UTF-8 JSON bytes, out-of-range timestamp) fails its chunk as a Data error rather than being silently dropped or repaired; the per-row pass then isolates it as an invariant violation (see below).

Error handling

Failures are handled at chunk granularity: the store splits each batch into chunks (sized by upsert_batch_size), runs them in parallel against PgStore::execute_chunk, and reports successful, transient-failed, and data-failed chunks separately so only the affected rows need retry.

  • Transient (connection loss, deadlock): retry just the failed chunks with exponential backoff (1s, 2s, 4s). Successful chunks are not re-executed. After 3 consecutive failures, signal unhealthy and shut down.
  • Saturation (local pool acquire timeout): the writer's own in-flight statements exceeded pool capacity — self-inflicted, so it retries with the same backoff but never counts toward the consecutive-failure shutdown. Escalating on it would let a slow-PG burst crash-loop the pod, and each restart drops the buffers and redelivers, worsening the load that caused it.
  • Data (constraint violation, invalid input): fall back to per-row inserts for the failed chunks' rows, isolating the bad records. Per-row upserts run with bounded concurrency (ROW_FALLBACK_CONCURRENCY). Successful chunks are not re-executed.
  • Non-transient row failure (constraint violation, unbindable field): an invariant violation — the leader admitted a record Postgres cannot apply, so admission has a gap. The flush halts via signal_failure without committing; Kafka redelivers after restart, and the alarm stands until the gap is fixed. Skipping is never an option: it would permanently diverge PG from the cache and changelog.
  • Chunk task panic: a spawned chunk task that panics cannot hand its persons back — the task's stack is unwound. The writer treats this as a fatal error, signals failure, and exits. Because Kafka offsets are committed only on full batch success, redelivery after restart recovers the records; the panic payload is captured in the error message for diagnosis.
  • Slow flush: the lane keeps its liveness heartbeat ticking while a flush runs, for about five minutes, so a slow database does not restart the pod. A longer flush reports the lane unhealthy, and the pod restarts unless the flush completes before the next health check.

User-facing size warnings are emitted by the leader at admission time, where the client also gets synchronous feedback; the writer emits none.

JSON handling

The leader serializes a parsed serde_json::Value into proto bytes, so they are valid JSON by construction. The writer passes them through as text[] in the UNNEST query and lets PostgreSQL cast ::jsonb — it never parses JSON itself. The tests/admission_weld.rs suite welds the leader's admission functions to this exact statement against a live Postgres.

Backpressure

Total in-flight statement concurrency is capped by a pod-wide permit budget (UPSERT_CONCURRENCY) shared across all writer lanes and both the chunk and per-row paths. A burst of chunks queues at the semaphore — a cheap, unbounded wait — instead of oversubscribing the pool and converting its own fan-out into acquire timeouts.

When the writer is slow (PG latency, pool saturation), size-triggered flushes stop being accepted, the lane buffer grows, and at the hard cap the consumer blocks on channel send, which stops Kafka consumption. Capped drains keep the resulting batches bounded, so the backlog releases at the writer's pace. This naturally limits memory usage and prevents unbounded buffering.

Kafka consumer configuration

  • Cooperative-sticky assignment: during autoscaling, only partitions that need to move are revoked
  • Static group membership: when KAFKA_CLIENT_ID is set, the broker holds partition assignments during pod restarts (requires StatefulSet for stable pod names)
  • Manual offset commits: offsets are committed only after a successful Postgres write, and only for partitions still assigned to this pod
  • Revocation: when the broker takes partitions away, their buffered rows are dropped, and a batch already queued for the writer is filtered the same way when the writer picks it up. Their offsets were never committed, so the new owner reads them from the last commit instead of both pods writing the same rows. A batch already executing against Postgres, including its retries, still completes

Metrics

Metric Type Description
personhog_writer_messages_consumed_total counter Messages decoded from Kafka
personhog_writer_decode_errors_total counter Proto decode failures
personhog_writer_kafka_errors_total counter Kafka recv errors
personhog_writer_flushes_total counter Total flush events
personhog_writer_flushes_by_trigger_total{trigger} counter Flush trigger breakdown (timer, size, backpressure, shutdown)
personhog_writer_rows_upserted_total{mode} counter Rows PG reported as affected (mode: chunk, row)
personhog_writer_rows_version_skipped_total{mode} counter Rows skipped by version guard (mode: chunk, row)
personhog_writer_upsert_errors_total{mode} counter PG write failures (mode: chunk, row)
personhog_writer_batch_fallback_total counter Batches that fell back to per-row
personhog_writer_unbindable_field_total{field} counter Rows whose field could not be bound losslessly (invariant violation)
personhog_writer_unapplyable_rows_total{kind} counter Rows PG refused non-transiently (invariant violation, halts the flush)
personhog_writer_chunk_fallback_rows_total counter Rows from data-failed chunks sent to per-row fallback
personhog_writer_chunk_retry_rows_total counter Rows from transient-failed chunks retried as a batch
personhog_writer_chunk_saturated_rows_total counter Rows from chunks that hit local pool saturation
personhog_writer_saturation_retries_total counter Retry rounds caused only by pool saturation (never escalate)
personhog_writer_chunk_fatal_total counter Chunk tasks that panicked or were cancelled
personhog_writer_row_fallback_duration_seconds histogram Duration of the per-row fallback for a batch
personhog_writer_row_fallback_in_flight gauge Concurrent per-row upserts in flight during fallback
personhog_writer_pg_pool_size gauge Total sqlx pool connections (sampled every 5s)
personhog_writer_pg_pool_idle gauge Idle sqlx pool connections (sampled every 5s)
personhog_writer_offset_commits_total counter Successful offset commits
personhog_writer_offset_commit_errors_total counter Failed offset commits
personhog_writer_offset_commits_skipped_total{reason} counter Partition offsets left uncommitted because the partition is no longer assigned
personhog_writer_partitions_revoked_total counter Partitions the broker revoked from this pod
personhog_writer_revoked_rows_dropped_total{stage} counter Rows dropped because their partition was revoked (stage: buffer, queued)
personhog_writer_flush_duration_seconds histogram PG write latency per flush
personhog_writer_flush_rows histogram Rows per flush
personhog_writer_channel_send_duration_seconds histogram Time waiting on the writer channel (backpressure indicator)
personhog_writer_e2e_latency_seconds histogram End-to-end latency from Kafka message creation to PG commit
personhog_writer_buffer_size gauge Current buffer entries
personhog_writer_partition_offset{partition} gauge Current offset per partition

Consumer lag per partition is monitored externally via KMinion (kminion_kafka_consumer_group_topic_lag).

Configuration

Variable Default Description
KAFKA_HOSTS localhost:9092 Kafka bootstrap servers
KAFKA_TLS false Enable TLS for Kafka
KAFKA_CLIENT_ID (empty) Pod name for static group membership
KAFKA_CLIENT_RACK (empty) Rack ID for rack-aware consumption
KAFKA_CONSUMER_GROUP personhog-writer Consumer group ID
ETCD_ENDPOINTS http://localhost:2379 etcd endpoints; the topic comes from config/changelog
ETCD_PREFIX /personhog/ etcd key prefix
DATABASE_URL (required) Postgres connection string
PG_MAX_CONNECTIONS 20 Connection pool size
PG_TARGET_TABLE personhog_person_tmp Target table (posthog_person for production cutover)
FLUSH_INTERVAL_MS 30000 Timer-based flush interval (longer = better dedup, higher latency)
FLUSH_BUFFER_SIZE 10000 Size-based flush trigger and per-flush drain cap
BUFFER_CAPACITY 100000 Hard cap for backpressure
UPSERT_BATCH_SIZE 5000 Max rows per INSERT statement (chunks execute in parallel)
UPSERT_CONCURRENCY 8 Pod-wide cap on concurrent upsert statements, shared across lanes
ROW_FALLBACK_CONCURRENCY 16 Max concurrent per-row upserts during per-row fallback
FLUSH_CHANNEL_CAPACITY 8 Channel capacity between tasks
METRICS_PORT 9103 Prometheus metrics HTTP port