:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(clickhouse): guide ingestion role choice (#129632)
Bryan Ciaraldie5bb73894b9b
diff --git a/.agents/skills/clickhouse-migrations/SKILL.md b/.agents/skills/clickhouse-migrations/SKILL.mdindex 349630b63747..cae5c87997aa 100644--- a/.agents/skills/clickhouse-migrations/SKILL.md+++ b/.agents/skills/clickhouse-migrations/SKILL.md@@ -35,7 +35,10 @@ Pick the role by **which cluster owns the object**, not by its type alone:tables in front of sharded or satellite tables, views, dictionaries. A non-sharded replicated table on`DATA` keeps a full copy on every main-cluster node, so it is the wrong home for a product rollup orpre-aggregate (see below).-- `[NodeRole.INGESTION_SMALL]`: writable tables, Kafka tables, materialized views on the ingestion layer+- `[NodeRole.INGESTION_SMALL]`, `[NodeRole.INGESTION_MEDIUM]`, `[NodeRole.INGESTION_EVENTS]`: writable tables, Kafka tables, materialized views on the ingestion layer.+ `INGESTION_EVENTS` is for consumers of all of `clickhouse_events_json`.+ For other topics, check where comparable consumers run in each environment, and use `INGESTION_SMALL` or `INGESTION_MEDIUM` to match.+ Use `INGESTION_SMALL` when you can't tell.- `[NodeRole.APM]`: the `ingestion-apm` nodes, the metrics ingestion layer. Kafka tables, `Null` inputtables, materialized views and writable `Distributed` tables that write into storage on `LOGS`.Stateless like the other ingestion nodes, so storage tables never go here.@@ -86,11 +89,13 @@ SETTINGS kafka_skip_broken_messages = 100,kafka_num_consumers = {kafka_num_consumers(1)},kafka_thread_per_consumer = 1,kafka_poll_timeout_ms = 10000,+ kafka_flush_interval_ms = 7500,kafka_max_block_size = 100000```- `kafka_skip_broken_messages` defaults to 0, where one malformed message stops the consumer for good.- `kafka_poll_timeout_ms` is 10000 for WarpStream, which does not support `fetch.min.bytes`.+- `kafka_flush_interval_ms` matches the default of 7500. Set it so the table shows its flush interval.- `kafka_max_block_size` defaults to about a million rows, so 100000 is a reduction; lower for widerows, not below ~1000.- `kafka_num_consumers` is per node, so the role's node count multiplies it; the total must notdiff --git a/posthog/clickhouse/migrations/AGENTS.md b/posthog/clickhouse/migrations/AGENTS.mdindex d6651d810d28..466baaf98a06 100644--- a/posthog/clickhouse/migrations/AGENTS.md+++ b/posthog/clickhouse/migrations/AGENTS.md@@ -337,6 +337,9 @@ run_sql_with_exceptions()```+`INGESTION_SMALL` is only the default.+Choose small, medium or events with the node role guide in `.agents/skills/clickhouse-migrations/SKILL.md`.+### Required settings`kafka_engine()` returns the ENGINE clause only, so each table appends its own `SETTINGS`.@@ -347,6 +350,7 @@ SETTINGS kafka_skip_broken_messages = 100,kafka_num_consumers = {kafka_num_consumers(1)},kafka_thread_per_consumer = 1,kafka_poll_timeout_ms = 10000,+ kafka_flush_interval_ms = 7500,kafka_max_block_size = 100000```@@ -357,8 +361,8 @@ SETTINGS kafka_skip_broken_messages = 100,`fetch.min.bytes`, so a short timeout buys many small fetches instead of a few large ones.- `kafka_thread_per_consumer` serializes flushes when off, and changes nothing unless`kafka_num_consumers` is above 1.-- Leave `kafka_flush_interval_ms` unset; it defaults to 7500. Leave `kafka_max_wait_ms` alone, it is- a user-profile setting.+- `kafka_flush_interval_ms` matches the default of 7500. Set it so the table shows its flush interval.+- Leave `kafka_max_wait_ms` alone, it is a user-profile setting.### Sizing@@ -371,8 +375,8 @@ about 1000 to avoid trading tiny parts for merge pressure.`kafka_num_consumers` is per node. Nodes of a role share the topic's partitions and a consumer withno partition still holds a thread, so the total across the role must not exceed the partition count— a role on many nodes reaches that ceiling at 1. Each consumer also takes a slot in-`background_message_broker_schedule_pool_size`, a server-level pool of 16 shared by every Kafka-table on the node. Resolve the count through `kafka_num_consumers()` in+`background_message_broker_schedule_pool_size`, a server-level pool shared by every Kafka+table on the node. Check its live size with Housekeeper. Resolve the count through `kafka_num_consumers()` in`posthog/clickhouse/kafka_engine.py`, which returns 1 outside deployed cloud.### Changing a Kafka table or its materialized view