Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(clickhouse): guide ingestion role choice (#129632)

Bryan Ciaraldie5bb73894b9b
diff --git a/.agents/skills/clickhouse-migrations/SKILL.md b/.agents/skills/clickhouse-migrations/SKILL.md
index 349630b63747..cae5c87997aa 100644
--- a/.agents/skills/clickhouse-migrations/SKILL.md
+++ b/.agents/skills/clickhouse-migrations/SKILL.md
@@ -35,7 +35,10 @@ Pick the role by **which cluster owns the object**, not by its type alone:
tables in front of sharded or satellite tables, views, dictionaries. A non-sharded replicated table on
`DATA` keeps a full copy on every main-cluster node, so it is the wrong home for a product rollup or
pre-aggregate (see below).
-- `[NodeRole.INGESTION_SMALL]`: writable tables, Kafka tables, materialized views on the ingestion layer
+- `[NodeRole.INGESTION_SMALL]`, `[NodeRole.INGESTION_MEDIUM]`, `[NodeRole.INGESTION_EVENTS]`: writable tables, Kafka tables, materialized views on the ingestion layer.
+ `INGESTION_EVENTS` is for consumers of all of `clickhouse_events_json`.
+ For other topics, check where comparable consumers run in each environment, and use `INGESTION_SMALL` or `INGESTION_MEDIUM` to match.
+ Use `INGESTION_SMALL` when you can't tell.
- `[NodeRole.APM]`: the `ingestion-apm` nodes, the metrics ingestion layer. Kafka tables, `Null` input
tables, materialized views and writable `Distributed` tables that write into storage on `LOGS`.
Stateless like the other ingestion nodes, so storage tables never go here.
@@ -86,11 +89,13 @@ SETTINGS kafka_skip_broken_messages = 100,
kafka_num_consumers = {kafka_num_consumers(1)},
kafka_thread_per_consumer = 1,
kafka_poll_timeout_ms = 10000,
+ kafka_flush_interval_ms = 7500,
kafka_max_block_size = 100000
```
- `kafka_skip_broken_messages` defaults to 0, where one malformed message stops the consumer for good.
- `kafka_poll_timeout_ms` is 10000 for WarpStream, which does not support `fetch.min.bytes`.
+- `kafka_flush_interval_ms` matches the default of 7500. Set it so the table shows its flush interval.
- `kafka_max_block_size` defaults to about a million rows, so 100000 is a reduction; lower for wide
rows, not below ~1000.
- `kafka_num_consumers` is per node, so the role's node count multiplies it; the total must not
diff --git a/posthog/clickhouse/migrations/AGENTS.md b/posthog/clickhouse/migrations/AGENTS.md
index d6651d810d28..466baaf98a06 100644
--- a/posthog/clickhouse/migrations/AGENTS.md
+++ b/posthog/clickhouse/migrations/AGENTS.md
@@ -337,6 +337,9 @@ run_sql_with_exceptions(
)
```
+`INGESTION_SMALL` is only the default.
+Choose small, medium or events with the node role guide in `.agents/skills/clickhouse-migrations/SKILL.md`.
+
### Required settings
`kafka_engine()` returns the ENGINE clause only, so each table appends its own `SETTINGS`.
@@ -347,6 +350,7 @@ SETTINGS kafka_skip_broken_messages = 100,
kafka_num_consumers = {kafka_num_consumers(1)},
kafka_thread_per_consumer = 1,
kafka_poll_timeout_ms = 10000,
+ kafka_flush_interval_ms = 7500,
kafka_max_block_size = 100000
```
@@ -357,8 +361,8 @@ SETTINGS kafka_skip_broken_messages = 100,
`fetch.min.bytes`, so a short timeout buys many small fetches instead of a few large ones.
- `kafka_thread_per_consumer` serializes flushes when off, and changes nothing unless
`kafka_num_consumers` is above 1.
-- Leave `kafka_flush_interval_ms` unset; it defaults to 7500. Leave `kafka_max_wait_ms` alone, it is
- a user-profile setting.
+- `kafka_flush_interval_ms` matches the default of 7500. Set it so the table shows its flush interval.
+- Leave `kafka_max_wait_ms` alone, it is a user-profile setting.
### Sizing
@@ -371,8 +375,8 @@ about 1000 to avoid trading tiny parts for merge pressure.
`kafka_num_consumers` is per node. Nodes of a role share the topic's partitions and a consumer with
no partition still holds a thread, so the total across the role must not exceed the partition count
— a role on many nodes reaches that ceiling at 1. Each consumer also takes a slot in
-`background_message_broker_schedule_pool_size`, a server-level pool of 16 shared by every Kafka
-table on the node. Resolve the count through `kafka_num_consumers()` in
+`background_message_broker_schedule_pool_size`, a server-level pool shared by every Kafka
+table on the node. Check its live size with Housekeeper. Resolve the count through `kafka_num_consumers()` in
`posthog/clickhouse/kafka_engine.py`, which returns 1 outside deployed cloud.
### Changing a Kafka table or its materialized view