:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
chore(clickhouse): guide agents to pick a query workload (#121453)
Bryan Ciaraldi132f85430027
diff --git a/.agents/skills/writing-clickhouse-queries/SKILL.md b/.agents/skills/writing-clickhouse-queries/SKILL.mdindex 48ceb3fe4a6e..3eada0eeda6a 100644--- a/.agents/skills/writing-clickhouse-queries/SKILL.md+++ b/.agents/skills/writing-clickhouse-queries/SKILL.md@@ -1,6 +1,6 @@---name: writing-clickhouse-queries-description: Guide for writing performant ClickHouse queries in PostHog product code. Use when writing HogQL query runners, designing a ClickHouse table for a new product, adding materialized columns or skip indexes, or choosing a row ID format. For optimizing an existing query that is already too slow, use `/optimizing-clickhouse-and-hogql-queries` instead.+description: Guide for writing performant ClickHouse queries in PostHog product code. Use when writing HogQL query runners, adding or reviewing a ClickHouse call site (`sync_execute`, `execute_hogql_query`, a `QueryRunner`) and choosing its `Workload` (ONLINE, OFFLINE, LOGS, ENDPOINTS), designing a ClickHouse table for a new product, adding materialized columns or skip indexes, or choosing a row ID format. For optimizing an existing query that is already too slow, use `/optimizing-clickhouse-and-hogql-queries` instead.---# Writing ClickHouse queries for new products@@ -18,6 +18,7 @@ Then pull in whichever related docs the task touches:## When to use- Writing or reviewing a `QueryRunner` subclass in `posthog/hogql_queries/` or `products/*/backend/`+- Adding or reviewing any ClickHouse call site, in request code, a Celery task, a Temporal activity, or a Dagster asset: pick its `Workload` with [`references/choosing-a-workload.md`](references/choosing-a-workload.md)- Adding a new ClickHouse table or ALTER for a product (`posthog/clickhouse/migrations/`)- Choosing a row ID format for a new table- Adding or removing materialized columns, skip indexes, or projections@@ -25,3 +26,11 @@ Then pull in whichever related docs the task touches:For investigating an existing slow query, debugging a `system.query_log` row, or reviewing a proposed HogQL printer change for performance, use [`/optimizing-clickhouse-and-hogql-queries`](../optimizing-clickhouse-and-hogql-queries/SKILL.md).Not the right skill for: customer-facing ad-hoc HogQL via Max / `posthog:execute-sql`, use `query-examples` for that. For migration mechanics (node roles, engines, replication), use `clickhouse-migrations`.++## Choose a workload on purpose++Every call site passes an explicit `workload=`, and the PR description says which one and why.+`ONLINE` is only for results a person waits for on screen.+Work that no person waits for (scheduled Celery tasks, most Temporal activities, Dagster assets, exports, backfills) is `OFFLINE`.+The workload is a tag of intent; the pod's `CLICKHOUSE_HOST` decides which replicas run the query.+Read [`references/choosing-a-workload.md`](references/choosing-a-workload.md) for the decision table, the overrides in `sync_execute`, and how to check where queries land.diff --git a/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md b/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.mdnew file mode 100644index 000000000000..2f459b368969--- /dev/null+++ b/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md@@ -0,0 +1,69 @@+# Choosing a ClickHouse workload++Every ClickHouse call takes a `Workload` from [`posthog/clickhouse/workload.py`](../../../../posthog/clickhouse/workload.py):+`sync_execute(..., workload=)`, `execute_hogql_query(..., workload=)`, and `QueryRunner(..., workload=)`.+The workload sets two things:+the cluster the query runs on, and the `workload` query tag that `query_log_archive.lc_workload` records.+Both feed capacity planning, so pick the workload on purpose.++## The rule++- Pass `workload=` explicitly at every new call site.+ Do not rely on `Workload.DEFAULT`, unless the code runs both in a request and in the background and must follow its caller.+- Say which workload you picked and why in the PR description.+ If the reason is not clear from where the code runs, add a one-line comment at the call site, for example+ `workload=Workload.OFFLINE, # nightly recompute, no user waits for the result`.+- When you review a diff that adds a ClickHouse call, check that its workload matches the questions below.++## Pick the workload++Answer in order and stop at the first yes.++| Question | Workload |+| ------------------------------------------------------------------------------- | -------------------- |+| Does the query read a table that exists only on the logs cluster? | `LOGS` |+| Does the query execute an Endpoints (the product) request? | `ENDPOINTS` |+| Does the query target the native events schema cluster? | `NEW_EVENTS_CLUSTER` |+| Does a person wait on screen for this result (an app request, a Max chat turn)? | `ONLINE` |+| Anything else: scheduled, batch, export, backfill, recompute, eval, digest | `OFFLINE` |++`ONLINE` is for latency a person feels.+A background job does not get faster with `ONLINE`.+It competes with interactive queries for the online replicas, or it runs on the offline replicas anyway and only gets the wrong tag (see below).++`OFFLINE` turns off hedged requests, so ClickHouse does not send a second copy of a slow query to another replica.+This keeps offline load off the online replicas when the offline replicas are busy.+Latency and failure rates are higher as a result, so offline work must tolerate retries.++## How a workload resolves++`sync_execute` in [`posthog/clickhouse/client/execute.py`](../../../../posthog/clickhouse/client/execute.py) can change the workload you pass:++1. `DEFAULT` with API-key auth (personal API key, project secret key) becomes `OFFLINE`.+2. `DEFAULT` in a Celery task becomes `OFFLINE`, except cache warming (`Feature.CACHE_WARMUP`), which stays `ONLINE`.+3. `process_query_task` forces `ONLINE` for app users and `OFFLINE` for API-key users, **even when you pass a workload explicitly**. Only `LOGS` is exempt.+4. A `workload=ENDPOINTS` query tag becomes `ENDPOINTS`.+5. Any `DEFAULT` that is left becomes the process default.+ The process default is `CLICKHOUSE_DEFAULT_WORKLOAD`, which is `ONLINE` unless the deployment sets it.+ Celery switches its process default to `OFFLINE` before each task.++## The tag and the host are separate++The workload is intent.+The host comes from the environment of the pod that runs the code, in [`get_kwargs_for_client`](../../../../posthog/clickhouse/client/connection.py):++- `ONLINE` connects to `CLICKHOUSE_HOST`.+- `OFFLINE` connects to `CLICKHOUSE_OFFLINE_CLUSTER_HOST` when the deployment sets it, and to `CLICKHOUSE_HOST` when it does not.+- `LOGS`, `ENDPOINTS`, and `NEW_EVENTS_CLUSTER` connect to their own hosts.++A background worker deployment can point `CLICKHOUSE_HOST` at the offline replicas and set `CLICKHOUSE_DEFAULT_WORKLOAD=OFFLINE`.+Then `ONLINE` and `OFFLINE` from those pods both run on the offline replicas.+Passing `ONLINE` there does not move the query to the online replicas.+It only tags an offline query as online, which hides it from per-workload capacity numbers.+Moving a query between clusters is a deployment change, not a code change.++## Check where queries land++`posthog.query_log_archive` records both the intent and the host.+Group recent initial queries by `lc_workload` and by whether `hostname` is an offline node.+An `ONLINE` tag on an offline host, from a service that is not in a request path, is a mislabeled call site.diff --git a/AGENTS.md b/AGENTS.mdindex 87224c9c1c94..0229afe2034d 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -321,6 +321,7 @@ ALWAYS invoke the matching skill **first** — do not skip it, and do not attemp- `/sending-notifications` — adding notification support- `/routing-outbound-api-calls` — adding or changing Python code that calls a third-party HTTP API, or changing any domain under `posthog/egress/` (budget, lanes, identity, metrics, headers)- `/authenticating-to-clickhouse` — adding a service, sidecar, or container that connects to ClickHouse, adding a `ClickHouseUser`, or building or changing a ClickHouse pool or client by hand; it must stay token-aware (a native pool carries the credential provider, an HTTP client resolves the token per call) or it silently uses the static password+- `/writing-clickhouse-queries` — adding or changing a ClickHouse call site (`sync_execute`, `execute_hogql_query`, a `QueryRunner`), including the `Workload` it passes- `/adding-activity-logging` — adding activity logging (the audit trail) to a model, writing or changing a `model_activity_signal` receiver or an activity describer, auditing which write paths of a model are logged, or debugging a change that is missing from the activity log- `/adding-inbound-webhooks` — adding a webhook endpoint for a third party that sends to PostHog, adding a consumer for a provider that already has an endpoint, or migrating a verifier the `inbound-webhooks-go-through-ingress` rule flags- `/writing-skills` — creating or updating skills in `.agents/skills/`diff --git a/posthog/clickhouse/workload.py b/posthog/clickhouse/workload.pyindex 24cc9db7cb9b..dade6763b568 100644--- a/posthog/clickhouse/workload.py+++ b/posthog/clickhouse/workload.py@@ -1,14 +1,16 @@from enum import StrEnum+# The workload records intent in the query tag. The pod's CLICKHOUSE_HOST decides which replicas run the query.+# See .agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md before you pick one.class Workload(StrEnum):- # Default workload+ # Follows the caller: sync_execute resolves it to ONLINE or OFFLINE. Avoid in new call sites.DEFAULT = "DEFAULT"- # Analytics queries, other 'lively' queries+ # A person waits on screen for the resultONLINE = "ONLINE"- # Historical exports, other long-running processes where latency is less critical+ # No person waits: scheduled jobs, exports, backfills, recomputes. Turns off hedged requests.OFFLINE = "OFFLINE"- # Logs queries+ # Tables that exist only on the logs clusterLOGS = "LOGS"NEW_EVENTS_CLUSTER = "NEW_EVENTS_CLUSTER"# Endpoints (the product) queries