Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(clickhouse): guide agents to pick a query workload (#121453)

Bryan Ciaraldi132f85430027
diff --git a/.agents/skills/writing-clickhouse-queries/SKILL.md b/.agents/skills/writing-clickhouse-queries/SKILL.md
index 48ceb3fe4a6e..3eada0eeda6a 100644
--- a/.agents/skills/writing-clickhouse-queries/SKILL.md
+++ b/.agents/skills/writing-clickhouse-queries/SKILL.md
@@ -1,6 +1,6 @@
---
name: writing-clickhouse-queries
-description: Guide for writing performant ClickHouse queries in PostHog product code. Use when writing HogQL query runners, designing a ClickHouse table for a new product, adding materialized columns or skip indexes, or choosing a row ID format. For optimizing an existing query that is already too slow, use `/optimizing-clickhouse-and-hogql-queries` instead.
+description: Guide for writing performant ClickHouse queries in PostHog product code. Use when writing HogQL query runners, adding or reviewing a ClickHouse call site (`sync_execute`, `execute_hogql_query`, a `QueryRunner`) and choosing its `Workload` (ONLINE, OFFLINE, LOGS, ENDPOINTS), designing a ClickHouse table for a new product, adding materialized columns or skip indexes, or choosing a row ID format. For optimizing an existing query that is already too slow, use `/optimizing-clickhouse-and-hogql-queries` instead.
---
# Writing ClickHouse queries for new products
@@ -18,6 +18,7 @@ Then pull in whichever related docs the task touches:
## When to use
- Writing or reviewing a `QueryRunner` subclass in `posthog/hogql_queries/` or `products/*/backend/`
+- Adding or reviewing any ClickHouse call site, in request code, a Celery task, a Temporal activity, or a Dagster asset: pick its `Workload` with [`references/choosing-a-workload.md`](references/choosing-a-workload.md)
- Adding a new ClickHouse table or ALTER for a product (`posthog/clickhouse/migrations/`)
- Choosing a row ID format for a new table
- Adding or removing materialized columns, skip indexes, or projections
@@ -25,3 +26,11 @@ Then pull in whichever related docs the task touches:
For investigating an existing slow query, debugging a `system.query_log` row, or reviewing a proposed HogQL printer change for performance, use [`/optimizing-clickhouse-and-hogql-queries`](../optimizing-clickhouse-and-hogql-queries/SKILL.md).
Not the right skill for: customer-facing ad-hoc HogQL via Max / `posthog:execute-sql`, use `query-examples` for that. For migration mechanics (node roles, engines, replication), use `clickhouse-migrations`.
+
+## Choose a workload on purpose
+
+Every call site passes an explicit `workload=`, and the PR description says which one and why.
+`ONLINE` is only for results a person waits for on screen.
+Work that no person waits for (scheduled Celery tasks, most Temporal activities, Dagster assets, exports, backfills) is `OFFLINE`.
+The workload is a tag of intent; the pod's `CLICKHOUSE_HOST` decides which replicas run the query.
+Read [`references/choosing-a-workload.md`](references/choosing-a-workload.md) for the decision table, the overrides in `sync_execute`, and how to check where queries land.
diff --git a/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md b/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md
new file mode 100644
index 000000000000..2f459b368969
--- /dev/null
+++ b/.agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md
@@ -0,0 +1,69 @@
+# Choosing a ClickHouse workload
+
+Every ClickHouse call takes a `Workload` from [`posthog/clickhouse/workload.py`](../../../../posthog/clickhouse/workload.py):
+`sync_execute(..., workload=)`, `execute_hogql_query(..., workload=)`, and `QueryRunner(..., workload=)`.
+The workload sets two things:
+the cluster the query runs on, and the `workload` query tag that `query_log_archive.lc_workload` records.
+Both feed capacity planning, so pick the workload on purpose.
+
+## The rule
+
+- Pass `workload=` explicitly at every new call site.
+ Do not rely on `Workload.DEFAULT`, unless the code runs both in a request and in the background and must follow its caller.
+- Say which workload you picked and why in the PR description.
+ If the reason is not clear from where the code runs, add a one-line comment at the call site, for example
+ `workload=Workload.OFFLINE, # nightly recompute, no user waits for the result`.
+- When you review a diff that adds a ClickHouse call, check that its workload matches the questions below.
+
+## Pick the workload
+
+Answer in order and stop at the first yes.
+
+| Question | Workload |
+| ------------------------------------------------------------------------------- | -------------------- |
+| Does the query read a table that exists only on the logs cluster? | `LOGS` |
+| Does the query execute an Endpoints (the product) request? | `ENDPOINTS` |
+| Does the query target the native events schema cluster? | `NEW_EVENTS_CLUSTER` |
+| Does a person wait on screen for this result (an app request, a Max chat turn)? | `ONLINE` |
+| Anything else: scheduled, batch, export, backfill, recompute, eval, digest | `OFFLINE` |
+
+`ONLINE` is for latency a person feels.
+A background job does not get faster with `ONLINE`.
+It competes with interactive queries for the online replicas, or it runs on the offline replicas anyway and only gets the wrong tag (see below).
+
+`OFFLINE` turns off hedged requests, so ClickHouse does not send a second copy of a slow query to another replica.
+This keeps offline load off the online replicas when the offline replicas are busy.
+Latency and failure rates are higher as a result, so offline work must tolerate retries.
+
+## How a workload resolves
+
+`sync_execute` in [`posthog/clickhouse/client/execute.py`](../../../../posthog/clickhouse/client/execute.py) can change the workload you pass:
+
+1. `DEFAULT` with API-key auth (personal API key, project secret key) becomes `OFFLINE`.
+2. `DEFAULT` in a Celery task becomes `OFFLINE`, except cache warming (`Feature.CACHE_WARMUP`), which stays `ONLINE`.
+3. `process_query_task` forces `ONLINE` for app users and `OFFLINE` for API-key users, **even when you pass a workload explicitly**. Only `LOGS` is exempt.
+4. A `workload=ENDPOINTS` query tag becomes `ENDPOINTS`.
+5. Any `DEFAULT` that is left becomes the process default.
+ The process default is `CLICKHOUSE_DEFAULT_WORKLOAD`, which is `ONLINE` unless the deployment sets it.
+ Celery switches its process default to `OFFLINE` before each task.
+
+## The tag and the host are separate
+
+The workload is intent.
+The host comes from the environment of the pod that runs the code, in [`get_kwargs_for_client`](../../../../posthog/clickhouse/client/connection.py):
+
+- `ONLINE` connects to `CLICKHOUSE_HOST`.
+- `OFFLINE` connects to `CLICKHOUSE_OFFLINE_CLUSTER_HOST` when the deployment sets it, and to `CLICKHOUSE_HOST` when it does not.
+- `LOGS`, `ENDPOINTS`, and `NEW_EVENTS_CLUSTER` connect to their own hosts.
+
+A background worker deployment can point `CLICKHOUSE_HOST` at the offline replicas and set `CLICKHOUSE_DEFAULT_WORKLOAD=OFFLINE`.
+Then `ONLINE` and `OFFLINE` from those pods both run on the offline replicas.
+Passing `ONLINE` there does not move the query to the online replicas.
+It only tags an offline query as online, which hides it from per-workload capacity numbers.
+Moving a query between clusters is a deployment change, not a code change.
+
+## Check where queries land
+
+`posthog.query_log_archive` records both the intent and the host.
+Group recent initial queries by `lc_workload` and by whether `hostname` is an offline node.
+An `ONLINE` tag on an offline host, from a service that is not in a request path, is a mislabeled call site.
diff --git a/AGENTS.md b/AGENTS.md
index 87224c9c1c94..0229afe2034d 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -321,6 +321,7 @@ ALWAYS invoke the matching skill **first** — do not skip it, and do not attemp
- `/sending-notifications` — adding notification support
- `/routing-outbound-api-calls` — adding or changing Python code that calls a third-party HTTP API, or changing any domain under `posthog/egress/` (budget, lanes, identity, metrics, headers)
- `/authenticating-to-clickhouse` — adding a service, sidecar, or container that connects to ClickHouse, adding a `ClickHouseUser`, or building or changing a ClickHouse pool or client by hand; it must stay token-aware (a native pool carries the credential provider, an HTTP client resolves the token per call) or it silently uses the static password
+- `/writing-clickhouse-queries` — adding or changing a ClickHouse call site (`sync_execute`, `execute_hogql_query`, a `QueryRunner`), including the `Workload` it passes
- `/adding-activity-logging` — adding activity logging (the audit trail) to a model, writing or changing a `model_activity_signal` receiver or an activity describer, auditing which write paths of a model are logged, or debugging a change that is missing from the activity log
- `/adding-inbound-webhooks` — adding a webhook endpoint for a third party that sends to PostHog, adding a consumer for a provider that already has an endpoint, or migrating a verifier the `inbound-webhooks-go-through-ingress` rule flags
- `/writing-skills` — creating or updating skills in `.agents/skills/`
diff --git a/posthog/clickhouse/workload.py b/posthog/clickhouse/workload.py
index 24cc9db7cb9b..dade6763b568 100644
--- a/posthog/clickhouse/workload.py
+++ b/posthog/clickhouse/workload.py
@@ -1,14 +1,16 @@
from enum import StrEnum
+# The workload records intent in the query tag. The pod's CLICKHOUSE_HOST decides which replicas run the query.
+# See .agents/skills/writing-clickhouse-queries/references/choosing-a-workload.md before you pick one.
class Workload(StrEnum):
- # Default workload
+ # Follows the caller: sync_execute resolves it to ONLINE or OFFLINE. Avoid in new call sites.
DEFAULT = "DEFAULT"
- # Analytics queries, other 'lively' queries
+ # A person waits on screen for the result
ONLINE = "ONLINE"
- # Historical exports, other long-running processes where latency is less critical
+ # No person waits: scheduled jobs, exports, backfills, recomputes. Turns off hedged requests.
OFFLINE = "OFFLINE"
- # Logs queries
+ # Tables that exist only on the logs cluster
LOGS = "LOGS"
NEW_EVENTS_CLUSTER = "NEW_EVENTS_CLUSTER"
# Endpoints (the product) queries