Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(experiments): document the recalculation clickhouse user (#119922)

Rodrigo Iloro88a6fad301f5
diff --git a/products/experiments/backend/temporal/ARCHITECTURE.md b/products/experiments/backend/temporal/ARCHITECTURE.md
index 9e6ce119da60..e2ad570a72f1 100644
--- a/products/experiments/backend/temporal/ARCHITECTURE.md
+++ b/products/experiments/backend/temporal/ARCHITECTURE.md
@@ -283,10 +283,19 @@ The split is intentional: Grafana tells you about the worker process, PostHog te
Temporal delivers a cancellation only in a heartbeat response, so a workflow cancel or a server-side timeout now reaches the running attempt and starts the cleanup above.
A dead worker misses its heartbeats, so Temporal retries the metric after 60s instead of after the full 300s attempt.
See [cancellation and result-write protection](../../../../docs/internal/experiment-metric-recalculation.md) for the locking contract and cleanup limits.
-- **Calc queries run on the offline ClickHouse nodes, as the default user.** The calc activity builds its `ExperimentQueryRunner` with `workload=Workload.ONLINE`. That workload uses `CLICKHOUSE_HOST`, and the recalculation worker deployments set `CLICKHOUSE_HOST` to the offline host. So the queries reach the offline replicas, not the replicas that serve interactive product traffic, and they count against the default user's per-node limit on concurrent queries. The activity task queue above keeps recalculation under that limit.
+- **Calc queries run on the offline ClickHouse nodes.** The calc activity builds its `ExperimentQueryRunner` with `workload=Workload.ONLINE`. That workload uses `CLICKHOUSE_HOST`, and the recalculation worker deployments set `CLICKHOUSE_HOST` to the offline host. So the queries reach the offline replicas, not the replicas that serve interactive product traffic. The activity task queue above caps how many of them run at once.
The workload stays set explicitly rather than left to the default. The base `QueryRunner` default is `Workload.DEFAULT`, which follows the process default (Celery pins every task to offline process-wide, and API-key traffic is forced offline). Passing `Workload.ONLINE` keeps the routing the same however the activity is invoked, and the worker's host setting decides where it goes.
+ Without the flag below, the queries run as the `default` user. The calc tags them with the cache-warming feature, so `sync_execute` asks for the `cache_warmup` user. The recalculation workers do not wire that user, so `get_clickhouse_creds` falls back to the pod's `default` user, and the queries count against its per-node limit on concurrent queries, which other products share.
+
+ The `experiment-recalculation-clickhouse-user` flag, targeted by organization, moves the queries to the dedicated `experiments` user (`ClickHouseUser.EXPERIMENTS`), which has its own per-node limit. The calc activity evaluates the flag on each attempt, locally and without flag events, and sets `ch_user` in the `tags_context` around the runner. Every query in that context that names no user takes it, the precompute inserts too. The tag wins over the cache-warming fallback. A flag error keeps the `default` routing. Some queries keep their own user: the HogQL metadata lookups name theirs, the query kill on cancellation uses the pod's user, and the live-progress read on the GET path runs as `app`.
+
+ Operating rules:
+ - Deploy order, per region: the ClickHouse user and its token grant for both worker identities, then the token mount with `CLICKHOUSE_EXPERIMENTS_USER` and `CLICKHOUSE_EXPERIMENTS_PASSWORD_FILE` on both worker deployments, then this code, then the flag for one organization at a time. A query sent as a user the cluster does not know fails, so the flag goes on only after the first two steps are live.
+ - Verify with the `user` column of `system.query_log` for the organization's recalculation queries, not with the flag state. A worker without `CLICKHOUSE_EXPERIMENTS_USER` or `CLICKHOUSE_EXPERIMENTS_PASSWORD_FILE` falls back to `default` with no error. A worker with both variables set but no file at that path still sends `experiments`, with an empty password, so its queries fail authentication and the activity retries them.
+ - To roll back, turn the flag off. The next attempt of each metric uses the `default` routing, with no deploy and no workflow change.
+
- **No saved-metric snapshot.** Each calc activity that processes a saved/shared metric re-queries the M2M through-model to resolve the metric definition. For a 5–10 metric experiment this is fine; for larger experiments the per-call query starts to add up. Tracked as a deferred optimization (snapshot resolved metric dicts on the recalc row at discovery time).
## What this doc doesn't cover