Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

chore(experiments): document the recalculation activity task queue (#119685)

Rodrigo Iloroba0fcb0f4bcd
diff --git a/products/experiments/backend/temporal/ARCHITECTURE.md b/products/experiments/backend/temporal/ARCHITECTURE.md
index 486db50f9520..f0abe9eea739 100644
--- a/products/experiments/backend/temporal/ARCHITECTURE.md
+++ b/products/experiments/backend/temporal/ARCHITECTURE.md
@@ -239,7 +239,13 @@ The split is intentional: Grafana tells you about the worker process, PostHog te
## Performance notes
-- **Fan out all metrics at once, Temporal owns retries.** The workflow schedules one calc activity per metric in a single `asyncio.gather` (`recalculation_workflow.py`), with no concurrency of its own. Pacing is owned by the layers that actually constrain it: the worker's activity slots (`MAX_CONCURRENT_ACTIVITIES`, autoscaled on task-queue backlog) bound compute and query fan-out. The per-org ClickHouse app-query limiter does not apply inside Temporal activities. Scheduling every metric up front also keeps the task-queue backlog honest for the autoscaler. The workflow runs on its own `experiments-recalculation-task-queue` (see `EXPERIMENTS_RECALCULATION_TASK_QUEUE`), served by a dedicated worker deployment, so recalc load can be scaled and throttled independently of the general-purpose worker.
+- **Fan out all metrics at once, Temporal owns retries.** The workflow schedules one calc activity per metric in a single `asyncio.gather` (`recalculation_workflow.py`), with no concurrency of its own. The activity slots of the worker that runs the calc activity set the pacing. The per-org ClickHouse app-query limiter does not apply inside Temporal activities. Scheduling every metric up front also keeps the task-queue backlog honest for the autoscaler. The workflow runs on its own `experiments-recalculation-task-queue` (see `EXPERIMENTS_RECALCULATION_TASK_QUEUE`), served by a dedicated worker deployment, so recalc load can be scaled and throttled independently of the general-purpose worker. That deployment autoscales on backlog, so its slots (`MAX_CONCURRENT_ACTIVITIES` times the pod count) do not cap the number of concurrent queries. The activity task queue below does.
+
+- **Activity task queue with a fixed cap.** A scheduled burst can send more concurrent queries than the ClickHouse default user allows, and the rejections then also hit other products that share that user. The calc activity is also registered on `experiments-recalculation-activity-task-queue` (`EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE`), with no workflows. A fixed worker with no autoscaling polls it in each region, so its pod count times its slot count is a hard cap on concurrent recalculation queries for the runs routed there. The workflow and its light Postgres activities stay on the main queue.
+
+ Routing is per run. `start_metrics_recalculation_workflow` evaluates the `experiment-recalculation-activity-queue` flag for the run's organization, locally and without flag events, and stores the queue in the `activity_task_queue` workflow input. A flag error leaves the input empty. The workflow passes the input as `task_queue=` to the calc activity, gated on `workflow.patched("recalc-activity-task-queue-2026-10")`, and an empty input keeps the workflow's own queue. The workflow never reads the flag, so replay stays deterministic, and a flag change never moves a run in flight: Temporal retries an activity on the queue it was first scheduled on.
+
+ Operating rules. Turn the flag on for an organization only after the activity worker polls the queue in that region, because a queue with no poller holds the calculations until `schedule_to_close` fails them. To roll back, turn the flag off, then remove the worker only after every recalculation workflow that started before the flag change has closed. A fixed wait is not enough: such a run keeps its queue choice and schedules its calculations only after discovery and the start step, which take up to about 20 minutes of attempts and have no limit on queue wait. The 20-minute calc budget (`METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS`) starts only then, and the workflow has no execution timeout.
Retries live in Temporal's `RetryPolicy` on the calc activity, not in the workflow. The policy is exponential: `initial_interval=5s`, `backoff_coefficient=2.0`, `maximum_interval=60s`, `maximum_attempts=MAX_METRIC_ATTEMPTS` (currently 8). So a metric backs off 5s, 10s, 20s, 40s, 60s, 60s, 60s across its retries, then the eighth attempt is final. `asyncio.gather(..., return_exceptions=True)` lets healthy metrics finish while a failing one retries; a retrying metric never blocks the others, because each activity retries on its own worker slot rather than holding a shared queue slot.
@@ -265,11 +271,9 @@ The split is intentional: Grafana tells you about the worker process, PostHog te
Temporal delivers a cancellation only in a heartbeat response, so a workflow cancel or a server-side timeout now reaches the running attempt and starts the cleanup above.
A dead worker misses its heartbeats, so Temporal retries the metric after 60s instead of after the full 300s attempt.
See [cancellation and result-write protection](../../../../docs/internal/experiment-metric-recalculation.md) for the locking contract and cleanup limits.
-- **Calc queries run on the online ClickHouse cluster.** The calc activity builds its `ExperimentQueryRunner` with `workload=Workload.ONLINE`, so the queries hit the same replicas that serve interactive product traffic rather than the offline cluster that heavy background jobs use.
-
- This is a deliberate trade-off against the offline default that most background scans take. The offline replicas can trail ingestion, disable hedged requests (higher and more variable latency, higher failure rate), and share one global concurrency limit across every product. A recalc is a user-initiated snapshot the person is waiting on, so it wants the freshest data and the most reliable, lowest-latency path, which is the online cluster. The cost is that recalc scans now compete with live user queries instead of being isolated from them; the worker's activity-slot cap and the per-org ClickHouse app-query limiter are what keep that load bounded.
+- **Calc queries run on the offline ClickHouse nodes, as the default user.** The calc activity builds its `ExperimentQueryRunner` with `workload=Workload.ONLINE`. That workload uses `CLICKHOUSE_HOST`, and the recalculation worker deployments set `CLICKHOUSE_HOST` to the offline host. So the queries reach the offline replicas, not the replicas that serve interactive product traffic, and they count against the default user's per-node limit on concurrent queries. The activity task queue above keeps recalculation under that limit.
- The workload is set explicitly rather than left to the default. The base `QueryRunner` default is `Workload.DEFAULT`, which resolves to online today but flips to offline whenever the process default is offline (Celery pins every task to offline process-wide, and API-key traffic is forced offline). Passing `Workload.ONLINE` keeps the routing stable regardless of how the activity is invoked. Note that only US Cloud has a real offline cluster; EU, self-hosted, dev, and test all resolve offline back to the online host, so this change is a no-op outside US Cloud.
+ The workload stays set explicitly rather than left to the default. The base `QueryRunner` default is `Workload.DEFAULT`, which follows the process default (Celery pins every task to offline process-wide, and API-key traffic is forced offline). Passing `Workload.ONLINE` keeps the routing the same however the activity is invoked, and the worker's host setting decides where it goes.
- **No saved-metric snapshot.** Each calc activity that processes a saved/shared metric re-queries the M2M through-model to resolve the metric definition. For a 5–10 metric experiment this is fine; for larger experiments the per-call query starts to add up. Tracked as a deferred optimization (snapshot resolved metric dicts on the recalc row at discovery time).