:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(experiments): pick the recalculation clickhouse user from its task queue (#131011)
Rodrigo Iloro3fae1f68f27f
diff --git a/products/experiments/backend/recalculation.py b/products/experiments/backend/recalculation.pyindex 24ee1a19b603..b6f117145e1c 100644--- a/products/experiments/backend/recalculation.py+++ b/products/experiments/backend/recalculation.py@@ -69,7 +69,6 @@ _QUEUED_RECALC_CEILING = timedelta(hours=1)MIN_USER_RECALCULATION_INTERVAL = timedelta(minutes=5)RECALCULATION_RATE_LIMIT_FEATURE_FLAG = "experiment-metric-recalculation-rate-limit"RECALCULATION_ACTIVITY_QUEUE_FEATURE_FLAG = "experiment-recalculation-activity-queue"-RECALCULATION_CLICKHOUSE_USER_FEATURE_FLAG = "experiment-recalculation-clickhouse-user"_RATE_LIMITED_TRIGGERS = frozenset({ExperimentMetricsRecalculation.Trigger.MANUAL,@@ -406,14 +405,13 @@ def _activity_task_queue(organization_id: str) -> str | None:return settings.EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE if enabled else None-def recalculation_clickhouse_user(organization_id: str) -> ClickHouseUser | None:- # None keeps today's routing, also on a flag error, because that routing always works.- try:- enabled = _organization_flag_enabled(RECALCULATION_CLICKHOUSE_USER_FEATURE_FLAG, organization_id)- except Exception as e:- capture_exception(e)- return None- return ClickHouseUser.EXPERIMENTS if enabled else None+def recalculation_clickhouse_user(task_queue: str) -> ClickHouseUser | None:+ # Only the slot-capped queues run as the experiments user; a queue that collapses into the main queue keeps the default.+ capped_task_queues = {+ settings.EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE,+ settings.EXPERIMENTS_RECALCULATION_PRECOMPUTE_TASK_QUEUE,+ } - {settings.EXPERIMENTS_RECALCULATION_TASK_QUEUE}+ return ClickHouseUser.EXPERIMENTS if task_queue in capped_task_queues else Nonedef _refresh_window_enforced(experiment: Experiment) -> bool:diff --git a/products/experiments/backend/temporal/ARCHITECTURE.md b/products/experiments/backend/temporal/ARCHITECTURE.mdindex 68a051f3d169..f2449d091fa8 100644--- a/products/experiments/backend/temporal/ARCHITECTURE.md+++ b/products/experiments/backend/temporal/ARCHITECTURE.md@@ -278,7 +278,7 @@ The split is intentional: Grafana tells you about the worker process, PostHog teAs a second check, the workflow uses the route only for a run with an `activity_task_queue`. Runs off the flag keep the workflow's own queue and the AUTO mode. The route needs no patch marker: on replay, sdk-core matches a scheduled activity only by its id and type (`on_activity_task_scheduled` in `activity_state_machine.rs`), not by its queue or arguments, and a discovery result from before routing has no route, so it dispatches as before. The DIRECT mode is a sixth positional argument of the calc activity, and an AUTO calc keeps five. In a rolling deploy, a worker that predates the argument fails a DIRECT calc with a retryable `TypeError` until it rolls.- Operating rules. Turn the flag on for an organization only after the activity worker and the precompute worker poll their queues in that region, because a queue with no poller holds the calculations, and a calculation on those queues has no `schedule_to_close` budget to fail it. To roll back, turn the flag off, then remove the worker only after every recalculation workflow that started before the flag change has closed. A fixed wait is not enough: such a run keeps its queue choice and schedules its calculations only after discovery and the start step, which take up to about 20 minutes of attempts and have no limit on queue wait. A calculation on the activity or precompute queue then has no limit on queue wait either, and the workflow has no execution timeout.+ Operating rules. The `experiment-recalculation-activity-queue` flag is the one rollout and rollback control for recalculation: the queue also picks the ClickHouse user (see "Calc queries run on the offline ClickHouse nodes"). Turn the flag on for an organization only after the activity worker and the precompute worker poll their queues in that region, because a queue with no poller holds the calculations, and a calculation on those queues has no `schedule_to_close` budget to fail it. To roll back, turn the flag off, then remove the worker only after every recalculation workflow that started before the flag change has closed. A fixed wait is not enough: such a run keeps its queue choice and schedules its calculations only after discovery and the start step, which take up to about 20 minutes of attempts and have no limit on queue wait. A calculation on the activity or precompute queue then has no limit on queue wait either, and the workflow has no execution timeout.Retries live in Temporal's `RetryPolicy` on the calc activity, not in the workflow. The policy is exponential: `initial_interval=5s`, `backoff_coefficient=2.0`, `maximum_interval=60s`, `maximum_attempts=MAX_METRIC_ATTEMPTS` (currently 8). So a metric backs off 5s, 10s, 20s, 40s, 60s, 60s, 60s across its retries, then the eighth attempt is final. `asyncio.gather(..., return_exceptions=True)` lets healthy metrics finish while a failing one retries; a retrying metric never blocks the others, because each activity retries on its own worker slot rather than holding a shared queue slot.@@ -309,14 +309,14 @@ The split is intentional: Grafana tells you about the worker process, PostHog teThe workload stays set explicitly rather than left to `Workload.DEFAULT`, which follows the process default and changes with how the activity is invoked.- Without the flag below, the queries run as the `default` user. The calc tags them with the cache-warming feature, so `sync_execute` asks for the `cache_warmup` user. The recalculation workers do not wire that user, so `get_clickhouse_creds` falls back to the pod's `default` user, and the queries count against its per-node limit on concurrent queries, which other products share.+ The ClickHouse user follows the task queue of the calc. The calc activity reads its queue from `activity.info().task_queue` and passes it to `recalculation_clickhouse_user`, which returns the dedicated `experiments` user (`ClickHouseUser.EXPERIMENTS`) for the activity queue and the precompute queue. Those queues have a fixed slot count, so the `experiments` user always runs under a cap on concurrent queries. The calc sets the user as `ch_user` in the `tags_context` around the runner. Every query in that context that names no user takes it, the precompute inserts too. The tag wins over the cache-warming fallback. Some queries keep their own user: the HogQL metadata lookups name theirs, the query kill on cancellation uses the pod's user, and the live-progress read on the GET path runs as `app`.- The `experiment-recalculation-clickhouse-user` flag, targeted by organization, moves the queries to the dedicated `experiments` user (`ClickHouseUser.EXPERIMENTS`), which has its own per-node limit. The calc activity evaluates the flag on each attempt, locally and without flag events, and sets `ch_user` in the `tags_context` around the runner. Every query in that context that names no user takes it, the precompute inserts too. The tag wins over the cache-warming fallback. A flag error keeps the `default` routing. Some queries keep their own user: the HogQL metadata lookups name theirs, the query kill on cancellation uses the pod's user, and the live-progress read on the GET path runs as `app`.+ A calc on the main queue sets no `ch_user`. The calc tags its queries with the cache-warming feature, so `sync_execute` asks for the `cache_warmup` user. The recalculation workers do not wire that user, so `get_clickhouse_creds` falls back to the pod's `default` user, and the queries count against its per-node limit on concurrent queries, which other products share. A queue that has the same name as the main queue counts as the main queue. Local development and single-queue mode collapse all three queues into one, so they run as `default`.Operating rules:- - Deploy order, per region: the ClickHouse user and its token grant for both worker identities, then the token mount with `CLICKHOUSE_EXPERIMENTS_USER` and `CLICKHOUSE_EXPERIMENTS_PASSWORD_FILE` on both worker deployments, then this code, then the flag for one organization at a time. A query sent as a user the cluster does not know fails, so the flag goes on only after the first two steps are live.- - Verify with the `user` column of `system.query_log` for the organization's recalculation queries, not with the flag state. A worker without `CLICKHOUSE_EXPERIMENTS_USER` or `CLICKHOUSE_EXPERIMENTS_PASSWORD_FILE` falls back to `default` with no error. A worker with both variables set but no file at that path still sends `experiments`, with an empty password, so its queries fail authentication and the activity retries them.- - To roll back, turn the flag off. The next attempt of each metric uses the `default` routing, with no deploy and no workflow change.+ - Both the activity worker and the precompute worker need the `experiments` user, its token grant, and the token mount (`CLICKHOUSE_EXPERIMENTS_USER` and `CLICKHOUSE_EXPERIMENTS_PASSWORD_FILE`). A query sent as a user the cluster does not know fails.+ - Verify with `query_log_archive`: group the calc queries by `lc_service_name` (the worker deployment, one per queue) and `user`. The activity and precompute workers send only `experiments`, the main worker only `default`. A worker without `CLICKHOUSE_EXPERIMENTS_USER` or `CLICKHOUSE_EXPERIMENTS_PASSWORD_FILE` falls back to `default` with no error. A worker with both variables set but no file at that path still sends `experiments`, with an empty password, so its queries fail authentication and the activity retries them.+ - To roll back, turn the `experiment-recalculation-activity-queue` flag off. New runs then stay on the main queue and run as `default`. A run in flight keeps its queue, and so its user, for all retries.- **No saved-metric snapshot.** Each calc activity that processes a saved/shared metric re-queries the M2M through-model to resolve the metric definition. For a 5–10 metric experiment this is fine; for larger experiments the per-call query starts to add up. Tracked as a deferred optimization (snapshot resolved metric dicts on the recalc row at discovery time).diff --git a/products/experiments/backend/temporal/recalculation_activities.py b/products/experiments/backend/temporal/recalculation_activities.pyindex e7556c1e691a..1db54f7e2aa6 100644--- a/products/experiments/backend/temporal/recalculation_activities.py+++ b/products/experiments/backend/temporal/recalculation_activities.py@@ -130,7 +130,8 @@ async def calculate_experiment_metric_for_recalculation(outcome once Temporal stops retrying. When Temporal ends the metric earlier (the schedule_to_close budget, or akill from the outside), no attempt persists it and the workflow runs record_metric_out_of_time_failure."""- attempt = temporalio.activity.info().attempt+ activity_info = temporalio.activity.info()+ attempt = activity_info.attemptis_final_attempt = attempt >= MAX_METRIC_ATTEMPTSdeadline = asyncio.get_running_loop().time() + METRIC_CALC_ACTIVITY_TIMEOUT_SECONDS# The heartbeat runs on the event loop beside the body's thread. Temporal delivers a cancellation (a workflow@@ -147,6 +148,7 @@ async def calculate_experiment_metric_for_recalculation(attempt,precomputation_mode,calculation_key_version,+ activity_info.task_queue,))try:diff --git a/products/experiments/backend/temporal/recalculation_logic.py b/products/experiments/backend/temporal/recalculation_logic.pyindex e08d13654e40..b1db81aaae70 100644--- a/products/experiments/backend/temporal/recalculation_logic.py+++ b/products/experiments/backend/temporal/recalculation_logic.py@@ -701,6 +701,7 @@ def _calculate_experiment_metric_for_recalculation_sync(attempt: int = 1,precomputation_mode: PrecomputationMode | None = None,calculation_key_version: int = 1,+ task_queue: str = "",) -> MetricRecalculationResult:close_old_connections()@@ -864,7 +865,7 @@ def _calculate_experiment_metric_for_recalculation_sync(feature=Feature.CACHE_WARMUP,client_query_id=client_query_id,# Also covers the precompute inserts, which run inside this context.- ch_user=recalculation_clickhouse_user(str(experiment.team.organization_id)),+ ch_user=recalculation_clickhouse_user(task_queue),):result = runner.run(execution_mode=ExecutionMode.CALCULATE_BLOCKING_ALWAYS)result_dict = sanitize_non_finite(result.model_dump(mode="json"))diff --git a/products/experiments/backend/temporal/test_recalculation_activities.py b/products/experiments/backend/temporal/test_recalculation_activities.pyindex f2b89bf0bce3..2664c9e83e87 100644--- a/products/experiments/backend/temporal/test_recalculation_activities.py+++ b/products/experiments/backend/temporal/test_recalculation_activities.py@@ -101,6 +101,7 @@ def _calculate(attempt: int = 1,precomputation_mode: PrecomputationMode | None = None,calculation_key_version: int = 1,+ task_queue: str = "",):with patch("products.experiments.backend.temporal.recalculation_logic.close_old_connections"):return _calculate_experiment_metric_for_recalculation_sync(@@ -113,6 +114,7 @@ def _calculate(attempt,precomputation_mode,calculation_key_version,+ task_queue,)@@ -1214,12 +1216,15 @@ class TestCalculateActivity(BaseTest):@parameterized.expand([- ("flag_on", {"return_value": True}, ClickHouseUser.EXPERIMENTS),- ("flag_off", {"return_value": False}, ClickHouseUser.CACHE_WARMUP),- ("flag_error", {"side_effect": RuntimeError("flags unavailable")}, ClickHouseUser.CACHE_WARMUP),+ ("activity_queue", "activity-q", "activity-q", ClickHouseUser.EXPERIMENTS),+ ("precompute_queue", "activity-q", "precompute-q", ClickHouseUser.EXPERIMENTS),+ ("main_queue", "activity-q", "main-q", ClickHouseUser.CACHE_WARMUP),+ ("collapsed_queues", "main-q", "main-q", ClickHouseUser.CACHE_WARMUP),])- def test_queries_run_as_the_flagged_clickhouse_user(self, name: str, flag: dict, expected_user: ClickHouseUser):+ def test_queries_run_as_the_clickhouse_user_of_their_task_queue(+ self, name: str, activity_task_queue: str, task_queue: str, expected_user: ClickHouseUser+ ):exp = self._experiment(flag_key=f"calc-ch-user-{name}", metrics=[_mean_metric("m1")])recalc = self._recalc(exp, metric_uuids=["m1"])@@ -1231,13 +1236,17 @@ class TestCalculateActivity(BaseTest):return MagicMock(model_dump=MagicMock(return_value={}))with (- patch("products.experiments.backend.recalculation.feature_enabled_or_false", **flag),+ override_settings(+ EXPERIMENTS_RECALCULATION_TASK_QUEUE="main-q",+ EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE=activity_task_queue,+ EXPERIMENTS_RECALCULATION_PRECOMPUTE_TASK_QUEUE="precompute-q",+ ),patch("posthog.clickhouse.client.execute.get_client_from_pool") as client_from_pool,_patched_runner() as mock_runner,):client_from_pool.return_value.__enter__.return_value.execute.return_value = []mock_runner.return_value.run.side_effect = run_metric_and_precompute_queries- _calculate(exp.id, "m1", str(recalc.id), _QUERY_TO)+ _calculate(exp.id, "m1", str(recalc.id), _QUERY_TO, task_queue=task_queue)assert [call.args[3] for call in client_from_pool.call_args_list] == [expected_user, expected_user]@@ -1786,6 +1795,16 @@ class TestCalculateActivityCancellation:assert result.success is Trueassert asyncio.all_tasks() - tasks_before == set()+ async def test_body_gets_the_task_queue_the_worker_polls(self) -> None:+ body = AsyncMock(return_value=MetricRecalculationResult(metric_uuid="m1", success=True))+ env = ActivityEnvironment()+ env.info = dataclasses.replace(env.info, task_queue="activity-q")++ with self._patched_boundaries(body, AsyncMock()):+ await _start_activity(env)++ assert body.call_args.args[-1] == "activity-q"+@contextmanagerdef _record_captures():diff --git a/products/experiments/backend/temporal/test_recalculation_workflow.py b/products/experiments/backend/temporal/test_recalculation_workflow.pyindex 6d229da40600..a20b9e827687 100644--- a/products/experiments/backend/temporal/test_recalculation_workflow.py+++ b/products/experiments/backend/temporal/test_recalculation_workflow.py@@ -354,6 +354,7 @@ class TestExperimentMetricsRecalculationWorkflow:attempt: int,precomputation_mode: PrecomputationMode | None,calculation_key_version: int = 1,+ task_queue: str = "",) -> MetricRecalculationResult:recorded_flags.append(is_final_attempt)raise RuntimeError("always fails")