:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(experiments): route recalc calcs to the precompute queue (#127841)
Rodrigo Iloro91408ee59e22
diff --git a/products/experiments/backend/temporal/ARCHITECTURE.md b/products/experiments/backend/temporal/ARCHITECTURE.mdindex ecd934c99fab..84138f51c960 100644--- a/products/experiments/backend/temporal/ARCHITECTURE.md+++ b/products/experiments/backend/temporal/ARCHITECTURE.md@@ -257,7 +257,11 @@ The split is intentional: Grafana tells you about the worker process, PostHog teRouting is per run. `start_metrics_recalculation_workflow` evaluates the `experiment-recalculation-activity-queue` flag for the run's organization, locally and without flag events, and stores the queue in the `activity_task_queue` workflow input. A flag error leaves the input empty. The workflow passes the input as `task_queue=` to the calc activity, gated on `workflow.patched("recalc-activity-task-queue-2026-10")`, and an empty input keeps the workflow's own queue. The workflow never reads the flag, so replay stays deterministic, and a flag change never moves a run in flight: Temporal retries an activity on the queue it was first scheduled on.- Operating rules. Turn the flag on for an organization only after the activity worker polls the queue in that region, because a queue with no poller holds the calculations until `schedule_to_close` fails them. To roll back, turn the flag off, then remove the worker only after every recalculation workflow that started before the flag change has closed. A fixed wait is not enough: such a run keeps its queue choice and schedules its calculations only after discovery and the start step, which take up to about 20 minutes of attempts and have no limit on queue wait. The 20-minute calc budget (`METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS`) starts only then, and the workflow has no execution timeout.+- **Precompute queue for calcs that read precomputed data.** Only the metric-events precompute table is on the aux ClickHouse cluster, so precompute builds and direct scans load different nodes. Discovery gives each metric a route only for an organization on the `experiment-recalculation-activity-queue` flag, with the same local flag check as the start step. When the flag is off, or the check fails, discovery builds no runner and sets no route, so every metric keeps the AUTO mode on the run's queue. A metric for which `ExposuresPrecompute.applicable` and `MetricEventsPrecompute.applicable` both hold goes to `experiments-recalculation-precompute-task-queue` (`EXPERIMENTS_RECALCULATION_PRECOMPUTE_TASK_QUEUE`) in the AUTO mode, and the runner checks the gates again when it calculates. Any other metric goes to the activity queue with `PrecomputationMode.DIRECT`, so it never starts a build there. A failed check routes to the precompute queue in the AUTO mode, which is what every run used before this routing.++ As a second check, the workflow uses the route only for a run with an `activity_task_queue`. Runs off the flag keep the workflow's own queue and the AUTO mode. The route needs no patch marker: on replay, sdk-core matches a scheduled activity only by its id and type (`on_activity_task_scheduled` in `activity_state_machine.rs`), not by its queue or arguments, and a discovery result from before routing has no route, so it dispatches as before. The DIRECT mode is a sixth positional argument of the calc activity, and an AUTO calc keeps five. In a rolling deploy, a worker that predates the argument fails a DIRECT calc with a retryable `TypeError` until it rolls.++ Operating rules. Turn the flag on for an organization only after the activity worker and the precompute worker poll their queues in that region, because a queue with no poller holds the calculations until `schedule_to_close` fails them. To roll back, turn the flag off, then remove the worker only after every recalculation workflow that started before the flag change has closed. A fixed wait is not enough: such a run keeps its queue choice and schedules its calculations only after discovery and the start step, which take up to about 20 minutes of attempts and have no limit on queue wait. The 20-minute calc budget (`METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS`) starts only then, and the workflow has no execution timeout.Retries live in Temporal's `RetryPolicy` on the calc activity, not in the workflow. The policy is exponential: `initial_interval=5s`, `backoff_coefficient=2.0`, `maximum_interval=60s`, `maximum_attempts=MAX_METRIC_ATTEMPTS` (currently 8). So a metric backs off 5s, 10s, 20s, 40s, 60s, 60s, 60s across its retries, then the eighth attempt is final. `asyncio.gather(..., return_exceptions=True)` lets healthy metrics finish while a failing one retries; a retrying metric never blocks the others, because each activity retries on its own worker slot rather than holding a shared queue slot.diff --git a/products/experiments/backend/temporal/models.py b/products/experiments/backend/temporal/models.pyindex ea2e9a53e604..e5056a241a62 100644--- a/products/experiments/backend/temporal/models.py+++ b/products/experiments/backend/temporal/models.py@@ -1,6 +1,8 @@import dataclassesfrom typing import Final, Literal+from posthog.schema import PrecomputationMode+from posthog.dataclasses import frozen# Shared by the workflow definition, the schedule, and the management command.@@ -128,6 +130,10 @@ class ExperimentMetricToRecalculate:metric_uuid: str# "primary" or "secondary": the metric's role in the experiment, not its kind.role: str+ # The calc activity's queue and precompute mode, chosen at discovery. The workflow uses them only for a run+ # that has an activity_task_queue. None keeps the run's activity_task_queue and the AUTO mode.+ task_queue: str | None = None+ precomputation_mode: PrecomputationMode | None = None@dataclasses.dataclassdiff --git a/products/experiments/backend/temporal/recalculation_activities.py b/products/experiments/backend/temporal/recalculation_activities.pyindex 34b27caf5790..cf2754405838 100644--- a/products/experiments/backend/temporal/recalculation_activities.py+++ b/products/experiments/backend/temporal/recalculation_activities.py@@ -10,6 +10,8 @@ import asyncioimport structlogimport temporalio.activity+from posthog.schema import PrecomputationMode+from posthog.temporal.common.heartbeat import Heartbeaterfrom posthog.temporal.common.utils import on_asyncify_pool@@ -107,6 +109,7 @@ async def calculate_experiment_metric_for_recalculation(recalculation_id: str,query_to: str,role: str = "primary",+ precomputation_mode: PrecomputationMode | None = None,) -> MetricRecalculationResult:"""Calculate one metric, write its recalc-fingerprinted result to ExperimentMetricResult, and fold theprogress update (counter + error) into the same job atomically. query_to is the run's shared data-window end.@@ -116,6 +119,10 @@ async def calculate_experiment_metric_for_recalculation(to resolve it. Defaults to "primary" so existing call sites and tests that don't pass it remain valid. Theworkflow passes these arguments by position, so recorded histories depend on their order, not their names.+ precomputation_mode comes from discovery. The workflow passes it only for a DIRECT calc on the activity queue,+ and None keeps the AUTO mode. A calc on the precompute queue also uses AUTO, because PRECOMPUTED is reserved+ for staff tools.+Finality is derived from activity.info().attempt against MAX_METRIC_ATTEMPTS, the same constant theworkflow's RetryPolicy is built from, so both sides agree on which attempt is the last. On the finalattempt a transient failure is persisted rather than re-raised silently, so the row reflects the real@@ -130,7 +137,14 @@ async def calculate_experiment_metric_for_recalculation(async with Heartbeater(factor=_HEARTBEAT_FACTOR):task = asyncio.ensure_future(on_asyncify_pool(_calculate_experiment_metric_for_recalculation_sync)(- experiment_id, metric_uuid, recalculation_id, query_to, role, is_final_attempt, attempt+ experiment_id,+ metric_uuid,+ recalculation_id,+ query_to,+ role,+ is_final_attempt,+ attempt,+ precomputation_mode,))try:diff --git a/products/experiments/backend/temporal/recalculation_logic.py b/products/experiments/backend/temporal/recalculation_logic.pyindex b7d0511f7a8c..91996851a526 100644--- a/products/experiments/backend/temporal/recalculation_logic.py+++ b/products/experiments/backend/temporal/recalculation_logic.py@@ -11,6 +11,7 @@ from concurrent.futures import ThreadPoolExecutorfrom datetime import datetime, timedeltafrom typing import Any+from django.conf import settingsfrom django.db import close_old_connections, transactionfrom django.db.models import Qfrom django.utils import timezone@@ -19,17 +20,19 @@ import structlogimport posthoganalyticsfrom temporalio.exceptions import ApplicationError-from posthog.schema import ExperimentQuery+from posthog.schema import ExperimentQuery, PrecomputationModefrom posthog.clickhouse.cancel import cancel_query_on_clusterfrom posthog.clickhouse.client.connection import Workloadfrom posthog.clickhouse.client.limit import ConcurrencyLimitExceededfrom posthog.clickhouse.query_tagging import Feature, Product, tags_context+from posthog.dataclasses import frozenfrom posthog.event_usage import groupsfrom posthog.exceptions import ClickHouseAtCapacityfrom posthog.exceptions_capture import capture_exceptionfrom posthog.hogql_queries.query_runner import ExecutionModefrom posthog.models.scoping import team_scope+from posthog.models.team.extensions import get_or_create_team_extensionfrom posthog.sync import database_sync_to_async_poolfrom posthog.temporal.common.errors import NonReportableApplicationError@@ -45,6 +48,7 @@ from products.experiments.backend.metric_calculation.config import get_metric_cafrom products.experiments.backend.metric_calculation.results import MetricResultStorefrom products.experiments.backend.metric_resolution import build_metric, get_metrics_for_calculationfrom products.experiments.backend.models.experiment import Experiment, ExperimentMetricsRecalculation+from products.experiments.backend.models.team_experiments_config import TeamExperimentsConfigfrom products.experiments.backend.temporal.models import (CAPACITY_MAX_METRIC_ATTEMPTS,CAPACITY_RETRY_BACKOFF_COEFFICIENT,@@ -121,6 +125,48 @@ def _get_recalc_state(recalculation_id: str) -> _RecalcState:)+@frozen+class _CalcRoute:+ """The queue and precompute mode of one metric's calc activity, for a run on the activity queue flag."""++ task_queue: str+ precomputation_mode: PrecomputationMode | None+++def _calc_route(+ experiment: Experiment, team_experiments_config: TeamExperimentsConfig, definition: dict[str, Any]+) -> _CalcRoute:+ """+ A metric that can read precomputed exposures and metric events goes to the precompute queue in the AUTO+ mode, and the runner checks the gates again when it calculates. The route does not use+ PrecomputationMode.PRECOMPUTED, which is reserved for staff tools, so for now the check runs twice. Any other+ metric goes to the activity queue and scans directly, so it never starts a precompute build there.+ """+ try:+ runner = ExperimentQueryRunner(+ query=ExperimentQuery(experiment_id=experiment.id, metric=build_metric(definition)),+ team=experiment.team,+ experiment=experiment,+ team_experiments_config=team_experiments_config,+ bypass_warehouse_access_control=True,+ user_facing=False,+ error_event_context=None,+ )+ reads_precomputed = runner._exposures_precompute_applicable() and runner._metric_events_precompute_applicable()+ except Exception as e:+ # The AUTO mode is what every run used before this routing, so a failed check falls back to it. Report+ # it, because a broken check sends every metric to the precompute queue.+ capture_exception(e, additional_properties={"tag": "experiment_recalculation_route_failed"})+ reads_precomputed = True++ if reads_precomputed:+ return _CalcRoute(task_queue=settings.EXPERIMENTS_RECALCULATION_PRECOMPUTE_TASK_QUEUE, precomputation_mode=None)+ return _CalcRoute(+ task_queue=settings.EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE,+ precomputation_mode=PrecomputationMode.DIRECT,+ )++@database_sync_to_async_pooldef _discover_experiment_metrics_sync(recalculation_id: str) -> list[ExperimentMetricToRecalculate]:close_old_connections()@@ -130,12 +176,36 @@ def _discover_experiment_metrics_sync(recalculation_id: str) -> list[ExperimentMrecalculation = ExperimentMetricsRecalculation.objects.select_related("experiment").get(id=recalculation_id)experiment = recalculation.experiment+ # Deferred: recalculation.py imports temporal.models, whose package __init__ imports this module.+ from products.experiments.backend.recalculation import _activity_task_queue # noqa: PLC0415++ # The same flag check as the start step, so an organization off the flag (or a failed check) keeps the+ # AUTO mode on the run's queue and pays for no routing.+ team_experiments_config = (+ get_or_create_team_extension(experiment.team, TeamExperimentsConfig)+ if _activity_task_queue(str(experiment.team.organization_id)) is not None+ else None+ )+ routes: dict[str, _CalcRoute] = {}+ metrics_to_recalculate = []# request_recalculation sets total_metrics from the same resolver, so the run's progress count# matches the metrics this activity schedules.- metrics_to_recalculate = [- ExperimentMetricToRecalculate(experiment_id=experiment.id, metric_uuid=metric.uuid, role=metric.role)- for metric in get_metrics_for_calculation(experiment)- ]+ for metric in get_metrics_for_calculation(experiment):+ route: _CalcRoute | None = None+ if team_experiments_config is not None:+ # Metrics that share a uuid share one definition, so they share one route.+ route = routes.get(metric.uuid)+ if route is None:+ route = routes[metric.uuid] = _calc_route(experiment, team_experiments_config, metric.definition)+ metrics_to_recalculate.append(+ ExperimentMetricToRecalculate(+ experiment_id=experiment.id,+ metric_uuid=metric.uuid,+ role=metric.role,+ task_queue=route.task_queue if route is not None else None,+ precomputation_mode=route.precomputation_mode if route is not None else None,+ )+ )recalculation.metric_uuids = [m.metric_uuid for m in metrics_to_recalculate]recalculation.save(update_fields=["metric_uuids"])@@ -626,6 +696,7 @@ def _calculate_experiment_metric_for_recalculation_sync(role: str = "primary",is_final_attempt: bool = True,attempt: int = 1,+ precomputation_mode: PrecomputationMode | None = None,) -> MetricRecalculationResult:close_old_connections()@@ -750,7 +821,11 @@ def _calculate_experiment_metric_for_recalculation_sync(try:runner = ExperimentQueryRunner(- query=ExperimentQuery(experiment_id=experiment_id, metric=build_metric(metric_dict)),+ query=ExperimentQuery(+ experiment_id=experiment_id,+ metric=build_metric(metric_dict),+ precomputation_mode=precomputation_mode,+ ),team=experiment.team,as_of=query_to_dt,workload=Workload.OFFLINE,diff --git a/products/experiments/backend/temporal/recalculation_metrics.py b/products/experiments/backend/temporal/recalculation_metrics.pyindex dbf05b093a33..bbe90a0af17c 100644--- a/products/experiments/backend/temporal/recalculation_metrics.py+++ b/products/experiments/backend/temporal/recalculation_metrics.py@@ -212,11 +212,12 @@ class ExperimentsRecalculationMetricsInterceptor(Interceptor):# Required by `is_task_queue_supported` in `posthog/temporal/common/interceptor.py` — without this attribute# the interceptor is filtered out of every worker and the metrics never emit. The recalc workflow + activities- # run on the first queue, and the metric calculation activity also runs on the second. Both registrations are in- # `posthog/management/commands/start_temporal_worker.py`.+ # run on the first queue, and the metric calculation activity also runs on the other two. The registrations are+ # in `posthog/management/commands/start_temporal_worker.py`.task_queue = (settings.EXPERIMENTS_RECALCULATION_TASK_QUEUE,settings.EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE,+ settings.EXPERIMENTS_RECALCULATION_PRECOMPUTE_TASK_QUEUE,)def intercept_activity(self, next: ActivityInboundInterceptor) -> ActivityInboundInterceptor:diff --git a/products/experiments/backend/temporal/recalculation_workflow.py b/products/experiments/backend/temporal/recalculation_workflow.pyindex c2af3789cb89..204b18c3453a 100644--- a/products/experiments/backend/temporal/recalculation_workflow.py+++ b/products/experiments/backend/temporal/recalculation_workflow.py@@ -10,9 +10,12 @@ from temporalio.exceptions import (TimeoutError as TemporalTimeoutError,)+from posthog.dataclasses import frozenfrom posthog.temporal.common.base import PostHogWorkflowwith temporalio.workflow.unsafe.imports_passed_through():+ from posthog.schema import PrecomputationMode+from products.experiments.backend.temporal.models import (MAX_METRIC_ATTEMPTS,METRIC_CALC_ACTIVITY_TIMEOUT_SECONDS,@@ -41,6 +44,31 @@ with temporalio.workflow.unsafe.imports_passed_through():from products.experiments.backend.temporal.recalculation_metrics import increment_workflow_finished+@frozen+class _CalcDispatch:+ """Where one metric's calc activity runs, and the precompute mode it passes."""++ task_queue: str | None+ precomputation_mode: PrecomputationMode | None+++def _calc_dispatch(metric: ExperimentMetricToRecalculate, activity_task_queue: str | None) -> _CalcDispatch:+ # Only a run on the activity queue flag uses the route that discovery set.+ if activity_task_queue and metric.task_queue:+ return _CalcDispatch(task_queue=metric.task_queue, precomputation_mode=metric.precomputation_mode)+ return _CalcDispatch(task_queue=activity_task_queue, precomputation_mode=None)+++def _calc_args(+ metric: ExperimentMetricToRecalculate, recalculation_id: str, query_to: str, dispatch: _CalcDispatch+) -> list[object]:+ args: list[object] = [metric.experiment_id, metric.metric_uuid, recalculation_id, query_to, metric.role]+ # An AUTO calc keeps the five arguments, so a worker that predates the mode argument still runs it.+ if dispatch.precomputation_mode is not None:+ args.append(dispatch.precomputation_mode)+ return args++def _ended_by_temporal(error: ActivityError) -> bool:"""True when Temporal ended the metric's retries from the outside, so no attempt persisted its outcome.@@ -164,27 +192,22 @@ class ExperimentMetricsRecalculationWorkflow(PostHogWorkflow):if inputs.activity_task_queue and temporalio.workflow.patched("recalc-activity-task-queue-2026-10")else None)+ dispatches = [_calc_dispatch(metric, activity_task_queue) for metric in metrics]results = await asyncio.gather(*[temporalio.workflow.execute_activity(calculate_experiment_metric_for_recalculation,- args=[- metric.experiment_id,- metric.metric_uuid,- recalculation_id,- query_to,- metric.role,- ],+ args=_calc_args(metric, recalculation_id, query_to, dispatch),start_to_close_timeout=timedelta(seconds=METRIC_CALC_ACTIVITY_TIMEOUT_SECONDS),schedule_to_close_timeout=timedelta(seconds=METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS),heartbeat_timeout=timedelta(seconds=METRIC_CALC_HEARTBEAT_TIMEOUT_SECONDS),retry_policy=calc_retry_policy,- task_queue=activity_task_queue,+ task_queue=dispatch.task_queue,# Round-robin dispatch across orgs under backlog; a no-op on namespaces without# fairness support.priority=Priority(fairness_key=inputs.fairness_key),)- for metric in metrics+ for metric, dispatch in zip(metrics, dispatches)],return_exceptions=True,)diff --git a/products/experiments/backend/temporal/test_recalculation_activities.py b/products/experiments/backend/temporal/test_recalculation_activities.pyindex 81708b382095..96740763e9b2 100644--- a/products/experiments/backend/temporal/test_recalculation_activities.py+++ b/products/experiments/backend/temporal/test_recalculation_activities.py@@ -12,7 +12,7 @@ import time_machinefrom posthog.test.base import BaseTestfrom unittest.mock import AsyncMock, MagicMock, patch-from django.test import SimpleTestCase+from django.test import SimpleTestCase, override_settingsfrom django.utils import timezonefrom clickhouse_driver.errors import ServerException@@ -21,15 +21,19 @@ from rest_framework.exceptions import ValidationErrorfrom temporalio.exceptions import ApplicationErrorfrom temporalio.testing import ActivityEnvironment+from posthog.schema import PrecomputationMode+from posthog.clickhouse.client import sync_executefrom posthog.clickhouse.client.connection import ClickHouseUser, Workloadfrom posthog.clickhouse.client.limit import ConcurrencyLimitExceededfrom posthog.clickhouse.query_tagging import Feature, Product, get_query_tags, tags_contextfrom posthog.exceptions import ClickHouseAtCapacity, ClickHouseQueryMemoryLimitExceeded, ClickHouseQueryTimeOut+from posthog.models.team.extensions import get_or_create_team_extensionfrom posthog.temporal.common.errors import NonReportableErrorfrom posthog.temporal.tests.utils.asyncify import restored_asyncify_executorfrom products.experiments.backend.facade.enums import ExposuresPath+from products.experiments.backend.hogql_queries.experiment_query_runner import ExperimentQueryRunnerfrom products.experiments.backend.metric_calculation.config import get_metric_calculation_configfrom products.experiments.backend.metric_calculation.results import _recalc_fingerprintfrom products.experiments.backend.models.experiment import (@@ -39,6 +43,7 @@ from products.experiments.backend.models.experiment import (ExperimentSavedMetric,ExperimentToSavedMetric,)+from products.experiments.backend.models.team_experiments_config import TeamExperimentsConfigfrom products.experiments.backend.temporal.models import (CAPACITY_MAX_METRIC_ATTEMPTS,MAX_METRIC_ATTEMPTS,@@ -94,10 +99,11 @@ def _calculate(role: str = "primary",is_final_attempt: bool = True,attempt: int = 1,+ precomputation_mode: PrecomputationMode | None = None,):with patch("products.experiments.backend.temporal.recalculation_logic.close_old_connections"):return _calculate_experiment_metric_for_recalculation_sync(- experiment_id, metric_uuid, recalculation_id, query_to, role, is_final_attempt, attempt+ experiment_id, metric_uuid, recalculation_id, query_to, role, is_final_attempt, attempt, precomputation_mode)@@ -210,6 +216,72 @@ class TestRecalculationActivities(BaseTest):assert {m.metric_uuid for m in metrics} == expected_uuidsassert {m.role for m in metrics} == expected_roles+ @parameterized.expand(+ [+ # name, activity queue flag, team_enabled, runtime_hours, funnel_order_type, route_check_fails, expected route+ ("eligible", "on", True, 36, "ordered", False, "precompute-q", None),+ ("team_disabled", "on", False, 36, "ordered", False, "activity-q", PrecomputationMode.DIRECT),+ ("below_min_runtime", "on", True, 6, "ordered", False, "activity-q", PrecomputationMode.DIRECT),+ ("metric_not_eligible", "on", True, 36, "unordered", False, "activity-q", PrecomputationMode.DIRECT),+ ("route_check_fails", "on", False, 36, "ordered", True, "precompute-q", None),+ ("flag_off", "off", True, 36, "ordered", False, None, None),+ ("flag_check_fails", "error", True, 36, "ordered", False, None, None),+ ]+ )+ @override_settings(+ EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE="activity-q",+ EXPERIMENTS_RECALCULATION_PRECOMPUTE_TASK_QUEUE="precompute-q",+ )+ @time_machine.travel("2026-04-30T12:00:00Z", tick=False)+ def test_discover_routes_each_metric(+ self,+ name: str,+ flag: str,+ team_enabled: bool,+ runtime_hours: int,+ funnel_order_type: str,+ route_check_fails: bool,+ expected_task_queue: str | None,+ expected_mode: PrecomputationMode | None,+ ):+ exp = self._experiment(flag_key=f"route-{name}")+ exp.start_date = timezone.now() - timedelta(hours=runtime_hours)+ exp.metrics = [+ {+ "uuid": "m1",+ "kind": "ExperimentMetric",+ "metric_type": "funnel",+ "series": [{"kind": "EventsNode", "event": "purchase"}],+ "funnel_order_type": funnel_order_type,+ }+ ]+ exp.save()+ config = get_or_create_team_extension(self.team, TeamExperimentsConfig)+ config.experiment_precomputation_enabled = team_enabled+ config.save()+ recalc = self._recalc(exp)++ with (+ patch(+ "products.experiments.backend.recalculation._organization_flag_enabled",+ return_value=flag == "on",+ side_effect=Exception("flags down") if flag == "error" else None,+ ),+ patch(+ "products.experiments.backend.temporal.recalculation_logic.ExperimentQueryRunner",+ side_effect=Exception("boom") if route_check_fails else ExperimentQueryRunner,+ ) as runner,+ patch(+ "products.experiments.backend.temporal.recalculation_logic.get_or_create_team_extension",+ wraps=get_or_create_team_extension,+ ) as config_read,+ ):+ metrics = _discover(str(recalc.id))++ assert [(m.task_queue, m.precomputation_mode) for m in metrics] == [(expected_task_queue, expected_mode)]+ assert runner.called is (flag == "on")+ assert config_read.called is (flag == "on")+@parameterized.expand([(@@ -1077,7 +1149,8 @@ class TestCalculateActivity(BaseTest):row = ExperimentMetricResult.objects.get(experiment=exp, metric_uuid="m1")assert row.status == ExperimentMetricResult.Status.FAILED- def test_query_to_is_passed_as_as_of_to_runner(self):+ @parameterized.expand([("auto", None), ("direct", PrecomputationMode.DIRECT)])+ def test_run_window_and_precompute_mode_reach_the_runner(self, _name: str, mode: PrecomputationMode | None):# The run's shared query_to MUST be threaded into the ClickHouse query bounds via the runner's# as_of, not just stored on the result row. Without as_of the runner falls back to its own now(),# so every metric in the run queries a slightly different time window — silently violating the@@ -1088,12 +1161,13 @@ class TestCalculateActivity(BaseTest):with _patched_runner() as mock_runner:mock_runner.return_value.run.return_value.model_dump.return_value = {}- _calculate(exp.id, "m1", str(recalc.id), _QUERY_TO)+ _calculate(exp.id, "m1", str(recalc.id), _QUERY_TO, precomputation_mode=mode)mock_runner.assert_called_once()kwargs = mock_runner.call_args.kwargsassert kwargs["as_of"] == datetime.fromisoformat(_QUERY_TO)assert kwargs["workload"] == Workload.OFFLINE+ assert kwargs["query"].precomputation_mode == modedef test_query_from_matches_experiment_start_date_on_result_row(self):# Companion to test_query_to_is_passed_as_as_of_to_runner: the lower bound of the run'sdiff --git a/products/experiments/backend/temporal/test_recalculation_metrics.py b/products/experiments/backend/temporal/test_recalculation_metrics.pyindex 7b710a73abc3..f22bebd2f914 100644--- a/products/experiments/backend/temporal/test_recalculation_metrics.py+++ b/products/experiments/backend/temporal/test_recalculation_metrics.py@@ -45,10 +45,11 @@ def test_registered_on_recalculation_queue():assert is_task_queue_supported(settings.EXPERIMENTS_RECALCULATION_TASK_QUEUE, ExperimentsRecalculationMetricsInterceptor)- # Both queue names alias to one value in test settings, so only the tuple shows a missing activity queue.+ # The queue names alias to one value in test settings, so only the tuple shows a missing queue.assert ExperimentsRecalculationMetricsInterceptor.task_queue == (settings.EXPERIMENTS_RECALCULATION_TASK_QUEUE,settings.EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE,+ settings.EXPERIMENTS_RECALCULATION_PRECOMPUTE_TASK_QUEUE,)diff --git a/products/experiments/backend/temporal/test_recalculation_workflow.py b/products/experiments/backend/temporal/test_recalculation_workflow.pyindex d0bb61a4560f..520e9df6518b 100644--- a/products/experiments/backend/temporal/test_recalculation_workflow.py+++ b/products/experiments/backend/temporal/test_recalculation_workflow.py@@ -16,6 +16,8 @@ from temporalio.exceptions import ApplicationErrorfrom temporalio.testing import WorkflowEnvironmentfrom temporalio.worker import Worker+from posthog.schema import PrecomputationMode+from posthog.dataclasses import frozenfrom products.experiments.backend.recalculation import _STALE_RECALC_THRESHOLD@@ -33,7 +35,11 @@ from products.experiments.backend.temporal.models import (RecalculationProgressUpdate,)from products.experiments.backend.temporal.recalculation_activities import calculate_experiment_metric_for_recalculation-from products.experiments.backend.temporal.recalculation_workflow import ExperimentMetricsRecalculationWorkflow+from products.experiments.backend.temporal.recalculation_workflow import (+ ExperimentMetricsRecalculationWorkflow,+ _calc_dispatch,+ _CalcDispatch,+)_START_QUERY_TO = "2026-05-29T12:00:00+00:00"_WORKFLOW_MODULE = "products.experiments.backend.temporal.recalculation_workflow"@@ -121,7 +127,10 @@ async def _run_workflow(activities) -> dict:)-def _recording_calculate(calls: list[str]) -> Callable:+_Call = tuple[str, PrecomputationMode | None]+++def _recording_calculate(calls: list[_Call]) -> Callable:@activity.defn(name="calculate_experiment_metric_for_recalculation")async def mock_calculate(experiment_id: int,@@ -129,27 +138,33 @@ def _recording_calculate(calls: list[str]) -> Callable:recalculation_id: str,query_to: str,role: str = "primary",+ precomputation_mode: PrecomputationMode | None = None,) -> MetricRecalculationResult:- calls.append(metric_uuid)+ calls.append((metric_uuid, precomputation_mode))return MetricRecalculationResult(metric_uuid=metric_uuid, success=True)return mock_calculate@frozen-class _TwoQueueRun:+class _QueueRun:result: dict- workflow_queue_calls: list[str]- activity_queue_calls: list[str]+ workflow_queue_calls: list[_Call]+ activity_queue_calls: list[_Call]+ precompute_queue_calls: list[_Call]history: WorkflowHistory-async def _run_with_activity_queue_worker(- activity_task_queue: str | None, activity_worker_task_queue: str-) -> _TwoQueueRun:- activities, _progress, _calls = _make_mock_activities(metrics=[_metric("m1")])- workflow_queue_calls: list[str] = []- activity_queue_calls: list[str] = []+async def _run_with_calc_queue_workers(+ activity_task_queue: str | None,+ activity_worker_task_queue: str,+ precompute_worker_task_queue: str | None = None,+ metric: ExperimentMetricToRecalculate | None = None,+) -> _QueueRun:+ activities, _progress, _calls = _make_mock_activities(metrics=[metric or _metric("m1")])+ workflow_queue_calls: list[_Call] = []+ activity_queue_calls: list[_Call] = []+ precompute_queue_calls: list[_Call] = []task_queue = str(uuid.uuid4())async with await WorkflowEnvironment.start_time_skipping() as env:async with (@@ -165,6 +180,11 @@ async def _run_with_activity_queue_worker(task_queue=activity_worker_task_queue,activities=[_recording_calculate(activity_queue_calls)],),+ Worker(+ env.client,+ task_queue=precompute_worker_task_queue or str(uuid.uuid4()),+ activities=[_recording_calculate(precompute_queue_calls)],+ ),):handle = await env.client.start_workflow(ExperimentMetricsRecalculationWorkflow.run,@@ -176,10 +196,11 @@ async def _run_with_activity_queue_worker()result = await handle.result()history = await handle.fetch_history()- return _TwoQueueRun(+ return _QueueRun(result=result,workflow_queue_calls=workflow_queue_calls,activity_queue_calls=activity_queue_calls,+ precompute_queue_calls=precompute_queue_calls,history=history,)@@ -288,6 +309,7 @@ class TestExperimentMetricsRecalculationWorkflow:role: str,is_final_attempt: bool,attempt: int,+ precomputation_mode: PrecomputationMode | None,) -> MetricRecalculationResult:recorded_flags.append(is_final_attempt)raise RuntimeError("always fails")@@ -625,19 +647,19 @@ class TestExperimentMetricsRecalculationWorkflow:@parameterized.expand([# name, set_field, expected_workflow_queue_calls, expected_activity_queue_calls- ("field_unset_stays_on_workflow_queue", False, ["m1"], []),- ("field_set_routes_to_activity_queue", True, [], ["m1"]),+ ("field_unset_stays_on_workflow_queue", False, [("m1", None)], []),+ ("field_set_routes_to_activity_queue", True, [], [("m1", None)]),])async def test_activity_task_queue_routes_the_calculation(self,name: str,set_field: bool,- expected_workflow_queue_calls: list[str],- expected_activity_queue_calls: list[str],+ expected_workflow_queue_calls: list[_Call],+ expected_activity_queue_calls: list[_Call],):activity_worker_task_queue = str(uuid.uuid4())- run = await _run_with_activity_queue_worker(+ run = await _run_with_calc_queue_workers(activity_task_queue=activity_worker_task_queue if set_field else None,activity_worker_task_queue=activity_worker_task_queue,)@@ -656,14 +678,69 @@ class TestExperimentMetricsRecalculationWorkflow:return False if patch_id == "recalc-activity-task-queue-2026-10" else original_patched(patch_id)with patch.object(temporalio.workflow, "patched", side_effect=pre_patch):- run = await _run_with_activity_queue_worker(+ run = await _run_with_calc_queue_workers(activity_task_queue=activity_worker_task_queue,activity_worker_task_queue=activity_worker_task_queue,)- assert run.workflow_queue_calls == ["m1"]+ assert run.workflow_queue_calls == [("m1", None)]assert run.activity_queue_calls == []await temporalio.worker.Replayer(workflows=[ExperimentMetricsRecalculationWorkflow],workflow_runner=temporalio.worker.UnsandboxedWorkflowRunner(),).replay_workflow(run.history)++ @parameterized.expand(+ [+ # name, routed_to_precompute, run_on_activity_queue, recorded_without_routes, expected calls per queue+ ("precompute_route", True, True, False, [], [], [("m1", None)]),+ ("direct_route", False, True, False, [], [("m1", PrecomputationMode.DIRECT)], []),+ ("run_off_the_flag_ignores_the_route", True, False, False, [("m1", None)], [], []),+ ("history_without_routes_replays_on_another_queue", True, True, True, [], [("m1", None)], []),+ ("history_without_routes_replays_with_another_arity", False, True, True, [], [("m1", None)], []),+ ]+ )+ async def test_discovery_route_dispatches_the_calculation(+ self,+ name: str,+ routed_to_precompute: bool,+ run_on_activity_queue: bool,+ recorded_without_routes: bool,+ expected_workflow_queue_calls: list[_Call],+ expected_activity_queue_calls: list[_Call],+ expected_precompute_queue_calls: list[_Call],+ ):+ activity_worker_task_queue = str(uuid.uuid4())+ precompute_worker_task_queue = str(uuid.uuid4())+ metric = ExperimentMetricToRecalculate(+ experiment_id=1,+ metric_uuid="m1",+ role="primary",+ task_queue=precompute_worker_task_queue if routed_to_precompute else activity_worker_task_queue,+ precomputation_mode=None if routed_to_precompute else PrecomputationMode.DIRECT,+ )++ def dispatch_without_routes(+ metric: ExperimentMetricToRecalculate, activity_task_queue: str | None+ ) -> _CalcDispatch:+ return _CalcDispatch(task_queue=activity_task_queue, precomputation_mode=None)++ with patch(+ f"{_WORKFLOW_MODULE}._calc_dispatch",+ side_effect=dispatch_without_routes if recorded_without_routes else _calc_dispatch,+ ):+ run = await _run_with_calc_queue_workers(+ activity_task_queue=activity_worker_task_queue if run_on_activity_queue else None,+ activity_worker_task_queue=activity_worker_task_queue,+ precompute_worker_task_queue=precompute_worker_task_queue,+ metric=metric,+ )++ assert run.result == {"total": 1, "succeeded": 1, "failed": 0}+ assert run.workflow_queue_calls == expected_workflow_queue_calls+ assert run.activity_queue_calls == expected_activity_queue_calls+ assert run.precompute_queue_calls == expected_precompute_queue_calls+ await temporalio.worker.Replayer(+ workflows=[ExperimentMetricsRecalculationWorkflow],+ workflow_runner=temporalio.worker.UnsandboxedWorkflowRunner(),+ ).replay_workflow(run.history)