Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(experiments): route recalc calcs to the precompute queue (#127841)

Rodrigo Iloro91408ee59e22
diff --git a/products/experiments/backend/temporal/ARCHITECTURE.md b/products/experiments/backend/temporal/ARCHITECTURE.md
index ecd934c99fab..84138f51c960 100644
--- a/products/experiments/backend/temporal/ARCHITECTURE.md
+++ b/products/experiments/backend/temporal/ARCHITECTURE.md
@@ -257,7 +257,11 @@ The split is intentional: Grafana tells you about the worker process, PostHog te
Routing is per run. `start_metrics_recalculation_workflow` evaluates the `experiment-recalculation-activity-queue` flag for the run's organization, locally and without flag events, and stores the queue in the `activity_task_queue` workflow input. A flag error leaves the input empty. The workflow passes the input as `task_queue=` to the calc activity, gated on `workflow.patched("recalc-activity-task-queue-2026-10")`, and an empty input keeps the workflow's own queue. The workflow never reads the flag, so replay stays deterministic, and a flag change never moves a run in flight: Temporal retries an activity on the queue it was first scheduled on.
- Operating rules. Turn the flag on for an organization only after the activity worker polls the queue in that region, because a queue with no poller holds the calculations until `schedule_to_close` fails them. To roll back, turn the flag off, then remove the worker only after every recalculation workflow that started before the flag change has closed. A fixed wait is not enough: such a run keeps its queue choice and schedules its calculations only after discovery and the start step, which take up to about 20 minutes of attempts and have no limit on queue wait. The 20-minute calc budget (`METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS`) starts only then, and the workflow has no execution timeout.
+- **Precompute queue for calcs that read precomputed data.** Only the metric-events precompute table is on the aux ClickHouse cluster, so precompute builds and direct scans load different nodes. Discovery gives each metric a route only for an organization on the `experiment-recalculation-activity-queue` flag, with the same local flag check as the start step. When the flag is off, or the check fails, discovery builds no runner and sets no route, so every metric keeps the AUTO mode on the run's queue. A metric for which `ExposuresPrecompute.applicable` and `MetricEventsPrecompute.applicable` both hold goes to `experiments-recalculation-precompute-task-queue` (`EXPERIMENTS_RECALCULATION_PRECOMPUTE_TASK_QUEUE`) in the AUTO mode, and the runner checks the gates again when it calculates. Any other metric goes to the activity queue with `PrecomputationMode.DIRECT`, so it never starts a build there. A failed check routes to the precompute queue in the AUTO mode, which is what every run used before this routing.
+
+ As a second check, the workflow uses the route only for a run with an `activity_task_queue`. Runs off the flag keep the workflow's own queue and the AUTO mode. The route needs no patch marker: on replay, sdk-core matches a scheduled activity only by its id and type (`on_activity_task_scheduled` in `activity_state_machine.rs`), not by its queue or arguments, and a discovery result from before routing has no route, so it dispatches as before. The DIRECT mode is a sixth positional argument of the calc activity, and an AUTO calc keeps five. In a rolling deploy, a worker that predates the argument fails a DIRECT calc with a retryable `TypeError` until it rolls.
+
+ Operating rules. Turn the flag on for an organization only after the activity worker and the precompute worker poll their queues in that region, because a queue with no poller holds the calculations until `schedule_to_close` fails them. To roll back, turn the flag off, then remove the worker only after every recalculation workflow that started before the flag change has closed. A fixed wait is not enough: such a run keeps its queue choice and schedules its calculations only after discovery and the start step, which take up to about 20 minutes of attempts and have no limit on queue wait. The 20-minute calc budget (`METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS`) starts only then, and the workflow has no execution timeout.
Retries live in Temporal's `RetryPolicy` on the calc activity, not in the workflow. The policy is exponential: `initial_interval=5s`, `backoff_coefficient=2.0`, `maximum_interval=60s`, `maximum_attempts=MAX_METRIC_ATTEMPTS` (currently 8). So a metric backs off 5s, 10s, 20s, 40s, 60s, 60s, 60s across its retries, then the eighth attempt is final. `asyncio.gather(..., return_exceptions=True)` lets healthy metrics finish while a failing one retries; a retrying metric never blocks the others, because each activity retries on its own worker slot rather than holding a shared queue slot.
diff --git a/products/experiments/backend/temporal/models.py b/products/experiments/backend/temporal/models.py
index ea2e9a53e604..e5056a241a62 100644
--- a/products/experiments/backend/temporal/models.py
+++ b/products/experiments/backend/temporal/models.py
@@ -1,6 +1,8 @@
import dataclasses
from typing import Final, Literal
+from posthog.schema import PrecomputationMode
+
from posthog.dataclasses import frozen
# Shared by the workflow definition, the schedule, and the management command.
@@ -128,6 +130,10 @@ class ExperimentMetricToRecalculate:
metric_uuid: str
# "primary" or "secondary": the metric's role in the experiment, not its kind.
role: str
+ # The calc activity's queue and precompute mode, chosen at discovery. The workflow uses them only for a run
+ # that has an activity_task_queue. None keeps the run's activity_task_queue and the AUTO mode.
+ task_queue: str | None = None
+ precomputation_mode: PrecomputationMode | None = None
@dataclasses.dataclass
diff --git a/products/experiments/backend/temporal/recalculation_activities.py b/products/experiments/backend/temporal/recalculation_activities.py
index 34b27caf5790..cf2754405838 100644
--- a/products/experiments/backend/temporal/recalculation_activities.py
+++ b/products/experiments/backend/temporal/recalculation_activities.py
@@ -10,6 +10,8 @@ import asyncio
import structlog
import temporalio.activity
+from posthog.schema import PrecomputationMode
+
from posthog.temporal.common.heartbeat import Heartbeater
from posthog.temporal.common.utils import on_asyncify_pool
@@ -107,6 +109,7 @@ async def calculate_experiment_metric_for_recalculation(
recalculation_id: str,
query_to: str,
role: str = "primary",
+ precomputation_mode: PrecomputationMode | None = None,
) -> MetricRecalculationResult:
"""Calculate one metric, write its recalc-fingerprinted result to ExperimentMetricResult, and fold the
progress update (counter + error) into the same job atomically. query_to is the run's shared data-window end.
@@ -116,6 +119,10 @@ async def calculate_experiment_metric_for_recalculation(
to resolve it. Defaults to "primary" so existing call sites and tests that don't pass it remain valid. The
workflow passes these arguments by position, so recorded histories depend on their order, not their names.
+ precomputation_mode comes from discovery. The workflow passes it only for a DIRECT calc on the activity queue,
+ and None keeps the AUTO mode. A calc on the precompute queue also uses AUTO, because PRECOMPUTED is reserved
+ for staff tools.
+
Finality is derived from activity.info().attempt against MAX_METRIC_ATTEMPTS, the same constant the
workflow's RetryPolicy is built from, so both sides agree on which attempt is the last. On the final
attempt a transient failure is persisted rather than re-raised silently, so the row reflects the real
@@ -130,7 +137,14 @@ async def calculate_experiment_metric_for_recalculation(
async with Heartbeater(factor=_HEARTBEAT_FACTOR):
task = asyncio.ensure_future(
on_asyncify_pool(_calculate_experiment_metric_for_recalculation_sync)(
- experiment_id, metric_uuid, recalculation_id, query_to, role, is_final_attempt, attempt
+ experiment_id,
+ metric_uuid,
+ recalculation_id,
+ query_to,
+ role,
+ is_final_attempt,
+ attempt,
+ precomputation_mode,
)
)
try:
diff --git a/products/experiments/backend/temporal/recalculation_logic.py b/products/experiments/backend/temporal/recalculation_logic.py
index b7d0511f7a8c..91996851a526 100644
--- a/products/experiments/backend/temporal/recalculation_logic.py
+++ b/products/experiments/backend/temporal/recalculation_logic.py
@@ -11,6 +11,7 @@ from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timedelta
from typing import Any
+from django.conf import settings
from django.db import close_old_connections, transaction
from django.db.models import Q
from django.utils import timezone
@@ -19,17 +20,19 @@ import structlog
import posthoganalytics
from temporalio.exceptions import ApplicationError
-from posthog.schema import ExperimentQuery
+from posthog.schema import ExperimentQuery, PrecomputationMode
from posthog.clickhouse.cancel import cancel_query_on_cluster
from posthog.clickhouse.client.connection import Workload
from posthog.clickhouse.client.limit import ConcurrencyLimitExceeded
from posthog.clickhouse.query_tagging import Feature, Product, tags_context
+from posthog.dataclasses import frozen
from posthog.event_usage import groups
from posthog.exceptions import ClickHouseAtCapacity
from posthog.exceptions_capture import capture_exception
from posthog.hogql_queries.query_runner import ExecutionMode
from posthog.models.scoping import team_scope
+from posthog.models.team.extensions import get_or_create_team_extension
from posthog.sync import database_sync_to_async_pool
from posthog.temporal.common.errors import NonReportableApplicationError
@@ -45,6 +48,7 @@ from products.experiments.backend.metric_calculation.config import get_metric_ca
from products.experiments.backend.metric_calculation.results import MetricResultStore
from products.experiments.backend.metric_resolution import build_metric, get_metrics_for_calculation
from products.experiments.backend.models.experiment import Experiment, ExperimentMetricsRecalculation
+from products.experiments.backend.models.team_experiments_config import TeamExperimentsConfig
from products.experiments.backend.temporal.models import (
CAPACITY_MAX_METRIC_ATTEMPTS,
CAPACITY_RETRY_BACKOFF_COEFFICIENT,
@@ -121,6 +125,48 @@ def _get_recalc_state(recalculation_id: str) -> _RecalcState:
)
+@frozen
+class _CalcRoute:
+ """The queue and precompute mode of one metric's calc activity, for a run on the activity queue flag."""
+
+ task_queue: str
+ precomputation_mode: PrecomputationMode | None
+
+
+def _calc_route(
+ experiment: Experiment, team_experiments_config: TeamExperimentsConfig, definition: dict[str, Any]
+) -> _CalcRoute:
+ """
+ A metric that can read precomputed exposures and metric events goes to the precompute queue in the AUTO
+ mode, and the runner checks the gates again when it calculates. The route does not use
+ PrecomputationMode.PRECOMPUTED, which is reserved for staff tools, so for now the check runs twice. Any other
+ metric goes to the activity queue and scans directly, so it never starts a precompute build there.
+ """
+ try:
+ runner = ExperimentQueryRunner(
+ query=ExperimentQuery(experiment_id=experiment.id, metric=build_metric(definition)),
+ team=experiment.team,
+ experiment=experiment,
+ team_experiments_config=team_experiments_config,
+ bypass_warehouse_access_control=True,
+ user_facing=False,
+ error_event_context=None,
+ )
+ reads_precomputed = runner._exposures_precompute_applicable() and runner._metric_events_precompute_applicable()
+ except Exception as e:
+ # The AUTO mode is what every run used before this routing, so a failed check falls back to it. Report
+ # it, because a broken check sends every metric to the precompute queue.
+ capture_exception(e, additional_properties={"tag": "experiment_recalculation_route_failed"})
+ reads_precomputed = True
+
+ if reads_precomputed:
+ return _CalcRoute(task_queue=settings.EXPERIMENTS_RECALCULATION_PRECOMPUTE_TASK_QUEUE, precomputation_mode=None)
+ return _CalcRoute(
+ task_queue=settings.EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE,
+ precomputation_mode=PrecomputationMode.DIRECT,
+ )
+
+
@database_sync_to_async_pool
def _discover_experiment_metrics_sync(recalculation_id: str) -> list[ExperimentMetricToRecalculate]:
close_old_connections()
@@ -130,12 +176,36 @@ def _discover_experiment_metrics_sync(recalculation_id: str) -> list[ExperimentM
recalculation = ExperimentMetricsRecalculation.objects.select_related("experiment").get(id=recalculation_id)
experiment = recalculation.experiment
+ # Deferred: recalculation.py imports temporal.models, whose package __init__ imports this module.
+ from products.experiments.backend.recalculation import _activity_task_queue # noqa: PLC0415
+
+ # The same flag check as the start step, so an organization off the flag (or a failed check) keeps the
+ # AUTO mode on the run's queue and pays for no routing.
+ team_experiments_config = (
+ get_or_create_team_extension(experiment.team, TeamExperimentsConfig)
+ if _activity_task_queue(str(experiment.team.organization_id)) is not None
+ else None
+ )
+ routes: dict[str, _CalcRoute] = {}
+ metrics_to_recalculate = []
# request_recalculation sets total_metrics from the same resolver, so the run's progress count
# matches the metrics this activity schedules.
- metrics_to_recalculate = [
- ExperimentMetricToRecalculate(experiment_id=experiment.id, metric_uuid=metric.uuid, role=metric.role)
- for metric in get_metrics_for_calculation(experiment)
- ]
+ for metric in get_metrics_for_calculation(experiment):
+ route: _CalcRoute | None = None
+ if team_experiments_config is not None:
+ # Metrics that share a uuid share one definition, so they share one route.
+ route = routes.get(metric.uuid)
+ if route is None:
+ route = routes[metric.uuid] = _calc_route(experiment, team_experiments_config, metric.definition)
+ metrics_to_recalculate.append(
+ ExperimentMetricToRecalculate(
+ experiment_id=experiment.id,
+ metric_uuid=metric.uuid,
+ role=metric.role,
+ task_queue=route.task_queue if route is not None else None,
+ precomputation_mode=route.precomputation_mode if route is not None else None,
+ )
+ )
recalculation.metric_uuids = [m.metric_uuid for m in metrics_to_recalculate]
recalculation.save(update_fields=["metric_uuids"])
@@ -626,6 +696,7 @@ def _calculate_experiment_metric_for_recalculation_sync(
role: str = "primary",
is_final_attempt: bool = True,
attempt: int = 1,
+ precomputation_mode: PrecomputationMode | None = None,
) -> MetricRecalculationResult:
close_old_connections()
@@ -750,7 +821,11 @@ def _calculate_experiment_metric_for_recalculation_sync(
try:
runner = ExperimentQueryRunner(
- query=ExperimentQuery(experiment_id=experiment_id, metric=build_metric(metric_dict)),
+ query=ExperimentQuery(
+ experiment_id=experiment_id,
+ metric=build_metric(metric_dict),
+ precomputation_mode=precomputation_mode,
+ ),
team=experiment.team,
as_of=query_to_dt,
workload=Workload.OFFLINE,
diff --git a/products/experiments/backend/temporal/recalculation_metrics.py b/products/experiments/backend/temporal/recalculation_metrics.py
index dbf05b093a33..bbe90a0af17c 100644
--- a/products/experiments/backend/temporal/recalculation_metrics.py
+++ b/products/experiments/backend/temporal/recalculation_metrics.py
@@ -212,11 +212,12 @@ class ExperimentsRecalculationMetricsInterceptor(Interceptor):
# Required by `is_task_queue_supported` in `posthog/temporal/common/interceptor.py` — without this attribute
# the interceptor is filtered out of every worker and the metrics never emit. The recalc workflow + activities
- # run on the first queue, and the metric calculation activity also runs on the second. Both registrations are in
- # `posthog/management/commands/start_temporal_worker.py`.
+ # run on the first queue, and the metric calculation activity also runs on the other two. The registrations are
+ # in `posthog/management/commands/start_temporal_worker.py`.
task_queue = (
settings.EXPERIMENTS_RECALCULATION_TASK_QUEUE,
settings.EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE,
+ settings.EXPERIMENTS_RECALCULATION_PRECOMPUTE_TASK_QUEUE,
)
def intercept_activity(self, next: ActivityInboundInterceptor) -> ActivityInboundInterceptor:
diff --git a/products/experiments/backend/temporal/recalculation_workflow.py b/products/experiments/backend/temporal/recalculation_workflow.py
index c2af3789cb89..204b18c3453a 100644
--- a/products/experiments/backend/temporal/recalculation_workflow.py
+++ b/products/experiments/backend/temporal/recalculation_workflow.py
@@ -10,9 +10,12 @@ from temporalio.exceptions import (
TimeoutError as TemporalTimeoutError,
)
+from posthog.dataclasses import frozen
from posthog.temporal.common.base import PostHogWorkflow
with temporalio.workflow.unsafe.imports_passed_through():
+ from posthog.schema import PrecomputationMode
+
from products.experiments.backend.temporal.models import (
MAX_METRIC_ATTEMPTS,
METRIC_CALC_ACTIVITY_TIMEOUT_SECONDS,
@@ -41,6 +44,31 @@ with temporalio.workflow.unsafe.imports_passed_through():
from products.experiments.backend.temporal.recalculation_metrics import increment_workflow_finished
+@frozen
+class _CalcDispatch:
+ """Where one metric's calc activity runs, and the precompute mode it passes."""
+
+ task_queue: str | None
+ precomputation_mode: PrecomputationMode | None
+
+
+def _calc_dispatch(metric: ExperimentMetricToRecalculate, activity_task_queue: str | None) -> _CalcDispatch:
+ # Only a run on the activity queue flag uses the route that discovery set.
+ if activity_task_queue and metric.task_queue:
+ return _CalcDispatch(task_queue=metric.task_queue, precomputation_mode=metric.precomputation_mode)
+ return _CalcDispatch(task_queue=activity_task_queue, precomputation_mode=None)
+
+
+def _calc_args(
+ metric: ExperimentMetricToRecalculate, recalculation_id: str, query_to: str, dispatch: _CalcDispatch
+) -> list[object]:
+ args: list[object] = [metric.experiment_id, metric.metric_uuid, recalculation_id, query_to, metric.role]
+ # An AUTO calc keeps the five arguments, so a worker that predates the mode argument still runs it.
+ if dispatch.precomputation_mode is not None:
+ args.append(dispatch.precomputation_mode)
+ return args
+
+
def _ended_by_temporal(error: ActivityError) -> bool:
"""True when Temporal ended the metric's retries from the outside, so no attempt persisted its outcome.
@@ -164,27 +192,22 @@ class ExperimentMetricsRecalculationWorkflow(PostHogWorkflow):
if inputs.activity_task_queue and temporalio.workflow.patched("recalc-activity-task-queue-2026-10")
else None
)
+ dispatches = [_calc_dispatch(metric, activity_task_queue) for metric in metrics]
results = await asyncio.gather(
*[
temporalio.workflow.execute_activity(
calculate_experiment_metric_for_recalculation,
- args=[
- metric.experiment_id,
- metric.metric_uuid,
- recalculation_id,
- query_to,
- metric.role,
- ],
+ args=_calc_args(metric, recalculation_id, query_to, dispatch),
start_to_close_timeout=timedelta(seconds=METRIC_CALC_ACTIVITY_TIMEOUT_SECONDS),
schedule_to_close_timeout=timedelta(seconds=METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS),
heartbeat_timeout=timedelta(seconds=METRIC_CALC_HEARTBEAT_TIMEOUT_SECONDS),
retry_policy=calc_retry_policy,
- task_queue=activity_task_queue,
+ task_queue=dispatch.task_queue,
# Round-robin dispatch across orgs under backlog; a no-op on namespaces without
# fairness support.
priority=Priority(fairness_key=inputs.fairness_key),
)
- for metric in metrics
+ for metric, dispatch in zip(metrics, dispatches)
],
return_exceptions=True,
)
diff --git a/products/experiments/backend/temporal/test_recalculation_activities.py b/products/experiments/backend/temporal/test_recalculation_activities.py
index 81708b382095..96740763e9b2 100644
--- a/products/experiments/backend/temporal/test_recalculation_activities.py
+++ b/products/experiments/backend/temporal/test_recalculation_activities.py
@@ -12,7 +12,7 @@ import time_machine
from posthog.test.base import BaseTest
from unittest.mock import AsyncMock, MagicMock, patch
-from django.test import SimpleTestCase
+from django.test import SimpleTestCase, override_settings
from django.utils import timezone
from clickhouse_driver.errors import ServerException
@@ -21,15 +21,19 @@ from rest_framework.exceptions import ValidationError
from temporalio.exceptions import ApplicationError
from temporalio.testing import ActivityEnvironment
+from posthog.schema import PrecomputationMode
+
from posthog.clickhouse.client import sync_execute
from posthog.clickhouse.client.connection import ClickHouseUser, Workload
from posthog.clickhouse.client.limit import ConcurrencyLimitExceeded
from posthog.clickhouse.query_tagging import Feature, Product, get_query_tags, tags_context
from posthog.exceptions import ClickHouseAtCapacity, ClickHouseQueryMemoryLimitExceeded, ClickHouseQueryTimeOut
+from posthog.models.team.extensions import get_or_create_team_extension
from posthog.temporal.common.errors import NonReportableError
from posthog.temporal.tests.utils.asyncify import restored_asyncify_executor
from products.experiments.backend.facade.enums import ExposuresPath
+from products.experiments.backend.hogql_queries.experiment_query_runner import ExperimentQueryRunner
from products.experiments.backend.metric_calculation.config import get_metric_calculation_config
from products.experiments.backend.metric_calculation.results import _recalc_fingerprint
from products.experiments.backend.models.experiment import (
@@ -39,6 +43,7 @@ from products.experiments.backend.models.experiment import (
ExperimentSavedMetric,
ExperimentToSavedMetric,
)
+from products.experiments.backend.models.team_experiments_config import TeamExperimentsConfig
from products.experiments.backend.temporal.models import (
CAPACITY_MAX_METRIC_ATTEMPTS,
MAX_METRIC_ATTEMPTS,
@@ -94,10 +99,11 @@ def _calculate(
role: str = "primary",
is_final_attempt: bool = True,
attempt: int = 1,
+ precomputation_mode: PrecomputationMode | None = None,
):
with patch("products.experiments.backend.temporal.recalculation_logic.close_old_connections"):
return _calculate_experiment_metric_for_recalculation_sync(
- experiment_id, metric_uuid, recalculation_id, query_to, role, is_final_attempt, attempt
+ experiment_id, metric_uuid, recalculation_id, query_to, role, is_final_attempt, attempt, precomputation_mode
)
@@ -210,6 +216,72 @@ class TestRecalculationActivities(BaseTest):
assert {m.metric_uuid for m in metrics} == expected_uuids
assert {m.role for m in metrics} == expected_roles
+ @parameterized.expand(
+ [
+ # name, activity queue flag, team_enabled, runtime_hours, funnel_order_type, route_check_fails, expected route
+ ("eligible", "on", True, 36, "ordered", False, "precompute-q", None),
+ ("team_disabled", "on", False, 36, "ordered", False, "activity-q", PrecomputationMode.DIRECT),
+ ("below_min_runtime", "on", True, 6, "ordered", False, "activity-q", PrecomputationMode.DIRECT),
+ ("metric_not_eligible", "on", True, 36, "unordered", False, "activity-q", PrecomputationMode.DIRECT),
+ ("route_check_fails", "on", False, 36, "ordered", True, "precompute-q", None),
+ ("flag_off", "off", True, 36, "ordered", False, None, None),
+ ("flag_check_fails", "error", True, 36, "ordered", False, None, None),
+ ]
+ )
+ @override_settings(
+ EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE="activity-q",
+ EXPERIMENTS_RECALCULATION_PRECOMPUTE_TASK_QUEUE="precompute-q",
+ )
+ @time_machine.travel("2026-04-30T12:00:00Z", tick=False)
+ def test_discover_routes_each_metric(
+ self,
+ name: str,
+ flag: str,
+ team_enabled: bool,
+ runtime_hours: int,
+ funnel_order_type: str,
+ route_check_fails: bool,
+ expected_task_queue: str | None,
+ expected_mode: PrecomputationMode | None,
+ ):
+ exp = self._experiment(flag_key=f"route-{name}")
+ exp.start_date = timezone.now() - timedelta(hours=runtime_hours)
+ exp.metrics = [
+ {
+ "uuid": "m1",
+ "kind": "ExperimentMetric",
+ "metric_type": "funnel",
+ "series": [{"kind": "EventsNode", "event": "purchase"}],
+ "funnel_order_type": funnel_order_type,
+ }
+ ]
+ exp.save()
+ config = get_or_create_team_extension(self.team, TeamExperimentsConfig)
+ config.experiment_precomputation_enabled = team_enabled
+ config.save()
+ recalc = self._recalc(exp)
+
+ with (
+ patch(
+ "products.experiments.backend.recalculation._organization_flag_enabled",
+ return_value=flag == "on",
+ side_effect=Exception("flags down") if flag == "error" else None,
+ ),
+ patch(
+ "products.experiments.backend.temporal.recalculation_logic.ExperimentQueryRunner",
+ side_effect=Exception("boom") if route_check_fails else ExperimentQueryRunner,
+ ) as runner,
+ patch(
+ "products.experiments.backend.temporal.recalculation_logic.get_or_create_team_extension",
+ wraps=get_or_create_team_extension,
+ ) as config_read,
+ ):
+ metrics = _discover(str(recalc.id))
+
+ assert [(m.task_queue, m.precomputation_mode) for m in metrics] == [(expected_task_queue, expected_mode)]
+ assert runner.called is (flag == "on")
+ assert config_read.called is (flag == "on")
+
@parameterized.expand(
[
(
@@ -1077,7 +1149,8 @@ class TestCalculateActivity(BaseTest):
row = ExperimentMetricResult.objects.get(experiment=exp, metric_uuid="m1")
assert row.status == ExperimentMetricResult.Status.FAILED
- def test_query_to_is_passed_as_as_of_to_runner(self):
+ @parameterized.expand([("auto", None), ("direct", PrecomputationMode.DIRECT)])
+ def test_run_window_and_precompute_mode_reach_the_runner(self, _name: str, mode: PrecomputationMode | None):
# The run's shared query_to MUST be threaded into the ClickHouse query bounds via the runner's
# as_of, not just stored on the result row. Without as_of the runner falls back to its own now(),
# so every metric in the run queries a slightly different time window — silently violating the
@@ -1088,12 +1161,13 @@ class TestCalculateActivity(BaseTest):
with _patched_runner() as mock_runner:
mock_runner.return_value.run.return_value.model_dump.return_value = {}
- _calculate(exp.id, "m1", str(recalc.id), _QUERY_TO)
+ _calculate(exp.id, "m1", str(recalc.id), _QUERY_TO, precomputation_mode=mode)
mock_runner.assert_called_once()
kwargs = mock_runner.call_args.kwargs
assert kwargs["as_of"] == datetime.fromisoformat(_QUERY_TO)
assert kwargs["workload"] == Workload.OFFLINE
+ assert kwargs["query"].precomputation_mode == mode
def test_query_from_matches_experiment_start_date_on_result_row(self):
# Companion to test_query_to_is_passed_as_as_of_to_runner: the lower bound of the run's
diff --git a/products/experiments/backend/temporal/test_recalculation_metrics.py b/products/experiments/backend/temporal/test_recalculation_metrics.py
index 7b710a73abc3..f22bebd2f914 100644
--- a/products/experiments/backend/temporal/test_recalculation_metrics.py
+++ b/products/experiments/backend/temporal/test_recalculation_metrics.py
@@ -45,10 +45,11 @@ def test_registered_on_recalculation_queue():
assert is_task_queue_supported(
settings.EXPERIMENTS_RECALCULATION_TASK_QUEUE, ExperimentsRecalculationMetricsInterceptor
)
- # Both queue names alias to one value in test settings, so only the tuple shows a missing activity queue.
+ # The queue names alias to one value in test settings, so only the tuple shows a missing queue.
assert ExperimentsRecalculationMetricsInterceptor.task_queue == (
settings.EXPERIMENTS_RECALCULATION_TASK_QUEUE,
settings.EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE,
+ settings.EXPERIMENTS_RECALCULATION_PRECOMPUTE_TASK_QUEUE,
)
diff --git a/products/experiments/backend/temporal/test_recalculation_workflow.py b/products/experiments/backend/temporal/test_recalculation_workflow.py
index d0bb61a4560f..520e9df6518b 100644
--- a/products/experiments/backend/temporal/test_recalculation_workflow.py
+++ b/products/experiments/backend/temporal/test_recalculation_workflow.py
@@ -16,6 +16,8 @@ from temporalio.exceptions import ApplicationError
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker
+from posthog.schema import PrecomputationMode
+
from posthog.dataclasses import frozen
from products.experiments.backend.recalculation import _STALE_RECALC_THRESHOLD
@@ -33,7 +35,11 @@ from products.experiments.backend.temporal.models import (
RecalculationProgressUpdate,
)
from products.experiments.backend.temporal.recalculation_activities import calculate_experiment_metric_for_recalculation
-from products.experiments.backend.temporal.recalculation_workflow import ExperimentMetricsRecalculationWorkflow
+from products.experiments.backend.temporal.recalculation_workflow import (
+ ExperimentMetricsRecalculationWorkflow,
+ _calc_dispatch,
+ _CalcDispatch,
+)
_START_QUERY_TO = "2026-05-29T12:00:00+00:00"
_WORKFLOW_MODULE = "products.experiments.backend.temporal.recalculation_workflow"
@@ -121,7 +127,10 @@ async def _run_workflow(activities) -> dict:
)
-def _recording_calculate(calls: list[str]) -> Callable:
+_Call = tuple[str, PrecomputationMode | None]
+
+
+def _recording_calculate(calls: list[_Call]) -> Callable:
@activity.defn(name="calculate_experiment_metric_for_recalculation")
async def mock_calculate(
experiment_id: int,
@@ -129,27 +138,33 @@ def _recording_calculate(calls: list[str]) -> Callable:
recalculation_id: str,
query_to: str,
role: str = "primary",
+ precomputation_mode: PrecomputationMode | None = None,
) -> MetricRecalculationResult:
- calls.append(metric_uuid)
+ calls.append((metric_uuid, precomputation_mode))
return MetricRecalculationResult(metric_uuid=metric_uuid, success=True)
return mock_calculate
@frozen
-class _TwoQueueRun:
+class _QueueRun:
result: dict
- workflow_queue_calls: list[str]
- activity_queue_calls: list[str]
+ workflow_queue_calls: list[_Call]
+ activity_queue_calls: list[_Call]
+ precompute_queue_calls: list[_Call]
history: WorkflowHistory
-async def _run_with_activity_queue_worker(
- activity_task_queue: str | None, activity_worker_task_queue: str
-) -> _TwoQueueRun:
- activities, _progress, _calls = _make_mock_activities(metrics=[_metric("m1")])
- workflow_queue_calls: list[str] = []
- activity_queue_calls: list[str] = []
+async def _run_with_calc_queue_workers(
+ activity_task_queue: str | None,
+ activity_worker_task_queue: str,
+ precompute_worker_task_queue: str | None = None,
+ metric: ExperimentMetricToRecalculate | None = None,
+) -> _QueueRun:
+ activities, _progress, _calls = _make_mock_activities(metrics=[metric or _metric("m1")])
+ workflow_queue_calls: list[_Call] = []
+ activity_queue_calls: list[_Call] = []
+ precompute_queue_calls: list[_Call] = []
task_queue = str(uuid.uuid4())
async with await WorkflowEnvironment.start_time_skipping() as env:
async with (
@@ -165,6 +180,11 @@ async def _run_with_activity_queue_worker(
task_queue=activity_worker_task_queue,
activities=[_recording_calculate(activity_queue_calls)],
),
+ Worker(
+ env.client,
+ task_queue=precompute_worker_task_queue or str(uuid.uuid4()),
+ activities=[_recording_calculate(precompute_queue_calls)],
+ ),
):
handle = await env.client.start_workflow(
ExperimentMetricsRecalculationWorkflow.run,
@@ -176,10 +196,11 @@ async def _run_with_activity_queue_worker(
)
result = await handle.result()
history = await handle.fetch_history()
- return _TwoQueueRun(
+ return _QueueRun(
result=result,
workflow_queue_calls=workflow_queue_calls,
activity_queue_calls=activity_queue_calls,
+ precompute_queue_calls=precompute_queue_calls,
history=history,
)
@@ -288,6 +309,7 @@ class TestExperimentMetricsRecalculationWorkflow:
role: str,
is_final_attempt: bool,
attempt: int,
+ precomputation_mode: PrecomputationMode | None,
) -> MetricRecalculationResult:
recorded_flags.append(is_final_attempt)
raise RuntimeError("always fails")
@@ -625,19 +647,19 @@ class TestExperimentMetricsRecalculationWorkflow:
@parameterized.expand(
[
# name, set_field, expected_workflow_queue_calls, expected_activity_queue_calls
- ("field_unset_stays_on_workflow_queue", False, ["m1"], []),
- ("field_set_routes_to_activity_queue", True, [], ["m1"]),
+ ("field_unset_stays_on_workflow_queue", False, [("m1", None)], []),
+ ("field_set_routes_to_activity_queue", True, [], [("m1", None)]),
]
)
async def test_activity_task_queue_routes_the_calculation(
self,
name: str,
set_field: bool,
- expected_workflow_queue_calls: list[str],
- expected_activity_queue_calls: list[str],
+ expected_workflow_queue_calls: list[_Call],
+ expected_activity_queue_calls: list[_Call],
):
activity_worker_task_queue = str(uuid.uuid4())
- run = await _run_with_activity_queue_worker(
+ run = await _run_with_calc_queue_workers(
activity_task_queue=activity_worker_task_queue if set_field else None,
activity_worker_task_queue=activity_worker_task_queue,
)
@@ -656,14 +678,69 @@ class TestExperimentMetricsRecalculationWorkflow:
return False if patch_id == "recalc-activity-task-queue-2026-10" else original_patched(patch_id)
with patch.object(temporalio.workflow, "patched", side_effect=pre_patch):
- run = await _run_with_activity_queue_worker(
+ run = await _run_with_calc_queue_workers(
activity_task_queue=activity_worker_task_queue,
activity_worker_task_queue=activity_worker_task_queue,
)
- assert run.workflow_queue_calls == ["m1"]
+ assert run.workflow_queue_calls == [("m1", None)]
assert run.activity_queue_calls == []
await temporalio.worker.Replayer(
workflows=[ExperimentMetricsRecalculationWorkflow],
workflow_runner=temporalio.worker.UnsandboxedWorkflowRunner(),
).replay_workflow(run.history)
+
+ @parameterized.expand(
+ [
+ # name, routed_to_precompute, run_on_activity_queue, recorded_without_routes, expected calls per queue
+ ("precompute_route", True, True, False, [], [], [("m1", None)]),
+ ("direct_route", False, True, False, [], [("m1", PrecomputationMode.DIRECT)], []),
+ ("run_off_the_flag_ignores_the_route", True, False, False, [("m1", None)], [], []),
+ ("history_without_routes_replays_on_another_queue", True, True, True, [], [("m1", None)], []),
+ ("history_without_routes_replays_with_another_arity", False, True, True, [], [("m1", None)], []),
+ ]
+ )
+ async def test_discovery_route_dispatches_the_calculation(
+ self,
+ name: str,
+ routed_to_precompute: bool,
+ run_on_activity_queue: bool,
+ recorded_without_routes: bool,
+ expected_workflow_queue_calls: list[_Call],
+ expected_activity_queue_calls: list[_Call],
+ expected_precompute_queue_calls: list[_Call],
+ ):
+ activity_worker_task_queue = str(uuid.uuid4())
+ precompute_worker_task_queue = str(uuid.uuid4())
+ metric = ExperimentMetricToRecalculate(
+ experiment_id=1,
+ metric_uuid="m1",
+ role="primary",
+ task_queue=precompute_worker_task_queue if routed_to_precompute else activity_worker_task_queue,
+ precomputation_mode=None if routed_to_precompute else PrecomputationMode.DIRECT,
+ )
+
+ def dispatch_without_routes(
+ metric: ExperimentMetricToRecalculate, activity_task_queue: str | None
+ ) -> _CalcDispatch:
+ return _CalcDispatch(task_queue=activity_task_queue, precomputation_mode=None)
+
+ with patch(
+ f"{_WORKFLOW_MODULE}._calc_dispatch",
+ side_effect=dispatch_without_routes if recorded_without_routes else _calc_dispatch,
+ ):
+ run = await _run_with_calc_queue_workers(
+ activity_task_queue=activity_worker_task_queue if run_on_activity_queue else None,
+ activity_worker_task_queue=activity_worker_task_queue,
+ precompute_worker_task_queue=precompute_worker_task_queue,
+ metric=metric,
+ )
+
+ assert run.result == {"total": 1, "succeeded": 1, "failed": 0}
+ assert run.workflow_queue_calls == expected_workflow_queue_calls
+ assert run.activity_queue_calls == expected_activity_queue_calls
+ assert run.precompute_queue_calls == expected_precompute_queue_calls
+ await temporalio.worker.Replayer(
+ workflows=[ExperimentMetricsRecalculationWorkflow],
+ workflow_runner=temporalio.worker.UnsandboxedWorkflowRunner(),
+ ).replay_workflow(run.history)