Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(experiments): let recalc calcs wait on the activity and precompute queues (#127946)

Rodrigo Ilorobab69c5a41f2
diff --git a/products/experiments/backend/temporal/ARCHITECTURE.md b/products/experiments/backend/temporal/ARCHITECTURE.md
index 84138f51c960..48ada1fb242b 100644
--- a/products/experiments/backend/temporal/ARCHITECTURE.md
+++ b/products/experiments/backend/temporal/ARCHITECTURE.md
@@ -261,7 +261,7 @@ The split is intentional: Grafana tells you about the worker process, PostHog te
As a second check, the workflow uses the route only for a run with an `activity_task_queue`. Runs off the flag keep the workflow's own queue and the AUTO mode. The route needs no patch marker: on replay, sdk-core matches a scheduled activity only by its id and type (`on_activity_task_scheduled` in `activity_state_machine.rs`), not by its queue or arguments, and a discovery result from before routing has no route, so it dispatches as before. The DIRECT mode is a sixth positional argument of the calc activity, and an AUTO calc keeps five. In a rolling deploy, a worker that predates the argument fails a DIRECT calc with a retryable `TypeError` until it rolls.
- Operating rules. Turn the flag on for an organization only after the activity worker and the precompute worker poll their queues in that region, because a queue with no poller holds the calculations until `schedule_to_close` fails them. To roll back, turn the flag off, then remove the worker only after every recalculation workflow that started before the flag change has closed. A fixed wait is not enough: such a run keeps its queue choice and schedules its calculations only after discovery and the start step, which take up to about 20 minutes of attempts and have no limit on queue wait. The 20-minute calc budget (`METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS`) starts only then, and the workflow has no execution timeout.
+ Operating rules. Turn the flag on for an organization only after the activity worker and the precompute worker poll their queues in that region, because a queue with no poller holds the calculations, and a calculation on those queues has no `schedule_to_close` budget to fail it. To roll back, turn the flag off, then remove the worker only after every recalculation workflow that started before the flag change has closed. A fixed wait is not enough: such a run keeps its queue choice and schedules its calculations only after discovery and the start step, which take up to about 20 minutes of attempts and have no limit on queue wait. A calculation on the activity or precompute queue then has no limit on queue wait either, and the workflow has no execution timeout.
Retries live in Temporal's `RetryPolicy` on the calc activity, not in the workflow. The policy is exponential: `initial_interval=5s`, `backoff_coefficient=2.0`, `maximum_interval=60s`, `maximum_attempts=MAX_METRIC_ATTEMPTS` (currently 8). So a metric backs off 5s, 10s, 20s, 40s, 60s, 60s, 60s across its retries, then the eighth attempt is final. `asyncio.gather(..., return_exceptions=True)` lets healthy metrics finish while a failing one retries; a retrying metric never blocks the others, because each activity retries on its own worker slot rather than holding a shared queue slot.
@@ -271,8 +271,9 @@ The split is intentional: Grafana tells you about the worker process, PostHog te
- **`is_final_attempt` is derived inside the activity.** Because Temporal now owns retries, the activity reads its own attempt count from `activity.info().attempt` and compares it against `MAX_METRIC_ATTEMPTS` (the same constant the workflow's `RetryPolicy` is built from, so both sides agree on which attempt is last). The activity persists a transient failure (FAILED row + `metric_errors`) only on the final attempt (or at the error class cap above), so a metric still being retried stays in its loading/dim state on the frontend instead of flashing an error that may yet resolve. Non-final transient state is tracked separately in the `metric_retries` column for the UI.
-- **Per-metric time budget.** The calc activity call sets `schedule_to_close_timeout` to `METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS` (20 minutes). It covers all attempts of one metric, the retry delays between them, and the time each attempt waits in the task queue. Without it, one metric could use 8 attempts of 300s plus backoff, which is more than the 30-minute staleness threshold, and a slow but healthy run could be marked stale while it still ran.
- The IN_PROGRESS phase is this budget, the failure backstop below (120s `schedule_to_close_timeout`, queue wait included), and the finish step (3 attempts of 90s), about 26 minutes in the worst case. `test_in_progress_phase_ends_before_the_staleness_ttl` keeps that sum below `_STALE_RECALC_THRESHOLD`.
+- **Per-metric time budget.** A calc on the workflow's own queue gets `schedule_to_close_timeout` set to `METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS` (20 minutes). It covers all attempts of one metric, the retry delays between them, and the time each attempt waits in the task queue. Without it, one metric could use 8 attempts of 300s plus backoff, which is more than the 30-minute staleness threshold, and a slow but healthy run could be marked stale while it still ran.
+ A calc on the activity queue or the precompute queue gets no `schedule_to_close_timeout` (`_calc_schedule_to_close_timeout` in `recalculation_workflow.py`). Those queues have a fixed slot count, so a full queue is backpressure: the calc waits for a slot instead of failing at the budget. `start_to_close`, the heartbeat timeout and the retry policy still bound each attempt and the number of attempts. Replay needs no patch marker, because sdk-core matches a scheduled activity only by its id and type. A run that waits longer than `_STALE_RECALC_THRESHOLD` can still be marked FAILED and cancelled by a new request.
+ On the workflow's own queue, the IN_PROGRESS phase is this budget, the failure backstop below (120s `schedule_to_close_timeout`, queue wait included), and the finish step (3 attempts of 90s), about 26 minutes in the worst case. `test_in_progress_phase_ends_before_the_staleness_ttl` keeps that sum below `_STALE_RECALC_THRESHOLD`.
With the budget, a metric with slow attempts can stop before `MAX_METRIC_ATTEMPTS`.
- **Failure backstop for metrics Temporal ends from the outside.** The activity can persist its own failure only when an attempt runs to the end. Temporal ends a metric without that in three cases: the budget has no room for the next retry, the last attempt passes `start_to_close_timeout`, or the last attempt misses its heartbeats (a dead worker). The workflow detects these from the activity error (a `TIMEOUT` retry state or a timeout cause) and runs `record_metric_out_of_time_failure` for each such metric. That activity writes the FAILED row, a `metric_errors` entry with `error_type="timeout"` and `retriable=True`, and the terminal `experiment metric error` event. It does nothing when the metric already has a `metric_errors` entry from its final attempt. It also keeps a COMPLETED row, because a last attempt can write its result and then die before it reports success. The check and both failure writes run under the result store's run-write locks. If the killed attempt's thread still finishes later, its result write removes the timeout entry from `metric_errors`, so the frontend shows the result. The new commands are gated on `workflow.patched("recalc-persist-out-of-time-metrics-2026-10")`, so executions that started before the gate replay without them.
diff --git a/products/experiments/backend/temporal/models.py b/products/experiments/backend/temporal/models.py
index e5056a241a62..84f50907e4f8 100644
--- a/products/experiments/backend/temporal/models.py
+++ b/products/experiments/backend/temporal/models.py
@@ -52,7 +52,7 @@ TIMEOUT_MAX_METRIC_ATTEMPTS = 2
# being polled constantly. The random position in the band spreads out metrics that bounced in the same burst,
# so their retries do not arrive together. The per-org app limiter skips Temporal activities, so this delay is
# what lets the cluster drain. The sum of the bands, at most 600 s, stays inside
-# METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS.
+# METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS where that budget applies.
CAPACITY_RETRY_FIRST_DELAY_MIN_SECONDS = 10
CAPACITY_RETRY_FIRST_DELAY_MAX_SECONDS = 20
CAPACITY_RETRY_BACKOFF_COEFFICIENT = 2.0
@@ -80,7 +80,9 @@ METRIC_CALC_ACTIVITY_TIMEOUT_SECONDS = 300
# Total budget for one metric across all its attempts and retry delays, queue wait included (the calc
# activity's schedule_to_close_timeout). It keeps the IN_PROGRESS phase (this budget, the failure backstop and
# the finish step) below the staleness TTL in recalculation.py, so a slow but healthy run is never treated as
-# stale. The budget ends a metric before MAX_METRIC_ATTEMPTS when its attempts are slow.
+# stale. The budget ends a metric before MAX_METRIC_ATTEMPTS when its attempts are slow. Only a calc on the
+# workflow's own queue gets it. A calc on the activity or precompute queue waits for a slot with no total budget,
+# so its run can outlast the staleness TTL.
METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS = 1200
# Without a heartbeat, Temporal can deliver a cancellation to the calc activity only after start_to_close,
diff --git a/products/experiments/backend/temporal/recalculation_workflow.py b/products/experiments/backend/temporal/recalculation_workflow.py
index 204b18c3453a..ba38b4d104cb 100644
--- a/products/experiments/backend/temporal/recalculation_workflow.py
+++ b/products/experiments/backend/temporal/recalculation_workflow.py
@@ -46,17 +46,32 @@ with temporalio.workflow.unsafe.imports_passed_through():
@frozen
class _CalcDispatch:
- """Where one metric's calc activity runs, and the precompute mode it passes."""
+ """Where one metric's calc activity runs, the precompute mode it passes, and its total time budget."""
task_queue: str | None
precomputation_mode: PrecomputationMode | None
+ schedule_to_close_timeout: timedelta | None
+
+
+def _calc_schedule_to_close_timeout(task_queue: str | None) -> timedelta | None:
+ # The activity and precompute queues have a fixed slot count, so a calc there waits for a slot instead of
+ # failing when the queue is full. Only a calc on the workflow's own queue keeps the total budget.
+ return None if task_queue else timedelta(seconds=METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS)
def _calc_dispatch(metric: ExperimentMetricToRecalculate, activity_task_queue: str | None) -> _CalcDispatch:
# Only a run on the activity queue flag uses the route that discovery set.
if activity_task_queue and metric.task_queue:
- return _CalcDispatch(task_queue=metric.task_queue, precomputation_mode=metric.precomputation_mode)
- return _CalcDispatch(task_queue=activity_task_queue, precomputation_mode=None)
+ return _CalcDispatch(
+ task_queue=metric.task_queue,
+ precomputation_mode=metric.precomputation_mode,
+ schedule_to_close_timeout=_calc_schedule_to_close_timeout(metric.task_queue),
+ )
+ return _CalcDispatch(
+ task_queue=activity_task_queue,
+ precomputation_mode=None,
+ schedule_to_close_timeout=_calc_schedule_to_close_timeout(activity_task_queue),
+ )
def _calc_args(
@@ -199,7 +214,7 @@ class ExperimentMetricsRecalculationWorkflow(PostHogWorkflow):
calculate_experiment_metric_for_recalculation,
args=_calc_args(metric, recalculation_id, query_to, dispatch),
start_to_close_timeout=timedelta(seconds=METRIC_CALC_ACTIVITY_TIMEOUT_SECONDS),
- schedule_to_close_timeout=timedelta(seconds=METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS),
+ schedule_to_close_timeout=dispatch.schedule_to_close_timeout,
heartbeat_timeout=timedelta(seconds=METRIC_CALC_HEARTBEAT_TIMEOUT_SECONDS),
retry_policy=calc_retry_policy,
task_queue=dispatch.task_queue,
diff --git a/products/experiments/backend/temporal/test_recalculation_workflow.py b/products/experiments/backend/temporal/test_recalculation_workflow.py
index 520e9df6518b..8f4153cfe4a2 100644
--- a/products/experiments/backend/temporal/test_recalculation_workflow.py
+++ b/products/experiments/backend/temporal/test_recalculation_workflow.py
@@ -42,6 +42,7 @@ from products.experiments.backend.temporal.recalculation_workflow import (
)
_START_QUERY_TO = "2026-05-29T12:00:00+00:00"
+_CALC_BUDGET = timedelta(seconds=METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS)
_WORKFLOW_MODULE = "products.experiments.backend.temporal.recalculation_workflow"
_ACTIVITIES_MODULE = "products.experiments.backend.temporal.recalculation_activities"
@@ -130,7 +131,7 @@ async def _run_workflow(activities) -> dict:
_Call = tuple[str, PrecomputationMode | None]
-def _recording_calculate(calls: list[_Call]) -> Callable:
+def _recording_calculate(calls: list[_Call], budgets: list[timedelta | None]) -> Callable:
@activity.defn(name="calculate_experiment_metric_for_recalculation")
async def mock_calculate(
experiment_id: int,
@@ -141,6 +142,7 @@ def _recording_calculate(calls: list[_Call]) -> Callable:
precomputation_mode: PrecomputationMode | None = None,
) -> MetricRecalculationResult:
calls.append((metric_uuid, precomputation_mode))
+ budgets.append(activity.info().schedule_to_close_timeout)
return MetricRecalculationResult(metric_uuid=metric_uuid, success=True)
return mock_calculate
@@ -152,6 +154,7 @@ class _QueueRun:
workflow_queue_calls: list[_Call]
activity_queue_calls: list[_Call]
precompute_queue_calls: list[_Call]
+ schedule_to_close_timeouts: list[timedelta | None]
history: WorkflowHistory
@@ -165,6 +168,7 @@ async def _run_with_calc_queue_workers(
workflow_queue_calls: list[_Call] = []
activity_queue_calls: list[_Call] = []
precompute_queue_calls: list[_Call] = []
+ budgets: list[timedelta | None] = []
task_queue = str(uuid.uuid4())
async with await WorkflowEnvironment.start_time_skipping() as env:
async with (
@@ -172,18 +176,18 @@ async def _run_with_calc_queue_workers(
env.client,
task_queue=task_queue,
workflows=[ExperimentMetricsRecalculationWorkflow],
- activities=[*activities[:2], _recording_calculate(workflow_queue_calls)],
+ activities=[*activities[:2], _recording_calculate(workflow_queue_calls, budgets)],
workflow_runner=temporalio.worker.UnsandboxedWorkflowRunner(),
),
Worker(
env.client,
task_queue=activity_worker_task_queue,
- activities=[_recording_calculate(activity_queue_calls)],
+ activities=[_recording_calculate(activity_queue_calls, budgets)],
),
Worker(
env.client,
task_queue=precompute_worker_task_queue or str(uuid.uuid4()),
- activities=[_recording_calculate(precompute_queue_calls)],
+ activities=[_recording_calculate(precompute_queue_calls, budgets)],
),
):
handle = await env.client.start_workflow(
@@ -201,6 +205,7 @@ async def _run_with_calc_queue_workers(
workflow_queue_calls=workflow_queue_calls,
activity_queue_calls=activity_queue_calls,
precompute_queue_calls=precompute_queue_calls,
+ schedule_to_close_timeouts=budgets,
history=history,
)
@@ -692,12 +697,22 @@ class TestExperimentMetricsRecalculationWorkflow:
@parameterized.expand(
[
- # name, routed_to_precompute, run_on_activity_queue, recorded_without_routes, expected calls per queue
- ("precompute_route", True, True, False, [], [], [("m1", None)]),
- ("direct_route", False, True, False, [], [("m1", PrecomputationMode.DIRECT)], []),
- ("run_off_the_flag_ignores_the_route", True, False, False, [("m1", None)], [], []),
- ("history_without_routes_replays_on_another_queue", True, True, True, [], [("m1", None)], []),
- ("history_without_routes_replays_with_another_arity", False, True, True, [], [("m1", None)], []),
+ # name, routed_to_precompute, run_on_activity_queue, recorded_without_routes, expected calls per queue,
+ # has the total budget
+ ("precompute_route", True, True, False, [], [], [("m1", None)], False),
+ ("direct_route", False, True, False, [], [("m1", PrecomputationMode.DIRECT)], [], False),
+ ("run_off_the_flag_ignores_the_route", True, False, False, [("m1", None)], [], [], True),
+ ("history_without_routes_replays_on_another_queue", True, True, True, [], [("m1", None)], [], True),
+ (
+ "history_without_routes_replays_with_another_arity",
+ False,
+ True,
+ True,
+ [],
+ [("m1", None)],
+ [],
+ True,
+ ),
]
)
async def test_discovery_route_dispatches_the_calculation(
@@ -709,6 +724,7 @@ class TestExperimentMetricsRecalculationWorkflow:
expected_workflow_queue_calls: list[_Call],
expected_activity_queue_calls: list[_Call],
expected_precompute_queue_calls: list[_Call],
+ has_total_budget: bool,
):
activity_worker_task_queue = str(uuid.uuid4())
precompute_worker_task_queue = str(uuid.uuid4())
@@ -723,7 +739,9 @@ class TestExperimentMetricsRecalculationWorkflow:
def dispatch_without_routes(
metric: ExperimentMetricToRecalculate, activity_task_queue: str | None
) -> _CalcDispatch:
- return _CalcDispatch(task_queue=activity_task_queue, precomputation_mode=None)
+ return _CalcDispatch(
+ task_queue=activity_task_queue, precomputation_mode=None, schedule_to_close_timeout=_CALC_BUDGET
+ )
with patch(
f"{_WORKFLOW_MODULE}._calc_dispatch",
@@ -740,6 +758,12 @@ class TestExperimentMetricsRecalculationWorkflow:
assert run.workflow_queue_calls == expected_workflow_queue_calls
assert run.activity_queue_calls == expected_activity_queue_calls
assert run.precompute_queue_calls == expected_precompute_queue_calls
+ assert len(run.schedule_to_close_timeouts) == 1
+ if has_total_budget:
+ assert run.schedule_to_close_timeouts[0] == _CALC_BUDGET
+ else:
+ # The test server reports an unset budget as a ten-year one.
+ assert run.schedule_to_close_timeouts[0] in (None, timedelta(days=3650))
await temporalio.worker.Replayer(
workflows=[ExperimentMetricsRecalculationWorkflow],
workflow_runner=temporalio.worker.UnsandboxedWorkflowRunner(),