Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(experiments): cap concurrent scheduled recalculation runs (#119373)

Vasco de Krijgerb5fe40e2a3c9
diff --git a/products/experiments/backend/temporal/ARCHITECTURE.md b/products/experiments/backend/temporal/ARCHITECTURE.md
index a2c4d264ac0b..9e6ce119da60 100644
--- a/products/experiments/backend/temporal/ARCHITECTURE.md
+++ b/products/experiments/backend/temporal/ARCHITECTURE.md
@@ -8,7 +8,7 @@ When a user opens an experiment and the cached results look stale, they click **
It's distinct from the daily timeseries workflows in two ways:
-- **On-demand, not scheduled.** Triggered by the user (or by experiment lifecycle events like launch/stop), not by a cron schedule.
+- **One experiment per run.** Triggered by the user, by experiment lifecycle events like launch/stop, or by the hourly scheduled recalculation for enabled teams. Every trigger starts the same per-experiment run; there is no sweep across experiments inside one workflow. See "Scheduled recalculation".
- **Snapshot per run, not cache warming.** Each click creates a new run with its own results, identifiable by `recalculation_id`. The timeseries family overwrites cached values; we preserve them. (Two exceptions: a stopped experiment has a fixed window, and a `METRIC_CONFIG_CHANGE` run reuses the previous window, so those runs share result rows rather than each getting a distinct snapshot. See "Snapshot semantics for the user".)
## End-to-end flow
@@ -220,6 +220,17 @@ An entry without the flag predates it and counts as non-retriable, so an old run
For an in-progress run, `get_live_query_progress` reads `system.processes` and `system.query_log` by client-query-id prefix to surface cumulative ClickHouse `rows_read` while the run executes. Nothing is stored: each metric query is tagged with a deterministic `client_query_id`, so the progress is reconstructable from ClickHouse alone. It is best-effort and decorative; a failure there returns null and never sinks the core status payload.
+### Scheduled recalculation
+
+`ScheduledExperimentRecalculationWorkflow` (`scheduled_recalculation_workflow.py`) runs on one hourly schedule at `:30`, after the daily timeseries schedule at `:00`. It is behind the `experiments-scheduled-recalculation` flag. Each firing starts ordinary recalculation runs with trigger `SCHEDULED`, through the same dispatch the API uses.
+
+- **Discovery** (`find_scheduled_recalculation_candidates`) selects running experiments whose team recalculates at the current UTC hour (`experiment_recalculation_times`, default 02:00). It skips, before any ClickHouse query, a flag that serves nobody, no computable metric, an active or recent run, and an exposure back-off. Candidates come oldest scheduled run first, never-run first, so an experiment deferred one day starts first the next day.
+- **Exposure gate.** `check_experiment_exposures` runs one cheap query per candidate and skips experiments below `MIN_TOTAL_EXPOSURES`.
+- **Admission cap.** One experiment holds a slot of `MAX_CONCURRENT_EXPERIMENTS` from its exposure check until its run's workflow finishes (`wait_for_scheduled_recalculation`). A run fans out all its metrics, so the multiplier on ClickHouse load is the number of runs in flight, not the metrics per run. Without the wait, one team with many running experiments starts every run within minutes and exhausts the per-user concurrency limit of its ClickHouse user for every other workload on it.
+- **Start window.** No run starts later than `RUN_START_WINDOW` after the firing. The deadline bounds the activity timeouts, and `start_scheduled_recalculation` checks it again before it creates the row, so a delayed retry cannot start a run late. Candidates left over are counted as `deferred`. The window plus the start budget plus `RUN_WAIT_TIMEOUT` (the staleness threshold) stays under the hourly interval, because the schedule's `SKIP` overlap policy would otherwise drop the next hour's teams. `test_last_run_ends_before_the_next_hourly_firing` keeps that sum below an hour.
+- **Fail closed on unknown outcomes.** A start activity that fails, a dispatch that raised after Temporal may have accepted the run (`dispatch_uncertain`), or a wait that cannot observe the run each stop new admissions for the rest of the batch. The run may still be live, so releasing its slot could exceed the cap.
+- **Replay safety.** The wait, the window and the stop are gated on `workflow.patched("scheduled-recalc-wait-for-runs-2026-10")`.
+
## Eviction
We don't currently delete old recalcs or their results. Power users compound rows fast: 50 clicks × 10 metrics × 90 days = 45k rows for one team. The plan is a bi-weekly Temporal Schedule that deletes both tables past a cutoff:
@@ -240,7 +251,7 @@ The split is intentional: Grafana tells you about the worker process, PostHog te
## Performance notes
-- **Fan out all metrics at once, Temporal owns retries.** The workflow schedules one calc activity per metric in a single `asyncio.gather` (`recalculation_workflow.py`), with no concurrency of its own. The activity slots of the worker that runs the calc activity set the pacing. The per-org ClickHouse app-query limiter does not apply inside Temporal activities. Scheduling every metric up front also keeps the task-queue backlog honest for the autoscaler. The workflow runs on its own `experiments-recalculation-task-queue` (see `EXPERIMENTS_RECALCULATION_TASK_QUEUE`), served by a dedicated worker deployment, so recalc load can be scaled and throttled independently of the general-purpose worker. That deployment autoscales on backlog, so its slots (`MAX_CONCURRENT_ACTIVITIES` times the pod count) do not cap the number of concurrent queries. The activity task queue below does.
+- **Fan out all metrics at once, Temporal owns retries.** The workflow schedules one calc activity per metric in a single `asyncio.gather` (`recalculation_workflow.py`), with no concurrency of its own. The activity slots of the worker that runs the calc activity set the pacing. The per-org ClickHouse app-query limiter does not apply inside Temporal activities. Scheduling every metric up front also keeps the task-queue backlog honest for the autoscaler. The per-run fan-out is unbounded by design; the scheduled path bounds how many runs it starts at once instead (see "Scheduled recalculation"). The workflow runs on its own `experiments-recalculation-task-queue` (see `EXPERIMENTS_RECALCULATION_TASK_QUEUE`), served by a dedicated worker deployment, so recalc load can be scaled and throttled independently of the general-purpose worker. That deployment autoscales on backlog, so its slots (`MAX_CONCURRENT_ACTIVITIES` times the pod count) do not cap the number of concurrent queries. The activity task queue below does.
- **Activity task queue with a fixed cap.** A scheduled burst can send more concurrent queries than the ClickHouse default user allows, and the rejections then also hit other products that share that user. The calc activity is also registered on `experiments-recalculation-activity-task-queue` (`EXPERIMENTS_RECALCULATION_ACTIVITY_TASK_QUEUE`), with no workflows. A fixed worker with no autoscaling polls it in each region, so its pod count times its slot count is a hard cap on concurrent recalculation queries for the runs routed there. The workflow and its light Postgres activities stay on the main queue.
diff --git a/products/experiments/backend/temporal/__init__.py b/products/experiments/backend/temporal/__init__.py
index f9291da62de8..1b4fedd55da4 100644
--- a/products/experiments/backend/temporal/__init__.py
+++ b/products/experiments/backend/temporal/__init__.py
@@ -19,6 +19,7 @@ from products.experiments.backend.temporal.scheduled_recalculation_activities im
check_experiment_exposures,
discover_scheduled_recalculation_candidates,
start_scheduled_recalculation,
+ wait_for_scheduled_recalculation,
)
from products.experiments.backend.temporal.scheduled_recalculation_workflow import (
ScheduledExperimentRecalculationWorkflow,
@@ -45,6 +46,7 @@ EXPERIMENT_SCHEDULED_RECALCULATION_ACTIVITIES = [
discover_scheduled_recalculation_candidates,
check_experiment_exposures,
start_scheduled_recalculation,
+ wait_for_scheduled_recalculation,
]
EXPERIMENT_RECALCULATION_ACTIVITY_QUEUE_ACTIVITIES = [
calculate_experiment_metric_for_recalculation,
@@ -91,4 +93,5 @@ __all__ = [
"sample_experiment_canary_targets",
"start_scheduled_recalculation",
"update_recalculation_progress",
+ "wait_for_scheduled_recalculation",
]
diff --git a/products/experiments/backend/temporal/scheduled_recalculation_activities.py b/products/experiments/backend/temporal/scheduled_recalculation_activities.py
index 2d02418fe959..cd7c33afef31 100644
--- a/products/experiments/backend/temporal/scheduled_recalculation_activities.py
+++ b/products/experiments/backend/temporal/scheduled_recalculation_activities.py
@@ -1,28 +1,35 @@
"""Activities for the scheduled experiment recalculation workflow.
Each one wraps `scheduled_recalculation_logic` and reports its outcome to product analytics.
-Every activity is fire and forget: a failure skips one experiment and never stops the batch,
-because the recalculation workflow it starts owns its own retries and error reporting.
+A failure skips one experiment and never stops the batch, because the recalculation workflow
+it starts owns its own retries and error reporting.
"""
+from datetime import datetime
from typing import Any
from django.db import close_old_connections
+from django.utils import timezone
import structlog
import posthoganalytics
import temporalio.activity
+from temporalio.client import WorkflowFailureError
from posthog.event_usage import groups
from posthog.sync import database_sync_to_async_pool
+from posthog.temporal.common.client import async_connect
+from posthog.temporal.common.heartbeat import Heartbeater
from products.experiments.backend.models.experiment import Experiment, ExperimentMetricsRecalculation
from products.experiments.backend.temporal.models import ScheduledRecalculationStartResult
from products.experiments.backend.temporal.scheduled_recalculation_logic import (
MIN_TOTAL_EXPOSURES,
SKIP_ACTIVE_RUN,
+ SKIP_DISPATCH_UNCERTAIN,
SKIP_EXPOSURE_QUERY_FAILED,
SKIP_INSUFFICIENT_EXPOSURES,
+ SKIP_START_WINDOW_ELAPSED,
ScheduledRecalculationDiscovery,
count_total_exposures,
find_scheduled_recalculation_candidates,
@@ -116,7 +123,14 @@ async def check_experiment_exposures(experiment_id: int, hour: int) -> bool:
@database_sync_to_async_pool
-def _start_scheduled_recalculation_sync(experiment_id: int, hour: int) -> ScheduledRecalculationStartResult:
+def _start_scheduled_recalculation_sync(
+ experiment_id: int, hour: int, start_before: datetime | None = None
+) -> ScheduledRecalculationStartResult:
+ if start_before is not None and timezone.now() >= start_before:
+ return ScheduledRecalculationStartResult(
+ experiment_id=experiment_id, started=False, skip_reason=SKIP_START_WINDOW_ELAPSED
+ )
+
# Deferred: recalculation.py imports temporal.recalculation_logic at module level, and this
# module is loaded from the temporal package's __init__, so a module-level import here closes
# a cycle back into a partially initialized module.
@@ -141,6 +155,11 @@ def _start_scheduled_recalculation_sync(experiment_id: int, hour: int) -> Schedu
capture_skip(experiment, hour, reason=skip.reason, detail=dict(skip.detail))
return ScheduledRecalculationStartResult(experiment_id=experiment_id, started=False, skip_reason=skip.reason)
+ if start_before is not None and timezone.now() >= start_before:
+ return ScheduledRecalculationStartResult(
+ experiment_id=experiment_id, started=False, skip_reason=SKIP_START_WINDOW_ELAPSED
+ )
+
try:
payload = request_recalculation(experiment, None, ExperimentMetricsRecalculation.Trigger.SCHEDULED)
except Exception:
@@ -174,7 +193,12 @@ def _start_scheduled_recalculation_sync(experiment_id: int, hour: int) -> Schedu
recalculation_id=recalculation_id,
exc_info=True,
)
- return ScheduledRecalculationStartResult(experiment_id=experiment_id, started=False)
+ return ScheduledRecalculationStartResult(
+ experiment_id=experiment_id,
+ started=False,
+ recalculation_id=recalculation_id,
+ skip_reason=SKIP_DISPATCH_UNCERTAIN,
+ )
capture_started(experiment, hour, recalculation_id)
return ScheduledRecalculationStartResult(
@@ -183,6 +207,27 @@ def _start_scheduled_recalculation_sync(experiment_id: int, hour: int) -> Schedu
@temporalio.activity.defn
-async def start_scheduled_recalculation(experiment_id: int, hour: int) -> ScheduledRecalculationStartResult:
+async def start_scheduled_recalculation(
+ experiment_id: int, hour: int, start_before: datetime | None = None
+) -> ScheduledRecalculationStartResult:
"""Create the recalculation row and dispatch its workflow, unless a recent or active run exists."""
- return await _start_scheduled_recalculation_sync(experiment_id, hour)
+ return await _start_scheduled_recalculation_sync(experiment_id, hour, start_before)
+
+
+async def wait_for_scheduled_recalculation(recalculation_id: str) -> None:
+ """Return once a started run's workflow has finished, whatever its outcome.
+
+ The run records its own outcome on its row, so a failed or cancelled run is not an error here.
+ """
+ from products.experiments.backend.recalculation import ( # noqa: PLC0415 — breaks that cycle
+ metrics_recalculation_workflow_id,
+ )
+
+ client = await async_connect()
+ handle = client.get_workflow_handle(metrics_recalculation_workflow_id(recalculation_id))
+ async with Heartbeater(factor=4):
+ try:
+ await handle.result()
+ except WorkflowFailureError:
+ pass
diff --git a/products/experiments/backend/temporal/scheduled_recalculation_logic.py b/products/experiments/backend/temporal/scheduled_recalculation_logic.py
index 4533f81e2490..0ea552e82960 100644
--- a/products/experiments/backend/temporal/scheduled_recalculation_logic.py
+++ b/products/experiments/backend/temporal/scheduled_recalculation_logic.py
@@ -8,7 +8,7 @@ import json
from datetime import UTC, datetime, timedelta
from typing import TYPE_CHECKING, Final
-from django.db.models import Prefetch, Q, QuerySet
+from django.db.models import Max, Prefetch, Q, QuerySet
from django.utils import timezone
import structlog
@@ -62,6 +62,10 @@ SKIP_RECENT_RUN: Final = "recent_run_exists"
SKIP_ZERO_ROLLOUT: Final = "zero_rollout"
SKIP_NO_METRICS: Final = "no_metrics"
SKIP_EXPOSURE_BACKOFF: Final = "exposure_backoff"
+SKIP_DISPATCH_UNCERTAIN: Final = "dispatch_uncertain"
+SKIP_START_WINDOW_ELAPSED: Final = "start_window_elapsed"
+
+NEVER_SCHEDULED = datetime.min.replace(tzinfo=UTC)
@frozen
@@ -131,6 +135,9 @@ def find_scheduled_recalculation_candidates() -> ScheduledRecalculationSelection
or an exposure back-off. The query count does not grow with the number of candidates.
An exposure-frozen experiment stays eligible, because its metric events keep arriving.
+
+ Candidates come oldest scheduled run first, so an experiment that the workflow deferred
+ yesterday starts before the ones that ran.
"""
# Deferred: importing this module runs posthog/temporal/experiments/__init__.py, which pulls
# activities.py, which imports back into products.experiments.backend.facade.timeseries.
@@ -205,6 +212,9 @@ def find_scheduled_recalculation_candidates() -> ScheduledRecalculationSelection
)
)
+ last_scheduled_runs = _last_scheduled_runs([c.experiment_id for c in candidates])
+ candidates.sort(key=lambda c: (last_scheduled_runs.get(c.experiment_id, NEVER_SCHEDULED), c.experiment_id))
+
logger.info("scheduled_recalculation_candidates", hour=hour, count=len(candidates), skipped=len(skips))
return ScheduledRecalculationSelection(
discovery=ScheduledRecalculationDiscovery(hour=hour, candidates=candidates), skips=skips
@@ -307,6 +317,18 @@ def _recent_run_skips(experiment_ids: list[int]) -> dict[int, SkipDecision]:
return skips
+def _last_scheduled_runs(experiment_ids: list[int]) -> dict[int, datetime]:
+ """When each experiment last got a scheduled run, in one query across teams."""
+ rows = (
+ ExperimentMetricsRecalculation.objects.unscoped()
+ .filter(experiment_id__in=experiment_ids, trigger=ExperimentMetricsRecalculation.Trigger.SCHEDULED)
+ .values("experiment_id")
+ .annotate(last_created_at=Max("created_at"))
+ .values_list("experiment_id", "last_created_at")
+ )
+ return dict(rows)
+
+
def _exposure_backoff_key(experiment_id: int) -> str:
return f"{EXPOSURE_BACKOFF_KEY_PREFIX}:{experiment_id}"
diff --git a/products/experiments/backend/temporal/scheduled_recalculation_workflow.py b/products/experiments/backend/temporal/scheduled_recalculation_workflow.py
index d31ae1f576f9..b7f6db0ec70f 100644
--- a/products/experiments/backend/temporal/scheduled_recalculation_workflow.py
+++ b/products/experiments/backend/temporal/scheduled_recalculation_workflow.py
@@ -3,6 +3,7 @@ from datetime import timedelta
import temporalio.workflow
from temporalio.common import RetryPolicy
+from temporalio.exceptions import ActivityError
from posthog.temporal.common.base import PostHogWorkflow
@@ -12,13 +13,24 @@ with temporalio.workflow.unsafe.imports_passed_through():
check_experiment_exposures,
discover_scheduled_recalculation_candidates,
start_scheduled_recalculation,
+ wait_for_scheduled_recalculation,
+ )
+ from products.experiments.backend.temporal.scheduled_recalculation_logic import (
+ SKIP_DISPATCH_UNCERTAIN,
+ SKIP_START_WINDOW_ELAPSED,
+ ScheduledRecalculationDiscovery,
)
- from products.experiments.backend.temporal.scheduled_recalculation_logic import ScheduledRecalculationDiscovery
-# Matches the daily timeseries workflows. Each unit here is one cheap query plus one workflow
-# start, so this bounds the exposure queries rather than the recalculations themselves.
MAX_CONCURRENT_EXPERIMENTS = 10
+SCHEDULE_INTERVAL = timedelta(hours=1)
+START_ACTIVITY_TIMEOUT = timedelta(minutes=2)
+START_ACTIVITY_MAX_ATTEMPTS = 2
+RUN_WAIT_TIMEOUT = timedelta(minutes=30)
+RUN_START_WINDOW = timedelta(minutes=20)
+
+WAIT_FOR_RUNS_PATCH = "scheduled-recalc-wait-for-runs-2026-10"
+
@temporalio.workflow.defn(name=SCHEDULED_RECALCULATION_WORKFLOW_NAME)
class ScheduledExperimentRecalculationWorkflow(PostHogWorkflow):
@@ -27,9 +39,12 @@ class ScheduledExperimentRecalculationWorkflow(PostHogWorkflow):
One hourly schedule drives it. Discovery resolves the hour from the clock and selects the
teams configured for it, so the workflow carries no input and no clock of its own.
- Fire and forget. Each started run is an ordinary `ExperimentMetricsRecalculationWorkflow`,
- dispatched the same way the API dispatches one, and this workflow never waits for it. The
- counts it returns are for an operator reading the run, not a source of truth: each run's
+ Each started run is an ordinary `ExperimentMetricsRecalculationWorkflow`, dispatched the same
+ way the API dispatches one. The workflow waits for a run to finish before it uses that slot
+ for the next experiment, so each batch admits at most MAX_CONCURRENT_EXPERIMENTS runs at once.
+ Experiments not started within RUN_START_WINDOW are deferred to their next scheduled hour, so
+ activity queueing and retries cannot consume the next hourly firing, which SKIP would drop.
+ The counts it returns are for an operator reading the run, not a source of truth: each run's
real outcome lands on its own `ExperimentMetricsRecalculation` row.
"""
@@ -39,45 +54,96 @@ class ScheduledExperimentRecalculationWorkflow(PostHogWorkflow):
@temporalio.workflow.run
async def run(self) -> dict[str, int]:
+ started_at = temporalio.workflow.now()
+ start_before = started_at + RUN_START_WINDOW
+ wait_for_runs = temporalio.workflow.patched(WAIT_FOR_RUNS_PATCH)
discovery: ScheduledRecalculationDiscovery = await temporalio.workflow.execute_activity(
discover_scheduled_recalculation_candidates,
start_to_close_timeout=timedelta(minutes=5),
+ schedule_to_close_timeout=RUN_START_WINDOW if wait_for_runs else None,
retry_policy=RetryPolicy(maximum_attempts=3),
)
hour, candidates = discovery.hour, discovery.candidates
if not candidates:
- return {"hour": hour, "candidates": 0, "started": 0, "skipped": 0}
+ return {"hour": hour, "candidates": 0, "started": 0, "skipped": 0, "deferred": 0}
semaphore = asyncio.Semaphore(MAX_CONCURRENT_EXPERIMENTS)
+ deferred = 0
+ stop_starting = False
+
+ def _should_defer() -> bool:
+ nonlocal deferred
+ if wait_for_runs and (stop_starting or temporalio.workflow.now() >= start_before):
+ deferred += 1
+ return True
+ return False
async def _process(experiment_id: int) -> bool:
+ nonlocal deferred, stop_starting
async with semaphore:
+ if _should_defer():
+ return False
has_exposures = await temporalio.workflow.execute_activity(
check_experiment_exposures,
args=[experiment_id, hour],
start_to_close_timeout=timedelta(minutes=10),
+ schedule_to_close_timeout=(start_before - temporalio.workflow.now()) if wait_for_runs else None,
retry_policy=RetryPolicy(maximum_attempts=2),
)
- if not has_exposures:
+ if not has_exposures or _should_defer():
return False
- result = await temporalio.workflow.execute_activity(
- start_scheduled_recalculation,
- args=[experiment_id, hour],
- start_to_close_timeout=timedelta(minutes=2),
- retry_policy=RetryPolicy(maximum_attempts=2),
- )
- return bool(result.started)
+ try:
+ result = await temporalio.workflow.execute_activity(
+ start_scheduled_recalculation,
+ args=[experiment_id, hour, start_before] if wait_for_runs else [experiment_id, hour],
+ start_to_close_timeout=START_ACTIVITY_TIMEOUT,
+ schedule_to_close_timeout=(
+ min(
+ START_ACTIVITY_TIMEOUT * START_ACTIVITY_MAX_ATTEMPTS,
+ start_before - temporalio.workflow.now(),
+ )
+ if wait_for_runs
+ else None
+ ),
+ retry_policy=RetryPolicy(maximum_attempts=START_ACTIVITY_MAX_ATTEMPTS),
+ )
+ except ActivityError:
+ stop_starting = True
+ raise
+ if not result.started:
+ if result.skip_reason == SKIP_START_WINDOW_ELAPSED:
+ deferred += 1
+ elif result.skip_reason == SKIP_DISPATCH_UNCERTAIN:
+ stop_starting = True
+ return False
+ if wait_for_runs and result.recalculation_id is not None:
+ try:
+ await temporalio.workflow.execute_activity(
+ wait_for_scheduled_recalculation,
+ result.recalculation_id,
+ schedule_to_close_timeout=RUN_WAIT_TIMEOUT,
+ heartbeat_timeout=timedelta(minutes=1),
+ )
+ except ActivityError:
+ stop_starting = True
+ temporalio.workflow.logger.warning(
+ f"scheduled recalculation {result.recalculation_id} wait ended before the run finished"
+ )
+ return True
outcomes = await asyncio.gather(
*[_process(candidate.experiment_id) for candidate in candidates], return_exceptions=True
)
started = sum(1 for outcome in outcomes if outcome is True)
- temporalio.workflow.logger.info(f"scheduled recalculation hour {hour}: {started} of {len(candidates)} started")
+ temporalio.workflow.logger.info(
+ f"scheduled recalculation hour {hour}: {started} of {len(candidates)} started, {deferred} deferred"
+ )
return {
"hour": hour,
"candidates": len(candidates),
"started": started,
"skipped": len(candidates) - started,
+ "deferred": deferred,
}
diff --git a/products/experiments/backend/temporal/test_registration.py b/products/experiments/backend/temporal/test_registration.py
index db09ace31803..97baff2ee891 100644
--- a/products/experiments/backend/temporal/test_registration.py
+++ b/products/experiments/backend/temporal/test_registration.py
@@ -14,6 +14,7 @@ from products.experiments.backend.temporal.scheduled_recalculation_activities im
check_experiment_exposures,
discover_scheduled_recalculation_candidates,
start_scheduled_recalculation,
+ wait_for_scheduled_recalculation,
)
from products.experiments.backend.temporal.scheduled_recalculation_workflow import (
ScheduledExperimentRecalculationWorkflow,
@@ -34,6 +35,7 @@ def test_activities_registered():
"discover_scheduled_recalculation_candidates",
"check_experiment_exposures",
"start_scheduled_recalculation",
+ "wait_for_scheduled_recalculation",
}
@@ -71,6 +73,7 @@ def test_scheduled_workflows_registered_on_general_purpose_queue():
assert discover_scheduled_recalculation_candidates in activities
assert check_experiment_exposures in activities
assert start_scheduled_recalculation in activities
+ assert wait_for_scheduled_recalculation in activities
def test_calc_activity_registered_on_recalculation_and_activity_queues():
diff --git a/products/experiments/backend/temporal/test_scheduled_recalculation_activities.py b/products/experiments/backend/temporal/test_scheduled_recalculation_activities.py
index 8755e6bcd7a6..5ed06e13786c 100644
--- a/products/experiments/backend/temporal/test_scheduled_recalculation_activities.py
+++ b/products/experiments/backend/temporal/test_scheduled_recalculation_activities.py
@@ -1,7 +1,8 @@
from collections.abc import Callable
-from datetime import timedelta
+from datetime import UTC, datetime, timedelta
from uuid import uuid4
+import pytest
import time_machine
from posthog.test.base import BaseTest
from unittest.mock import patch
@@ -30,11 +31,13 @@ from products.experiments.backend.temporal.scheduled_recalculation_activities im
)
from products.experiments.backend.temporal.scheduled_recalculation_logic import (
SKIP_ACTIVE_RUN,
+ SKIP_DISPATCH_UNCERTAIN,
SKIP_EXPOSURE_BACKOFF,
SKIP_EXPOSURE_QUERY_FAILED,
SKIP_INSUFFICIENT_EXPOSURES,
SKIP_NO_METRICS,
SKIP_RECENT_RUN,
+ SKIP_START_WINDOW_ELAPSED,
SKIP_ZERO_ROLLOUT,
ScheduledRecalculationDiscovery,
)
@@ -57,9 +60,11 @@ def _check_exposures(experiment_id: int, hour: int) -> bool:
return _check_exposures_raw(experiment_id, hour)
-def _start_recalculation(experiment_id: int, hour: int) -> ScheduledRecalculationStartResult:
+def _start_recalculation(
+ experiment_id: int, hour: int, start_before: datetime | None = None
+) -> ScheduledRecalculationStartResult:
with patch(f"{MODULE}.close_old_connections"):
- return _start_recalculation_raw(experiment_id, hour)
+ return _start_recalculation_raw(experiment_id, hour, start_before)
def _discover() -> ScheduledRecalculationDiscovery:
@@ -225,9 +230,12 @@ class TestScheduledRecalculationActivities(BaseTest):
):
assert _check_exposures(self.experiment.id, 12) is expected
- def test_start_creates_a_row_and_dispatches_the_workflow(self):
+ @parameterized.expand([("legacy", False), ("within_deadline", True)])
+ def test_start_creates_a_row_and_dispatches_the_workflow(self, _name: str, with_deadline: bool):
with patch(f"{RECALCULATION}.start_metrics_recalculation_workflow") as dispatch:
- result = _start_recalculation(self.experiment.id, 2)
+ result = _start_recalculation(
+ self.experiment.id, 2, timezone.now() + timedelta(minutes=1) if with_deadline else None
+ )
assert result.started is True
assert result.recalculation_id is not None
with team_scope(self.team.id, canonical=True):
@@ -273,6 +281,8 @@ class TestScheduledRecalculationActivities(BaseTest):
):
result = _start_recalculation(self.experiment.id, 2)
assert result.started is False
+ assert result.skip_reason == SKIP_DISPATCH_UNCERTAIN
+ assert result.recalculation_id is not None
started_event.assert_not_called()
def test_deleted_experiment_is_skipped_not_raised(self):
@@ -289,3 +299,19 @@ class TestScheduledRecalculationActivities(BaseTest):
result = _start_recalculation(self.experiment.id, 2)
assert result.started is False
dispatch.assert_not_called()
+
+
[email protected]("seconds_late", [0, 60])
+def test_start_picked_up_after_deadline_does_not_create_a_run(seconds_late):
+ deadline = datetime(2026, 10, 19, 6, 50, tzinfo=UTC)
+ with (
+ time_machine.travel(deadline + timedelta(seconds=seconds_late), tick=False),
+ patch(f"{MODULE}.close_old_connections"),
+ patch(f"{RECALCULATION}.request_recalculation") as request,
+ patch(f"{RECALCULATION}.start_metrics_recalculation_workflow") as dispatch,
+ ):
+ result = _start_recalculation_raw(1, 2, deadline)
+ assert result.started is False
+ assert result.skip_reason == SKIP_START_WINDOW_ELAPSED
+ request.assert_not_called()
+ dispatch.assert_not_called()
diff --git a/products/experiments/backend/temporal/test_scheduled_recalculation_logic.py b/products/experiments/backend/temporal/test_scheduled_recalculation_logic.py
index b5f5997efd61..d2190df91168 100644
--- a/products/experiments/backend/temporal/test_scheduled_recalculation_logic.py
+++ b/products/experiments/backend/temporal/test_scheduled_recalculation_logic.py
@@ -395,6 +395,24 @@ class TestScheduledRecalculationLogic(BaseTest):
)
self._assert_discovery_and_start_skip(experiment, None)
+ def test_candidates_come_oldest_scheduled_run_first(self):
+ ran_yesterday, never_ran, ran_two_days_ago = self._experiment(), self._experiment(), self._experiment()
+ with team_scope(self.team.id, canonical=True):
+ for experiment, age in [(ran_yesterday, timedelta(days=1)), (ran_two_days_ago, timedelta(days=2))]:
+ recalc = ExperimentMetricsRecalculation.objects.create(
+ team=self.team,
+ experiment=experiment,
+ status=ExperimentMetricsRecalculation.Status.COMPLETED,
+ trigger=ExperimentMetricsRecalculation.Trigger.SCHEDULED,
+ completed_at=timezone.now() - age,
+ )
+ ExperimentMetricsRecalculation.objects.filter(id=recalc.id).update(created_at=timezone.now() - age)
+
+ with patch(FLAG_PATH, return_value=True):
+ candidates = find_scheduled_recalculation_candidates().discovery.candidates
+
+ assert [c.experiment_id for c in candidates] == [never_ran.id, ran_two_days_ago.id, ran_yesterday.id]
+
def test_flag_helper_passes_group_properties(self):
with patch(
"products.experiments.backend.temporal.scheduled_recalculation_logic.feature_enabled_or_false",
diff --git a/products/experiments/backend/temporal/test_scheduled_recalculation_workflow.py b/products/experiments/backend/temporal/test_scheduled_recalculation_workflow.py
index 5b4fe6cc310d..2c2fae8a060d 100644
--- a/products/experiments/backend/temporal/test_scheduled_recalculation_workflow.py
+++ b/products/experiments/backend/temporal/test_scheduled_recalculation_workflow.py
@@ -1,5 +1,7 @@
import uuid
-from datetime import timedelta
+import asyncio
+from contextlib import nullcontext
+from datetime import UTC, datetime, timedelta
import pytest
import time_machine
@@ -11,23 +13,33 @@ from django.utils import timezone
from asgiref.sync import async_to_sync
from temporalio import activity
from temporalio.client import WorkflowFailureError
+from temporalio.exceptions import ApplicationError
from temporalio.testing import WorkflowEnvironment
-from temporalio.worker import UnsandboxedWorkflowRunner, Worker
+from temporalio.worker import Replayer, UnsandboxedWorkflowRunner, Worker
from products.experiments.backend.models.experiment import Experiment
+from products.experiments.backend.recalculation import _STALE_RECALC_THRESHOLD
from products.experiments.backend.temporal.models import ScheduledRecalculationStartResult
from products.experiments.backend.temporal.scheduled_recalculation_activities import (
discover_scheduled_recalculation_candidates,
)
from products.experiments.backend.temporal.scheduled_recalculation_logic import (
+ SKIP_DISPATCH_UNCERTAIN,
ScheduledRecalculationCandidate,
ScheduledRecalculationDiscovery,
)
from products.experiments.backend.temporal.scheduled_recalculation_workflow import (
+ RUN_START_WINDOW,
+ RUN_WAIT_TIMEOUT,
+ SCHEDULE_INTERVAL,
+ START_ACTIVITY_MAX_ATTEMPTS,
+ START_ACTIVITY_TIMEOUT,
ScheduledExperimentRecalculationWorkflow,
)
from products.feature_flags.backend.models.feature_flag import FeatureFlag
+WORKFLOW_MODULE = "products.experiments.backend.temporal.scheduled_recalculation_workflow"
+
def _candidates(*experiment_ids: int) -> list[ScheduledRecalculationCandidate]:
return [
@@ -35,47 +47,78 @@ def _candidates(*experiment_ids: int) -> list[ScheduledRecalculationCandidate]:
]
-async def _run(discover, exposures, start, hour: int = 2) -> dict:
+async def _no_wait(recalculation_id: str) -> None:
+ return None
+
+
+def _started(experiment_id: int) -> ScheduledRecalculationStartResult:
+ return ScheduledRecalculationStartResult(
+ experiment_id=experiment_id, started=True, recalculation_id=str(uuid.uuid4())
+ )
+
+
+async def _run(discover, exposures, start, hour: int = 2, wait=_no_wait, replay_legacy=False) -> dict:
@activity.defn(name="discover_scheduled_recalculation_candidates")
async def discover_activity() -> ScheduledRecalculationDiscovery:
return ScheduledRecalculationDiscovery(hour=hour, candidates=discover(hour))
- return await _execute(discover_activity, exposures, start)
+ return await _execute(discover_activity, exposures, start, wait, replay_legacy=replay_legacy)
-async def _execute(discover_activity, exposures, start) -> dict:
+async def _execute(discover_activity, exposures, start, wait=_no_wait, replay_legacy=False) -> dict:
@activity.defn(name="check_experiment_exposures")
async def exposures_activity(experiment_id: int, hour: int) -> bool:
return exposures(experiment_id)
@activity.defn(name="start_scheduled_recalculation")
- async def start_activity(experiment_id: int, hour: int) -> ScheduledRecalculationStartResult:
+ async def start_activity(
+ experiment_id: int, hour: int, start_before: datetime | None = None
+ ) -> ScheduledRecalculationStartResult:
return start(experiment_id)
+ @activity.defn(name="wait_for_scheduled_recalculation")
+ async def wait_activity(recalculation_id: str) -> None:
+ await wait(recalculation_id)
+
async with await WorkflowEnvironment.start_time_skipping() as env:
task_queue = f"test-{uuid.uuid4()}"
async with Worker(
env.client,
task_queue=task_queue,
workflows=[ScheduledExperimentRecalculationWorkflow],
- activities=[discover_activity, exposures_activity, start_activity],
+ activities=[discover_activity, exposures_activity, start_activity, wait_activity],
workflow_runner=UnsandboxedWorkflowRunner(),
):
- return await env.client.execute_workflow(
- ScheduledExperimentRecalculationWorkflow.run,
- id=str(uuid.uuid4()),
- task_queue=task_queue,
- )
+ with (
+ patch(f"{WORKFLOW_MODULE}.temporalio.workflow.patched", return_value=False)
+ if replay_legacy
+ else nullcontext(),
+ ):
+ handle = await env.client.start_workflow(
+ ScheduledExperimentRecalculationWorkflow.run,
+ id=str(uuid.uuid4()),
+ task_queue=task_queue,
+ execution_timeout=SCHEDULE_INTERVAL,
+ )
+ result = await handle.result()
+ history = await handle.fetch_history()
+ if replay_legacy:
+ assert not any(event.HasField("marker_recorded_event_attributes") for event in history.events)
+ await Replayer(
+ workflows=[ScheduledExperimentRecalculationWorkflow], workflow_runner=UnsandboxedWorkflowRunner()
+ ).replay_workflow(history)
+ return result
@pytest.mark.asyncio
async def test_no_candidates_returns_zeroes():
result = await _run(lambda hour: [], lambda eid: True, lambda eid: None)
- assert result == {"hour": 2, "candidates": 0, "started": 0, "skipped": 0}
+ assert result == {"hour": 2, "candidates": 0, "started": 0, "skipped": 0, "deferred": 0}
@pytest.mark.asyncio
-async def test_every_candidate_starts():
[email protected]("replay_legacy", [False, True])
+async def test_every_candidate_starts(replay_legacy):
started: list[int] = []
def start(experiment_id: int) -> ScheduledRecalculationStartResult:
@@ -84,13 +127,105 @@ async def test_every_candidate_starts():
experiment_id=experiment_id, started=True, recalculation_id=str(uuid.uuid4())
)
- result = await _run(lambda hour: _candidates(1, 2, 3), lambda eid: True, start)
+ result = await _run(lambda hour: _candidates(1, 2, 3), lambda eid: True, start, replay_legacy=replay_legacy)
assert result["candidates"] == 3
assert result["started"] == 3
assert result["skipped"] == 0
assert sorted(started) == [1, 2, 3]
+async def test_runs_in_flight_never_exceed_the_cap():
+ running = 0
+ peak = 0
+
+ def start(experiment_id: int) -> ScheduledRecalculationStartResult:
+ nonlocal running, peak
+ running += 1
+ peak = max(peak, running)
+ return _started(experiment_id)
+
+ async def wait(recalculation_id: str) -> None:
+ nonlocal running
+ await asyncio.sleep(0.01)
+ running -= 1
+
+ with patch(f"{WORKFLOW_MODULE}.MAX_CONCURRENT_EXPERIMENTS", 3):
+ result = await _run(lambda hour: _candidates(*range(1, 13)), lambda eid: True, start, wait=wait)
+
+ assert result["started"] == 12
+ assert peak <= 3
+
+
[email protected]("failure", [None, "wait", "start", "dispatch"])
+async def test_experiments_left_after_the_start_window_are_deferred(failure):
+ checked: list[int] = []
+ started: list[int] = []
+
+ def exposures(experiment_id: int) -> bool:
+ checked.append(experiment_id)
+ return True
+
+ def start(experiment_id: int) -> ScheduledRecalculationStartResult:
+ started.append(experiment_id)
+ if failure == "start":
+ raise ApplicationError("unknown dispatch outcome", non_retryable=True)
+ if failure == "dispatch":
+ return ScheduledRecalculationStartResult(
+ experiment_id=experiment_id,
+ started=False,
+ recalculation_id=str(uuid.uuid4()),
+ skip_reason=SKIP_DISPATCH_UNCERTAIN,
+ )
+ return _started(experiment_id)
+
+ now = datetime(2026, 10, 18, 6, 30, tzinfo=UTC)
+
+ async def wait(recalculation_id):
+ nonlocal now
+ if failure == "wait":
+ raise ApplicationError("cannot observe run", non_retryable=True)
+ now += RUN_START_WINDOW
+
+ with (
+ patch(f"{WORKFLOW_MODULE}.MAX_CONCURRENT_EXPERIMENTS", 1),
+ patch(f"{WORKFLOW_MODULE}.temporalio.workflow.now", lambda: now),
+ ):
+ result = await _run(lambda hour: _candidates(1, 2, 3), exposures, start, wait=wait)
+
+ assert checked == [1]
+ assert started == [1]
+ count = 0 if failure in ("start", "dispatch") else 1
+ assert result == {"hour": 2, "candidates": 3, "started": count, "skipped": 3 - count, "deferred": 2}
+
+
[email protected]("delayed_activity", ["exposures", "start"])
+async def test_retry_backoff_cannot_start_another_attempt_after_the_window(delayed_activity):
+ attempts = []
+
+ def delayed(experiment_id):
+ attempts.append(experiment_id)
+ raise ApplicationError("worker unavailable", next_retry_delay=RUN_START_WINDOW + timedelta(minutes=1))
+
+ result = await _run(
+ lambda hour: _candidates(1),
+ delayed if delayed_activity == "exposures" else lambda eid: True,
+ delayed if delayed_activity == "start" else _started,
+ )
+ assert result["started"] == 0
+ assert result["skipped"] == 1
+ assert attempts == [1]
+
+
+def test_last_run_ends_before_the_next_hourly_firing():
+ worst_case = RUN_START_WINDOW + START_ACTIVITY_TIMEOUT * START_ACTIVITY_MAX_ATTEMPTS + RUN_WAIT_TIMEOUT
+
+ assert RUN_WAIT_TIMEOUT >= _STALE_RECALC_THRESHOLD
+ assert worst_case < SCHEDULE_INTERVAL
+
+
@pytest.mark.asyncio
async def test_exposure_gate_stops_the_start_activity():
started: list[int] = []