:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(experiments): back off capacity retries exponentially from 10 seconds (#123216)
Juraj Majerik210c29f95462
diff --git a/products/experiments/backend/temporal/ARCHITECTURE.md b/products/experiments/backend/temporal/ARCHITECTURE.mdindex d01d02f53d7b..e73ed0952a2b 100644--- a/products/experiments/backend/temporal/ARCHITECTURE.md+++ b/products/experiments/backend/temporal/ARCHITECTURE.md@@ -261,9 +261,9 @@ The split is intentional: Grafana tells you about the worker process, PostHog teRetries live in Temporal's `RetryPolicy` on the calc activity, not in the workflow. The policy is exponential: `initial_interval=5s`, `backoff_coefficient=2.0`, `maximum_interval=60s`, `maximum_attempts=MAX_METRIC_ATTEMPTS` (currently 8). So a metric backs off 5s, 10s, 20s, 40s, 60s, 60s, 60s across its retries, then the eighth attempt is final. `asyncio.gather(..., return_exceptions=True)` lets healthy metrics finish while a failing one retries; a retrying metric never blocks the others, because each activity retries on its own worker slot rather than holding a shared queue slot.- A permanent failure fails fast rather than burning the retry budget. `NON_RETRYABLE_ERROR_TYPES` (`out_of_memory`, `byte_limit`, `validation_error`, `recalculation_window_too_long`) plus `ValueError` are raised non-retryable, so the metric is terminal on the first trip. A different bucket, the cluster at-capacity error (`ClickHouseAtCapacity`, from `TOO_MANY_SIMULTANEOUS_QUERIES` or `CANNOT_SCHEDULE_TASK`) and `ConcurrencyLimitExceeded`, is retried after a random delay between 2 and 5 minutes (`CONCURRENCY_LIMIT_RETRY_DELAY_MIN_SECONDS` and `CONCURRENCY_LIMIT_RETRY_DELAY_MAX_SECONDS`) via `ApplicationError(next_retry_delay=...)`, out-of-band from the exponential schedule, because that error means "come back later", not "this query is wrong". The random delay spreads out metrics that bounced in the same burst, so their retries do not arrive together and hit the same saturated cluster again.+ A permanent failure fails fast rather than burning the retry budget. `NON_RETRYABLE_ERROR_TYPES` (`out_of_memory`, `byte_limit`, `validation_error`, `recalculation_window_too_long`) plus `ValueError` are raised non-retryable, so the metric is terminal on the first trip. A different bucket, the cluster at-capacity error (`ClickHouseAtCapacity`, from `TOO_MANY_SIMULTANEOUS_QUERIES` or `CANNOT_SCHEDULE_TASK`) and `ConcurrencyLimitExceeded`, is retried after a random delay in a band that doubles with each attempt, 10 to 20 seconds first and 160 to 300 seconds at most (the `CAPACITY_RETRY_*` constants), via `ApplicationError(next_retry_delay=...)`, out-of-band from the RetryPolicy schedule, because that error means "come back later", not "this query is wrong". A refused query costs nothing, so the first retry comes fast and catches the common case of a pool that frees up within seconds; the later, longer waits keep a sustained full pool from being polled constantly. The random position in the band spreads out metrics that bounced in the same burst, so their retries do not arrive together and hit the same saturated cluster again.- Two error classes have a lower attempt cap than `MAX_METRIC_ATTEMPTS`, because their retries rarely succeed and each one costs ClickHouse capacity: capacity errors stop at `CAPACITY_MAX_METRIC_ATTEMPTS` (3) and the `timeout` class at `TIMEOUT_MAX_METRIC_ATTEMPTS` (2). The activity compares the attempt number with the cap of the error it just got. At the cap it persists the failure with `retriable=True` and raises non-retryable, so Temporal stops early without a change to the workflow's `RetryPolicy`. A user retry can recover the metric later.+ Two error classes have a lower attempt cap than `MAX_METRIC_ATTEMPTS`, because their retries rarely succeed within one run: capacity errors stop at `CAPACITY_MAX_METRIC_ATTEMPTS` (6, one attempt per band of the delay schedule) and the `timeout` class at `TIMEOUT_MAX_METRIC_ATTEMPTS` (2). The activity compares the attempt number with the cap of the error it just got. At the cap it persists the failure with `retriable=True` and raises non-retryable, so Temporal stops early without a change to the workflow's `RetryPolicy`. A user retry can recover the metric later.- **`is_final_attempt` is derived inside the activity.** Because Temporal now owns retries, the activity reads its own attempt count from `activity.info().attempt` and compares it against `MAX_METRIC_ATTEMPTS` (the same constant the workflow's `RetryPolicy` is built from, so both sides agree on which attempt is last). The activity persists a transient failure (FAILED row + `metric_errors`) only on the final attempt (or at the error class cap above), so a metric still being retried stays in its loading/dim state on the frontend instead of flashing an error that may yet resolve. Non-final transient state is tracked separately in the `metric_retries` column for the UI.diff --git a/products/experiments/backend/temporal/models.py b/products/experiments/backend/temporal/models.pyindex 9723faa894af..ea2e9a53e604 100644--- a/products/experiments/backend/temporal/models.py+++ b/products/experiments/backend/temporal/models.py@@ -37,14 +37,24 @@ MAX_METRIC_ATTEMPTS = 8# Below MAX_METRIC_ATTEMPTS for errors where a retry rarely succeeds and adds ClickHouse load. At the cap the# activity fails the metric non-retryable, so the RetryPolicy keeps MAX_METRIC_ATTEMPTS.-CAPACITY_MAX_METRIC_ATTEMPTS = 3+# A capacity rejection costs nothing, because ClickHouse refuses the query before it runs, so it gets more+# attempts than a timeout. 6 attempts are 5 waits, one per band of the CAPACITY_RETRY_* schedule below.+CAPACITY_MAX_METRIC_ATTEMPTS = 6# The timeout class also holds SOCKET_TIMEOUT, so one retry stays for network failures.TIMEOUT_MAX_METRIC_ATTEMPTS = 2-# Jittered delay after a capacity rejection (ClickHouseAtCapacity or ConcurrencyLimitExceeded), set via-# next_retry_delay. The per-org app limiter skips Temporal activities, so the delay lets the cluster drain.-CONCURRENCY_LIMIT_RETRY_DELAY_MIN_SECONDS = 120-CONCURRENCY_LIMIT_RETRY_DELAY_MAX_SECONDS = 300+# Delay before the retry of an attempt that a capacity rejection ended (ClickHouseAtCapacity or+# ConcurrencyLimitExceeded), set via next_retry_delay. The delay is a random value in a band, and the band+# doubles with each attempt: 10-20 s, 20-40 s, 40-80 s, 80-160 s, then 160-300 s. The short first wait fits the+# common case, a pool that frees up within seconds. The later, longer waits stop a sustained full pool from+# being polled constantly. The random position in the band spreads out metrics that bounced in the same burst,+# so their retries do not arrive together. The per-org app limiter skips Temporal activities, so this delay is+# what lets the cluster drain. The sum of the bands, at most 600 s, stays inside+# METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS.+CAPACITY_RETRY_FIRST_DELAY_MIN_SECONDS = 10+CAPACITY_RETRY_FIRST_DELAY_MAX_SECONDS = 20+CAPACITY_RETRY_BACKOFF_COEFFICIENT = 2.0+CAPACITY_RETRY_MAX_DELAY_SECONDS = 300RECALCULATION_RETRY_INITIAL_INTERVAL_SECONDS = 5RECALCULATION_RETRY_BACKOFF_COEFFICIENT = 2.0diff --git a/products/experiments/backend/temporal/recalculation_logic.py b/products/experiments/backend/temporal/recalculation_logic.pyindex 5267988b4454..74a397473fc3 100644--- a/products/experiments/backend/temporal/recalculation_logic.py+++ b/products/experiments/backend/temporal/recalculation_logic.py@@ -46,8 +46,10 @@ from products.experiments.backend.metric_resolution import build_metric, get_metfrom products.experiments.backend.models.experiment import Experiment, ExperimentMetricsRecalculationfrom products.experiments.backend.temporal.models import (CAPACITY_MAX_METRIC_ATTEMPTS,- CONCURRENCY_LIMIT_RETRY_DELAY_MAX_SECONDS,- CONCURRENCY_LIMIT_RETRY_DELAY_MIN_SECONDS,+ CAPACITY_RETRY_BACKOFF_COEFFICIENT,+ CAPACITY_RETRY_FIRST_DELAY_MAX_SECONDS,+ CAPACITY_RETRY_FIRST_DELAY_MIN_SECONDS,+ CAPACITY_RETRY_MAX_DELAY_SECONDS,MAX_METRIC_ATTEMPTS,METRIC_CALC_MAX_EXECUTION_TIME_SECONDS,NON_RETRYABLE_ERROR_TYPES,@@ -426,8 +428,13 @@ def _estimated_retry_delay_seconds(attempt: int) -> float:)-def _concurrency_limit_retry_delay_seconds() -> float:- return random.uniform(CONCURRENCY_LIMIT_RETRY_DELAY_MIN_SECONDS, CONCURRENCY_LIMIT_RETRY_DELAY_MAX_SECONDS)+def _capacity_retry_delay_seconds(attempt: int) -> float:+ """Random delay in the band for the attempt that just failed. The band doubles per attempt, up to the cap."""+ growth = CAPACITY_RETRY_BACKOFF_COEFFICIENT ** (max(attempt, 1) - 1)+ return random.uniform(+ min(CAPACITY_RETRY_FIRST_DELAY_MIN_SECONDS * growth, CAPACITY_RETRY_MAX_DELAY_SECONDS),+ min(CAPACITY_RETRY_FIRST_DELAY_MAX_SECONDS * growth, CAPACITY_RETRY_MAX_DELAY_SECONDS),+ )_RETRY_SAFE_MESSAGES: dict[str, str] = {@@ -818,7 +825,7 @@ def _calculate_experiment_metric_for_recalculation_sync(except (ConcurrencyLimitExceeded, ClickHouseAtCapacity) as e:message = str(e)[:_MAX_ERROR_MESSAGE_LENGTH]- retry_delay_seconds = _concurrency_limit_retry_delay_seconds()+ retry_delay_seconds = _capacity_retry_delay_seconds(attempt)capacity_attempts_exhausted = attempt >= CAPACITY_MAX_METRIC_ATTEMPTSis_last_attempt = is_final_attempt or capacity_attempts_exhaustedif is_last_attempt:diff --git a/products/experiments/backend/temporal/test_recalculation_activities.py b/products/experiments/backend/temporal/test_recalculation_activities.pyindex fd8d704c9ee6..464327178fdb 100644--- a/products/experiments/backend/temporal/test_recalculation_activities.py+++ b/products/experiments/backend/temporal/test_recalculation_activities.py@@ -11,6 +11,7 @@ import time_machinefrom posthog.test.base import BaseTestfrom unittest.mock import AsyncMock, MagicMock, patch+from django.test import SimpleTestCasefrom django.utils import timezonefrom clickhouse_driver.errors import ServerException@@ -47,6 +48,7 @@ from products.experiments.backend.temporal.recalculation_activities import calcufrom products.experiments.backend.temporal.recalculation_logic import (_calculate_experiment_metric_for_recalculation_sync,_cancel_metric_query_sync,+ _capacity_retry_delay_seconds,_discover_experiment_metrics_sync,_record_metric_out_of_time_failure_sync,_update_recalculation_progress_sync,@@ -966,7 +968,7 @@ class TestCalculateActivity(BaseTest):def test_backpressure_bounce_defers_with_retry_delay_and_no_capture(self, name: str, exc: Exception):# A query bouncing off the per-org ClickHouse limiter or the cluster's at-capacity guard is# backpressure, not a fault: it must re-raise as an ApplicationError carrying next_retry_delay- # (overriding the retry policy's 5s exponential schedule with the flat quota-wait delay), must never+ # (overriding the retry policy's 5s schedule with the capacity backoff band), must never# reach capture_exception (a saturated org would spam error tracking with expected bounces), and must# persist nothing before the final attempt.exp = self._experiment(flag_key=f"calc-quota-{name}", metrics=[_mean_metric("m1")])@@ -1384,6 +1386,25 @@ class TestCalculateActivity(BaseTest):)+class TestCapacityRetryDelay(SimpleTestCase):+ @parameterized.expand(+ [+ ("first_attempt", 1, 10, 20),+ ("third_attempt", 3, 40, 80),+ ("beyond_the_cap", 7, 300, 300),+ ]+ )+ def test_band_doubles_per_attempt_up_to_the_cap(self, name: str, attempt: int, low: float, high: float):+ # The first retry must stay fast and the band must grow, or a brief contention costs minutes again+ # and a sustained one is polled constantly. The cap keeps the sum of the waits inside the metric's+ # schedule_to_close budget.+ with patch(+ "products.experiments.backend.temporal.recalculation_logic.random.uniform", return_value=low+ ) as uniform:+ assert _capacity_retry_delay_seconds(attempt) == low+ uniform.assert_called_once_with(low, high)++@pytest.mark.django_db(transaction=True)class TestMissingRecalcRow:# All three activities go through _get_recalc_state at the top. A missing row is deterministic