Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(experiments): keep metric writes consistent with executed settings (#127790)

Anders35b37ebcf8ce
diff --git a/docs/internal/experiment-metric-recalculation.md b/docs/internal/experiment-metric-recalculation.md
index ceca4f326137..6ebd38d83301 100644
--- a/docs/internal/experiment-metric-recalculation.md
+++ b/docs/internal/experiment-metric-recalculation.md
@@ -26,3 +26,9 @@ Precomputation and the query runner do not cancel cooperatively, so the thread c
Result-write protection must remain independent of cancellation delivery.
See the [workflow architecture](../../products/experiments/backend/temporal/ARCHITECTURE.md) and [Temporal's cancellation contract](https://github.com/temporalio/sdk-python#heartbeating-and-cancellation).
+
+## Superseded timeseries backfills
+
+A request with the same metric and calculation key reuses the active backfill. A different key cancels the old request and starts a replacement. Each daily write locks the experiment and checks the request status in the same transaction, preventing a superseded worker from overwriting the replacement's days. Queries run outside the transaction.
+
+The request stores the metric's current definition and key, and ignores a definition or key in the request body. Until backfills consume a frozen calculation config, each day's runner must give the request's key, in the key version of that key, before executing. Settings drift cancels the request instead of storing a result under settings it did not use.
diff --git a/posthog/temporal/experiments/README.md b/posthog/temporal/experiments/README.md
index d265bd3f37e8..b8217720ff8d 100644
--- a/posthog/temporal/experiments/README.md
+++ b/posthog/temporal/experiments/README.md
@@ -40,6 +40,11 @@ The write looks the row up on `(experiment, metric_uuid, query_to)` and stores t
A row that already holds that window under another fingerprint, such as a recalculation result at the same window, is updated in place instead of failing on the unique constraint.
A failed write is the exception: it leaves a completed row under another fingerprint in place, so a transient failure cannot replace a recalculation result.
+Discovery queues each metric with its calculation key, and the activity reads the settings again when it builds the query runner.
+Before the query runs, the activity checks that the runner's settings still give the queued key, in the key version of that key.
+When they do not, the activity skips the metric: it stores no result or failure, sends no notification, and returns `skipped=True` without a retry.
+The workflow counts such metrics as skipped, not as failed, and the next scheduled run computes them under the new settings.
+
The significance notification compares a new daily result with the metric's newest earlier completed point that is compatible with it: a point under either key version of the metric's config, or whose display key is the version 1 key.
So neither a key version change nor a settings change that version 1 does not cover sends a repeated notification.
Without any earlier point, every significant variant counts as newly significant, as on the first point of a new experiment.
diff --git a/posthog/temporal/experiments/activities.py b/posthog/temporal/experiments/activities.py
index 196d45bc6ec2..be579a319512 100644
--- a/posthog/temporal/experiments/activities.py
+++ b/posthog/temporal/experiments/activities.py
@@ -1,5 +1,5 @@
from datetime import datetime, timedelta
-from typing import Any
+from typing import TYPE_CHECKING, Any
from zoneinfo import ZoneInfo
from django.db import close_old_connections
@@ -41,6 +41,7 @@ from products.experiments.backend.facade.timeseries import (
is_daily_timeseries_metric,
record_daily_metric_failure,
record_daily_metric_result,
+ saved_metric_link_for_calculation_key,
sync_timeseries_recalculation,
)
from products.experiments.backend.hogql_queries.base_query_utils import experiment_window_end
@@ -53,6 +54,9 @@ from products.experiments.backend.hogql_queries.utils import sanitize_non_finite
from products.experiments.backend.models.experiment import Experiment, ExperimentToSavedMetric
from products.experiments.stats.shared.statistics import StatisticError
+if TYPE_CHECKING:
+ from products.experiments.backend.metric_calculation.config import MetricCalculationConfig
+
logger = structlog.get_logger(__name__)
EXPERIMENT_RECALCULATION_MAX_AGE_DAYS = 60
@@ -69,6 +73,34 @@ def _build_metric_validated(
raise ValidationError(str(e)) from e
+def _queued_calculation_config(
+ query_runner: ExperimentQueryRunner, *, metric_uuid: str, definition: dict[str, Any], calculation_key: str
+) -> "MetricCalculationConfig | None":
+ """The config that the runner computes, in the key version of the queued key, or None when the runner's settings
+ no longer give that key.
+
+ The runner reads the experiment and the team settings when it is built, after the discovery computed the key. The
+ stored definition builds the config, as in the discovery: the runner's own config reads the definition through the
+ metric model, which adds null fields and so gives another key.
+ """
+ runtime_config = query_runner.calculation_config.settings.build_metric_config(
+ metric_id=metric_uuid, role="primary", definition=definition
+ )
+ key_version = runtime_config.key_version_of(calculation_key)
+ return runtime_config.with_key_version(key_version) if key_version is not None else None
+
+
+def _log_settings_drift(experiment_id: int, metric_uuid: str) -> None:
+ logger.info(
+ "Skipping experiment metric calculation, because the settings changed after discovery",
+ experiment_id=experiment_id,
+ metric_uuid=metric_uuid,
+ )
+
+
+_SETTINGS_DRIFT_MESSAGE = "The calculation settings changed after the run was scheduled"
+
+
@database_sync_to_async
def _get_experiment_regular_metrics_for_hour_sync(hour: int) -> list[ExperimentRegularMetricInput]:
close_old_connections()
@@ -177,6 +209,7 @@ def _calculate_experiment_regular_metric_sync(
now_utc = datetime.now(ZoneInfo("UTC"))
query_to_utc = experiment_window_end(experiment, now_utc)
+ display_key: str | None = None
try:
# Inside the try so a malformed stored metric dict follows the same
# failure path as a query error instead of escaping the activity.
@@ -202,6 +235,20 @@ def _calculate_experiment_regular_metric_sync(
# .run() writes to the response cache. The "warming/*" trigger tells
# run() this is a scheduled job, not a user query, so it skips logging
# the events as "used by this team."
+ calculation_config = _queued_calculation_config(
+ query_runner, metric_uuid=metric_uuid, definition=metric_dict, calculation_key=fingerprint
+ )
+ if calculation_config is None:
+ _log_settings_drift(experiment_id, metric_uuid)
+ return ExperimentRegularMetricResult(
+ experiment_id=experiment_id,
+ metric_uuid=metric_uuid,
+ fingerprint=fingerprint,
+ success=False,
+ error_message=_SETTINGS_DRIFT_MESSAGE,
+ skipped=True,
+ )
+ display_key = calculation_config.legacy_key()
tag_queries(trigger="warming/experiment_timeseries")
result = query_runner.run(execution_mode=ExecutionMode.CALCULATE_BLOCKING_ALWAYS)
result_dict = sanitize_non_finite(result.model_dump(mode="json"))
@@ -213,9 +260,12 @@ def _calculate_experiment_regular_metric_sync(
window=query_to_utc,
query_from=query_from_utc,
result=result_dict,
+ display_key=display_key,
)
- check_significance_transition(experiment, metric_uuid, fingerprint, result_dict, query_to_utc)
+ check_significance_transition(
+ experiment, metric_uuid, fingerprint, result_dict, query_to_utc, display_key=display_key
+ )
logger.info(
"Successfully calculated experiment metric",
@@ -238,6 +288,7 @@ def _calculate_experiment_regular_metric_sync(
window=query_to_utc,
query_from=query_from_utc,
error_message=str(e),
+ display_key=display_key,
)
logger.warning(
@@ -279,6 +330,7 @@ def _calculate_experiment_regular_metric_sync(
window=query_to_utc,
query_from=query_from_utc,
error_message=str(e),
+ display_key=display_key,
)
if is_permanent:
@@ -398,15 +450,12 @@ def _calculate_experiment_saved_metric_sync(
error_message=f"Experiment {experiment_id} not found or deleted",
)
- saved_metric = None
- saved_metric_metadata: dict = {}
- for exp_to_sm in experiment.experimenttosavedmetric_set.select_related("saved_metric").all():
- if exp_to_sm.saved_metric.query.get("uuid") == metric_uuid:
- saved_metric = exp_to_sm.saved_metric
- saved_metric_metadata = exp_to_sm.metadata or {}
- break
-
- if not saved_metric:
+ links = [
+ link
+ for link in experiment.experimenttosavedmetric_set.select_related("saved_metric").order_by("id")
+ if link.saved_metric.query.get("uuid") == metric_uuid
+ ]
+ if not links:
return ExperimentSavedMetricResult(
experiment_id=experiment_id,
metric_uuid=metric_uuid,
@@ -415,13 +464,32 @@ def _calculate_experiment_saved_metric_sync(
error_message=f"Saved metric {metric_uuid} not found for experiment {experiment_id}",
)
+ # Each link that shares this uuid was queued under its own key.
+ if len(links) == 1:
+ link: ExperimentToSavedMetric | None = links[0]
+ else:
+ link_id = saved_metric_link_for_calculation_key(
+ experiment_id, team_id=experiment.team_id, metric_uuid=metric_uuid, calculation_key=fingerprint
+ )
+ link = next((candidate for candidate in links if candidate.id == link_id), None)
+ if link is None:
+ _log_settings_drift(experiment_id, metric_uuid)
+ return ExperimentSavedMetricResult(
+ experiment_id=experiment_id,
+ metric_uuid=metric_uuid,
+ fingerprint=fingerprint,
+ success=False,
+ error_message=_SETTINGS_DRIFT_MESSAGE,
+ skipped=True,
+ )
+
# The frontend receives saved metrics with two extra fields injected before
# they get posted back to /query: the link overrides (via resolveSharedMetric
# in experiments/utils.ts, which mirrors apply_saved_metric_overrides) and
# a fingerprint (added by the experiment API serializer). The activity must
# apply both or the response cache key diverges from /query's.
query = {
- **apply_saved_metric_overrides(saved_metric.query, saved_metric_metadata),
+ **apply_saved_metric_overrides(link.saved_metric.query, link.metadata or {}),
"fingerprint": fingerprint,
}
metric_type = query.get("metric_type")
@@ -447,6 +515,7 @@ def _calculate_experiment_saved_metric_sync(
now_utc = datetime.now(ZoneInfo("UTC"))
query_to_utc = experiment_window_end(experiment, now_utc)
+ display_key: str | None = None
try:
# Inside the try so a malformed stored metric dict follows the same
# failure path as a query error instead of escaping the activity.
@@ -472,6 +541,20 @@ def _calculate_experiment_saved_metric_sync(
# .run() writes to the response cache. The "warming/*" trigger tells
# run() this is a scheduled job, not a user query, so it skips logging
# the events as "used by this team."
+ calculation_config = _queued_calculation_config(
+ query_runner, metric_uuid=metric_uuid, definition=query, calculation_key=fingerprint
+ )
+ if calculation_config is None:
+ _log_settings_drift(experiment_id, metric_uuid)
+ return ExperimentSavedMetricResult(
+ experiment_id=experiment_id,
+ metric_uuid=metric_uuid,
+ fingerprint=fingerprint,
+ success=False,
+ error_message=_SETTINGS_DRIFT_MESSAGE,
+ skipped=True,
+ )
+ display_key = calculation_config.legacy_key()
tag_queries(trigger="warming/experiment_timeseries")
result = query_runner.run(execution_mode=ExecutionMode.CALCULATE_BLOCKING_ALWAYS)
result_dict = sanitize_non_finite(result.model_dump(mode="json"))
@@ -483,9 +566,12 @@ def _calculate_experiment_saved_metric_sync(
window=query_to_utc,
query_from=query_from_utc,
result=result_dict,
+ display_key=display_key,
)
- check_significance_transition(experiment, metric_uuid, fingerprint, result_dict, query_to_utc)
+ check_significance_transition(
+ experiment, metric_uuid, fingerprint, result_dict, query_to_utc, display_key=display_key
+ )
logger.info(
"Successfully calculated experiment saved metric",
@@ -508,6 +594,7 @@ def _calculate_experiment_saved_metric_sync(
window=query_to_utc,
query_from=query_from_utc,
error_message=str(e),
+ display_key=display_key,
)
logger.warning(
@@ -549,6 +636,7 @@ def _calculate_experiment_saved_metric_sync(
window=query_to_utc,
query_from=query_from_utc,
error_message=str(e),
+ display_key=display_key,
)
if is_permanent:
diff --git a/posthog/temporal/experiments/models.py b/posthog/temporal/experiments/models.py
index 2d2ed4872b4d..4dbe44cedd9f 100644
--- a/posthog/temporal/experiments/models.py
+++ b/posthog/temporal/experiments/models.py
@@ -26,7 +26,7 @@ class ExperimentRegularMetricInput:
team_id: int | None = None
[email protected](frozen=False)
class ExperimentRegularMetricResult:
"""Result from calculating a single experiment-metric."""
@@ -35,6 +35,7 @@ class ExperimentRegularMetricResult:
fingerprint: str
success: bool
error_message: str | None = None
+ skipped: bool = False
@dataclasses.dataclass
@@ -54,7 +55,7 @@ class ExperimentSavedMetricInput:
team_id: int | None = None
[email protected](frozen=False)
class ExperimentSavedMetricResult:
"""Result from calculating a single experiment-saved metric."""
@@ -63,6 +64,7 @@ class ExperimentSavedMetricResult:
fingerprint: str
success: bool
error_message: str | None = None
+ skipped: bool = False
@dataclasses.dataclass
diff --git a/posthog/temporal/experiments/test_cache_warming.py b/posthog/temporal/experiments/test_cache_warming.py
index a79f3dfaa499..7e623f3323f1 100644
--- a/posthog/temporal/experiments/test_cache_warming.py
+++ b/posthog/temporal/experiments/test_cache_warming.py
@@ -9,7 +9,6 @@ from django.core.cache import cache
from django.test import override_settings
from parameterized import parameterized
-from rest_framework.exceptions import ValidationError
from posthog.schema import CachedExperimentQueryResponse, EventsNode, ExperimentMeanMetric, ExperimentQuery
@@ -21,7 +20,6 @@ from posthog.temporal.experiments.activities import (
_calculate_experiment_regular_metric_sync,
_calculate_experiment_saved_metric_sync,
)
-from posthog.temporal.experiments.models import TIMESERIES_METRIC_MAX_ATTEMPTS
from products.access_control.backend.facade.contracts import PropertyAccessLevel
from products.access_control.backend.models.property_access_control import PropertyAccessControl
@@ -34,101 +32,9 @@ from products.experiments.backend.models.experiment import (
ExperimentToSavedMetric,
)
-# A recalculation of the experiment can hold the daily window under its own fingerprint. The daily write must take
-# that row over instead of failing on the (experiment, metric_uuid, query_to) unique constraint.
-_WINDOW_HOLDERS = [("free_window", None), ("window_held_by_another_fingerprint", "recalculation-fingerprint")]
-
@override_settings(IN_UNIT_TESTING=True)
class TestTemporalRecalcWarmsResponseCache(ExperimentQueryRunnerBaseTest):
- def _hold_window(self, experiment_id: int, metric_uuid: str, query_to: datetime, fingerprint: str | None) -> None:
- if fingerprint is not None:
- ExperimentMetricResult.objects.create(
- experiment_id=experiment_id,
- metric_uuid=metric_uuid,
- fingerprint=fingerprint,
- query_from=datetime.fromisoformat("2020-01-01T00:00:00+00:00"),
- query_to=query_to,
- status=ExperimentMetricResult.Status.COMPLETED,
- result={"stale": True},
- )
-
- @parameterized.expand(_WINDOW_HOLDERS)
- @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
- def test_regular_metric_activity_pins_runner_to_stored_query_to(self, _name: str, holder: str | None):
- feature_flag = self.create_feature_flag()
- experiment = self.create_experiment(
- feature_flag=feature_flag,
- start_date=datetime(2020, 1, 1, 0, 0, 0),
- end_date=datetime(2020, 1, 5, 0, 0, 0),
- )
- metric = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase"))
- metric_dict = metric.model_dump(mode="json")
- experiment.metrics = [metric_dict]
- experiment.save()
- expected_query_to = datetime.fromisoformat("2020-01-05T00:00:00+00:00")
- self._hold_window(experiment.id, metric_dict["uuid"], expected_query_to, holder)
-
- with (
- patch("posthog.temporal.experiments.activities.close_old_connections"),
- patch("posthog.temporal.experiments.activities.ExperimentQueryRunner") as mock_runner_class,
- ):
- mock_runner_class.return_value.run.return_value.model_dump.return_value = {"variant_results": []}
-
- activity_result = _calculate_experiment_regular_metric_sync.func( # type: ignore[attr-defined]
- experiment.id, metric_dict["uuid"], "fingerprint"
- )
-
- self.assertTrue(activity_result.success, msg=activity_result.error_message)
- mock_runner_class.assert_called_once()
- assert mock_runner_class.call_args.kwargs["as_of"] == expected_query_to
- result_row = ExperimentMetricResult.objects.get(experiment=experiment, metric_uuid=metric_dict["uuid"])
- assert result_row.query_to == expected_query_to
- assert (result_row.fingerprint, result_row.result) == ("fingerprint", {"variant_results": []})
-
- @parameterized.expand(_WINDOW_HOLDERS)
- @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
- def test_saved_metric_activity_pins_runner_to_stored_query_to(self, _name: str, holder: str | None):
- feature_flag = self.create_feature_flag()
- experiment = self.create_experiment(
- feature_flag=feature_flag,
- start_date=datetime(2020, 1, 1, 0, 0, 0),
- end_date=datetime(2020, 1, 5, 0, 0, 0),
- )
- metric = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase"))
- metric_dict = metric.model_dump(mode="json")
- expected_query_to = datetime.fromisoformat("2020-01-05T00:00:00+00:00")
-
- saved_metric = ExperimentSavedMetric.objects.create(
- name="test saved metric",
- team=self.team,
- query=metric_dict,
- created_by=self.user,
- )
- ExperimentToSavedMetric.objects.create(
- experiment=experiment,
- saved_metric=saved_metric,
- metadata={"type": "primary"},
- )
- self._hold_window(experiment.id, metric_dict["uuid"], expected_query_to, holder)
-
- with (
- patch("posthog.temporal.experiments.activities.close_old_connections"),
- patch("posthog.temporal.experiments.activities.ExperimentQueryRunner") as mock_runner_class,
- ):
- mock_runner_class.return_value.run.return_value.model_dump.return_value = {"variant_results": []}
-
- activity_result = _calculate_experiment_saved_metric_sync.func( # type: ignore[attr-defined]
- experiment.id, metric_dict["uuid"], "fingerprint"
- )
-
- self.assertTrue(activity_result.success, msg=activity_result.error_message)
- mock_runner_class.assert_called_once()
- assert mock_runner_class.call_args.kwargs["as_of"] == expected_query_to
- result_row = ExperimentMetricResult.objects.get(experiment=experiment, metric_uuid=metric_dict["uuid"])
- assert result_row.query_to == expected_query_to
- assert (result_row.fingerprint, result_row.result) == ("fingerprint", {"variant_results": []})
-
@parameterized.expand([("creator_is_member", True), ("creator_left_organization", False)])
@time_machine.travel("2020-01-10T12:00:00Z", tick=False)
def test_temporal_activity_warms_query_cache(self, _name: str, creator_is_member: bool):
@@ -305,74 +211,6 @@ class TestTemporalRecalcWarmsResponseCache(ExperimentQueryRunnerBaseTest):
assert isinstance(warm_response, CachedExperimentQueryResponse)
self.assertTrue(warm_response.is_cached)
- @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
- def test_regular_metric_activity_returns_permanent_failure_on_validation_error(self):
- feature_flag = self.create_feature_flag()
- experiment = self.create_experiment(
- feature_flag=feature_flag,
- start_date=datetime(2020, 1, 1, 0, 0, 0),
- end_date=datetime(2020, 1, 5, 0, 0, 0),
- )
- metric = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase"))
- metric_dict = metric.model_dump(mode="json")
- experiment.metrics = [metric_dict]
- experiment.save()
-
- with (
- patch("posthog.temporal.experiments.activities.close_old_connections"),
- patch("posthog.temporal.experiments.activities.ExperimentQueryRunner") as mock_runner_class,
- patch("posthog.temporal.experiments.activities.capture_experiment_metric_error_event") as mock_event,
- ):
- mock_runner_class.return_value.run.side_effect = ValidationError("compound expressions not supported")
-
- activity_result = _calculate_experiment_regular_metric_sync.func( # type: ignore[attr-defined]
- experiment.id, metric_dict["uuid"], "fingerprint"
- )
-
- self.assertFalse(activity_result.success)
- result_row = ExperimentMetricResult.objects.get(experiment=experiment, metric_uuid=metric_dict["uuid"])
- assert result_row.status == ExperimentMetricResult.Status.FAILED
- mock_event.assert_called_once()
-
- @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
- def test_saved_metric_activity_returns_permanent_failure_on_validation_error(self):
- feature_flag = self.create_feature_flag()
- experiment = self.create_experiment(
- feature_flag=feature_flag,
- start_date=datetime(2020, 1, 1, 0, 0, 0),
- end_date=datetime(2020, 1, 5, 0, 0, 0),
- )
- metric = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase"))
- metric_dict = metric.model_dump(mode="json")
-
- saved_metric = ExperimentSavedMetric.objects.create(
- name="test saved metric",
- team=self.team,
- query=metric_dict,
- created_by=self.user,
- )
- ExperimentToSavedMetric.objects.create(
- experiment=experiment,
- saved_metric=saved_metric,
- metadata={"type": "primary"},
- )
-
- with (
- patch("posthog.temporal.experiments.activities.close_old_connections"),
- patch("posthog.temporal.experiments.activities.ExperimentQueryRunner") as mock_runner_class,
- patch("posthog.temporal.experiments.activities.capture_experiment_metric_error_event") as mock_event,
- ):
- mock_runner_class.return_value.run.side_effect = ValidationError("compound expressions not supported")
-
- activity_result = _calculate_experiment_saved_metric_sync.func( # type: ignore[attr-defined]
- experiment.id, metric_dict["uuid"], "fingerprint"
- )
-
- self.assertFalse(activity_result.success)
- result_row = ExperimentMetricResult.objects.get(experiment=experiment, metric_uuid=metric_dict["uuid"])
- assert result_row.status == ExperimentMetricResult.Status.FAILED
- mock_event.assert_called_once()
-
@time_machine.travel("2020-01-10T12:00:00Z", tick=False)
def test_regular_metric_activity_returns_permanent_failure_on_malformed_config(self):
feature_flag = self.create_feature_flag()
@@ -432,77 +270,3 @@ class TestTemporalRecalcWarmsResponseCache(ExperimentQueryRunnerBaseTest):
result_row = ExperimentMetricResult.objects.get(experiment=experiment, metric_uuid=bad_metric["uuid"])
assert result_row.status == ExperimentMetricResult.Status.FAILED
mock_event.assert_called_once()
-
- @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
- def test_regular_metric_activity_still_raises_transient_errors(self):
- feature_flag = self.create_feature_flag()
- experiment = self.create_experiment(
- feature_flag=feature_flag,
- start_date=datetime(2020, 1, 1, 0, 0, 0),
- end_date=datetime(2020, 1, 5, 0, 0, 0),
- )
- metric = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase"))
- metric_dict = metric.model_dump(mode="json")
- experiment.metrics = [metric_dict]
- experiment.save()
-
- with (
- patch("posthog.temporal.experiments.activities.close_old_connections"),
- patch("posthog.temporal.experiments.activities.ExperimentQueryRunner") as mock_runner_class,
- patch("posthog.temporal.experiments.activities.capture_experiment_metric_error_event") as mock_event,
- ):
- mock_runner_class.return_value.run.side_effect = Exception("transient query failure")
-
- with self.assertRaises(Exception):
- _calculate_experiment_regular_metric_sync.func( # type: ignore[attr-defined]
- experiment.id, metric_dict["uuid"], "fingerprint", attempt=1
- )
- mock_event.assert_not_called()
-
- with self.assertRaises(Exception):
- _calculate_experiment_regular_metric_sync.func( # type: ignore[attr-defined]
- experiment.id, metric_dict["uuid"], "fingerprint", attempt=TIMESERIES_METRIC_MAX_ATTEMPTS
- )
- mock_event.assert_called_once()
-
- @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
- def test_saved_metric_activity_still_raises_transient_errors(self):
- feature_flag = self.create_feature_flag()
- experiment = self.create_experiment(
- feature_flag=feature_flag,
- start_date=datetime(2020, 1, 1, 0, 0, 0),
- end_date=datetime(2020, 1, 5, 0, 0, 0),
- )
- metric = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase"))
- metric_dict = metric.model_dump(mode="json")
-
- saved_metric = ExperimentSavedMetric.objects.create(
- name="test saved metric",
- team=self.team,
- query=metric_dict,
- created_by=self.user,
- )
- ExperimentToSavedMetric.objects.create(
- experiment=experiment,
- saved_metric=saved_metric,
- metadata={"type": "primary"},
- )
-
- with (
- patch("posthog.temporal.experiments.activities.close_old_connections"),
- patch("posthog.temporal.experiments.activities.ExperimentQueryRunner") as mock_runner_class,
- patch("posthog.temporal.experiments.activities.capture_experiment_metric_error_event") as mock_event,
- ):
- mock_runner_class.return_value.run.side_effect = Exception("transient query failure")
-
- with self.assertRaises(Exception):
- _calculate_experiment_saved_metric_sync.func( # type: ignore[attr-defined]
- experiment.id, metric_dict["uuid"], "fingerprint", attempt=1
- )
- mock_event.assert_not_called()
-
- with self.assertRaises(Exception):
- _calculate_experiment_saved_metric_sync.func( # type: ignore[attr-defined]
- experiment.id, metric_dict["uuid"], "fingerprint", attempt=TIMESERIES_METRIC_MAX_ATTEMPTS
- )
- mock_event.assert_called_once()
diff --git a/posthog/temporal/experiments/test_per_experiment_publish.py b/posthog/temporal/experiments/test_per_experiment_publish.py
index a01791549719..484fe000c5e0 100644
--- a/posthog/temporal/experiments/test_per_experiment_publish.py
+++ b/posthog/temporal/experiments/test_per_experiment_publish.py
@@ -1,7 +1,8 @@
import pytest
from unittest.mock import patch
-from posthog.temporal.experiments.workflows import _record_publish_outcome
+from posthog.temporal.experiments.models import ExperimentRegularMetricResult
+from posthog.temporal.experiments.workflows import _MetricOutcomes, _record_publish_outcome
@pytest.mark.parametrize(
@@ -28,3 +29,16 @@ def test_publish_outcome_counter_separates_published_from_missing(
{"workflow_type": "experiment-saved-metrics-workflow", "status": expected_status}
)
with_attributes.return_value.create_counter.return_value.add.assert_called_once_with(1)
+
+
+def test_metric_outcomes_count_skipped_metrics_apart_from_failures() -> None:
+ def result(*, success: bool, skipped: bool = False) -> ExperimentRegularMetricResult:
+ return ExperimentRegularMetricResult(
+ experiment_id=1, metric_uuid="m", fingerprint="k", success=success, skipped=skipped
+ )
+
+ outcomes = _MetricOutcomes.count(
+ [result(success=True), result(success=False), result(success=False, skipped=True), RuntimeError("lost")]
+ )
+
+ assert outcomes == _MetricOutcomes(succeeded=1, failed=2, skipped=1)
diff --git a/posthog/temporal/experiments/workflows.py b/posthog/temporal/experiments/workflows.py
index f2ed9713972f..ef825777fd56 100644
--- a/posthog/temporal/experiments/workflows.py
+++ b/posthog/temporal/experiments/workflows.py
@@ -7,6 +7,7 @@ import temporalio.workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ActivityError
+from posthog.dataclasses import frozen
from posthog.temporal.common.base import PostHogWorkflow
with temporalio.workflow.unsafe.imports_passed_through():
@@ -36,6 +37,22 @@ type MetricInput = ExperimentRegularMetricInput | ExperimentSavedMetricInput
type MetricResult = ExperimentRegularMetricResult | ExperimentSavedMetricResult
+@frozen
+class _MetricOutcomes:
+ succeeded: int
+ failed: int
+ # Metrics whose settings changed after discovery. The next scheduled run computes them, so they are no failure.
+ skipped: int
+
+ @classmethod
+ def count(cls, results: Sequence[MetricResult | BaseException]) -> "_MetricOutcomes":
+ succeeded = sum(1 for result in results if not isinstance(result, BaseException) and result.success)
+ skipped = sum(
+ 1 for result in results if not isinstance(result, BaseException) and not result.success and result.skipped
+ )
+ return cls(succeeded=succeeded, failed=len(results) - succeeded - skipped, skipped=skipped)
+
+
def _record_publish_outcome(succeeded: int, recalculations_synced: int) -> None:
"""Per-run publish counter for the executions that still publish.
@@ -200,6 +217,7 @@ class ExperimentRegularMetricsWorkflow(PostHogWorkflow):
"total": 0,
"succeeded": 0,
"failed": 0,
+ "skipped": 0,
}
# Step 2: Calculate every metric, bounded by one shared concurrency limit.
@@ -227,25 +245,17 @@ class ExperimentRegularMetricsWorkflow(PostHogWorkflow):
)
# Step 3: Summarize
- succeeded = 0
- failed = 0
-
- for result in results:
- if isinstance(result, BaseException):
- failed += 1
- elif result.success:
- succeeded += 1
- else:
- failed += 1
+ outcomes = _MetricOutcomes.count(results)
if published is not None:
- _record_publish_outcome(succeeded, published)
+ _record_publish_outcome(outcomes.succeeded, published)
return {
"hour": inputs.hour,
"total": len(experiment_metrics),
- "succeeded": succeeded,
- "failed": failed,
+ "succeeded": outcomes.succeeded,
+ "failed": outcomes.failed,
+ "skipped": outcomes.skipped,
}
@@ -280,6 +290,7 @@ class ExperimentSavedMetricsWorkflow(PostHogWorkflow):
"total": 0,
"succeeded": 0,
"failed": 0,
+ "skipped": 0,
}
# Step 2: Calculate every metric, bounded by one shared concurrency limit.
@@ -307,25 +318,17 @@ class ExperimentSavedMetricsWorkflow(PostHogWorkflow):
)
# Step 3: Summarize
- succeeded = 0
- failed = 0
-
- for result in results:
- if isinstance(result, BaseException):
- failed += 1
- elif result.success:
- succeeded += 1
- else:
- failed += 1
+ outcomes = _MetricOutcomes.count(results)
if published is not None:
- _record_publish_outcome(succeeded, published)
+ _record_publish_outcome(outcomes.succeeded, published)
return {
"hour": inputs.hour,
"total": len(experiment_metrics),
- "succeeded": succeeded,
- "failed": failed,
+ "succeeded": outcomes.succeeded,
+ "failed": outcomes.failed,
+ "skipped": outcomes.skipped,
}
diff --git a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt b/posthog/test/repo_invariants/dataclass_frozen_baseline.txt
index fd60bbb8e484..35575354ffff 100644
--- a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt
+++ b/posthog/test/repo_invariants/dataclass_frozen_baseline.txt
@@ -623,7 +623,7 @@
6 products/marketing_analytics/backend/services/mapping_suggester.py
6 products/tasks/backend/temporal/task_management/workflow.py
7 posthog/session_recordings/synthetic_playlists.py
-7 posthog/temporal/experiments/models.py
+5 posthog/temporal/experiments/models.py
7 products/batch_exports/backend/temporal/monitoring.py
7 products/growth/backend/sdk_health.py
7 products/marketing_analytics/backend/services/types.py
diff --git a/products/experiments/backend/experiment_service.py b/products/experiments/backend/experiment_service.py
index 15b22f4c8d56..3028bd148133 100644
--- a/products/experiments/backend/experiment_service.py
+++ b/products/experiments/backend/experiment_service.py
@@ -4661,16 +4661,7 @@ class ExperimentService:
overall_status = "partial"
active_recalculation = (
- ExperimentTimeseriesRecalculation.objects.filter(
- experiment=experiment,
- fingerprint=calculation_config.calculation_key(),
- status__in=[
- ExperimentTimeseriesRecalculation.Status.PENDING,
- ExperimentTimeseriesRecalculation.Status.IN_PROGRESS,
- ],
- ).first()
- if calculation_config is not None
- else None
+ self._active_timeseries_backfill(experiment, metric_uuid) if calculation_config is not None else None
)
response = {
@@ -4692,51 +4683,66 @@ class ExperimentService:
).format()
return response
- def request_timeseries_recalculation(
- self,
- experiment: Experiment,
- *,
- metric: dict,
- fingerprint: str,
- ) -> dict:
- """Create an idempotent recalculation request for experiment timeseries data."""
+ def request_timeseries_recalculation(self, experiment: Experiment, *, metric_uuid: str) -> dict:
+ """Create an idempotent recalculation request for experiment timeseries data.
+
+ The backfill computes the metric's current definition and files the days under its calculation key, both
+ from the experiment, so the chart finds them.
+ """
if not experiment.is_launched:
raise ValidationError("Cannot recalculate timeseries for experiment that hasn't started")
+ calculation_config = get_metric_calculation_config(experiment, metric_uuid)
+ if calculation_config is None:
+ raise ValidationError(
+ "This metric is not on the experiment, or it uses an older metric format that has no daily results. "
+ "Pick a metric from the experiment's metrics."
+ )
+ metric = calculation_config.definition
+ fingerprint = calculation_config.calculation_key()
+
# The backfill runs this metric without a user, so the caller's warehouse access is checked here.
validate_warehouse_metrics([metric], team=self.team, user=self.user)
- existing_recalculation = ExperimentTimeseriesRecalculation.objects.filter(
- experiment=experiment,
- fingerprint=fingerprint,
- status__in=[
- ExperimentTimeseriesRecalculation.Status.PENDING,
- ExperimentTimeseriesRecalculation.Status.IN_PROGRESS,
- ],
- ).first()
-
- if existing_recalculation:
- return {
- "id": existing_recalculation.id,
- "experiment_id": experiment.id,
- "metric_uuid": existing_recalculation.metric.get("uuid"),
- "fingerprint": fingerprint,
- "status": existing_recalculation.status,
- "created_at": existing_recalculation.created_at.isoformat(),
- "is_existing": True,
- }
+ with transaction.atomic():
+ # Serialize supersession with each backfill's status check and result write.
+ Experiment.objects.select_for_update(no_key=True).filter(id=experiment.id, team=self.team).exists()
+ # A failed backfill can still start again on a Temporal retry. The active or new backfill replaces it, so
+ # a retry must not run beside it or recompute the days again.
+ ExperimentTimeseriesRecalculation.objects.filter(
+ experiment=experiment,
+ metric__uuid=calculation_config.metric_id,
+ status=ExperimentTimeseriesRecalculation.Status.FAILED,
+ ).update(status=ExperimentTimeseriesRecalculation.Status.CANCELED)
+ existing_recalculation = self._active_timeseries_backfill(experiment, calculation_config.metric_id)
+
+ if existing_recalculation and existing_recalculation.fingerprint != fingerprint:
+ existing_recalculation.status = ExperimentTimeseriesRecalculation.Status.CANCELED
+ existing_recalculation.save(update_fields=["status"])
+ existing_recalculation = None
+
+ if existing_recalculation:
+ return {
+ "id": existing_recalculation.id,
+ "experiment_id": experiment.id,
+ "metric_uuid": existing_recalculation.metric.get("uuid"),
+ "fingerprint": existing_recalculation.fingerprint,
+ "status": existing_recalculation.status,
+ "created_at": existing_recalculation.created_at.isoformat(),
+ "is_existing": True,
+ }
- metric_uuid = metric.get("uuid")
- if metric_uuid:
- MetricResultStore(experiment_id=experiment.id).delete_daily_points(metric_uuid, fingerprint)
+ MetricResultStore(experiment_id=experiment.id).delete_daily_points(
+ calculation_config.metric_id, fingerprint
+ )
- recalculation_request = ExperimentTimeseriesRecalculation.objects.create(
- team=experiment.team,
- experiment=experiment,
- metric=metric,
- fingerprint=fingerprint,
- status=ExperimentTimeseriesRecalculation.Status.PENDING,
- )
+ recalculation_request = ExperimentTimeseriesRecalculation.objects.create(
+ team=experiment.team,
+ experiment=experiment,
+ metric=metric,
+ fingerprint=fingerprint,
+ status=ExperimentTimeseriesRecalculation.Status.PENDING,
+ )
return {
"id": recalculation_request.id,
@@ -4748,6 +4754,28 @@ class ExperimentService:
"is_existing": False,
}
+ @staticmethod
+ def _active_timeseries_backfill(
+ experiment: Experiment, metric_uuid: str
+ ) -> ExperimentTimeseriesRecalculation | None:
+ """The pending or running backfill of the metric with this uuid, under any key.
+
+ A different key must be canceled before its replacement starts. Workers check this status under the
+ experiment lock before writing, so a superseded query cannot overwrite the replacement's results.
+ """
+ return (
+ ExperimentTimeseriesRecalculation.objects.filter(
+ experiment=experiment,
+ metric__uuid=metric_uuid,
+ status__in=[
+ ExperimentTimeseriesRecalculation.Status.PENDING,
+ ExperimentTimeseriesRecalculation.Status.IN_PROGRESS,
+ ],
+ )
+ .order_by("-created_at")
+ .first()
+ )
+
# ------------------------------------------------------------------
# Velocity stats
# ------------------------------------------------------------------
diff --git a/products/experiments/backend/facade/timeseries.py b/products/experiments/backend/facade/timeseries.py
index d1fda80fb66d..28a154f79ccc 100644
--- a/products/experiments/backend/facade/timeseries.py
+++ b/products/experiments/backend/facade/timeseries.py
@@ -4,6 +4,7 @@ from products.experiments.backend.metric_calculation.keys import (
daily_inline_metric_calculations,
daily_saved_metric_calculations,
metric_calculation_keys,
+ saved_metric_link_for_calculation_key,
)
from products.experiments.backend.metric_calculation.results import (
previous_completed_metric_result,
@@ -33,5 +34,6 @@ __all__ = [
"record_daily_metric_failure",
"record_daily_metric_result",
"apply_saved_metric_overrides",
+ "saved_metric_link_for_calculation_key",
"sync_timeseries_recalculation",
]
diff --git a/products/experiments/backend/hogql_queries/experiment_query_runner.py b/products/experiments/backend/hogql_queries/experiment_query_runner.py
index d0d93ec732db..7702cf819bfd 100644
--- a/products/experiments/backend/hogql_queries/experiment_query_runner.py
+++ b/products/experiments/backend/hogql_queries/experiment_query_runner.py
@@ -69,6 +69,10 @@ from products.experiments.backend.hogql_queries.utils import (
get_variant_results,
split_baseline_and_test_variants,
)
+from products.experiments.backend.metric_calculation.config import (
+ ExperimentCalculationSettings,
+ MetricCalculationConfig,
+)
from products.experiments.backend.metric_resolution import (
metric_reads_data_warehouse,
metric_uses_warehouse_native_source,
@@ -712,6 +716,18 @@ class ExperimentQueryRunner(ExperimentResultsCacheMixin, QueryRunner):
f":max_execution_time={self.max_execution_time}"
)
+ @cached_property
+ def calculation_config(self) -> MetricCalculationConfig:
+ self.experiment.team = self.team
+ settings = ExperimentCalculationSettings.from_experiment(
+ self.experiment, team_config=self._team_experiments_config
+ )
+ return settings.build_metric_config(
+ metric_id=self.query.metric.uuid or "",
+ role="primary",
+ definition=self.query.metric.model_dump(mode="json", exclude_none=True),
+ )
+
def get_cache_payload(self) -> dict:
payload = super().get_cache_payload()
payload["experiment_response_version"] = 2
diff --git a/products/experiments/backend/metric_calculation/config.py b/products/experiments/backend/metric_calculation/config.py
index fd23875b5924..cbd2c0463496 100644
--- a/products/experiments/backend/metric_calculation/config.py
+++ b/products/experiments/backend/metric_calculation/config.py
@@ -525,13 +525,22 @@ def build_primary_calculation_configs(
return sorted(primary, key=lambda calculation_config: rank(calculation_config.metric_id))
-def get_metric_calculation_config(experiment: Experiment, metric_uuid: str) -> MetricCalculationConfig | None:
+def get_metric_calculation_config(
+ experiment: Experiment,
+ metric_uuid: str,
+ *,
+ team_config: TeamExperimentsConfig | None = None,
+ key_version: int | None = None,
+) -> MetricCalculationConfig | None:
"""The config that `build_calculation_configs` builds for one metric, or None when the experiment
- calculates no metric with this uuid."""
+ calculates no metric with this uuid. `team_config` and `key_version` are as for
+ `ExperimentCalculationSettings.from_experiment`."""
metric = next((metric for metric in get_metrics_for_calculation(experiment) if metric.uuid == metric_uuid), None)
if metric is None:
return None
- settings = ExperimentCalculationSettings.from_experiment(experiment)
+ settings = ExperimentCalculationSettings.from_experiment(
+ experiment, team_config=team_config, key_version=key_version
+ )
return settings.build_metric_config(metric_id=metric.uuid, role=metric.role, definition=metric.definition)
diff --git a/products/experiments/backend/metric_calculation/keys.py b/products/experiments/backend/metric_calculation/keys.py
index f4435baab72f..a90affeda2b9 100644
--- a/products/experiments/backend/metric_calculation/keys.py
+++ b/products/experiments/backend/metric_calculation/keys.py
@@ -18,7 +18,12 @@ from products.experiments.backend.metric_calculation.config import (
saved_metric_calculation_keys,
team_experiments_configs,
)
-from products.experiments.backend.metric_resolution import is_scheduled_metric, saved_metric_links
+from products.experiments.backend.metric_resolution import (
+ apply_saved_metric_overrides,
+ is_scheduled_metric,
+ saved_metric_link_role,
+ saved_metric_links,
+)
from products.experiments.backend.models.experiment import Experiment, ExperimentToSavedMetric
logger = structlog.get_logger(__name__)
@@ -114,6 +119,35 @@ def daily_saved_metric_calculations(team_id_by_experiment_id: Mapping[int, int])
return calculations
+def saved_metric_link_for_calculation_key(
+ experiment_id: int, *, team_id: int, metric_uuid: str, calculation_key: str
+) -> int | None:
+ """The id of the experiment's saved-metric link with this uuid whose effective definition gives `calculation_key`
+ in some key version under the current settings, or None when no link does.
+
+ Two links of one experiment can carry the same saved-metric uuid with different overrides, and the daily discovery
+ queues each of them under its own key. This tells the calculation which of them a queued key belongs to.
+ """
+ experiment = (
+ Experiment.objects.select_related("team", "feature_flag").filter(id=experiment_id, team_id=team_id).first()
+ )
+ if experiment is None:
+ return None
+ settings = ExperimentCalculationSettings.from_experiment(experiment)
+ for link in experiment.experimenttosavedmetric_set.select_related("saved_metric").order_by("id"):
+ query = link.saved_metric.query
+ if not isinstance(query, dict) or query.get("uuid") != metric_uuid:
+ continue
+ calculation_config = settings.build_metric_config(
+ metric_id=metric_uuid,
+ role=saved_metric_link_role(link),
+ definition=apply_saved_metric_overrides(query, link.metadata),
+ )
+ if calculation_config.key_version_of(calculation_key) is not None:
+ return link.id
+ return None
+
+
def metric_calculation_keys(experiment_id: int, *, team_id: int) -> MetricCalculationKeys:
"""The key of every addressable metric of one experiment, including metrics that cannot be scheduled.
The maps are empty when the team has no experiment with this id."""
diff --git a/products/experiments/backend/migrations/0050_metric_scoped_timeseries_backfills.py b/products/experiments/backend/migrations/0050_metric_scoped_timeseries_backfills.py
new file mode 100644
index 000000000000..f5f35df8520f
--- /dev/null
+++ b/products/experiments/backend/migrations/0050_metric_scoped_timeseries_backfills.py
@@ -0,0 +1,58 @@
+# Generated by Django 5.2.18 on 2026-10-15 11:20
+
+import django.db.models.fields.json
+from django.db import migrations, models
+
+from posthog.migration_helpers import CreateIndexConcurrently, DropIndexConcurrently
+
+
+class Migration(migrations.Migration):
+ atomic = False
+
+ dependencies = [
+ ("experiments", "0049_experimentmetricresult_display_key"),
+ ("posthog", "1397_uploaded_media_dimensions_and_archive"),
+ ]
+
+ operations = [
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.AddConstraint(
+ model_name="experimenttimeseriesrecalculation",
+ constraint=models.UniqueConstraint(
+ models.F("experiment"),
+ django.db.models.fields.json.KeyTextTransform("uuid", "metric"),
+ models.F("fingerprint"),
+ condition=models.Q(status__in=["pending", "in_progress"]),
+ name="unique_active_backfill_per_metric_key",
+ ),
+ ),
+ ],
+ database_operations=[
+ CreateIndexConcurrently(
+ index_name="unique_active_backfill_per_metric_key",
+ table_name="posthog_experimenttimeseriesrecalculation",
+ columns="(experiment_id, (metric ->> 'uuid'), fingerprint)",
+ unique=True,
+ where="WHERE status IN ('pending', 'in_progress')",
+ ),
+ ],
+ ),
+ migrations.SeparateDatabaseAndState(
+ state_operations=[
+ migrations.RemoveConstraint(
+ model_name="experimenttimeseriesrecalculation",
+ name="unique_active_recalculation_per_experiment_metric",
+ ),
+ ],
+ database_operations=[
+ DropIndexConcurrently(
+ index_name="unique_active_recalculation_per_experiment_metric",
+ table_name="posthog_experimenttimeseriesrecalculation",
+ columns="(experiment_id, fingerprint)",
+ unique=True,
+ where="WHERE status IN ('pending', 'in_progress')",
+ ),
+ ],
+ ),
+ ]
diff --git a/products/experiments/backend/migrations/0051_canceled_timeseries_recalculation.py b/products/experiments/backend/migrations/0051_canceled_timeseries_recalculation.py
new file mode 100644
index 000000000000..59dfe9aff75a
--- /dev/null
+++ b/products/experiments/backend/migrations/0051_canceled_timeseries_recalculation.py
@@ -0,0 +1,27 @@
+# Generated by Django 5.2.18 on 2026-10-19 16:32
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+ dependencies = [
+ ("experiments", "0050_metric_scoped_timeseries_backfills"),
+ ]
+
+ operations = [
+ migrations.AlterField(
+ model_name="experimenttimeseriesrecalculation",
+ name="status",
+ field=models.CharField(
+ choices=[
+ ("pending", "Pending"),
+ ("in_progress", "In Progress"),
+ ("completed", "Completed"),
+ ("failed", "Failed"),
+ ("canceled", "Canceled"),
+ ],
+ default="pending",
+ max_length=20,
+ ),
+ ),
+ ]
diff --git a/products/experiments/backend/migrations/max_migration.txt b/products/experiments/backend/migrations/max_migration.txt
index 2bf3bcfa5223..75f02cf45f04 100644
--- a/products/experiments/backend/migrations/max_migration.txt
+++ b/products/experiments/backend/migrations/max_migration.txt
@@ -1 +1 @@
-0049_experimentmetricresult_display_key
+0051_canceled_timeseries_recalculation
diff --git a/products/experiments/backend/models/experiment.py b/products/experiments/backend/models/experiment.py
index cbea7ee11e7e..60db565eb24f 100644
--- a/products/experiments/backend/models/experiment.py
+++ b/products/experiments/backend/models/experiment.py
@@ -4,6 +4,7 @@ from typing import TYPE_CHECKING, Any, Literal
from django.contrib.postgres.fields import ArrayField
from django.db import models
from django.db.models import Exists, OuterRef, QuerySet
+from django.db.models.fields.json import KeyTextTransform
from django.utils import timezone
from posthog.models.activity_logging.model_activity import ModelActivityMixin
@@ -505,6 +506,7 @@ class ExperimentTimeseriesRecalculation(UUIDModel):
IN_PROGRESS = "in_progress", "In Progress"
COMPLETED = "completed", "Completed"
FAILED = "failed", "Failed"
+ CANCELED = "canceled", "Canceled"
team = models.ForeignKey("posthog.Team", on_delete=models.CASCADE, related_name="+")
experiment = models.ForeignKey("Experiment", on_delete=models.CASCADE)
@@ -522,9 +524,11 @@ class ExperimentTimeseriesRecalculation(UUIDModel):
]
constraints = [
models.UniqueConstraint(
- fields=["experiment", "fingerprint"],
+ models.F("experiment"),
+ KeyTextTransform("uuid", "metric"),
+ models.F("fingerprint"),
condition=models.Q(status__in=["pending", "in_progress"]),
- name="unique_active_recalculation_per_experiment_metric",
+ name="unique_active_backfill_per_metric_key",
),
]
db_table = "posthog_experimenttimeseriesrecalculation"
diff --git a/products/experiments/backend/presentation/views.py b/products/experiments/backend/presentation/views.py
index a67b4bdfaebf..115fc5cd727c 100644
--- a/products/experiments/backend/presentation/views.py
+++ b/products/experiments/backend/presentation/views.py
@@ -1597,16 +1597,15 @@ class EnterpriseExperimentsViewSet(
def recalculate_timeseries(self, request: Request, *args: Any, **kwargs: Any) -> Response:
experiment = self.get_object()
+ # Only the metric's uuid is read, and a `fingerprint` in the body is ignored: the service derives the
+ # definition and the key from the experiment.
metric = request.data.get("metric")
- fingerprint = request.data.get("fingerprint")
- if not metric:
+ if not isinstance(metric, dict) or not metric:
raise ValidationError("metric is required")
- if not fingerprint:
- raise ValidationError("fingerprint is required")
service = ExperimentService(team=self.team, user=request.user)
- result = service.request_timeseries_recalculation(experiment, metric=metric, fingerprint=fingerprint)
+ result = service.request_timeseries_recalculation(experiment, metric_uuid=metric.get("uuid") or "")
is_existing = result.pop("is_existing", False)
if not is_existing:
diff --git a/products/experiments/backend/temporal/ARCHITECTURE.md b/products/experiments/backend/temporal/ARCHITECTURE.md
index 927c5881c5a0..68a051f3d169 100644
--- a/products/experiments/backend/temporal/ARCHITECTURE.md
+++ b/products/experiments/backend/temporal/ARCHITECTURE.md
@@ -75,6 +75,8 @@ Every result row is keyed by a `recalc_fp = sha256(config_fp + "recalculation")`
Two key versions exist, and `calculation_key()` returns the one of `settings.key_version`. `resolve_calculation_key_version` decides the version when an operation starts; queued work carries it in its key, so a worker never re-decides it. Key version 1 (`legacy_key()`) hashes only the effective metric definition, the start date, the stats method, the stored exposure criteria, maturity and the excluded variants.
+A metrics recalculation run carries its version in the `calculation_key_version` workflow input, and the workflow passes it to the calc activity only when it is not 1. It is the seventh positional argument, after the precompute mode, so a version 2 calc sends the mode too, as `None` for AUTO. Every worker in a region must run that code before any team uses version 2: an older activity worker rejects the extra argument with a `TypeError`, and an older workflow worker drops the field and runs the run as version 1.
+
Key version 2 hashes the effective metric definition and every analytical setting of the calculation config: start date, flag key, entity, variants, the resolved baseline, excluded variants, the exposure criteria, test account filters, the resolved statistics method and settings, CUPED with team defaults applied, maturity, and the team timezone. Every metric type keys on the timezone, because date property filters, the CUPED lookback and retention windows all read it. Floats are rounded to 12 significant digits first, so stored float noise such as `1 - 0.95` does not split a key. Nested settings enter through explicit field lists, so a schema change that adds a defaulted field to a model changes no key. The key version is part of the hashed payload: a change that makes the engine compute a different result for an existing config bumps `CALCULATION_KEY_VERSION`, so older results stop counting for reuse.
Key version 2 reads the exposure criteria as stored, with the product defaults applied (`normalize_exposure`). It does not read the event that the `experiment-exposure-event` flag picks as the default exposure event, because each process evaluates that flag locally: two pods that disagree would file and look up one experiment's results under two keys, and each rollout step would re-key every affected experiment. A stored result therefore stays reusable for its window when the flag changes for a team, as with key version 1. Shipping the new event to every team is an engine change, which bumps `CALCULATION_KEY_VERSION` if older results must stop counting for reuse.
@@ -128,6 +130,8 @@ A successful daily write then takes over the recalculation's row, and the next r
A failed daily write leaves a completed row under another fingerprint in place, because it has no result to replace it with.
The calc activity's write keeps its lock order and its terminal-run guard (see [cancellation and result-write protection](../../../../docs/internal/experiment-metric-recalculation.md)).
+Backfill requests lock the experiment while checking for an active job by metric UUID, deleting its daily results, and creating the new job in one transaction. Failed job creation preserves the existing results. The active-job unique index includes the metric UUID as well as the experiment and calculation key, so equivalent metrics can backfill independently. Keeping the key in the index allows existing jobs for one UUID under different keys to coexist during a rolling deployment; the locked UUID lookup prevents new requests from adding another job for that metric.
+
### Counters are derived, not stored
`completed_metrics` and `failed_metrics` aren't columns on the recalc row. They're computed at read time from `ExperimentMetricResult` rows (`status=COMPLETED` for completed, `status=FAILED` plus discovery-step failures from `metric_errors` for failed). This eliminates a class of bug: when Temporal retries an activity, there's no counter to double-increment. The only thing that matters is whether the result row exists.
diff --git a/products/experiments/backend/temporal/models.py b/products/experiments/backend/temporal/models.py
index 84f50907e4f8..683c8f77d55c 100644
--- a/products/experiments/backend/temporal/models.py
+++ b/products/experiments/backend/temporal/models.py
@@ -118,6 +118,8 @@ class ExperimentMetricsRecalculationWorkflowInputs:
# Task queue for the metric calculation activity, chosen when the run starts so replay stays deterministic.
# None keeps the activity on the workflow's own queue.
activity_task_queue: str | None = None
+ # The key version that the run files its results under, chosen when the run is requested.
+ calculation_key_version: int = 1
@dataclasses.dataclass(frozen=False)
@@ -161,6 +163,7 @@ class MetricOutOfTimeFailure:
experiment_id: int
metric_uuid: str
role: str # "primary" or "secondary"
+ calculation_key_version: int = 1
@dataclasses.dataclass
diff --git a/products/experiments/backend/temporal/recalculation_activities.py b/products/experiments/backend/temporal/recalculation_activities.py
index cf2754405838..e7556c1e691a 100644
--- a/products/experiments/backend/temporal/recalculation_activities.py
+++ b/products/experiments/backend/temporal/recalculation_activities.py
@@ -110,6 +110,7 @@ async def calculate_experiment_metric_for_recalculation(
query_to: str,
role: str = "primary",
precomputation_mode: PrecomputationMode | None = None,
+ calculation_key_version: int = 1,
) -> MetricRecalculationResult:
"""Calculate one metric, write its recalc-fingerprinted result to ExperimentMetricResult, and fold the
progress update (counter + error) into the same job atomically. query_to is the run's shared data-window end.
@@ -119,8 +120,8 @@ async def calculate_experiment_metric_for_recalculation(
to resolve it. Defaults to "primary" so existing call sites and tests that don't pass it remain valid. The
workflow passes these arguments by position, so recorded histories depend on their order, not their names.
- precomputation_mode comes from discovery. The workflow passes it only for a DIRECT calc on the activity queue,
- and None keeps the AUTO mode. A calc on the precompute queue also uses AUTO, because PRECOMPUTED is reserved
+ precomputation_mode comes from discovery. The workflow passes it for a DIRECT calc on the activity queue, and
+ as None ahead of a calculation_key_version other than 1. None keeps the AUTO mode. A calc on the precompute queue also uses AUTO, because PRECOMPUTED is reserved
for staff tools.
Finality is derived from activity.info().attempt against MAX_METRIC_ATTEMPTS, the same constant the
@@ -145,6 +146,7 @@ async def calculate_experiment_metric_for_recalculation(
is_final_attempt,
attempt,
precomputation_mode,
+ calculation_key_version,
)
)
try:
diff --git a/products/experiments/backend/temporal/recalculation_logic.py b/products/experiments/backend/temporal/recalculation_logic.py
index 91996851a526..8d6676a09ad5 100644
--- a/products/experiments/backend/temporal/recalculation_logic.py
+++ b/products/experiments/backend/temporal/recalculation_logic.py
@@ -44,7 +44,10 @@ from products.experiments.backend.hogql_queries.error_handling import (
)
from products.experiments.backend.hogql_queries.experiment_query_runner import ExperimentQueryRunner
from products.experiments.backend.hogql_queries.utils import sanitize_non_finite
-from products.experiments.backend.metric_calculation.config import get_metric_calculation_config
+from products.experiments.backend.metric_calculation.config import (
+ get_metric_calculation_config,
+ team_experiments_configs,
+)
from products.experiments.backend.metric_calculation.results import MetricResultStore
from products.experiments.backend.metric_resolution import build_metric, get_metrics_for_calculation
from products.experiments.backend.models.experiment import Experiment, ExperimentMetricsRecalculation
@@ -697,6 +700,7 @@ def _calculate_experiment_metric_for_recalculation_sync(
is_final_attempt: bool = True,
attempt: int = 1,
precomputation_mode: PrecomputationMode | None = None,
+ calculation_key_version: int = 1,
) -> MetricRecalculationResult:
close_old_connections()
@@ -750,7 +754,12 @@ def _calculate_experiment_metric_for_recalculation_sync(
error_type="validation_error",
)
- calculation_config = get_metric_calculation_config(experiment, metric_uuid)
+ # One read of the settings serves the reuse check, the query and the write, so the result is filed under the key
+ # of the settings it was computed from.
+ team_config = team_experiments_configs([experiment.team_id])[experiment.team_id]
+ calculation_config = get_metric_calculation_config(
+ experiment, metric_uuid, team_config=team_config, key_version=calculation_key_version
+ )
if calculation_config is None:
return _fail(
recalculation_id,
@@ -839,6 +848,8 @@ def _calculate_experiment_metric_for_recalculation_sync(
error_event_context=None,
# Must fail typed inside the activity before Temporal's start_to_close kill — see models.py.
max_execution_time=METRIC_CALC_MAX_EXECUTION_TIME_SECONDS,
+ experiment=experiment,
+ team_experiments_config=team_config,
)
# Deferred: recalculation.py imports temporal.models, whose package __init__ imports this module.
@@ -1049,7 +1060,11 @@ def _record_metric_out_of_time_failure_sync(failure: MetricOutOfTimeFailure) ->
return
experiment = Experiment.objects.filter(id=failure.experiment_id, deleted=False).first()
- calculation_config = get_metric_calculation_config(experiment, failure.metric_uuid) if experiment else None
+ calculation_config = (
+ get_metric_calculation_config(experiment, failure.metric_uuid, key_version=failure.calculation_key_version)
+ if experiment
+ else None
+ )
if experiment is None or calculation_config is None or not experiment.start_date or state.query_to is None:
_fail(
failure.recalculation_id, failure.metric_uuid, "calculation", _OUT_OF_TIME_MESSAGE, error_type="timeout"
diff --git a/products/experiments/backend/temporal/recalculation_workflow.py b/products/experiments/backend/temporal/recalculation_workflow.py
index 82c09fe2614b..fccbf402047c 100644
--- a/products/experiments/backend/temporal/recalculation_workflow.py
+++ b/products/experiments/backend/temporal/recalculation_workflow.py
@@ -175,9 +175,13 @@ class ExperimentMetricsRecalculationWorkflow(PostHogWorkflow):
None if task_queue else timedelta(seconds=METRIC_CALC_SCHEDULE_TO_CLOSE_TIMEOUT_SECONDS)
)
args: list[object] = [metric.experiment_id, metric.metric_uuid, recalculation_id, query_to, metric.role]
- # An AUTO calc keeps the five arguments, so a worker that predates the mode argument still runs it.
- if routed and metric.precomputation_mode is not None:
- args.append(metric.precomputation_mode)
+ # An argument goes on the wire only when its default does not cover it, so a worker that predates the
+ # mode or the key version argument still runs an AUTO, version 1 calc. The version follows the mode.
+ precomputation_mode = metric.precomputation_mode if routed else None
+ if precomputation_mode is not None or inputs.calculation_key_version != 1:
+ args.append(precomputation_mode)
+ if inputs.calculation_key_version != 1:
+ args.append(inputs.calculation_key_version)
calcs.append(
temporalio.workflow.execute_activity(
calculate_experiment_metric_for_recalculation,
@@ -200,7 +204,7 @@ class ExperimentMetricsRecalculationWorkflow(PostHogWorkflow):
if isinstance(result, ActivityError) and _ended_by_temporal(result)
]
if out_of_time and temporalio.workflow.patched("recalc-persist-out-of-time-metrics-2026-10"):
- await self._record_out_of_time_failures(recalculation_id, out_of_time)
+ await self._record_out_of_time_failures(recalculation_id, out_of_time, inputs.calculation_key_version)
succeeded = sum(1 for result in results if not isinstance(result, BaseException) and result.success)
failed = len(results) - succeeded
@@ -239,7 +243,7 @@ class ExperimentMetricsRecalculationWorkflow(PostHogWorkflow):
return {"total": len(metrics), "succeeded": succeeded, "failed": failed}
async def _record_out_of_time_failures(
- self, recalculation_id: str, metrics: list[ExperimentMetricToRecalculate]
+ self, recalculation_id: str, metrics: list[ExperimentMetricToRecalculate], calculation_key_version: int
) -> None:
# Best effort: a backstop failure must not stop the finish step, which still counts these metrics failed.
await asyncio.gather(
@@ -251,6 +255,7 @@ class ExperimentMetricsRecalculationWorkflow(PostHogWorkflow):
experiment_id=metric.experiment_id,
metric_uuid=metric.metric_uuid,
role=metric.role,
+ calculation_key_version=calculation_key_version,
),
start_to_close_timeout=timedelta(seconds=METRIC_FAILURE_BACKSTOP_ACTIVITY_TIMEOUT_SECONDS),
schedule_to_close_timeout=timedelta(
diff --git a/products/experiments/backend/temporal/test_discovery_fingerprints.py b/products/experiments/backend/temporal/test_discovery_fingerprints.py
index c0ac0b4472f8..d67e89828eca 100644
--- a/products/experiments/backend/temporal/test_discovery_fingerprints.py
+++ b/products/experiments/backend/temporal/test_discovery_fingerprints.py
@@ -151,6 +151,9 @@ class TestDiscoveryFingerprints:
patch("posthog.temporal.experiments.activities.ExperimentQueryRunner") as mock_runner_class,
):
mock_runner_class.return_value.run.return_value.model_dump.return_value = {"variant_results": []}
+ mock_runner_class.return_value.calculation_config = get_metric_calculation_config(
+ experiment, SAVED_FUNNEL["uuid"]
+ )
results = _raw_saved_sync(hour=2)
[discovered] = [r for r in results if r.experiment_id == experiment.id]
outcome = _raw_saved_calc(experiment.id, discovered.metric_uuid, discovered.fingerprint)
diff --git a/products/experiments/backend/temporal/test_recalculation_activities.py b/products/experiments/backend/temporal/test_recalculation_activities.py
index 96740763e9b2..f2b89bf0bce3 100644
--- a/products/experiments/backend/temporal/test_recalculation_activities.py
+++ b/products/experiments/backend/temporal/test_recalculation_activities.py
@@ -100,10 +100,19 @@ def _calculate(
is_final_attempt: bool = True,
attempt: int = 1,
precomputation_mode: PrecomputationMode | None = None,
+ calculation_key_version: int = 1,
):
with patch("products.experiments.backend.temporal.recalculation_logic.close_old_connections"):
return _calculate_experiment_metric_for_recalculation_sync(
- experiment_id, metric_uuid, recalculation_id, query_to, role, is_final_attempt, attempt, precomputation_mode
+ experiment_id,
+ metric_uuid,
+ recalculation_id,
+ query_to,
+ role,
+ is_final_attempt,
+ attempt,
+ precomputation_mode,
+ calculation_key_version,
)
@@ -1333,6 +1342,45 @@ class TestCalculateActivity(BaseTest):
mock_runner.assert_called_once()
+ @parameterized.expand([("key_version_1", 1), ("key_version_2", 2)])
+ def test_a_settings_change_before_the_runner_reads_them_is_not_stored_under_the_earlier_key(
+ self, _name: str, key_version: int
+ ) -> None:
+ metric = _mean_metric("m1")
+ exp = self._experiment(flag_key="calc-settings-race", metrics=[metric])
+ recalc = self._recalc(exp, metric_uuids=["m1"])
+ team_config = get_or_create_team_extension(self.team, TeamExperimentsConfig)
+ runners: list[ExperimentQueryRunner] = []
+
+ def change_settings_then_build_runner(*args, **kwargs) -> ExperimentQueryRunner:
+ # Version 1 hashes the stats method, and version 2 also hashes the CUPED team default.
+ Experiment.objects.filter(id=exp.id).update(stats_config={"method": "frequentist"})
+ team_config.default_cuped_enabled = not team_config.default_cuped_enabled
+ team_config.save()
+ runner = ExperimentQueryRunner(*args, **kwargs)
+ runners.append(runner)
+ return runner
+
+ computed = MagicMock()
+ computed.model_dump.return_value = {"variant_results": []}
+ with (
+ patch(
+ "products.experiments.backend.temporal.recalculation_logic.ExperimentQueryRunner",
+ side_effect=change_settings_then_build_runner,
+ ),
+ patch.object(ExperimentQueryRunner, "run", return_value=computed),
+ ):
+ _calculate(exp.id, "m1", str(recalc.id), _QUERY_TO, calculation_key_version=key_version)
+
+ [runner] = runners
+ executed_key = runner.calculation_config.settings.build_metric_config(
+ metric_id="m1", role="primary", definition=metric
+ ).key_for_version(key_version)
+ stored = ExperimentMetricResult.objects.filter(experiment=exp, metric_uuid="m1").values_list(
+ "fingerprint", flat=True
+ )
+ assert list(stored) == [_recalc_fingerprint(executed_key)]
+
@parameterized.expand(
[
# (name, metric_uuid, metrics_on_experiment, run_with_mocked_failure, expected_result_rows)
diff --git a/products/experiments/backend/temporal/test_recalculation_workflow.py b/products/experiments/backend/temporal/test_recalculation_workflow.py
index 4acad509db80..6d229da40600 100644
--- a/products/experiments/backend/temporal/test_recalculation_workflow.py
+++ b/products/experiments/backend/temporal/test_recalculation_workflow.py
@@ -129,7 +129,9 @@ async def _run_workflow(activities) -> dict:
_Call = tuple[str, PrecomputationMode | None]
-def _recording_calculate(calls: list[_Call], budgets: list[timedelta | None]) -> Callable:
+def _recording_calculate(
+ calls: list[_Call], budgets: list[timedelta | None], key_versions: list[int] | None = None
+) -> Callable:
@activity.defn(name="calculate_experiment_metric_for_recalculation")
async def mock_calculate(
experiment_id: int,
@@ -138,9 +140,12 @@ def _recording_calculate(calls: list[_Call], budgets: list[timedelta | None]) ->
query_to: str,
role: str = "primary",
precomputation_mode: PrecomputationMode | None = None,
+ calculation_key_version: int = 1,
) -> MetricRecalculationResult:
calls.append((metric_uuid, precomputation_mode))
budgets.append(activity.info().schedule_to_close_timeout)
+ if key_versions is not None:
+ key_versions.append(calculation_key_version)
return MetricRecalculationResult(metric_uuid=metric_uuid, success=True)
return mock_calculate
@@ -153,6 +158,7 @@ class _QueueRun:
activity_queue_calls: list[_Call]
precompute_queue_calls: list[_Call]
schedule_to_close_timeouts: list[timedelta | None]
+ key_versions: list[int]
history: WorkflowHistory
@@ -161,12 +167,14 @@ async def _run_with_calc_queue_workers(
activity_worker_task_queue: str,
precompute_worker_task_queue: str | None = None,
metric: ExperimentMetricToRecalculate | None = None,
+ calculation_key_version: int = 1,
) -> _QueueRun:
activities, _progress, _calls = _make_mock_activities(metrics=[metric or _metric("m1")])
workflow_queue_calls: list[_Call] = []
activity_queue_calls: list[_Call] = []
precompute_queue_calls: list[_Call] = []
budgets: list[timedelta | None] = []
+ key_versions: list[int] = []
task_queue = str(uuid.uuid4())
async with await WorkflowEnvironment.start_time_skipping() as env:
async with (
@@ -174,24 +182,26 @@ async def _run_with_calc_queue_workers(
env.client,
task_queue=task_queue,
workflows=[ExperimentMetricsRecalculationWorkflow],
- activities=[*activities[:2], _recording_calculate(workflow_queue_calls, budgets)],
+ activities=[*activities[:2], _recording_calculate(workflow_queue_calls, budgets, key_versions)],
workflow_runner=temporalio.worker.UnsandboxedWorkflowRunner(),
),
Worker(
env.client,
task_queue=activity_worker_task_queue,
- activities=[_recording_calculate(activity_queue_calls, budgets)],
+ activities=[_recording_calculate(activity_queue_calls, budgets, key_versions)],
),
Worker(
env.client,
task_queue=precompute_worker_task_queue or str(uuid.uuid4()),
- activities=[_recording_calculate(precompute_queue_calls, budgets)],
+ activities=[_recording_calculate(precompute_queue_calls, budgets, key_versions)],
),
):
handle = await env.client.start_workflow(
ExperimentMetricsRecalculationWorkflow.run,
ExperimentMetricsRecalculationWorkflowInputs(
- recalculation_id=str(uuid.uuid4()), activity_task_queue=activity_task_queue
+ recalculation_id=str(uuid.uuid4()),
+ activity_task_queue=activity_task_queue,
+ calculation_key_version=calculation_key_version,
),
id=str(uuid.uuid4()),
task_queue=task_queue,
@@ -204,6 +214,7 @@ async def _run_with_calc_queue_workers(
activity_queue_calls=activity_queue_calls,
precompute_queue_calls=precompute_queue_calls,
schedule_to_close_timeouts=budgets,
+ key_versions=key_versions,
history=history,
)
@@ -342,6 +353,7 @@ class TestExperimentMetricsRecalculationWorkflow:
is_final_attempt: bool,
attempt: int,
precomputation_mode: PrecomputationMode | None,
+ calculation_key_version: int = 1,
) -> MetricRecalculationResult:
recorded_flags.append(is_final_attempt)
raise RuntimeError("always fails")
@@ -788,3 +800,38 @@ class TestExperimentMetricsRecalculationWorkflow:
workflows=[ExperimentMetricsRecalculationWorkflow],
workflow_runner=temporalio.worker.UnsandboxedWorkflowRunner(),
).replay_workflow(run.history)
+
+ @parameterized.expand(
+ [
+ # name, routed_direct, expected workflow queue calls, expected activity queue calls
+ ("direct_route", True, [], [("m1", PrecomputationMode.DIRECT)]),
+ ("run_off_the_flag", False, [("m1", None)], []),
+ ]
+ )
+ async def test_key_version_reaches_the_calculation_after_the_mode(
+ self,
+ name: str,
+ routed_direct: bool,
+ expected_workflow_queue_calls: list[_Call],
+ expected_activity_queue_calls: list[_Call],
+ ):
+ activity_worker_task_queue = str(uuid.uuid4())
+ metric = ExperimentMetricToRecalculate(
+ experiment_id=1,
+ metric_uuid="m1",
+ role="primary",
+ task_queue=activity_worker_task_queue,
+ precomputation_mode=PrecomputationMode.DIRECT,
+ )
+
+ run = await _run_with_calc_queue_workers(
+ activity_task_queue=activity_worker_task_queue if routed_direct else None,
+ activity_worker_task_queue=activity_worker_task_queue,
+ metric=metric,
+ calculation_key_version=2,
+ )
+
+ assert run.result == {"total": 1, "succeeded": 1, "failed": 0}
+ assert run.workflow_queue_calls == expected_workflow_queue_calls
+ assert run.activity_queue_calls == expected_activity_queue_calls
+ assert run.key_versions == [2]
diff --git a/products/experiments/backend/test/test_daily_metric_activities.py b/products/experiments/backend/test/test_daily_metric_activities.py
new file mode 100644
index 000000000000..87f8d939174b
--- /dev/null
+++ b/products/experiments/backend/test/test_daily_metric_activities.py
@@ -0,0 +1,420 @@
+from collections.abc import Iterator
+from contextlib import contextmanager
+from datetime import datetime
+from uuid import uuid4
+from zoneinfo import ZoneInfo
+
+import time_machine
+from unittest.mock import MagicMock, patch
+
+from django.test import override_settings
+
+from parameterized import parameterized
+from rest_framework.exceptions import ValidationError
+
+from posthog.schema import EventsNode, ExperimentMeanMetric
+
+from posthog.models.team.extensions import get_or_create_team_extension
+from posthog.temporal.experiments.activities import (
+ _calculate_experiment_regular_metric_sync,
+ _calculate_experiment_saved_metric_sync,
+)
+from posthog.temporal.experiments.models import TIMESERIES_METRIC_MAX_ATTEMPTS
+
+from products.experiments.backend.facade.enums import ExperimentResultConfigurationStatus
+from products.experiments.backend.facade.timeseries import metric_calculation_keys
+from products.experiments.backend.hogql_queries.experiment_query_runner import ExperimentQueryRunner
+from products.experiments.backend.hogql_queries.test.experiment_query_runner.base import ExperimentQueryRunnerBaseTest
+from products.experiments.backend.metric_calculation.config import get_metric_calculation_config
+from products.experiments.backend.metric_calculation.results import MetricResultStore
+from products.experiments.backend.models.experiment import (
+ Experiment,
+ ExperimentMetricResult,
+ ExperimentSavedMetric,
+ ExperimentToSavedMetric,
+)
+from products.experiments.backend.models.team_experiments_config import TeamExperimentsConfig
+
+# A recalculation of the experiment can hold the daily window under its own fingerprint. The daily write must take
+# that row over instead of failing on the (experiment, metric_uuid, query_to) unique constraint.
+_WINDOW_HOLDERS = [("free_window", None), ("window_held_by_another_fingerprint", "recalculation-fingerprint")]
+
+
+_ACTIVITY_FOR_KIND = {
+ "regular": _calculate_experiment_regular_metric_sync,
+ "saved": _calculate_experiment_saved_metric_sync,
+}
+
+
+@override_settings(IN_UNIT_TESTING=True)
+class TestDailyMetricActivities(ExperimentQueryRunnerBaseTest):
+ def _queued_key(self, experiment: Experiment, metric_uuid: str) -> str:
+ calculation_config = get_metric_calculation_config(experiment, metric_uuid)
+ assert calculation_config is not None
+ return calculation_config.calculation_key()
+
+ @contextmanager
+ def _mocked_runner(self, experiment: Experiment, metric_uuid: str) -> Iterator[MagicMock]:
+ # The activity compares the key of the runner's settings with the queued key before it runs the query.
+ with patch("posthog.temporal.experiments.activities.ExperimentQueryRunner") as mock_runner_class:
+ mock_runner_class.return_value.calculation_config = get_metric_calculation_config(experiment, metric_uuid)
+ yield mock_runner_class
+
+ def _hold_window(self, experiment_id: int, metric_uuid: str, query_to: datetime, fingerprint: str | None) -> None:
+ if fingerprint is not None:
+ ExperimentMetricResult.objects.create(
+ experiment_id=experiment_id,
+ metric_uuid=metric_uuid,
+ fingerprint=fingerprint,
+ query_from=datetime.fromisoformat("2020-01-01T00:00:00+00:00"),
+ query_to=query_to,
+ status=ExperimentMetricResult.Status.COMPLETED,
+ result={"stale": True},
+ )
+
+ @parameterized.expand(_WINDOW_HOLDERS)
+ @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
+ def test_regular_metric_activity_pins_runner_to_stored_query_to(self, _name: str, holder: str | None):
+ feature_flag = self.create_feature_flag()
+ experiment = self.create_experiment(
+ feature_flag=feature_flag,
+ start_date=datetime(2020, 1, 1, 0, 0, 0),
+ end_date=datetime(2020, 1, 5, 0, 0, 0),
+ )
+ metric = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase"))
+ metric_dict = metric.model_dump(mode="json")
+ experiment.metrics = [metric_dict]
+ experiment.save()
+ expected_query_to = datetime.fromisoformat("2020-01-05T00:00:00+00:00")
+ self._hold_window(experiment.id, metric_dict["uuid"], expected_query_to, holder)
+
+ with (
+ patch("posthog.temporal.experiments.activities.close_old_connections"),
+ self._mocked_runner(experiment, metric_dict["uuid"]) as mock_runner_class,
+ ):
+ mock_runner_class.return_value.run.return_value.model_dump.return_value = {"variant_results": []}
+
+ activity_result = _calculate_experiment_regular_metric_sync.func( # type: ignore[attr-defined]
+ experiment.id, metric_dict["uuid"], self._queued_key(experiment, metric_dict["uuid"])
+ )
+
+ self.assertTrue(activity_result.success, msg=activity_result.error_message)
+ mock_runner_class.assert_called_once()
+ assert mock_runner_class.call_args.kwargs["as_of"] == expected_query_to
+ result_row = ExperimentMetricResult.objects.get(experiment=experiment, metric_uuid=metric_dict["uuid"])
+ assert result_row.query_to == expected_query_to
+ assert (result_row.fingerprint, result_row.display_key, result_row.result) == (
+ self._queued_key(experiment, metric_dict["uuid"]),
+ self._queued_key(experiment, metric_dict["uuid"]),
+ {"variant_results": []},
+ )
+
+ @parameterized.expand(_WINDOW_HOLDERS)
+ @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
+ def test_saved_metric_activity_pins_runner_to_stored_query_to(self, _name: str, holder: str | None):
+ feature_flag = self.create_feature_flag()
+ experiment = self.create_experiment(
+ feature_flag=feature_flag,
+ start_date=datetime(2020, 1, 1, 0, 0, 0),
+ end_date=datetime(2020, 1, 5, 0, 0, 0),
+ )
+ metric = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase"))
+ metric_dict = metric.model_dump(mode="json")
+ expected_query_to = datetime.fromisoformat("2020-01-05T00:00:00+00:00")
+
+ saved_metric = ExperimentSavedMetric.objects.create(
+ name="test saved metric",
+ team=self.team,
+ query=metric_dict,
+ created_by=self.user,
+ )
+ ExperimentToSavedMetric.objects.create(
+ experiment=experiment,
+ saved_metric=saved_metric,
+ metadata={"type": "primary"},
+ )
+ self._hold_window(experiment.id, metric_dict["uuid"], expected_query_to, holder)
+
+ with (
+ patch("posthog.temporal.experiments.activities.close_old_connections"),
+ self._mocked_runner(experiment, metric_dict["uuid"]) as mock_runner_class,
+ ):
+ mock_runner_class.return_value.run.return_value.model_dump.return_value = {"variant_results": []}
+
+ activity_result = _calculate_experiment_saved_metric_sync.func( # type: ignore[attr-defined]
+ experiment.id, metric_dict["uuid"], self._queued_key(experiment, metric_dict["uuid"])
+ )
+
+ self.assertTrue(activity_result.success, msg=activity_result.error_message)
+ mock_runner_class.assert_called_once()
+ assert mock_runner_class.call_args.kwargs["as_of"] == expected_query_to
+ result_row = ExperimentMetricResult.objects.get(experiment=experiment, metric_uuid=metric_dict["uuid"])
+ assert result_row.query_to == expected_query_to
+ assert (result_row.fingerprint, result_row.display_key, result_row.result) == (
+ self._queued_key(experiment, metric_dict["uuid"]),
+ self._queued_key(experiment, metric_dict["uuid"]),
+ {"variant_results": []},
+ )
+
+ @parameterized.expand(
+ [
+ (f"{kind}_{change}", kind, key_version, change)
+ for kind in ("regular", "saved")
+ for key_version, change in ((1, "stats_method"), (2, "team_cuped_default"), (2, None))
+ ]
+ )
+ @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
+ def test_activity_stores_a_result_only_under_the_key_the_runner_computes(
+ self, _name: str, kind: str, key_version: int, change: str | None
+ ) -> None:
+ feature_flag = self.create_feature_flag()
+ experiment = self.create_experiment(
+ feature_flag=feature_flag,
+ start_date=datetime(2020, 1, 1, 0, 0, 0),
+ end_date=datetime(2020, 1, 5, 0, 0, 0),
+ )
+ metric_dict = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase")).model_dump(
+ mode="json"
+ )
+ if kind == "regular":
+ experiment.metrics = [metric_dict]
+ experiment.save()
+ else:
+ ExperimentToSavedMetric.objects.create(
+ experiment=experiment,
+ saved_metric=ExperimentSavedMetric.objects.create(
+ name="saved", team=self.team, query=metric_dict, created_by=self.user
+ ),
+ metadata={"type": "primary"},
+ )
+ settings_a = get_metric_calculation_config(experiment, metric_dict["uuid"])
+ assert settings_a is not None
+ queued_key = settings_a.key_for_version(key_version)
+ team_config = get_or_create_team_extension(self.team, TeamExperimentsConfig)
+
+ def apply_settings(*, changed: bool) -> None:
+ if change == "stats_method":
+ experiment.stats_config = {"method": "frequentist"} if changed else None
+ experiment.save()
+ elif change == "team_cuped_default":
+ team_config.default_cuped_enabled = changed
+ team_config.save()
+
+ apply_settings(changed=True)
+ significant_result = MagicMock()
+ significant_result.model_dump.return_value = {
+ "variant_results": [{"key": "test", "significant": True, "chance_to_win": 0.99}]
+ }
+ with (
+ patch("posthog.temporal.experiments.activities.close_old_connections"),
+ patch.object(ExperimentQueryRunner, "run", return_value=significant_result),
+ patch("posthog.temporal.experiments.utils.produce_internal_event") as mock_produce,
+ ):
+ activity_result = _ACTIVITY_FOR_KIND[kind].func(experiment.id, metric_dict["uuid"], queued_key) # type: ignore[attr-defined]
+
+ rows = ExperimentMetricResult.objects.filter(experiment=experiment, metric_uuid=metric_dict["uuid"])
+ if change is None:
+ assert (activity_result.success, activity_result.skipped) == (True, False)
+ assert [(row.fingerprint, row.display_key) for row in rows] == [(queued_key, settings_a.legacy_key())]
+ mock_produce.assert_called_once()
+ return
+
+ assert (activity_result.success, activity_result.skipped) == (False, True)
+ assert not rows.exists()
+ mock_produce.assert_not_called()
+
+ apply_settings(changed=False)
+ experiment.refresh_from_db()
+ calculation_config = get_metric_calculation_config(experiment, metric_dict["uuid"])
+ assert calculation_config is not None and calculation_config.key_for_version(key_version) == queued_key
+ timeseries = MetricResultStore(experiment_id=experiment.id).timeseries(
+ calculation_config, timezone=ZoneInfo("UTC")
+ )
+ assert ExperimentResultConfigurationStatus.CURRENT not in timeseries.configuration_by_day.values()
+
+ @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
+ def test_saved_metric_activity_computes_the_link_of_the_queued_key(self) -> None:
+ # Two links can carry one saved metric with different breakdowns, and discovery queues each under its own key.
+ experiment = self.create_experiment(
+ feature_flag=self.create_feature_flag(),
+ start_date=datetime(2020, 1, 1, 0, 0, 0),
+ end_date=datetime(2020, 1, 5, 0, 0, 0),
+ )
+ metric_dict = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase")).model_dump(
+ mode="json"
+ )
+ saved_metric = ExperimentSavedMetric.objects.create(
+ name="saved", team=self.team, query=metric_dict, created_by=self.user
+ )
+ ExperimentToSavedMetric.objects.create(
+ experiment=experiment, saved_metric=saved_metric, metadata={"type": "primary"}
+ )
+ broken_down = ExperimentToSavedMetric.objects.create(
+ experiment=experiment,
+ saved_metric=saved_metric,
+ metadata={"type": "secondary", "breakdowns": [{"property": "$browser", "type": "event"}]},
+ )
+ queued_key = metric_calculation_keys(experiment.id, team_id=self.team.id).saved[broken_down.id]
+ computed = MagicMock()
+ computed.model_dump.return_value = {"variant_results": []}
+
+ with (
+ patch("posthog.temporal.experiments.activities.close_old_connections"),
+ patch.object(ExperimentQueryRunner, "run", return_value=computed),
+ ):
+ activity_result = _calculate_experiment_saved_metric_sync.func( # type: ignore[attr-defined]
+ experiment.id, metric_dict["uuid"], queued_key
+ )
+
+ assert (activity_result.success, activity_result.skipped) == (True, False)
+ assert list(
+ ExperimentMetricResult.objects.filter(experiment=experiment).values_list("fingerprint", flat=True)
+ ) == [queued_key]
+
+ @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
+ def test_regular_metric_activity_returns_permanent_failure_on_validation_error(self):
+ feature_flag = self.create_feature_flag()
+ experiment = self.create_experiment(
+ feature_flag=feature_flag,
+ start_date=datetime(2020, 1, 1, 0, 0, 0),
+ end_date=datetime(2020, 1, 5, 0, 0, 0),
+ )
+ metric = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase"))
+ metric_dict = metric.model_dump(mode="json")
+ experiment.metrics = [metric_dict]
+ experiment.save()
+
+ with (
+ patch("posthog.temporal.experiments.activities.close_old_connections"),
+ self._mocked_runner(experiment, metric_dict["uuid"]) as mock_runner_class,
+ patch("posthog.temporal.experiments.activities.capture_experiment_metric_error_event") as mock_event,
+ ):
+ mock_runner_class.return_value.run.side_effect = ValidationError("compound expressions not supported")
+
+ activity_result = _calculate_experiment_regular_metric_sync.func( # type: ignore[attr-defined]
+ experiment.id, metric_dict["uuid"], self._queued_key(experiment, metric_dict["uuid"])
+ )
+
+ self.assertFalse(activity_result.success)
+ result_row = ExperimentMetricResult.objects.get(experiment=experiment, metric_uuid=metric_dict["uuid"])
+ assert result_row.status == ExperimentMetricResult.Status.FAILED
+ mock_event.assert_called_once()
+
+ @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
+ def test_saved_metric_activity_returns_permanent_failure_on_validation_error(self):
+ feature_flag = self.create_feature_flag()
+ experiment = self.create_experiment(
+ feature_flag=feature_flag,
+ start_date=datetime(2020, 1, 1, 0, 0, 0),
+ end_date=datetime(2020, 1, 5, 0, 0, 0),
+ )
+ metric = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase"))
+ metric_dict = metric.model_dump(mode="json")
+
+ saved_metric = ExperimentSavedMetric.objects.create(
+ name="test saved metric",
+ team=self.team,
+ query=metric_dict,
+ created_by=self.user,
+ )
+ ExperimentToSavedMetric.objects.create(
+ experiment=experiment,
+ saved_metric=saved_metric,
+ metadata={"type": "primary"},
+ )
+
+ with (
+ patch("posthog.temporal.experiments.activities.close_old_connections"),
+ self._mocked_runner(experiment, metric_dict["uuid"]) as mock_runner_class,
+ patch("posthog.temporal.experiments.activities.capture_experiment_metric_error_event") as mock_event,
+ ):
+ mock_runner_class.return_value.run.side_effect = ValidationError("compound expressions not supported")
+
+ activity_result = _calculate_experiment_saved_metric_sync.func( # type: ignore[attr-defined]
+ experiment.id, metric_dict["uuid"], self._queued_key(experiment, metric_dict["uuid"])
+ )
+
+ self.assertFalse(activity_result.success)
+ result_row = ExperimentMetricResult.objects.get(experiment=experiment, metric_uuid=metric_dict["uuid"])
+ assert result_row.status == ExperimentMetricResult.Status.FAILED
+ mock_event.assert_called_once()
+
+ @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
+ def test_regular_metric_activity_still_raises_transient_errors(self):
+ feature_flag = self.create_feature_flag()
+ experiment = self.create_experiment(
+ feature_flag=feature_flag,
+ start_date=datetime(2020, 1, 1, 0, 0, 0),
+ end_date=datetime(2020, 1, 5, 0, 0, 0),
+ )
+ metric = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase"))
+ metric_dict = metric.model_dump(mode="json")
+ experiment.metrics = [metric_dict]
+ experiment.save()
+
+ with (
+ patch("posthog.temporal.experiments.activities.close_old_connections"),
+ self._mocked_runner(experiment, metric_dict["uuid"]) as mock_runner_class,
+ patch("posthog.temporal.experiments.activities.capture_experiment_metric_error_event") as mock_event,
+ ):
+ mock_runner_class.return_value.run.side_effect = Exception("transient query failure")
+
+ with self.assertRaises(Exception):
+ _calculate_experiment_regular_metric_sync.func( # type: ignore[attr-defined]
+ experiment.id, metric_dict["uuid"], self._queued_key(experiment, metric_dict["uuid"]), attempt=1
+ )
+ mock_event.assert_not_called()
+
+ with self.assertRaises(Exception):
+ _calculate_experiment_regular_metric_sync.func( # type: ignore[attr-defined]
+ experiment.id,
+ metric_dict["uuid"],
+ self._queued_key(experiment, metric_dict["uuid"]),
+ attempt=TIMESERIES_METRIC_MAX_ATTEMPTS,
+ )
+ mock_event.assert_called_once()
+
+ @time_machine.travel("2020-01-10T12:00:00Z", tick=False)
+ def test_saved_metric_activity_still_raises_transient_errors(self):
+ feature_flag = self.create_feature_flag()
+ experiment = self.create_experiment(
+ feature_flag=feature_flag,
+ start_date=datetime(2020, 1, 1, 0, 0, 0),
+ end_date=datetime(2020, 1, 5, 0, 0, 0),
+ )
+ metric = ExperimentMeanMetric(uuid=str(uuid4()), source=EventsNode(event="purchase"))
+ metric_dict = metric.model_dump(mode="json")
+
+ saved_metric = ExperimentSavedMetric.objects.create(
+ name="test saved metric",
+ team=self.team,
+ query=metric_dict,
+ created_by=self.user,
+ )
+ ExperimentToSavedMetric.objects.create(
+ experiment=experiment,
+ saved_metric=saved_metric,
+ metadata={"type": "primary"},
+ )
+
+ with (
+ patch("posthog.temporal.experiments.activities.close_old_connections"),
+ self._mocked_runner(experiment, metric_dict["uuid"]) as mock_runner_class,
+ patch("posthog.temporal.experiments.activities.capture_experiment_metric_error_event") as mock_event,
+ ):
+ mock_runner_class.return_value.run.side_effect = Exception("transient query failure")
+
+ with self.assertRaises(Exception):
+ _calculate_experiment_saved_metric_sync.func( # type: ignore[attr-defined]
+ experiment.id, metric_dict["uuid"], self._queued_key(experiment, metric_dict["uuid"]), attempt=1
+ )
+ mock_event.assert_not_called()
+
+ with self.assertRaises(Exception):
+ _calculate_experiment_saved_metric_sync.func( # type: ignore[attr-defined]
+ experiment.id,
+ metric_dict["uuid"],
+ self._queued_key(experiment, metric_dict["uuid"]),
+ attempt=TIMESERIES_METRIC_MAX_ATTEMPTS,
+ )
+ mock_event.assert_called_once()
diff --git a/products/experiments/backend/test/test_experiment_service.py b/products/experiments/backend/test/test_experiment_service.py
index 6bd7df9798fe..3d3fafe4f06b 100644
--- a/products/experiments/backend/test/test_experiment_service.py
+++ b/products/experiments/backend/test/test_experiment_service.py
@@ -12,6 +12,7 @@ import time_machine
from posthog.test.base import APIBaseTest
from unittest.mock import MagicMock, PropertyMock, patch
+from django.db import IntegrityError
from django.db.models import F
from django.test import SimpleTestCase
from django.utils import timezone
@@ -67,9 +68,11 @@ from products.experiments.backend.models.experiment import (
ExperimentMetricResult,
ExperimentSavedMetric,
ExperimentTimeseriesRecalculation,
+ ExperimentToSavedMetric,
)
from products.experiments.backend.models.team_experiments_config import TeamExperimentsConfig
from products.experiments.backend.test.utils import patch_exposure_event_flag
+from products.experiments.backend.timeseries_backfill import backfill_experiment_timeseries
from products.feature_flags.backend.facade.api import set_flag_active, update_flag
from products.feature_flags.backend.models.feature_flag import FeatureFlag
from products.surveys.backend.models import Survey
@@ -5664,43 +5667,109 @@ class TestExperimentService(APIBaseTest):
name="Recalc",
feature_flag_key="ts-recalc",
start_date=timezone.now(),
+ metrics=[self._DEFAULT_METRIC],
+ allow_unknown_events=True,
)
- result = service.request_timeseries_recalculation(
- experiment, metric={"uuid": "m1", "kind": "ExperimentMetric"}, fingerprint="fp1"
- )
+ result = service.request_timeseries_recalculation(experiment, metric_uuid="m1")
assert result["experiment_id"] == experiment.id
assert result["metric_uuid"] == "m1"
assert result["status"] == ExperimentTimeseriesRecalculation.Status.PENDING
assert result["is_existing"] is False
+ request = ExperimentTimeseriesRecalculation.objects.get(id=result["id"])
+ assert (request.fingerprint, request.metric["source"]["event"]) == (
+ _calculation_key(experiment, "m1"),
+ "$pageview",
+ )
- def test_request_timeseries_recalculation_idempotent(self):
- self._create_flag(key="ts-idempotent")
+ @parameterized.expand(
+ [
+ ("same_request", False, False, False),
+ # A backfill under the other key version writes the same days, so a second one would race it.
+ ("backfill_requested_under_the_other_key_version", True, False, False),
+ # A settings edit changes the key, but the running backfill still writes the same days.
+ ("settings_edited_while_the_backfill_runs", False, True, False),
+ # A Temporal retry can start a failed backfill again, beside its replacement under the same key.
+ ("backfill_failed_before_the_request", False, False, True),
+ ]
+ )
+ def test_request_timeseries_recalculation_idempotent(
+ self, _name: str, first_under_other_key_version: bool, edit_settings_after_first: bool, first_failed: bool
+ ) -> None:
+ self._create_flag(key=f"ts-idempotent-{_name}")
service = self._service()
experiment = service.create_experiment(
name="Idempotent",
- feature_flag_key="ts-idempotent",
+ feature_flag_key=f"ts-idempotent-{_name}",
start_date=timezone.now(),
+ metrics=[self._DEFAULT_METRIC],
+ allow_unknown_events=True,
)
+ if first_under_other_key_version:
+ calculation_config = get_metric_calculation_config(experiment, "m1")
+ assert calculation_config is not None
+ first_id = ExperimentTimeseriesRecalculation.objects.create(
+ team=self.team,
+ experiment=experiment,
+ metric=calculation_config.definition,
+ fingerprint=calculation_config.key_for_version(2),
+ status=ExperimentTimeseriesRecalculation.Status.IN_PROGRESS,
+ ).id
+ else:
+ first_id = service.request_timeseries_recalculation(experiment, metric_uuid="m1")["id"]
+ if first_failed:
+ ExperimentTimeseriesRecalculation.objects.filter(id=first_id).update(
+ status=ExperimentTimeseriesRecalculation.Status.FAILED
+ )
+ if edit_settings_after_first:
+ key_before_edit = _calculation_key(experiment, "m1")
+ experiment.stats_config = {**(experiment.stats_config or {}), "method": "frequentist"}
+ experiment.save()
+ assert _calculation_key(experiment, "m1") != key_before_edit
- result1 = service.request_timeseries_recalculation(experiment, metric={"uuid": "m1"}, fingerprint="fp1")
- result2 = service.request_timeseries_recalculation(experiment, metric={"uuid": "m1"}, fingerprint="fp1")
+ second = service.request_timeseries_recalculation(experiment, metric_uuid="m1")
- assert result1["id"] == result2["id"]
- assert result2["is_existing"] is True
+ superseded = first_under_other_key_version or edit_settings_after_first or first_failed
+ assert second["is_existing"] is not superseded
+ assert (second["id"] == first_id) is not superseded
+ assert second["fingerprint"] == _calculation_key(experiment, "m1")
+ assert ExperimentTimeseriesRecalculation.objects.filter(experiment=experiment).count() == (
+ 2 if superseded else 1
+ )
+ if superseded:
+ assert ExperimentTimeseriesRecalculation.objects.get(id=first_id).status == "canceled"
+ retried = backfill_experiment_timeseries(str(first_id))
+ assert (retried["status"], retried["days_processed"]) == ("canceled", 0)
+ assert ExperimentTimeseriesRecalculation.objects.get(id=first_id).status == "canceled"
+ chart = service.get_timeseries_results(experiment, metric_uuid="m1")
+ assert chart["recalculation_status"] is not None
- def test_request_timeseries_recalculation_not_started_raises(self):
- self._create_flag(key="ts-not-started")
+ @parameterized.expand(
+ [("not_started", False, "m1", "hasn't started"), ("unknown_metric", True, "nope", "not on the experiment")]
+ )
+ def test_request_timeseries_recalculation_rejects(
+ self, _name: str, launched: bool, metric_uuid: str, message: str
+ ) -> None:
+ self._create_flag(key=f"ts-{_name}")
service = self._service()
- experiment = service.create_experiment(name="Not Started", feature_flag_key="ts-not-started")
+ experiment = service.create_experiment(
+ name="Rejected",
+ feature_flag_key=f"ts-{_name}",
+ start_date=timezone.now() if launched else None,
+ metrics=[self._DEFAULT_METRIC],
+ allow_unknown_events=True,
+ )
with self.assertRaises(ValidationError) as ctx:
- service.request_timeseries_recalculation(experiment, metric={"uuid": "m1"}, fingerprint="fp1")
+ service.request_timeseries_recalculation(experiment, metric_uuid=metric_uuid)
- assert "hasn't started" in str(ctx.exception)
+ assert message in str(ctx.exception)
- def test_request_timeseries_recalculation_deletes_old_results(self):
+ @parameterized.expand([("created", False), ("creation_failed", True)])
+ def test_request_timeseries_recalculation_deletes_old_results_only_when_created(
+ self, _name: str, creation_fails: bool
+ ) -> None:
self._create_flag(key="ts-delete-old")
service = self._service()
now = timezone.now()
@@ -5708,12 +5777,14 @@ class TestExperimentService(APIBaseTest):
name="Delete Old",
feature_flag_key="ts-delete-old",
start_date=now,
+ metrics=[self._DEFAULT_METRIC],
+ allow_unknown_events=True,
)
ExperimentMetricResult.objects.create(
experiment=experiment,
metric_uuid="m1",
- fingerprint="fp1",
+ fingerprint=_calculation_key(experiment, "m1"),
query_from=now,
query_to=now + timedelta(days=1),
status="completed",
@@ -5722,9 +5793,54 @@ class TestExperimentService(APIBaseTest):
)
assert ExperimentMetricResult.objects.filter(experiment=experiment, metric_uuid="m1").count() == 1
- service.request_timeseries_recalculation(experiment, metric={"uuid": "m1"}, fingerprint="fp1")
+ if creation_fails:
+ with (
+ patch.object(
+ ExperimentTimeseriesRecalculation.objects, "create", side_effect=IntegrityError("insert failed")
+ ),
+ self.assertRaises(IntegrityError),
+ ):
+ service.request_timeseries_recalculation(experiment, metric_uuid="m1")
+ else:
+ service.request_timeseries_recalculation(experiment, metric_uuid="m1")
+
+ assert ExperimentMetricResult.objects.filter(experiment=experiment, metric_uuid="m1").count() == int(
+ creation_fails
+ )
+ assert ExperimentTimeseriesRecalculation.objects.filter(experiment=experiment).count() == int(
+ not creation_fails
+ )
+
+ @parameterized.expand([("inline", False), ("saved", True)])
+ def test_request_timeseries_recalculation_distinguishes_equivalent_metrics(self, _name: str, saved: bool) -> None:
+ self._create_flag(key="ts-equivalent")
+ service = self._service()
+ second_metric = {**self._DEFAULT_METRIC, "uuid": "m2"}
+ experiment = service.create_experiment(
+ name="Equivalent metrics",
+ feature_flag_key="ts-equivalent",
+ start_date=timezone.now(),
+ metrics=[self._DEFAULT_METRIC] if saved else [self._DEFAULT_METRIC, second_metric],
+ allow_unknown_events=True,
+ )
+ if saved:
+ saved_metric = ExperimentSavedMetric.objects.create(
+ team=self.team, name="Equivalent metric", query=second_metric
+ )
+ ExperimentToSavedMetric.objects.create(
+ experiment=experiment, saved_metric=saved_metric, metadata={"type": "secondary"}
+ )
+
+ first = service.request_timeseries_recalculation(experiment, metric_uuid="m1")
+ second = service.request_timeseries_recalculation(experiment, metric_uuid="m2")
- assert ExperimentMetricResult.objects.filter(experiment=experiment, metric_uuid="m1").count() == 0
+ assert first["fingerprint"] == second["fingerprint"]
+ assert first["id"] != second["id"]
+ assert (first["metric_uuid"], second["metric_uuid"]) == ("m1", "m2")
+ assert not second["is_existing"]
+ assert service.request_timeseries_recalculation(experiment, metric_uuid="m1")["id"] == first["id"]
+ assert service.request_timeseries_recalculation(experiment, metric_uuid="m2")["id"] == second["id"]
+ assert ExperimentTimeseriesRecalculation.objects.filter(experiment=experiment).count() == 2
# ------------------------------------------------------------------
# Experiment list/querying
@@ -7962,11 +8078,16 @@ class TestExperimentServiceWarehouseMetricAccess(APIBaseTest):
filters={"groups": [{"properties": [], "rollout_percentage": 100}]},
)
experiment = Experiment.objects.create(
- team=self.team, created_by=self.user, name="E", feature_flag=flag, start_date=timezone.now()
+ team=self.team,
+ created_by=self.user,
+ name="E",
+ feature_flag=flag,
+ start_date=timezone.now(),
+ metrics=[{**self._dw_metric(), "uuid": "dw-metric"}],
)
service = ExperimentService(team=self.team, user=self.user)
with pytest.raises(PermissionDenied):
- service.request_timeseries_recalculation(experiment, metric=self._dw_metric(), fingerprint="fp1")
+ service.request_timeseries_recalculation(experiment, metric_uuid="dw-metric")
assert not ExperimentTimeseriesRecalculation.objects.filter(experiment=experiment).exists()
diff --git a/products/experiments/backend/test/test_presentation_api.py b/products/experiments/backend/test/test_presentation_api.py
index 951104ca9205..c1ba3f1a2f1a 100644
--- a/products/experiments/backend/test/test_presentation_api.py
+++ b/products/experiments/backend/test/test_presentation_api.py
@@ -7,7 +7,7 @@ from uuid import uuid4
import unittest
import time_machine
from posthog.test.base import ClickhouseTestMixin, FuzzyInt, _create_event, _create_person, flush_persons_and_events
-from unittest.mock import ANY, MagicMock, patch
+from unittest.mock import ANY, AsyncMock, MagicMock, patch
from django.core.cache import cache
from django.db import connection
@@ -59,6 +59,7 @@ from products.experiments.backend.models.experiment import (
ExperimentHoldout,
ExperimentMetricResult,
ExperimentSavedMetric,
+ ExperimentTimeseriesRecalculation,
ExperimentToSavedMetric,
)
from products.experiments.backend.models.team_experiments_config import TeamExperimentsConfig
@@ -5676,6 +5677,46 @@ class TestExperimentCRUD(_HoistFlagConfigClientMixin, APILicensedTest):
assert response.status_code == status.HTTP_200_OK, response.content
assert response.json()["timeseries"] == {start.date().isoformat(): {"variant_results": []}}
+ @parameterized.expand([("stale_client_definition", {"fingerprint": "0" * 64}), ("uuid_only", {})])
+ def test_recalculate_timeseries_backfills_the_experiment_definition_under_its_key(
+ self, _name: str, extra_body: dict
+ ) -> None:
+ flag = FeatureFlag.objects.create(team=self.team, created_by=self.user, key="backfill-key-flag")
+ metric = {
+ "kind": "ExperimentMetric",
+ "metric_type": "mean",
+ "uuid": "m1",
+ "source": {"kind": "EventsNode", "event": "$pageview"},
+ }
+ experiment = Experiment.objects.create(
+ team=self.team,
+ created_by=self.user,
+ feature_flag=flag,
+ name="Backfill key",
+ start_date=timezone.now() - timedelta(days=2),
+ metrics=[metric],
+ )
+ posted_metric = (
+ {**metric, "source": {"kind": "EventsNode", "event": "stale_event"}} if extra_body else {"uuid": "m1"}
+ )
+
+ with patch("products.experiments.backend.presentation.views.sync_connect") as mock_connect:
+ mock_connect.return_value.start_workflow = AsyncMock()
+ response = self.client.post(
+ f"/api/projects/{self.team.id}/experiments/{experiment.id}/recalculate_timeseries/",
+ {"metric": posted_metric, **extra_body},
+ format="json",
+ )
+
+ assert response.status_code == status.HTTP_201_CREATED, response.content
+ calculation_config = get_metric_calculation_config(experiment, "m1")
+ assert calculation_config is not None
+ request = ExperimentTimeseriesRecalculation.objects.get(id=response.json()["id"])
+ assert (request.fingerprint, request.metric["source"]["event"]) == (
+ calculation_config.calculation_key(),
+ "$pageview",
+ )
+
def test_creating_draft_experiment_sets_status_draft(self):
response = self.client.post(
f"/api/projects/{self.team.id}/experiments/",
diff --git a/products/experiments/backend/test/test_timeseries_backfill.py b/products/experiments/backend/test/test_timeseries_backfill.py
index 88e6b26192ad..2d404abe9502 100644
--- a/products/experiments/backend/test/test_timeseries_backfill.py
+++ b/products/experiments/backend/test/test_timeseries_backfill.py
@@ -5,10 +5,13 @@ import pytest
from posthog.test.base import BaseTest
from unittest.mock import MagicMock, patch
+from parameterized import parameterized
+
from posthog.schema import ExperimentQueryResponse, ExperimentStatsBaseValidated, ExperimentVariantResultFrequentist
from posthog.models import Organization, Team, User
+from products.experiments.backend.metric_calculation.config import get_metric_calculation_config
from products.experiments.backend.models.experiment import (
Experiment,
ExperimentMetricResult,
@@ -26,24 +29,28 @@ class TestBackfillExperimentTimeseries(BaseTest):
user = User.objects.create(email="[email protected]")
flag = FeatureFlag.objects.create(team=team, key="test-flag", created_by=user)
+ metric_data = {
+ "kind": "ExperimentMetric",
+ "metric_type": "mean",
+ "uuid": "test-metric-uuid",
+ "source": {"kind": "EventsNode", "event": "test_event"},
+ }
experiment = Experiment.objects.create(
name="Test Experiment",
team=team,
feature_flag=flag,
start_date=datetime.datetime(2024, 12, 25, 10, 0, 0, tzinfo=ZoneInfo("UTC")),
end_date=datetime.datetime(2024, 12, 27, 10, 0, 0, tzinfo=ZoneInfo("UTC")),
+ metrics=[metric_data],
)
+ calculation_config = get_metric_calculation_config(experiment, "test-metric-uuid")
+ assert calculation_config is not None
- metric_data = {
- "metric_type": "mean",
- "uuid": "test-metric-uuid",
- "source": {"kind": "EventsNode", "event": "test_event"},
- }
recalculation_request = ExperimentTimeseriesRecalculation.objects.create(
team=team,
experiment=experiment,
metric=metric_data,
- fingerprint="test-fingerprint",
+ fingerprint=calculation_config.calculation_key(),
status=ExperimentTimeseriesRecalculation.Status.PENDING,
)
@@ -58,6 +65,7 @@ class TestBackfillExperimentTimeseries(BaseTest):
with patch("products.experiments.backend.timeseries_backfill.ExperimentQueryRunner") as mock_query_runner_class:
mock_query_runner = MagicMock()
+ mock_query_runner.calculation_config = calculation_config
mock_query_runner._calculate.return_value = mock_result
mock_query_runner_class.return_value = mock_query_runner
@@ -85,6 +93,7 @@ class TestBackfillExperimentTimeseries(BaseTest):
assert metric_result.query_from == experiment.start_date
assert metric_result.status == ExperimentMetricResult.Status.COMPLETED
assert metric_result.result == mock_result.model_dump()
+ assert metric_result.display_key == calculation_config.legacy_key()
assert result["recalculation_id"] == str(recalculation_request.id)
assert result["experiment_id"] == experiment.id
@@ -121,6 +130,13 @@ class TestBackfillExperimentTimeseries(BaseTest):
status=ExperimentTimeseriesRecalculation.Status.PENDING,
)
+ experiment.metrics = [recalculation_request.metric]
+ experiment.save()
+ calculation_config = get_metric_calculation_config(experiment, "test-metric-uuid")
+ assert calculation_config is not None
+ recalculation_request.fingerprint = calculation_config.calculation_key()
+ recalculation_request.save()
+
mock_result = ExperimentQueryResponse(
baseline=ExperimentStatsBaseValidated(key="control", number_of_samples=100, sum=1000, sum_squares=10000),
variant_results=[],
@@ -128,6 +144,7 @@ class TestBackfillExperimentTimeseries(BaseTest):
with patch("products.experiments.backend.timeseries_backfill.ExperimentQueryRunner") as mock_query_runner_class:
mock_query_runner = MagicMock()
+ mock_query_runner.calculation_config = calculation_config
mock_query_runner._calculate.return_value = mock_result
mock_query_runner_class.return_value = mock_query_runner
@@ -140,3 +157,81 @@ class TestBackfillExperimentTimeseries(BaseTest):
recalculation_request.refresh_from_db()
assert recalculation_request.status == ExperimentTimeseriesRecalculation.Status.COMPLETED
assert recalculation_request.last_successful_date == datetime.date(2024, 10, 27)
+
+ @parameterized.expand(
+ [
+ ("before_start", 1),
+ ("before_query", 1),
+ ("between_days", 1),
+ ("superseded_during_query", 1),
+ ("superseded_then_query_fails", 1),
+ ("between_days", 2),
+ ("unchanged", 2),
+ ]
+ )
+ def test_settings_drift_and_supersession_do_not_write_mislabeled_results(
+ self, moment: str, key_version: int
+ ) -> None:
+ # A version 2 request runs on a worker whose new operations use version 1. It keeps its own key version.
+ metric = {"uuid": "m1", "metric_type": "mean", "source": {"kind": "EventsNode", "event": "$pageview"}}
+ flag = FeatureFlag.objects.create(team=self.team, key="drift-flag", created_by=self.user)
+ experiment = Experiment.objects.create(
+ team=self.team,
+ feature_flag=flag,
+ metrics=[metric],
+ start_date=datetime.datetime(2024, 1, 1, tzinfo=ZoneInfo("UTC")),
+ end_date=datetime.datetime(2024, 1, 3, tzinfo=ZoneInfo("UTC")),
+ )
+ config = get_metric_calculation_config(experiment, "m1")
+ assert config is not None
+ key = config.key_for_version(key_version)
+ request = ExperimentTimeseriesRecalculation.objects.create(
+ team=self.team, experiment=experiment, metric=metric, fingerprint=key
+ )
+ result = ExperimentQueryResponse(
+ baseline=ExperimentStatsBaseValidated(key="control", number_of_samples=100, sum=1000, sum_squares=10000),
+ variant_results=[],
+ )
+
+ def change_settings() -> None:
+ # Both key versions hash the stats method.
+ experiment.stats_config = {"method": "frequentist"}
+ experiment.save()
+
+ def calculate() -> ExperimentQueryResponse:
+ if moment == "between_days":
+ change_settings()
+ if moment.startswith("superseded"):
+ ExperimentTimeseriesRecalculation.objects.filter(id=request.id).update(status="canceled")
+ if moment == "superseded_then_query_fails":
+ raise RuntimeError("query failed")
+ return result
+
+ def runner(**kwargs):
+ if moment == "before_query":
+ change_settings()
+ mock = MagicMock()
+ mock.calculation_config = get_metric_calculation_config(Experiment.objects.get(id=experiment.id), "m1")
+ mock._calculate.side_effect = calculate
+ return mock
+
+ if moment == "before_start":
+ change_settings()
+ with patch("products.experiments.backend.timeseries_backfill.ExperimentQueryRunner", side_effect=runner):
+ if moment == "superseded_then_query_fails":
+ with pytest.raises(RuntimeError):
+ backfill_experiment_timeseries(str(request.id))
+ output = {"days_processed": 0}
+ else:
+ output = backfill_experiment_timeseries(str(request.id))
+ # A Temporal retry of the activity must not start a canceled request again.
+ retried = backfill_experiment_timeseries(str(request.id)) if moment != "unchanged" else None
+ request.refresh_from_db()
+ assert request.status == ("completed" if moment == "unchanged" else "canceled")
+ if retried is not None:
+ assert retried["days_processed"] == 0
+ written_days = {"between_days": 1, "unchanged": 3}.get(moment, 0)
+ assert output["days_processed"] == written_days
+ rows = ExperimentMetricResult.objects.filter(experiment=experiment)
+ assert rows.count() == written_days
+ assert {(row.fingerprint, row.display_key) for row in rows} <= {(key, config.legacy_key())}
diff --git a/products/experiments/backend/timeseries_backfill.py b/products/experiments/backend/timeseries_backfill.py
index 4ff2f97f128c..e1f93e92d218 100644
--- a/products/experiments/backend/timeseries_backfill.py
+++ b/products/experiments/backend/timeseries_backfill.py
@@ -10,6 +10,8 @@ from datetime import date, datetime, time, timedelta
from typing import Any
from zoneinfo import ZoneInfo
+from django.db import transaction
+
import structlog
from posthog.schema import ExperimentQuery
@@ -18,9 +20,10 @@ from posthog.clickhouse.client.connection import Workload
from products.experiments.backend.hogql_queries.experiment_query_runner import ExperimentQueryRunner
from products.experiments.backend.hogql_queries.utils import sanitize_non_finite
+from products.experiments.backend.metric_calculation.config import find_calculation_config_by_key
from products.experiments.backend.metric_calculation.results import MetricResultStore
from products.experiments.backend.metric_resolution import build_metric
-from products.experiments.backend.models.experiment import ExperimentTimeseriesRecalculation
+from products.experiments.backend.models.experiment import Experiment, ExperimentTimeseriesRecalculation
logger = structlog.get_logger(__name__)
@@ -40,12 +43,21 @@ def backfill_experiment_timeseries(recalculation_id: str, *, backfill_until: dat
except ExperimentTimeseriesRecalculation.DoesNotExist:
raise ValueError(f"Recalculation request {recalculation_id} not found")
+ ExperimentTimeseriesRecalculation.objects.filter(
+ id=recalculation_id,
+ status__in=[ExperimentTimeseriesRecalculation.Status.PENDING, ExperimentTimeseriesRecalculation.Status.FAILED],
+ ).update(status=ExperimentTimeseriesRecalculation.Status.IN_PROGRESS)
+ recalculation_request.refresh_from_db()
+
if recalculation_request.status in (
- ExperimentTimeseriesRecalculation.Status.PENDING,
- ExperimentTimeseriesRecalculation.Status.FAILED,
+ ExperimentTimeseriesRecalculation.Status.CANCELED,
+ ExperimentTimeseriesRecalculation.Status.COMPLETED,
):
- recalculation_request.status = ExperimentTimeseriesRecalculation.Status.IN_PROGRESS
- recalculation_request.save(update_fields=["status"])
+ return {
+ "recalculation_id": str(recalculation_request.id),
+ "days_processed": 0,
+ "status": recalculation_request.status,
+ }
experiment = recalculation_request.experiment
if not experiment.start_date:
@@ -72,9 +84,17 @@ def backfill_experiment_timeseries(recalculation_id: str, *, backfill_until: dat
metric_obj = build_metric(recalculation_request.metric)
experiment_query = ExperimentQuery(experiment_id=experiment.id, metric=metric_obj)
fingerprint = recalculation_request.fingerprint
+ calculation_config = find_calculation_config_by_key(
+ experiment, recalculation_request.metric.get("uuid") or "", fingerprint
+ )
results = MetricResultStore(experiment_id=experiment.id)
days_processed = 0
+ if calculation_config is None:
+ ExperimentTimeseriesRecalculation.objects.filter(id=recalculation_id).update(
+ status=ExperimentTimeseriesRecalculation.Status.CANCELED
+ )
+ return {"recalculation_id": str(recalculation_request.id), "days_processed": 0, "status": "canceled"}
while current_date <= end_date:
try:
@@ -83,6 +103,7 @@ def backfill_experiment_timeseries(recalculation_id: str, *, backfill_until: dat
)
query_to_utc = end_of_day_team_tz.astimezone(ZoneInfo("UTC"))
+ experiment.team.refresh_from_db()
query_runner = ExperimentQueryRunner(
query=experiment_query,
team=experiment.team,
@@ -94,19 +115,37 @@ def backfill_experiment_timeseries(recalculation_id: str, *, backfill_until: dat
# Backfilling historical points is not user-visible pain — no terminal error event.
error_event_context=None,
)
- result = query_runner._calculate()
-
- results.record_daily_point(
- recalculation_request.metric["uuid"],
- fingerprint,
- window=query_to_utc,
- query_from=experiment.start_date,
- result=sanitize_non_finite(result.model_dump()),
+ runtime_config = query_runner.calculation_config.settings.build_metric_config(
+ metric_id=calculation_config.metric_id,
+ role=calculation_config.role,
+ definition=recalculation_request.metric,
)
+ if runtime_config.key_for_version(calculation_config.settings.key_version) != fingerprint:
+ ExperimentTimeseriesRecalculation.objects.filter(id=recalculation_id).update(
+ status=ExperimentTimeseriesRecalculation.Status.CANCELED
+ )
+ break
+ result = query_runner._calculate()
- recalculation_request.last_successful_date = current_date
- recalculation_request.save(update_fields=["last_successful_date"])
- days_processed += 1
+ with transaction.atomic():
+ Experiment.objects.select_for_update(no_key=True).filter(
+ id=experiment.id, team_id=experiment.team_id
+ ).exists()
+ current_request = ExperimentTimeseriesRecalculation.objects.select_for_update().get(id=recalculation_id)
+ if current_request.status != ExperimentTimeseriesRecalculation.Status.IN_PROGRESS:
+ break
+ results.record_daily_point(
+ recalculation_request.metric["uuid"],
+ fingerprint,
+ display_key=calculation_config.legacy_key(),
+ window=query_to_utc,
+ query_from=experiment.start_date,
+ result=sanitize_non_finite(result.model_dump()),
+ )
+
+ recalculation_request.last_successful_date = current_date
+ recalculation_request.save(update_fields=["last_successful_date"])
+ days_processed += 1
except Exception:
logger.exception(
@@ -114,17 +153,21 @@ def backfill_experiment_timeseries(recalculation_id: str, *, backfill_until: dat
recalculation_id=recalculation_id,
failed_date=str(current_date),
)
- recalculation_request.status = ExperimentTimeseriesRecalculation.Status.FAILED
- recalculation_request.save(update_fields=["status"])
+ ExperimentTimeseriesRecalculation.objects.filter(
+ id=recalculation_id, status=ExperimentTimeseriesRecalculation.Status.IN_PROGRESS
+ ).update(status=ExperimentTimeseriesRecalculation.Status.FAILED)
raise
current_date += timedelta(days=1)
- recalculation_request.status = ExperimentTimeseriesRecalculation.Status.COMPLETED
- recalculation_request.save(update_fields=["status"])
+ ExperimentTimeseriesRecalculation.objects.filter(
+ id=recalculation_id, status=ExperimentTimeseriesRecalculation.Status.IN_PROGRESS
+ ).update(status=ExperimentTimeseriesRecalculation.Status.COMPLETED)
+ recalculation_request.refresh_from_db(fields=["status"])
logger.info(
- "Timeseries recalculation completed",
+ "Timeseries recalculation finished",
+ status=recalculation_request.status,
recalculation_id=recalculation_id,
days_processed=days_processed,
)
@@ -136,4 +179,5 @@ def backfill_experiment_timeseries(recalculation_id: str, *, backfill_until: dat
"days_processed": days_processed,
"start_date": start_date.isoformat(),
"end_date": end_date.isoformat(),
+ "status": recalculation_request.status,
}