:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(experiments): rebuild recent precompute windows less often (#130758)
Juraj Majerik716d7064b9f9
diff --git a/products/experiments/backend/hogql_queries/LAZY_COMPUTATION.md b/products/experiments/backend/hogql_queries/LAZY_COMPUTATION.mdindex 9bb6dad6873a..64bfd6cbeb8c 100644--- a/products/experiments/backend/hogql_queries/LAZY_COMPUTATION.md+++ b/products/experiments/backend/hogql_queries/LAZY_COMPUTATION.md@@ -259,7 +259,7 @@ GROUP BY variant2. **Query hash determines cache sharing**: Different feature flags, variants, or exposure criteria produce different hashes. Each experiment typically has its own lazy-computed data.-3. **TTL and expiration**: Jobs use variable TTL based on data age (current day: 15 min, yesterday: 1 hour, 2-4 days ago: 18 hours, 5+ days: 60 days). The 2-4 day band stays recomputable so the daily warmer folds in late-arriving exposure events before a window freezes — a frozen window keeps a stale `first_exposure_time` if an earlier exposure event arrives after it was computed. This avoids recomputing genuinely frozen historical data while still re-folding the late-arrival tail. The system ignores jobs expiring within 1 hour to avoid race conditions. ClickHouse TTL automatically deletes expired rows.+3. **TTL and expiration**: Jobs use variable TTL based on data age (current day: 30 min, yesterday: 6 hours, 2-4 days ago: 18 hours, 5+ days: 60 days). The 2-4 day band stays recomputable so the daily warmer folds in late-arriving exposure events before a window freezes — a frozen window keeps a stale `first_exposure_time` if an earlier exposure event arrives after it was computed. This avoids recomputing genuinely frozen historical data while still re-folding the late-arrival tail. The system ignores jobs expiring within 1 hour to avoid race conditions. ClickHouse TTL automatically deletes expired rows.4. **Job width cap**: Missing ranges are split into jobs of at most `PRECOMPUTE_MAX_WINDOW_DAYS` (7) days via `TtlSchedule.max_window_days` (see `experiment_precompute_ttl_schedule`). Without the cap, the frozen band (everything older than 4 days shares one TTL) merges into a single INSERT, so a cold or expired long-running experiment would scan its entire history in one query and hit the per-query bytes-to-read cap, timeout, or memory limit on high-volume teams. Completed chunk jobs persist even when a later chunk fails, so wide backfills converge across successive runs.diff --git a/products/experiments/backend/hogql_queries/precompute_policy.py b/products/experiments/backend/hogql_queries/precompute_policy.pyindex 2f0de135186d..2b827aba8d80 100644--- a/products/experiments/backend/hogql_queries/precompute_policy.py+++ b/products/experiments/backend/hogql_queries/precompute_policy.py@@ -30,9 +30,19 @@ from products.experiments.backend.models.team_experiments_config import TeamExpe# recomputable (18h, just under the ~24h warmer cadence) so the daily warmer# folds in late-arriving exposure events before a window freezes. Freezing too# early keeps a stale first_exposure_time and undercounts the metric window.+#+# A closed day changes only through late-arriving events and person merges, so+# yesterday's window needs far fewer rebuilds than today's. A run rebuilds every+# expired window before it reads it, and each rebuild scans raw events once per+# metric. A TTL shorter than the usual gap between runs makes every run pay for+# that scan.+#+# Exposures and metric events share this schedule. If exposures refreshed more+# often than metric events, a run would count newly exposed users without their+# conversions.DEFAULT_EXPOSURE_TTL_SECONDS = {- "0d": 15 * 60, # 15 min- "1d": 60 * 60, # 1 hour+ "0d": 30 * 60, # 30 min+ "1d": 6 * 60 * 60, # 6 hours"4d": 18 * 60 * 60, # 18 hours; covers windows 2-4 days old"default": 60 * 24 * 60 * 60, # 60 days - data frozen}@@ -94,7 +104,7 @@ def experiment_precompute_ttl_schedule(team_timezone: str) -> TtlSchedule:# Minimum experiment runtime before auto-enabling precomputation. The-# today-window TTL (DEFAULT_EXPOSURE_TTL_SECONDS["0d"], 15 min) caches the+# today-window TTL (DEFAULT_EXPOSURE_TTL_SECONDS["0d"], 30 min) caches the# entire current-day result, so for very young experiments freshly arriving# exposures stay invisible until that TTL elapses. The gate scopes that# trade-off to experiments where most of the data is already in past@@ -102,7 +112,7 @@ def experiment_precompute_ttl_schedule(team_timezone: str) -> TtlSchedule:# query override.## Why 12h: it covers the workday in which users launch experiments. Users-# refresh the results page most often then, so the 15-min cache lag is most+# refresh the results page most often then, so that cache lag is most# noticeable. After that, refreshes drop off and the cost saving from# precomputation matters more than the freshness gap. 12h is also short enough# that experiments hit the precomputed (fast) path on day two.diff --git a/products/experiments/backend/replay_linkage.py b/products/experiments/backend/replay_linkage.pyindex 087156c51316..eee68ed4b027 100644--- a/products/experiments/backend/replay_linkage.py+++ b/products/experiments/backend/replay_linkage.py@@ -482,8 +482,8 @@ def _resolve_exposure_read(team: Team, experiment: Experiment, context: Experime"""config = get_or_create_team_extension(team, TeamExperimentsConfig)# Below the minimum runtime the analysis skips precomputation too: the scan window is- # hours wide (cheap on any team), and the current-day cache TTL would hide exposures- # from the tab for up to 15 minutes right when users watch it most.+ # hours wide (cheap on any team), and the current-day cache TTL would hide new+ # exposures from the tab until it expires, right when users watch it most.if not _precomputation_covers_full_window(config, experiment):tag_queries(experiment_exposures_path="direct_scan")return _ExposureRead(preaggregation_job_ids=None, live_scan_max_memory_bytes=None)diff --git a/products/experiments/skills/diagnosing-experiment-health/references/empty-experiment.md b/products/experiments/skills/diagnosing-experiment-health/references/empty-experiment.mdindex 814bfd23f8ef..1cd707f805b5 100644--- a/products/experiments/skills/diagnosing-experiment-health/references/empty-experiment.md+++ b/products/experiments/skills/diagnosing-experiment-health/references/empty-experiment.md@@ -84,8 +84,8 @@ Check directly, in this order, and ask only what the tools cannot answer:- Stored results may not exist yet for a fresh experiment.A read of the stored results that returns nothing in the first hours is expected.The scheduled calculation skips an experiment younger than 12 hours or with fewer than 50 exposures in total, so an empty experiment has stored results only when a calculation was started another way, for example from the experiment page.-- In a project with precomputed exposures, an experiment older than 12 hours can lag today's new exposures by up to 15 minutes.- The total is 0 because of that only when every exposure is younger than 15 minutes.+- In a project with precomputed exposures, an experiment older than 12 hours can lag today's new exposures by up to 30 minutes.+ The total is 0 because of that only when every exposure is younger than 30 minutes.<!-- Source for maintainers (may rot): MIN_PRECOMPUTATION_DURATION_SECONDS and DEFAULT_EXPOSURE_TTL_SECONDS inproducts/experiments/backend/hogql_queries/precompute_policy.py; ExperimentResultsCacheMixin in