Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(web-analytics): monitor dimensional precompute coverage and run it every 20 minutes (#119943)

Lucas Ricoy4aed61c0a025
diff --git a/products/web_analytics/PRECOMPUTATION.md b/products/web_analytics/PRECOMPUTATION.md
index 698de95f0dc6..94276bdf40de 100644
--- a/products/web_analytics/PRECOMPUTATION.md
+++ b/products/web_analytics/PRECOMPUTATION.md
@@ -32,14 +32,19 @@ Two ClickHouse tables carry the same fixed dimension set as v2 (`host`, `device_
Both are `ReplacingMergeTree` (version column `computed_at`), sharded by `sipHash64(job_id)` on the AUX cluster, partitioned by `toYYYYMMDD(expires_at)`.
- **Owned by**: web analytics team, on the analytics_platform framework.
-- **Population**: the hourly `web_dimensional_precompute` Dagster job (`products/web_analytics/dags/web_dimensional_precompute.py`).
- For each team on a region-scoped allowlist it drives `ensure_precomputed` over a rolling 90-day window, chunked to one UTC day per INSERT.
+- **Population**: the `web_dimensional_precompute_job` Dagster job, every 20 minutes (`products/web_analytics/dags/web_dimensional_precompute.py`).
+ For each team on a region-scoped allowlist it drives `ensure_precomputed` over a rolling 92-day window, chunked to one UTC day per INSERT.
+ 92 UTC days hold the start of a `-90d` read in every team timezone.
Every window is a `PreaggregationJob` (Postgres) with a `job_id` and a TTL by age (today 1h, ≤2 days 1 day, older 90 days); re-runs recompute only expired windows.
+ The TTL bands start at UTC midnight, like the windows, so the current UTC day keeps its 1h TTL in every team timezone.
The allowlist defaults per Cloud region (`DEFAULT_ROLLOUT_TEAM_IDS_BY_REGION`) and is overridable via `WEB_DIMENSIONAL_PRECOMPUTE_TEAM_IDS`; an empty list disables the job.
The inserts carry the legacy job's memory and time ceilings, because one UTC day of a very large team outgrew the interactive defaults (#117339).
They also narrow HogQL's 3-day session lookback to 25 hours, after the wider read ran the shared sessions node out of memory (#117872).
The job has no stale grace or forced refresh of its own: today's window rebuilds only when a run finds its job expired.
- So the today TTL must be clearly shorter than the cron period, or a run skips a still-fresh window that then expires before the next run. Today the two are equal.
+ So today's data can lag by up to the TTL plus one cron period (20 minutes).
+- **Monitoring**: after the writers, a second op reads the READY jobs back for each team and table, merges their intervals, and fails the run when a UTC day in the writers' window has no fresh job or the newest job for the writers' current day is older than the TTL plus two runs. The check takes the window and the current day from the writers, so a run that crosses UTC midnight does not report the new day as missing.
+ A failed run alerts `#alerts-web-analytics` through `notify_slack_on_failure`, which routes on the job's `owner` tag.
+ The op also pushes `web_dimensional_precompute_coverage_missing_days` and `web_dimensional_precompute_today_lag_seconds` gauges, labeled by `team_id` and `table`, to the Prometheus push gateway.
- **Coverage**: the same stats-breakdown and overview shapes v2 serves — about 45% of user-facing web-analytics reads by shape (stats breakdowns on the stored dimensions, plus overview).
Not goals, vitals, frustration metrics, external clicks, or the other engines (Trends / Retention / HogQL).
Its differentiator over lazy: arbitrary multi-dimension filters within the fixed set resolve from one table, with no per-filter-combination warming.
diff --git a/products/web_analytics/backend/hogql_queries/test/test_web_dimensional_precompute.py b/products/web_analytics/backend/hogql_queries/test/test_web_dimensional_precompute.py
index 68e072be1cd5..30d4bd9f8325 100644
--- a/products/web_analytics/backend/hogql_queries/test/test_web_dimensional_precompute.py
+++ b/products/web_analytics/backend/hogql_queries/test/test_web_dimensional_precompute.py
@@ -6,6 +6,8 @@ from posthog.test.base import APIBaseTest, ClickhouseTestMixin, _create_event, _
from django.test import override_settings
+from parameterized import parameterized
+
from posthog.hogql import ast
from posthog.hogql.parser import parse_select
@@ -138,6 +140,22 @@ class TestWebDimensionalPrecompute(ClickhouseTestMixin, APIBaseTest):
assert jobs_after_second == jobs_after_first
assert set(second.job_ids) == set(first.job_ids)
+ @parameterized.expand([("UTC",), ("Pacific/Honolulu",), ("Asia/Tokyo",)])
+ def test_current_utc_day_refreshes_hourly_in_any_team_timezone(self, timezone: str):
+ # 16:00 UTC is after local midnight in both non-UTC zones, so team-timezone TTL bands would
+ # put the current UTC day in the 1-day band and skip the second build.
+ self.team.timezone = timezone
+ self.team.save()
+ today_start, today_end = datetime(2024, 1, 15, tzinfo=UTC), datetime(2024, 1, 16, tzinfo=UTC)
+
+ with time_machine.travel("2024-01-15T16:00:00Z", tick=False):
+ first = ensure_web_stats_dimensional_precomputed(self.team, today_start, today_end)
+ with time_machine.travel("2024-01-15T17:01:00Z", tick=False):
+ second = ensure_web_stats_dimensional_precomputed(self.team, today_start, today_end)
+
+ assert first.ready and second.ready
+ assert set(second.job_ids).isdisjoint(first.job_ids)
+
@time_machine.travel("2024-01-15T12:00:00Z", tick=False)
def test_bounces_ensure_creates_jobs_and_correct_totals(self):
self._seed_two_sessions()
diff --git a/products/web_analytics/backend/hogql_queries/web_dimensional_precompute.py b/products/web_analytics/backend/hogql_queries/web_dimensional_precompute.py
index c02f324dcc83..6b35231551a4 100644
--- a/products/web_analytics/backend/hogql_queries/web_dimensional_precompute.py
+++ b/products/web_analytics/backend/hogql_queries/web_dimensional_precompute.py
@@ -36,7 +36,9 @@ from posthog.models.web_preaggregated.sql import is_eu_cluster
from products.analytics_platform.backend.lazy_computation.lazy_computation_executor import (
LazyComputationResult,
LazyComputationTable,
+ TtlSchedule,
ensure_precomputed,
+ parse_ttl_schedule,
)
from products.web_analytics.backend.hogql_queries.web_analytics_lazy_precompute import SESSION_FORWARD_PAD_MINUTES
@@ -51,6 +53,14 @@ DIMENSIONAL_TTL_SECONDS: dict[str, int] = {
}
+def dimensional_ttl_schedule() -> TtlSchedule:
+ # The jobs cover UTC days, so the TTL bands must start at UTC midnight too. With bands in the
+ # team timezone, the current UTC day moves into the 1-day band when the team's local day
+ # starts, and that day then stops refreshing until the next UTC midnight. The band cutoffs are
+ # relative to now, so build the schedule per call.
+ return parse_ttl_schedule(DIMENSIONAL_TTL_SECONDS, "UTC")
+
+
# Per-session row, then aggregated into hourly buckets. `host`/`device_type`/
# `pathname`/`browser`/`os`/`viewport_*`/geoip are event-level group keys; the
# remaining dimensions are session-level and pulled via `any(session.…)`.
@@ -342,7 +352,7 @@ def ensure_web_stats_dimensional_precomputed(
insert_query=STATS_INSERT_TEMPLATE,
time_range_start=time_range_start,
time_range_end=time_range_end,
- ttl_seconds=DIMENSIONAL_TTL_SECONDS,
+ ttl_seconds=dimensional_ttl_schedule(),
table=LazyComputationTable.WEB_STATS_DIMENSIONAL_PREAGGREGATED,
placeholders=_base_placeholders(),
query_type="web_stats_dimensional_insert",
@@ -362,7 +372,7 @@ def ensure_web_bounces_dimensional_precomputed(
insert_query=BOUNCES_INSERT_TEMPLATE,
time_range_start=time_range_start,
time_range_end=time_range_end,
- ttl_seconds=DIMENSIONAL_TTL_SECONDS,
+ ttl_seconds=dimensional_ttl_schedule(),
table=LazyComputationTable.WEB_BOUNCES_DIMENSIONAL_PREAGGREGATED,
placeholders=_base_placeholders(),
query_type="web_bounces_dimensional_insert",
diff --git a/products/web_analytics/dags/tests/test_web_dimensional_precompute.py b/products/web_analytics/dags/tests/test_web_dimensional_precompute.py
index 283b4947f473..95020dcad7aa 100644
--- a/products/web_analytics/dags/tests/test_web_dimensional_precompute.py
+++ b/products/web_analytics/dags/tests/test_web_dimensional_precompute.py
@@ -1,26 +1,64 @@
import os
-from datetime import UTC, datetime
+import uuid
+from datetime import UTC, date, datetime, time, timedelta
+from zoneinfo import ZoneInfo
+import pytest
import time_machine
from posthog.test.base import APIBaseTest
from unittest.mock import patch
+from django.test import SimpleTestCase
+
import dagster
from parameterized import parameterized
from posthog.models import Organization, Team
+from products.analytics_platform.backend.lazy_computation.lazy_computation_executor import LazyComputationResult
+from products.analytics_platform.backend.models.preaggregation_job import PreaggregationJob
from products.web_analytics.dags.web_dimensional_precompute import (
+ BOUNCES_TABLE,
DEFAULT_ROLLOUT_TEAM_IDS_BY_REGION,
PRECOMPUTE_CHUNK_DAYS,
PRECOMPUTE_WINDOW_DAYS,
SELECTED_TEAM_IDS_ENV_VAR,
+ STATS_TABLE,
+ DimensionalPrecomputeRun,
+ TimeSpan,
+ check_web_dimensional_precompute_coverage_op,
chunk_ranges,
ensure_web_dimensional_precompute_op,
get_selected_team_ids,
+ precompute_window,
+ summarize_coverage,
web_dimensional_precompute_job,
)
+_READY = LazyComputationResult(ready=True, job_ids=[uuid.uuid4()])
+_NOT_READY = LazyComputationResult(ready=False, job_ids=[], errors=["Max retries (1) exceeded: boom"])
+
+_NOW = datetime(2026, 10, 19, 12, 0, tzinfo=UTC)
+_TODAY = datetime(2026, 10, 19, tzinfo=UTC)
+
+
+def _day(offset: int) -> datetime:
+ return _TODAY + timedelta(days=offset)
+
+
+def _ready_job(start: int, end: int, built_minutes_ago: int, ttl_minutes: int) -> PreaggregationJob:
+ built_at = _NOW - timedelta(minutes=built_minutes_ago)
+ return PreaggregationJob(
+ team_id=1,
+ time_range_start=_day(start),
+ time_range_end=_day(end),
+ query_hash="hash",
+ status=PreaggregationJob.Status.READY,
+ computed_at=built_at,
+ expires_at=built_at + timedelta(minutes=ttl_minutes),
+ )
+
+
_IS_CLOUD = "products.web_analytics.dags.web_dimensional_precompute.is_cloud"
_CLOUD_DEPLOYMENT = "products.web_analytics.dags.web_dimensional_precompute.settings.CLOUD_DEPLOYMENT"
@@ -51,6 +89,70 @@ class TestChunkRanges:
assert PRECOMPUTE_CHUNK_DAYS == 1
+class TestPrecomputeWindow:
+ @pytest.mark.parametrize(
+ "timezone", ["UTC", "Etc/GMT+12", "America/Los_Angeles", "Asia/Tokyo", "Pacific/Kiritimati"]
+ )
+ @pytest.mark.parametrize("utc_hour", [0, 5, 11, 13, 23])
+ def test_window_holds_a_minus_90d_read_in_any_timezone(self, timezone: str, utc_hour: int):
+ now = datetime(2026, 10, 19, utc_hour, 30, tzinfo=UTC)
+ tz = ZoneInfo(timezone)
+ local_today = now.astimezone(tz).date()
+ read_start = datetime.combine(local_today - timedelta(days=90), time(), tzinfo=tz)
+
+ window = precompute_window(now)
+
+ assert window.start <= read_start
+ assert window.end > now
+ assert window.end == window.start + timedelta(days=PRECOMPUTE_WINDOW_DAYS)
+
+
+class TestSummarizeCoverage(SimpleTestCase):
+ @parameterized.expand(
+ [
+ (
+ "two_day_job_covers_both_days",
+ [_ready_job(-3, -1, 600, 90 * 24 * 60), _ready_job(-1, 0, 600, 24 * 60), _ready_job(0, 1, 10, 60)],
+ (),
+ 10 * 60,
+ ),
+ (
+ "gap_reports_the_missing_day",
+ [_ready_job(-3, -2, 600, 90 * 24 * 60), _ready_job(-1, 0, 600, 24 * 60), _ready_job(0, 1, 10, 60)],
+ (date(2026, 10, 17),),
+ 10 * 60,
+ ),
+ (
+ "expired_job_outside_the_grace_is_missing",
+ [_ready_job(-3, -1, 600, 90 * 24 * 60), _ready_job(-1, 0, 48 * 60, 24 * 60), _ready_job(0, 1, 10, 60)],
+ (date(2026, 10, 18),),
+ 10 * 60,
+ ),
+ (
+ "stale_today_reports_lag",
+ [_ready_job(-3, 0, 600, 90 * 24 * 60), _ready_job(0, 1, 180, 60), _ready_job(0, 1, 240, 60)],
+ (date(2026, 10, 19),),
+ 180 * 60,
+ ),
+ ("no_jobs", [], (date(2026, 10, 16), date(2026, 10, 17), date(2026, 10, 18), date(2026, 10, 19)), None),
+ ]
+ )
+ def test_reports_missing_days_and_today_lag(self, _name, jobs, expected_missing, expected_lag):
+ coverage = summarize_coverage(
+ team_id=1,
+ table=STATS_TABLE,
+ ready_jobs=jobs,
+ window=TimeSpan(start=_day(-3), end=_day(1)),
+ current_day=_NOW.date(),
+ now=_NOW,
+ )
+
+ assert coverage.missing_days == expected_missing
+ assert coverage.today_lag_seconds == expected_lag
+ assert coverage.covered_days == 4 - len(expected_missing)
+ assert coverage.healthy == (not expected_missing and expected_lag is not None)
+
+
class TestGetSelectedTeamIds:
def test_env_override_parses_comma_separated_ids(self):
with patch.dict(os.environ, {SELECTED_TEAM_IDS_ENV_VAR: "2, 47074 ,55348"}):
@@ -99,8 +201,14 @@ class TestEnsureWebDimensionalPrecomputeOp(APIBaseTest):
org = Organization.objects.create(name=name)
return Team.objects.create(organization=org, name=f"{name}-team")
- @patch("products.web_analytics.dags.web_dimensional_precompute.ensure_web_bounces_dimensional_precomputed")
- @patch("products.web_analytics.dags.web_dimensional_precompute.ensure_web_stats_dimensional_precomputed")
+ @patch(
+ "products.web_analytics.dags.web_dimensional_precompute.ensure_web_bounces_dimensional_precomputed",
+ return_value=_READY,
+ )
+ @patch(
+ "products.web_analytics.dags.web_dimensional_precompute.ensure_web_stats_dimensional_precomputed",
+ return_value=_READY,
+ )
@patch(_SINGLE_CHUNK, PRECOMPUTE_WINDOW_DAYS) # one chunk → one ensure call per team/table
def test_drives_both_tables_for_every_selected_team(self, stats_mock, bounces_mock):
t1 = self._make_team("A")
@@ -108,7 +216,13 @@ class TestEnsureWebDimensionalPrecomputeOp(APIBaseTest):
with patch.dict(os.environ, {SELECTED_TEAM_IDS_ENV_VAR: f"{t1.pk},{t2.pk}"}):
result = ensure_web_dimensional_precompute_op(dagster.build_op_context())
- assert result == {"teams": 2, "failures": 0}
+ assert (result.teams, result.failures, result.team_ids) == (2, 0, (t1.pk, t2.pk))
+ assert set(result.sample_job_ids) == {
+ (t1.pk, STATS_TABLE),
+ (t1.pk, BOUNCES_TABLE),
+ (t2.pk, STATS_TABLE),
+ (t2.pk, BOUNCES_TABLE),
+ }
assert stats_mock.call_count == 2
assert bounces_mock.call_count == 2
warmed_teams = {call.args[0].pk for call in stats_mock.call_args_list}
@@ -116,8 +230,14 @@ class TestEnsureWebDimensionalPrecomputeOp(APIBaseTest):
start, end = stats_mock.call_args_list[0].args[1], stats_mock.call_args_list[0].args[2]
assert (end - start).days == PRECOMPUTE_WINDOW_DAYS
- @patch("products.web_analytics.dags.web_dimensional_precompute.ensure_web_bounces_dimensional_precomputed")
- @patch("products.web_analytics.dags.web_dimensional_precompute.ensure_web_stats_dimensional_precomputed")
+ @patch(
+ "products.web_analytics.dags.web_dimensional_precompute.ensure_web_bounces_dimensional_precomputed",
+ return_value=_READY,
+ )
+ @patch(
+ "products.web_analytics.dags.web_dimensional_precompute.ensure_web_stats_dimensional_precomputed",
+ return_value=_READY,
+ )
def test_chunking_issues_multiple_bounded_calls(self, stats_mock, bounces_mock):
# Chunking bounds each ensure call to <= chunk_days so no single INSERT
# scans the whole window. Exercised here with a 7-day chunk for clarity.
@@ -135,46 +255,135 @@ class TestEnsureWebDimensionalPrecomputeOp(APIBaseTest):
# INSERT, so every boundary must sit on a UTC midnight even when the run starts mid-day.
assert all(b == b.replace(hour=0, minute=0, second=0, microsecond=0) for chunk in chunks for b in chunk)
- @patch("products.web_analytics.dags.web_dimensional_precompute.ensure_web_bounces_dimensional_precomputed")
+ @parameterized.expand([("raises", RuntimeError("boom")), ("returns_not_ready", _NOT_READY)])
+ @patch(
+ "products.web_analytics.dags.web_dimensional_precompute.ensure_web_bounces_dimensional_precomputed",
+ return_value=_READY,
+ )
@patch("products.web_analytics.dags.web_dimensional_precompute.ensure_web_stats_dimensional_precomputed")
@patch(_SINGLE_CHUNK, PRECOMPUTE_WINDOW_DAYS)
- def test_one_team_failure_does_not_poison_others(self, stats_mock, bounces_mock):
+ def test_one_team_failure_does_not_poison_others(self, _name, failure, stats_mock, bounces_mock):
t1 = self._make_team("A")
t2 = self._make_team("B")
def stats_side_effect(team, start, end):
- if team.pk == t1.pk:
- raise RuntimeError("boom")
+ if team.pk != t1.pk:
+ return _READY
+ if isinstance(failure, Exception):
+ raise failure
+ return failure
stats_mock.side_effect = stats_side_effect
with patch.dict(os.environ, {SELECTED_TEAM_IDS_ENV_VAR: f"{t1.pk},{t2.pk}"}):
result = ensure_web_dimensional_precompute_op(dagster.build_op_context())
- assert result["teams"] == 2
- assert result["failures"] == 1
+ assert (result.teams, result.failures) == (2, 1)
+ assert (t1.pk, STATS_TABLE) not in result.sample_job_ids
+ assert (t2.pk, STATS_TABLE) in result.sample_job_ids
assert stats_mock.call_count == 2
assert bounces_mock.call_count == 2
+ @parameterized.expand([("empty_allowlist", ""), ("missing_team", "999999999")])
@patch("products.web_analytics.dags.web_dimensional_precompute.ensure_web_bounces_dimensional_precomputed")
@patch("products.web_analytics.dags.web_dimensional_precompute.ensure_web_stats_dimensional_precomputed")
- def test_empty_allowlist_is_a_noop(self, stats_mock, bounces_mock):
- with patch.dict(os.environ, {SELECTED_TEAM_IDS_ENV_VAR: ""}):
- result = ensure_web_dimensional_precompute_op(dagster.build_op_context())
- assert result == {"teams": 0, "failures": 0}
- stats_mock.assert_not_called()
- bounces_mock.assert_not_called()
-
- @patch("products.web_analytics.dags.web_dimensional_precompute.ensure_web_bounces_dimensional_precomputed")
- @patch("products.web_analytics.dags.web_dimensional_precompute.ensure_web_stats_dimensional_precomputed")
- def test_missing_team_is_skipped(self, stats_mock, bounces_mock):
+ def test_no_enrolled_team_is_a_noop(self, _name, allowlist, stats_mock, bounces_mock):
# A non-existent team id in the allowlist is logged and skipped, not fatal.
- with patch.dict(os.environ, {SELECTED_TEAM_IDS_ENV_VAR: "999999999"}):
+ with patch.dict(os.environ, {SELECTED_TEAM_IDS_ENV_VAR: allowlist}):
result = ensure_web_dimensional_precompute_op(dagster.build_op_context())
- assert result == {"teams": 0, "failures": 0}
+ assert (result.teams, result.failures, result.team_ids) == (0, 0, ())
stats_mock.assert_not_called()
+ bounces_mock.assert_not_called()
def test_job_has_owner_and_runtime_tags(self):
tags = web_dimensional_precompute_job.tags
assert tags["owner"] == "team-web-analytics"
assert "dagster/max_runtime" in tags
+
+
+class TestCheckWebDimensionalPrecomputeCoverageOp(APIBaseTest):
+ def _create_jobs(self, query_hash: str, today_built_minutes_ago: int, now: datetime = _NOW) -> PreaggregationJob:
+ window = precompute_window(now)
+ today = window.end - timedelta(days=1)
+ PreaggregationJob.objects.create(
+ team=self.team,
+ time_range_start=window.start,
+ time_range_end=today,
+ query_hash=query_hash,
+ status=PreaggregationJob.Status.READY,
+ computed_at=now - timedelta(days=1),
+ expires_at=now + timedelta(days=60),
+ )
+ built_at = now - timedelta(minutes=today_built_minutes_ago)
+ return PreaggregationJob.objects.create(
+ team=self.team,
+ time_range_start=today,
+ time_range_end=window.end,
+ query_hash=query_hash,
+ status=PreaggregationJob.Status.READY,
+ computed_at=built_at,
+ expires_at=built_at + timedelta(hours=1),
+ )
+
+ def _run(self, sample_job_ids: dict[tuple[int, str], str], failures: int = 0) -> DimensionalPrecomputeRun:
+ return DimensionalPrecomputeRun(
+ teams=1,
+ failures=failures,
+ team_ids=(self.team.pk,),
+ sample_job_ids=sample_job_ids,
+ window=precompute_window(_NOW),
+ current_day=_NOW.date(),
+ )
+
+ @parameterized.expand([("fresh_today", 10, False), ("today_not_rebuilt", 3 * 60, True)])
+ def test_fails_the_run_on_incomplete_coverage(self, _name, today_built_minutes_ago, expect_failure):
+ stats_job = self._create_jobs("stats-hash", 10)
+ bounces_job = self._create_jobs("bounces-hash", today_built_minutes_ago)
+ run = self._run(
+ {
+ (self.team.pk, STATS_TABLE): str(stats_job.id),
+ (self.team.pk, BOUNCES_TABLE): str(bounces_job.id),
+ }
+ )
+
+ with time_machine.travel(_NOW, tick=False):
+ if expect_failure:
+ with pytest.raises(dagster.Failure, match=f"team={self.team.pk} table={BOUNCES_TABLE}"):
+ check_web_dimensional_precompute_coverage_op(dagster.build_op_context(), run)
+ else:
+ check_web_dimensional_precompute_coverage_op(dagster.build_op_context(), run)
+
+ def test_table_without_a_ready_result_this_run_fails(self):
+ stats_job = self._create_jobs("stats-hash", 10)
+ self._create_jobs("bounces-hash", 10)
+ run = self._run({(self.team.pk, STATS_TABLE): str(stats_job.id)}, failures=1)
+
+ with time_machine.travel(_NOW, tick=False), pytest.raises(dagster.Failure, match=BOUNCES_TABLE):
+ check_web_dimensional_precompute_coverage_op(dagster.build_op_context(), run)
+
+ @patch(_SINGLE_CHUNK, PRECOMPUTE_WINDOW_DAYS)
+ def test_run_crossing_utc_midnight_checks_the_writers_day(self):
+ # The writers finish before midnight and the check runs after it. The new UTC day has no job
+ # yet, and the check must not report it as missing.
+ writer_now = datetime(2026, 10, 19, 23, 50, tzinfo=UTC)
+ stats_job = self._create_jobs("stats-hash", 10, now=writer_now)
+ bounces_job = self._create_jobs("bounces-hash", 10, now=writer_now)
+
+ with (
+ time_machine.travel(writer_now, tick=False),
+ patch(
+ "products.web_analytics.dags.web_dimensional_precompute.ensure_web_stats_dimensional_precomputed",
+ return_value=LazyComputationResult(ready=True, job_ids=[stats_job.id]),
+ ),
+ patch(
+ "products.web_analytics.dags.web_dimensional_precompute.ensure_web_bounces_dimensional_precomputed",
+ return_value=LazyComputationResult(ready=True, job_ids=[bounces_job.id]),
+ ),
+ patch.dict(os.environ, {SELECTED_TEAM_IDS_ENV_VAR: str(self.team.pk)}),
+ ):
+ run = ensure_web_dimensional_precompute_op(dagster.build_op_context())
+
+ assert (run.window.end, run.current_day) == (datetime(2026, 10, 20, tzinfo=UTC), date(2026, 10, 19))
+
+ with time_machine.travel(writer_now + timedelta(minutes=15), tick=False):
+ check_web_dimensional_precompute_coverage_op(dagster.build_op_context(), run)
diff --git a/products/web_analytics/dags/web_dimensional_precompute.py b/products/web_analytics/dags/web_dimensional_precompute.py
index bedb01eef5cf..6700dd893aa8 100644
--- a/products/web_analytics/dags/web_dimensional_precompute.py
+++ b/products/web_analytics/dags/web_dimensional_precompute.py
@@ -1,7 +1,7 @@
"""Scheduled population of the fixed-dimension web precompute tables.
This job drives the precomputation framework: for each selected team it calls
-`ensure_*_dimensional_precomputed` over a rolling 90-day window, and the
+`ensure_*_dimensional_precomputed` over a rolling window, and the
framework splits that into daily jobs, tracks them in Postgres
(`PreaggregationJob`) and only (re)computes windows whose jobs have expired. So
the heavy raw-table scan happens on a schedule for a known audience rather than
@@ -18,22 +18,31 @@ is no dependency on the v2 team-selection dictionary or flag.
The write path is not yet wired into any query runner — this job only populates
the tables (so the new output can be compared with v2's side by side).
+
+After the writers, a coverage check reads the READY jobs back from Postgres and
+fails the run when a team has a day without a fresh job or a stale current day.
+A failed run alerts the job owner through `notify_slack_on_failure`.
"""
import os
-from datetime import UTC, datetime, timedelta
+from collections.abc import Iterable
+from datetime import UTC, date, datetime, time, timedelta
from django.conf import settings
import dagster
import structlog
-from prometheus_client import Counter
+from prometheus_client import Counter, Gauge
from posthog.cloud_utils import is_cloud
from posthog.dags.common import JobOwners, chunk_ranges, skip_on_kill_switch
+from posthog.dataclasses import frozen
+from posthog.metrics import pushed_metrics_registry
from posthog.models import Team
+from products.analytics_platform.backend.models.preaggregation_job import PreaggregationJob
from products.web_analytics.backend.hogql_queries.web_dimensional_precompute import (
+ DIMENSIONAL_TTL_SECONDS,
ensure_web_bounces_dimensional_precomputed,
ensure_web_stats_dimensional_precomputed,
)
@@ -46,7 +55,10 @@ logger = structlog.get_logger(__name__)
# `MAX_PRECOMPUTE_DAYS`, because this window is a fixed cost the dag pays for
# every enrolled team, not a per-shape span the tenant's own demand bounds.
# Widen it through the env var once the storage cost is agreed.
-PRECOMPUTE_WINDOW_DAYS = int(os.getenv("WEB_DIMENSIONAL_PRECOMPUTE_WINDOW_DAYS", "90"))
+# A `-90d` read starts at the team's local midnight 90 days back, and the window ends at the next UTC
+# midnight. 92 UTC days hold that start for every UTC offset from -12h to +14h. 91 days hold it only
+# for a team in UTC.
+PRECOMPUTE_WINDOW_DAYS = int(os.getenv("WEB_DIMENSIONAL_PRECOMPUTE_WINDOW_DAYS", "92"))
# Each ensure_precomputed call covers at most this many days. The framework merges
# a fully-missing range into ONE INSERT, so without chunking a cold backfill would
@@ -74,6 +86,25 @@ DEFAULT_ROLLOUT_TEAM_IDS_BY_REGION: dict[str, list[int]] = {
# set to empty to disable the job entirely.
SELECTED_TEAM_IDS_ENV_VAR = "WEB_DIMENSIONAL_PRECOMPUTE_TEAM_IDS"
+SCHEDULE_INTERVAL_MINUTES = 20
+
+# A day counts as covered while its newest READY job is at most two runs past its expiry. The check
+# runs after the writers, so a job that expires between the write and the check must not fail it.
+COVERAGE_GRACE_SECONDS = 2 * SCHEDULE_INTERVAL_MINUTES * 60
+
+# The current day's job stays fresh for its whole TTL, so its age reaches the TTL in a healthy run.
+# Alert only when it is also two runs past that.
+TODAY_MAX_LAG_SECONDS = DIMENSIONAL_TTL_SECONDS["0d"] + COVERAGE_GRACE_SECONDS
+
+STATS_TABLE = "web_stats_dimensional_preaggregated"
+BOUNCES_TABLE = "web_bounces_dimensional_preaggregated"
+
+# Run pods exit after the run, so nothing scrapes their in-process registry. The gauges go through
+# the push gateway instead, under this group name. A push replaces every gauge of the group.
+COVERAGE_METRICS_JOB = "web_dimensional_precompute_coverage"
+COVERAGE_MISSING_DAYS_GAUGE = "web_dimensional_precompute_coverage_missing_days"
+COVERAGE_TODAY_LAG_GAUGE = "web_dimensional_precompute_today_lag_seconds"
+
def get_selected_team_ids() -> list[int]:
"""Resolve the team allowlist.
@@ -104,8 +135,188 @@ WEB_DIMENSIONAL_PRECOMPUTE_TEAM_FAILED = Counter(
)
+@frozen
+class TimeSpan:
+ start: datetime
+ end: datetime
+
+
+def precompute_window(now: datetime) -> TimeSpan:
+ # chunk_ranges slices backwards from `end`, so ending on a UTC midnight makes every chunk one UTC
+ # day. The framework builds one job per UTC day, and a chunk that straddles midnight merges two
+ # missing days into a single INSERT that reads twice the data.
+ end = now.astimezone(UTC).replace(hour=0, minute=0, second=0, microsecond=0) + timedelta(days=1)
+ return TimeSpan(start=end - timedelta(days=PRECOMPUTE_WINDOW_DAYS), end=end)
+
+
+@frozen
+class DimensionalPrecomputeRun:
+ teams: int
+ failures: int
+ team_ids: tuple[int, ...]
+ # The window the writers requested and the UTC day they treated as current. The coverage check
+ # reads these instead of its own clock: a run that crosses UTC midnight would otherwise look for
+ # a day no writer was asked to build.
+ window: TimeSpan
+ current_day: date
+ # One job id per (team id, table) from the writers of this run. The coverage check reads its
+ # query_hash, because the executor derives the hash from the INSERT template and has no public
+ # helper for it.
+ sample_job_ids: dict[tuple[int, str], str]
+
+
+@frozen
+class TableCoverage:
+ team_id: int
+ table: str
+ window_days: int
+ missing_days: tuple[date, ...]
+ # None when no READY job covers the writer's current UTC day.
+ today_lag_seconds: float | None
+
+ @property
+ def covered_days(self) -> int:
+ return self.window_days - len(self.missing_days)
+
+ @property
+ def healthy(self) -> bool:
+ return (
+ not self.missing_days
+ and self.today_lag_seconds is not None
+ and self.today_lag_seconds <= TODAY_MAX_LAG_SECONDS
+ )
+
+ def describe(self) -> str:
+ lag = "none" if self.today_lag_seconds is None else f"{round(self.today_lag_seconds / 60)}m"
+ missing = ",".join(d.isoformat() for d in self.missing_days[:10])
+ if len(self.missing_days) > 10:
+ missing += ",..."
+ return (
+ f"team={self.team_id} table={self.table} covered={self.covered_days}/{self.window_days} "
+ f"missing=[{missing}] today_lag={lag}"
+ )
+
+
+def _merge_spans(spans: Iterable[TimeSpan]) -> list[TimeSpan]:
+ merged: list[TimeSpan] = []
+ for span in sorted(spans, key=lambda s: s.start):
+ if merged and span.start <= merged[-1].end:
+ merged[-1] = TimeSpan(start=merged[-1].start, end=max(merged[-1].end, span.end))
+ else:
+ merged.append(span)
+ return merged
+
+
+def summarize_coverage(
+ *,
+ team_id: int,
+ table: str,
+ ready_jobs: Iterable[PreaggregationJob],
+ window: TimeSpan,
+ current_day: date,
+ now: datetime,
+) -> TableCoverage:
+ """Report which UTC days of the window have a fresh READY job, and how old the current day's job is.
+
+ `window` and `current_day` come from the writer; `now` is the check's own clock and only drives
+ freshness and lag. Jobs can span more than one day, so the check merges the job intervals instead
+ of counting job start dates.
+ """
+ jobs = list(ready_jobs)
+ fresh_cutoff = now - timedelta(seconds=COVERAGE_GRACE_SECONDS)
+ merged = _merge_spans(
+ TimeSpan(start=job.time_range_start, end=job.time_range_end)
+ for job in jobs
+ if job.expires_at is not None and job.expires_at >= fresh_cutoff
+ )
+
+ missing: list[date] = []
+ day = window.start
+ while day < window.end:
+ day_end = day + timedelta(days=1)
+ if not any(span.start <= day and span.end >= day_end for span in merged):
+ missing.append(day.date())
+ day = day_end
+
+ today_start = datetime.combine(current_day, time(), tzinfo=UTC)
+ today_end = today_start + timedelta(days=1)
+ today_built_at = [
+ job.computed_at or job.created_at
+ for job in jobs
+ if job.time_range_start <= today_start and job.time_range_end >= today_end
+ ]
+ today_lag = (now - max(today_built_at)).total_seconds() if today_built_at else None
+
+ return TableCoverage(
+ team_id=team_id,
+ table=table,
+ window_days=(window.end - window.start).days,
+ missing_days=tuple(missing),
+ today_lag_seconds=today_lag,
+ )
+
+
+def find_table_coverage(
+ *,
+ team_id: int,
+ table: str,
+ query_hash: str | None,
+ window: TimeSpan,
+ current_day: date,
+ now: datetime,
+) -> TableCoverage:
+ # Expired READY jobs stay in the result, so the current day's lag still shows the age of the
+ # newest build after its job expired.
+ ready_jobs: list[PreaggregationJob] = []
+ if query_hash is not None:
+ ready_jobs = list(
+ PreaggregationJob.objects.filter(
+ team_id=team_id,
+ query_hash=query_hash,
+ status=PreaggregationJob.Status.READY,
+ time_range_start__lt=window.end,
+ time_range_end__gt=window.start,
+ )
+ )
+ return summarize_coverage(
+ team_id=team_id,
+ table=table,
+ ready_jobs=ready_jobs,
+ window=window,
+ current_day=current_day,
+ now=now,
+ )
+
+
+def _push_coverage_gauges(coverages: list[TableCoverage]) -> None:
+ with pushed_metrics_registry(COVERAGE_METRICS_JOB) as registry:
+ missing_days = Gauge(
+ COVERAGE_MISSING_DAYS_GAUGE,
+ "UTC days in the dimensional precompute window without a fresh READY job.",
+ ["team_id", "table"],
+ registry=registry,
+ )
+ today_lag = Gauge(
+ COVERAGE_TODAY_LAG_GAUGE,
+ "Age of the newest READY dimensional precompute job that covers the writer's current UTC day.",
+ ["team_id", "table"],
+ registry=registry,
+ )
+ for coverage in coverages:
+ labels = {"team_id": str(coverage.team_id), "table": coverage.table}
+ missing_days.labels(**labels).set(len(coverage.missing_days))
+ # A current day without any job reads as infinitely stale, so a lag alert fires on it too.
+ today_lag.labels(**labels).set(
+ coverage.today_lag_seconds if coverage.today_lag_seconds is not None else float("inf")
+ )
+
+
def _ensure_for_team(
- context: dagster.OpExecutionContext, team: Team, start: datetime, end: datetime, chunk_days: int
+ context: dagster.OpExecutionContext,
+ team: Team,
+ window: TimeSpan,
+ chunk_days: int,
+ sample_job_ids: dict[tuple[int, str], str],
) -> int:
"""Ensure both dimensional tables for one team, one bounded chunk at a time.
@@ -116,13 +327,12 @@ def _ensure_for_team(
"""
failures = 0
for table_label, ensure_fn in (
- ("web_stats_dimensional_preaggregated", ensure_web_stats_dimensional_precomputed),
- ("web_bounces_dimensional_preaggregated", ensure_web_bounces_dimensional_precomputed),
+ (STATS_TABLE, ensure_web_stats_dimensional_precomputed),
+ (BOUNCES_TABLE, ensure_web_bounces_dimensional_precomputed),
):
- for chunk_start, chunk_end in chunk_ranges(start, end, chunk_days):
+ for chunk_start, chunk_end in chunk_ranges(window.start, window.end, chunk_days):
try:
- ensure_fn(team, chunk_start, chunk_end)
- WEB_DIMENSIONAL_PRECOMPUTE_TEAM_DONE.labels(table=table_label).inc()
+ result = ensure_fn(team, chunk_start, chunk_end)
except Exception as exc:
WEB_DIMENSIONAL_PRECOMPUTE_TEAM_FAILED.labels(table=table_label, error_type=type(exc).__name__).inc()
context.log.exception(
@@ -130,45 +340,113 @@ def _ensure_for_team(
f"chunk=[{chunk_start}, {chunk_end})"
)
failures += 1
+ continue
+ # The executor catches a failed INSERT, marks the job FAILED and returns ready=False
+ # instead of raising.
+ if not result.ready:
+ error_type = "memory_limit_exceeded" if result.memory_exceeded else "not_ready"
+ WEB_DIMENSIONAL_PRECOMPUTE_TEAM_FAILED.labels(table=table_label, error_type=error_type).inc()
+ context.log.warning(
+ f"web_dimensional_precompute_not_ready team={team.pk} table={table_label} "
+ f"chunk=[{chunk_start}, {chunk_end}) errors={result.errors}"
+ )
+ failures += 1
+ continue
+ WEB_DIMENSIONAL_PRECOMPUTE_TEAM_DONE.labels(table=table_label).inc()
+ if result.job_ids:
+ sample_job_ids.setdefault((team.pk, table_label), str(result.job_ids[0]))
return failures
@dagster.op
-def ensure_web_dimensional_precompute_op(context: dagster.OpExecutionContext) -> dict[str, int]:
+def ensure_web_dimensional_precompute_op(context: dagster.OpExecutionContext) -> DimensionalPrecomputeRun:
"""Drive `ensure_*_dimensional_precomputed` over the rolling window for each selected team."""
- # chunk_ranges slices backwards from `end`, so ending on a UTC midnight makes every chunk one UTC
- # day. The framework builds one job per UTC day, and a chunk that straddles midnight merges two
- # missing days into a single INSERT that reads twice the data.
- end = datetime.now(UTC).replace(hour=0, minute=0, second=0, microsecond=0) + timedelta(days=1)
- start = end - timedelta(days=PRECOMPUTE_WINDOW_DAYS)
+ started_at = datetime.now(UTC)
+ window = precompute_window(started_at)
+ current_day = started_at.date()
team_ids = get_selected_team_ids()
context.log.info(
- f"web_dimensional_precompute_start teams={len(team_ids)} window=[{start}, {end}) "
+ f"web_dimensional_precompute_start teams={len(team_ids)} window=[{window.start}, {window.end}) "
f"chunk_days={PRECOMPUTE_CHUNK_DAYS}"
)
if not team_ids:
context.log.info(f"web_dimensional_precompute_noop ({SELECTED_TEAM_IDS_ENV_VAR} is empty)")
- result = {"teams": 0, "failures": 0}
- context.add_output_metadata(result)
- return result
+ context.add_output_metadata({"teams": 0, "failures": 0})
+ return DimensionalPrecomputeRun(
+ teams=0, failures=0, team_ids=(), sample_job_ids={}, window=window, current_day=current_day
+ )
teams_by_id = {t.pk: t for t in Team.objects.filter(pk__in=team_ids)}
failures = 0
- processed = 0
+ processed: list[int] = []
+ sample_job_ids: dict[tuple[int, str], str] = {}
for team_id in team_ids:
team = teams_by_id.get(team_id)
if team is None:
context.log.warning(f"web_dimensional_precompute_team_missing team_id={team_id}")
continue
- failures += _ensure_for_team(context, team, start, end, PRECOMPUTE_CHUNK_DAYS)
- processed += 1
+ failures += _ensure_for_team(context, team, window, PRECOMPUTE_CHUNK_DAYS, sample_job_ids)
+ processed.append(team_id)
+
+ context.log.info(f"web_dimensional_precompute_complete teams={len(processed)} failures={failures}")
+ context.add_output_metadata({"teams": len(processed), "failures": failures})
+ return DimensionalPrecomputeRun(
+ teams=len(processed),
+ failures=failures,
+ team_ids=tuple(processed),
+ sample_job_ids=sample_job_ids,
+ window=window,
+ current_day=current_day,
+ )
+
+
+def check_web_dimensional_precompute_coverage_op(
+ context: dagster.OpExecutionContext, run: DimensionalPrecomputeRun
+) -> None:
+ """Fail the run when a processed team has a missing day or a stale current day in either table."""
+ now = datetime.now(UTC)
+ query_hash_by_job_id = {
+ str(job_id): query_hash
+ for job_id, query_hash in PreaggregationJob.objects.filter(
+ id__in=list(run.sample_job_ids.values())
+ ).values_list("id", "query_hash")
+ }
+
+ coverages: list[TableCoverage] = []
+ for team_id in run.team_ids:
+ for table in (STATS_TABLE, BOUNCES_TABLE):
+ job_id = run.sample_job_ids.get((team_id, table))
+ coverages.append(
+ find_table_coverage(
+ team_id=team_id,
+ table=table,
+ query_hash=query_hash_by_job_id.get(job_id) if job_id is not None else None,
+ window=run.window,
+ current_day=run.current_day,
+ now=now,
+ )
+ )
- context.log.info(f"web_dimensional_precompute_complete teams={processed} failures={failures}")
- result = {"teams": processed, "failures": failures}
- context.add_output_metadata(result)
- return result
+ _push_coverage_gauges(coverages)
+ unhealthy = [coverage for coverage in coverages if not coverage.healthy]
+ metadata: dict[str, dagster.MetadataValue] = {
+ "checked": dagster.MetadataValue.int(len(coverages)),
+ "unhealthy": dagster.MetadataValue.int(len(unhealthy)),
+ "details": dagster.MetadataValue.md("\n".join(f"- {c.describe()}" for c in coverages) or "none"),
+ }
+ if unhealthy:
+ raise dagster.Failure(
+ description=(
+ f"Dimensional precompute coverage is incomplete for {len(unhealthy)} of {len(coverages)} "
+ "team tables: " + "; ".join(c.describe() for c in unhealthy)
+ ),
+ metadata=metadata,
+ )
+ context.log.info(f"web_dimensional_precompute_coverage_ok checked={len(coverages)}")
+ context.add_output_metadata(metadata)
@dagster.job(
@@ -176,7 +454,9 @@ def ensure_web_dimensional_precompute_op(context: dagster.OpExecutionContext) ->
f"Populates the fixed-dimension web precompute tables (web_stats_dimensional_preaggregated / "
f"web_bounces_dimensional_preaggregated) over the trailing {PRECOMPUTE_WINDOW_DAYS} days for the teams in "
f"the {SELECTED_TEAM_IDS_ENV_VAR} allowlist, by driving the precomputation framework's ensure_precomputed. "
- f"No-op when the allowlist is empty. Re-runs only recompute windows whose jobs have expired."
+ f"No-op when the allowlist is empty. Re-runs only recompute windows whose jobs have expired. "
+ f"Fails when a team has a UTC day without a fresh job, or a current-day job older than "
+ f"{TODAY_MAX_LAG_SECONDS // 60} minutes."
),
tags={
"owner": JobOwners.TEAM_WEB_ANALYTICS.value,
@@ -184,13 +464,16 @@ def ensure_web_dimensional_precompute_op(context: dagster.OpExecutionContext) ->
},
)
def web_dimensional_precompute_job():
- ensure_web_dimensional_precompute_op()
+ check_web_dimensional_precompute_coverage_op(ensure_web_dimensional_precompute_op())
@dagster.schedule(
- # Hourly. Recent windows carry a short TTL (see DIMENSIONAL_TTL_SECONDS), so an
- # hourly cadence keeps today fresh; older windows are computed once and skipped.
- cron_schedule="20 * * * *",
+ # The current day's job expires after an hour (see DIMENSIONAL_TTL_SECONDS), and a run rebuilds
+ # it only once it has expired. A 20-minute cadence bounds that gap to one interval; older
+ # windows are computed once and skipped. The minute offsets keep clear of the sibling web
+ # analytics schedules. check_for_concurrent_runs skips a tick while a run is still active, so a
+ # slow run does not stack up behind it.
+ cron_schedule="7,27,47 * * * *",
job=web_dimensional_precompute_job,
execution_timezone="UTC",
tags={"owner": JobOwners.TEAM_WEB_ANALYTICS.value},