Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

fix(eng-analytics): seed stored ci views to 30 days (#129745)

Raúl Negrón-Oterof1fe044ad41f
diff --git a/products/engineering_analytics/SPEC.md b/products/engineering_analytics/SPEC.md
index c2f997c355cb..6a86be28a20a 100644
--- a/products/engineering_analytics/SPEC.md
+++ b/products/engineering_analytics/SPEC.md
@@ -146,9 +146,9 @@ The per-PR friction view and the stored CI rows are the exceptions, below: they
- The stored CI rows: the output of the runs builder and of the jobs builder with the cost columns, as two materialized views that data modeling updates in place every 15 minutes (`logic/views/ci_rows.py`). A run upserts the rows created since two days before the newest stored row, keyed on the parsed `created_at`. Runs use `(source_id, ci_engine, id)` as their unique key, so a retry replaces the previous snapshot. Jobs use `(source_id, ci_engine, native_job_id, run_attempt)`, so a Depot job retry replaces its carried-over copy even though the execution ID changes. A row that settles later than two days after its creation is read again only by a full build, which a definition change triggers. Until then its stored snapshot keeps the earlier conclusion and duration in every historical aggregate and cost total. A rerun of a run created more than two days earlier is stored as a new attempt without rewriting the run row, and a rerun whose first attempt is more than nine days before the window (the two-day lookback plus the seven-day duplicate scan) is counted as an execution because the scan no longer sees the copy it re-lists.
- Two grains, never joined rows. A re-run mutates the run row and adds job rows with new ids, so a stored joined row would go stale. A read joins them on `(ci_engine, run_id)`, as `job_costs` does.
- `is_handoff_shell` is a flag on both stored grains, not a filter. A jobs build resolves the flag from raw run and handoff evidence in the same query, across run history because a new retry job can belong to an old run. A reader filters each grain on its own flag, so independently refreshed runs cannot remove jobs before their replacement jobs arrive. This is the one exception to the §6 rule that a shell is filtered at the substrate: the stored rows are storage under the read layer, and the read that swaps onto them applies the flag, so every consumer still sees the same suppression.
-- The views name the run's window start as `{incremental_window_start}` (`logic/views/incremental_floor.py`) and floor their primary scans on the raw `created_at` string formatted from it, because a predicate on the parsed `created_at` cannot prune the parquet scan. The duplicate scan and the runs build's handoff scan start seven days earlier. The jobs build resolves suppression across run history so a recent job from an older run still gets its flag. On a full build the placeholder is NULL and the floors keep every row.
+- The views name the run's window start as `{incremental_window_start}` (`logic/views/incremental_floor.py`) and floor their primary scans on the raw `created_at` string formatted from it, because a predicate on the parsed `created_at` cannot prune the parquet scan. The duplicate scan and the runs build's handoff scan start seven days earlier. The jobs build resolves suppression across run history so a recent job from an older run still gets its flag. On a full build the placeholder is NULL and the floors seed the rows created in the last 39 days (the 30-day default read window plus the widest scan-floor slack), because a full build runs as one ClickHouse query under the shared time limit and a busy repository's whole job history does not fit in it. Each incremental run then adds the rows created since. A view read at query time, outside data modeling, shows the same 39 days.
- Rows carry `source_id` and `repository`, so a product read filters on the one source the caller may use, like the friction view. They spend the team's warehouse compute every 15 minutes, so they exist only for the organizations the friction flag targets. When flag evaluation gives no answer, each existing managed view is preserved individually and no new one is enabled. A team whose GitHub source syncs less often than every 15 minutes cannot be fed at that cadence; data modeling then leaves the views computed at query time.
-- Historical aggregates and cost reads use this stored pair through the shared curated layer when the friction flag is enabled. Each view's newest completed build must be at most 30 minutes old and carry the fingerprint of the current definition; a refresh that is running or failed leaves the previous build serving until that limit passes. The reader checks permission on the selected raw sources and both views, then filters on source and repository. Any failed readiness check sends the pair back to raw queries.
+- Historical aggregates and cost reads use this stored pair through the shared curated layer when the friction flag is enabled. Each view's newest completed build must be at most 30 minutes old and carry the fingerprint of the current definition; a refresh that is running or failed leaves the previous build serving until that limit passes. The reader checks permission on the selected raw sources and both views, then filters on source and repository. Any failed readiness check sends the pair back to raw queries. Each read also names its window start. The stored pair serves it only when that start is at least nine days after the coverage start the newest full build recorded, so the widest scan floor stays inside the stored rows. An unbounded read, or one that starts earlier, uses the raw tables; the per-pull-request cost and push reads are unbounded, and a previous-period comparison reaches back twice its window.
- Current CI status, single-run and single-PR detail, engine identity checks, and raw job steps stay live. Their latest-attempt lookup also stays live, so a newer rerun cannot select an older attempt's jobs. Stored reads report conservative source import timestamps from before the builds, rather than later source syncs, and their caches include the materialization versions. The producer's two-day reconciliation bound above still applies to historical aggregates.
## 6. Locked decisions
diff --git a/products/engineering_analytics/backend/logic/_shared.py b/products/engineering_analytics/backend/logic/_shared.py
index 9fd6c1ce71f0..c5e6d907584a 100644
--- a/products/engineering_analytics/backend/logic/_shared.py
+++ b/products/engineering_analytics/backend/logic/_shared.py
@@ -1,6 +1,6 @@
"""Caller-input resolution shared by the domain orchestration modules."""
-from datetime import datetime
+from datetime import datetime, timedelta
from posthog.dataclasses import frozen
from posthog.models.team import Team
@@ -8,7 +8,8 @@ from posthog.utils import relative_date_parse
# Default recency window when a caller omits date_from. Relative strings (-30d) and
# ISO8601 are both accepted and resolved against the team's timezone.
-_DEFAULT_WINDOW = "-30d"
+DEFAULT_WINDOW = timedelta(days=30)
+_DEFAULT_WINDOW = f"-{DEFAULT_WINDOW.days}d"
# workflow_health zero-fills one daily entry per workflow per day in the window, so an
# unbounded range would materialize an enormous response. A year is plenty for trends.
diff --git a/products/engineering_analytics/backend/logic/materialized_ci.py b/products/engineering_analytics/backend/logic/materialized_ci.py
index 3645061d080f..7dc4e04b44ba 100644
--- a/products/engineering_analytics/backend/logic/materialized_ci.py
+++ b/products/engineering_analytics/backend/logic/materialized_ci.py
@@ -13,14 +13,17 @@ from posthog.dataclasses import frozen
from products.data_modeling.backend.facade.api import (
IncrementalConfig,
definition_fingerprint,
+ deserialize_watermark,
get_incremental_config,
get_incremental_state,
is_materialization_fresh,
)
from products.data_modeling.backend.facade.models import DataModelingJob, DataWarehouseSavedQuery
from products.engineering_analytics.backend.facade.contracts import MATERIALIZED_VIEWS_FEATURE_FLAG, CIDataFreshness
+from products.engineering_analytics.backend.logic.queries._workflow_filters import WIDEST_SCAN_FLOOR_SLACK
from products.engineering_analytics.backend.logic.sources import GitHubTables, resolve_job_source_tables
from products.engineering_analytics.backend.logic.views import ci_rows, depot_ci, workflow_jobs, workflow_runs
+from products.engineering_analytics.backend.logic.views.incremental_floor import SEED_LOOKBACK
from products.warehouse_sources.backend.facade.api import import_started_before
from products.warehouse_sources.backend.facade.types import DataWarehouseManagedViewSetKind
@@ -74,6 +77,9 @@ class MaterializedCI:
repository: str
freshness: CIDataFreshness
cache_scope: str
+ # The creation time of the oldest row both views hold. A full build seeds SEED_LOOKBACK before
+ # it starts, and incremental runs only add newer rows.
+ covers_from: datetime
@classmethod
def resolve(
@@ -132,6 +138,7 @@ class MaterializedCI:
expected = ci_rows.build_views(sources)
materialized_at: dict[str, datetime] = {}
started_at: dict[str, datetime] = {}
+ seeded_at: list[datetime] = []
for name, fields, config, sql in (
(ci_rows.RUNS_VIEW_NAME, ci_rows.RUN_FIELDS, ci_rows.RUN_INCREMENTAL_CONFIG, expected.runs),
(ci_rows.JOBS_VIEW_NAME, ci_rows.JOB_FIELDS, ci_rows.JOB_INCREMENTAL_CONFIG, expected.jobs),
@@ -161,13 +168,20 @@ class MaterializedCI:
return None
incremental = get_incremental_config(view)
fingerprint = _definition_fingerprint(sql, incremental) if incremental is not None else None
- if fingerprint is None or fingerprint != get_incremental_state(view).definition_fingerprint:
+ state = get_incremental_state(view)
+ if fingerprint is None or fingerprint != state.definition_fingerprint:
+ return None
+ # Data modeling records this instant when the full build completes, after the build
+ # computed its seed floor, so SEED_LOOKBACK before it never reaches past the seed.
+ full_refresh_at = deserialize_watermark(state.last_full_refresh_at, "datetime")
+ if full_refresh_at is None:
return None
completed_at = job.last_run_at
if not is_materialization_fresh(completed_at, int(MAX_MATERIALIZATION_AGE.total_seconds())):
return None
materialized_at[name] = completed_at
started_at[name] = job.created_at
+ seeded_at.append(full_refresh_at)
runs_at = materialized_at[ci_rows.RUNS_VIEW_NAME]
jobs_at = materialized_at[ci_rows.JOBS_VIEW_NAME]
@@ -186,8 +200,20 @@ class MaterializedCI:
runs_synced_at=synced_at(tables.workflow_runs), jobs_synced_at=synced_at(tables.workflow_jobs)
),
cache_scope=f"materialized:{runs_at.isoformat()}:{jobs_at.isoformat()}",
+ covers_from=max(seeded_at) - SEED_LOOKBACK,
)
+ def covers(self, since: datetime) -> bool:
+ """True when every row a read windowed from ``since`` scans was created inside the coverage.
+
+ Coverage is by creation time, so a run created before it and rerun inside the window is
+ missing from the stored rows. SPEC §5 accepts that. A naive ``since`` cannot be compared with
+ the coverage start, so it reads the raw tables.
+ """
+ if since.tzinfo is None:
+ return False
+ return since - WIDEST_SCAN_FLOOR_SLACK >= self.covers_from
+
def _scope(self) -> str:
return (
f"source_id = {escape_hogql_string(self.source_id)} AND repository = {escape_hogql_string(self.repository)}"
diff --git a/products/engineering_analytics/backend/logic/quarantine.py b/products/engineering_analytics/backend/logic/quarantine.py
index 092fd47c8218..6349770cfe67 100644
--- a/products/engineering_analytics/backend/logic/quarantine.py
+++ b/products/engineering_analytics/backend/logic/quarantine.py
@@ -268,11 +268,12 @@ def _most_active_repo(
# Raises GitHubSourceNotConnectedError when the team has no source (caught upstream and turned
# into available=false) or ValueError for a bad source_id (propagated to a 4xx by the view).
curated = _curated.CuratedGitHubSource.for_team(team, source_id=source_id, user_access_control=user_access_control)
- sql = _REPO_SELECT.replace("__RUNS_SOURCE__", curated.run_source())
+ date_from = datetime.now(UTC) - timedelta(days=30)
+ sql = _REPO_SELECT.replace("__RUNS_SOURCE__", curated.run_source(since=date_from))
response = curated.run(
sql,
query_type="engineering_analytics.quarantine_repo",
- placeholders={"date_from": ast.Constant(value=datetime.now(UTC) - timedelta(days=30))},
+ placeholders={"date_from": ast.Constant(value=date_from)},
)
if not response.results:
return None
diff --git a/products/engineering_analytics/backend/logic/queries/_curated.py b/products/engineering_analytics/backend/logic/queries/_curated.py
index 8ebf2609fcc7..f48914d352e4 100644
--- a/products/engineering_analytics/backend/logic/queries/_curated.py
+++ b/products/engineering_analytics/backend/logic/queries/_curated.py
@@ -20,6 +20,7 @@ from collections.abc import Callable, Iterator, Sequence
from concurrent.futures import Future
from contextlib import contextmanager
from dataclasses import dataclass
+from datetime import datetime
from functools import partial
from typing import TYPE_CHECKING
@@ -243,11 +244,13 @@ class CuratedGitHubSource:
"""Curated pull-requests ``SELECT``, parenthesised for use as a subquery."""
return f"({pull_requests.build_query(self._tables.pull_requests)})"
- def run_source(self, *, started_floor: bool = False, live: bool = False) -> str:
+ def run_source(self, *, started_floor: bool = False, since: datetime | None = None, live: bool = False) -> str:
"""Curated workflow-runs ``SELECT``, parenthesised for use as a subquery. ``started_floor``
adds the raw-string scan floor — callers must register {run_started_floor} (see
- run_started_floor_constant). ``live`` preserves current-status reads and raw step lookups."""
- if not live and (stored := self._stored_ci()):
+ run_started_floor_constant). ``since`` is the read's window start; the stored rows serve a
+ read only when they cover it, and an unbounded read never (see ``MaterializedCI.covers``).
+ ``live`` preserves current-status reads and raw step lookups."""
+ if not live and (stored := self._stored_ci_covering(since)):
return f"({stored.runs(started_floor=started_floor)})"
query = workflow_runs.build_query(
self._runs_table(),
@@ -256,17 +259,20 @@ class CuratedGitHubSource:
)
return f"({query})"
- def jobs_source(self, *, created_floor: bool = False, include_steps: bool = False) -> str | None:
+ def jobs_source(
+ self, *, created_floor: bool = False, since: datetime | None = None, include_steps: bool = False
+ ) -> str | None:
"""Curated workflow-jobs ``SELECT`` subquery, or None when the optional jobs table isn't synced.
``created_floor`` adds the raw-string scan floor inside the builder — callers must register
{job_created_floor} (see run_started_floor_constant). A windowed caller needs it: the builder's
``is_rerun_copy`` duplicate scan reads no ``created_at_raw``, so only the floor bounds it.
+ ``since`` is the read's window start, as for ``run_source``.
``include_steps`` appends the raw ``steps`` JSON column, for a read bounded to one run."""
if not self._tables.workflow_jobs:
return None
- if not include_steps and (stored := self._stored_ci()):
+ if not include_steps and (stored := self._stored_ci_covering(since)):
return f"({stored.jobs(created_floor=created_floor)})"
query = workflow_jobs.build_query(
self._jobs_table(self._tables.workflow_jobs), created_floor=created_floor, include_steps=include_steps
@@ -323,6 +329,13 @@ class CuratedGitHubSource:
team=self._team, tables=self._tables, user_access_control=self._user_access_control
)
+ def _stored_ci_covering(self, since: datetime | None) -> MaterializedCI | None:
+ # Resolving the views queries the database, and an unbounded read cannot use them.
+ if since is None:
+ return None
+ stored = self._stored_ci()
+ return stored if stored is not None and stored.covers(since) else None
+
def _stored_ci(self) -> MaterializedCI | None:
catalog = self._catalog()
depot = self._depot_job_attempts()
@@ -475,7 +488,7 @@ class CuratedGitHubSource:
)
"""
- def job_cost_source(self, *, created_floor: bool = False) -> str | None:
+ def job_cost_source(self, *, created_floor: bool = False, since: datetime | None = None) -> str | None:
"""Per-job cost ``SELECT`` subquery: the rows ``engineering_analytics_job_costs`` exposes, with
the pass-through columns the public view leaves out (``run_started_at`` and ``run_head_branch``
among them). None when the jobs table isn't synced, exactly like ``jobs_source``.
@@ -490,10 +503,11 @@ class CuratedGitHubSource:
the run-windowed predicates every cost query uses). Every windowed caller wants it: the cost
source's window predicates read the RUN's columns and so can never prune the jobs scan, and
the ``is_rerun_copy`` duplicate scan would otherwise aggregate the full history on every call.
+ ``since`` is the read's window start, as for ``run_source``.
"""
if not self._tables.workflow_jobs:
return None
- if stored := self._stored_ci():
+ if stored := self._stored_ci_covering(since):
query = job_costs.build_attributed_query(
costed_jobs=stored.jobs(created_floor=created_floor, with_cost=True),
runs=stored.runs(include_shells=True),
diff --git a/products/engineering_analytics/backend/logic/queries/_test_spans.py b/products/engineering_analytics/backend/logic/queries/_test_spans.py
index e786b31bb9b3..cf0914d8bbcb 100644
--- a/products/engineering_analytics/backend/logic/queries/_test_spans.py
+++ b/products/engineering_analytics/backend/logic/queries/_test_spans.py
@@ -304,7 +304,8 @@ def query_setup_breaks(
so no test loses its evidence on the strength of the span attributes alone, and a team whose jobs
table is unsynced drops nothing.
"""
- jobs_source = curated.jobs_source(created_floor=True)
+ floor_from = scan_from if scan_from is not None else date_from
+ jobs_source = curated.jobs_source(created_floor=True, since=floor_from)
if not curated.repository or jobs_source is None:
return SetupBreaks(run_attempts=(), job_attempts=())
@@ -350,7 +351,7 @@ def query_setup_breaks(
placeholders={
"candidate_run_ids": ast.Constant(value=candidate_run_ids),
"candidate_workflow_keys": ast.Constant(value=candidate_workflow_keys),
- "job_created_floor": job_created_floor_constant(scan_from if scan_from is not None else date_from),
+ "job_created_floor": job_created_floor_constant(floor_from),
},
)
}
diff --git a/products/engineering_analytics/backend/logic/queries/_workflow_filters.py b/products/engineering_analytics/backend/logic/queries/_workflow_filters.py
index 161662efd7a7..616658b6f7a1 100644
--- a/products/engineering_analytics/backend/logic/queries/_workflow_filters.py
+++ b/products/engineering_analytics/backend/logic/queries/_workflow_filters.py
@@ -89,6 +89,13 @@ LATEST_COMPLETED_RUN_FAILED = (
f"argMaxIf(conclusion IN ({DECISIVE_FAILURE_CONCLUSIONS_SQL}), {LATEST_RUN_ORDER}, status = 'completed')"
)
+# The slack a date-only floor subtracts from the window start.
+_DATE_FLOOR_SLACK = timedelta(days=1)
+
+# A date-only floor reaches back past the slack it subtracts: up to a day for the day rounding, and up
+# to a day more for the window's offset from the UTC strings it compares with.
+_DATE_FLOOR_REACH = timedelta(days=2)
+
def run_started_floor_constant(window_start: datetime) -> ast.Constant:
"""Raw-string scan floor for the runs builder's {run_started_floor} placeholder: a date-only
@@ -96,7 +103,7 @@ def run_started_floor_constant(window_start: datetime) -> ast.Constant:
ISO timestamp ('2026-07-11' < '2026-07-11T...'), and the one-day slack absorbs any timezone
offset between the window's zone and the UTC strings GitHub lands, so the coarse floor can
never cut rows the precise parsed {date_from} filter would keep."""
- return ast.Constant(value=(window_start - timedelta(days=1)).strftime("%Y-%m-%d"))
+ return _date_floor(window_start, _DATE_FLOOR_SLACK)
# How far below a window start the jobs-scan floor sits, by what the window actually filters.
@@ -118,9 +125,13 @@ def run_started_floor_constant(window_start: datetime) -> ast.Constant:
# the copy is only recognisable while its original attempt is inside the scan, so a tight floor under a
# late re-run would turn the copy back into an execution. Queries that keep copies (red/green reads)
# can use the tight floor.
-_JOB_FLOOR_SLACK_ON_JOB_CREATED = timedelta(days=1)
+_JOB_FLOOR_SLACK_ON_JOB_CREATED = _DATE_FLOOR_SLACK
JOB_FLOOR_SLACK_ON_RUN_STARTED = timedelta(days=7)
+# The furthest any scan floor reaches below its window start. A read whose window start is at least
+# this far after the stored coverage start scans only stored rows.
+WIDEST_SCAN_FLOOR_SLACK = JOB_FLOOR_SLACK_ON_RUN_STARTED + _DATE_FLOOR_REACH
+
def _date_floor(window_start: datetime, slack: timedelta) -> ast.Constant:
"""A date-only string ``slack`` below the window start. Date-only on purpose: it compares
diff --git a/products/engineering_analytics/backend/logic/queries/broken_tests.py b/products/engineering_analytics/backend/logic/queries/broken_tests.py
index e431f682f8dc..5702605c4d1a 100644
--- a/products/engineering_analytics/backend/logic/queries/broken_tests.py
+++ b/products/engineering_analytics/backend/logic/queries/broken_tests.py
@@ -292,7 +292,7 @@ def query_broken_tests(
# through. Keyed on (workflow_name, job_name) so a job name shared across workflows doesn't collapse.
master_by_key: dict[tuple[str | None, str, str], tuple[str | None, int | None]] = {}
breaking_master_jobs: set[str] = set()
- jobs_source = curated.jobs_source(created_floor=True)
+ jobs_source = curated.jobs_source(created_floor=True, since=date_from)
if jobs_source is not None:
master_rows = (
curated.run(
diff --git a/products/engineering_analytics/backend/logic/queries/delivery_pipeline.py b/products/engineering_analytics/backend/logic/queries/delivery_pipeline.py
index 823e8a88a7ea..a7463bc0d7a1 100644
--- a/products/engineering_analytics/backend/logic/queries/delivery_pipeline.py
+++ b/products/engineering_analytics/backend/logic/queries/delivery_pipeline.py
@@ -111,7 +111,7 @@ def query_delivery_pipeline(
}
date_to_merged_clause = date_to_filter_clause(date_to, placeholders, column="pr.merged_at")
sql = (
- _PIPELINE_SELECT.replace("__RUNS_SOURCE__", curated.run_source(started_floor=True))
+ _PIPELINE_SELECT.replace("__RUNS_SOURCE__", curated.run_source(started_floor=True, since=gate_from))
.replace("__PR_SOURCE__", curated.pr_source())
.replace("__DATE_TO_MERGED__", date_to_merged_clause)
)
diff --git a/products/engineering_analytics/backend/logic/queries/delivery_summary.py b/products/engineering_analytics/backend/logic/queries/delivery_summary.py
index e8d0637d04c8..795e99939a53 100644
--- a/products/engineering_analytics/backend/logic/queries/delivery_summary.py
+++ b/products/engineering_analytics/backend/logic/queries/delivery_summary.py
@@ -379,7 +379,7 @@ def query_ready_to_merge_facts(
def _query_pushes(curated: CuratedGitHubSource, pr_numbers: list[int], run_from: datetime) -> dict[int, list[datetime]]:
push_rows = push_rows_select(
- runs_source=curated.run_source(started_floor=True),
+ runs_source=curated.run_source(started_floor=True, since=run_from),
run_filter="pr_number IN {pr_numbers} AND run_started_at >= {run_from}",
)
response = curated.run(
@@ -400,7 +400,7 @@ def _query_gate_attempts(
gate_from = date_from - GATE_RUN_LOOKBACK
response = curated.run(
gate_attempts_sql(
- runs_source=curated.run_source(started_floor=True),
+ runs_source=curated.run_source(started_floor=True, since=gate_from),
pull_requests_source=curated.pr_source(),
pull_request_filter="pr.number IN {pr_numbers} AND pr.merged_at IS NOT NULL",
decisive_failure_conclusions_sql=DECISIVE_FAILURE_CONCLUSIONS_SQL,
diff --git a/products/engineering_analytics/backend/logic/queries/dora.py b/products/engineering_analytics/backend/logic/queries/dora.py
index 39cf2e357eb4..ee6ade167887 100644
--- a/products/engineering_analytics/backend/logic/queries/dora.py
+++ b/products/engineering_analytics/backend/logic/queries/dora.py
@@ -455,6 +455,8 @@ class _DoraScan:
placeholders: dict[str, ast.Expr]
date_from: datetime
date_to: datetime | None
+ # The earliest run start the merge-attribution scan reads, so the runs source must cover it.
+ runs_from: datetime
def run(self, sql: str, *, query_type: str, placeholders: dict[str, ast.Expr] | None = None) -> HogQLQueryResponse:
"""Run ``sql`` with the shared placeholders plus ``placeholders`` that only this read binds."""
@@ -470,7 +472,7 @@ class _DoraScan:
f"{self.deploys_cte}, {_DEPLOY_HEADS_CTE}, {_DEPLOYED_PRS_CTE}".replace(
"__PR_SOURCE__", self.curated.pr_source()
)
- .replace("__RUNS_SOURCE__", self.curated.run_source(started_floor=True))
+ .replace("__RUNS_SOURCE__", self.curated.run_source(started_floor=True, since=self.runs_from))
.replace("__TEAM_FILTER__", pr_filter)
)
@@ -729,13 +731,14 @@ def _scan(
"""Resolve the environment scope and bind the deploys CTE and placeholders every deploy read shares."""
end = date_to or datetime.now(tz=date_from.tzinfo)
prev_from = date_from - (end - date_from)
+ runs_from = prev_from - _MERGE_SCAN_LOOKBACK
placeholders: dict[str, ast.Expr] = {
"date_from": ast.Constant(value=date_from),
"prev_from": ast.Constant(value=prev_from),
"deploy_scan_floor": ast.Constant(value=prev_from - _DEPLOY_SCAN_SLACK),
- "merge_scan_floor": ast.Constant(value=prev_from - _MERGE_SCAN_LOOKBACK),
- "run_started_floor": run_started_floor_constant(prev_from - _MERGE_SCAN_LOOKBACK),
+ "merge_scan_floor": ast.Constant(value=runs_from),
+ "run_started_floor": run_started_floor_constant(runs_from),
}
if date_to is not None:
placeholders["date_to"] = ast.Constant(value=date_to)
@@ -762,6 +765,7 @@ def _scan(
placeholders=placeholders,
date_from=date_from,
date_to=date_to,
+ runs_from=runs_from,
)
diff --git a/products/engineering_analytics/backend/logic/queries/job_aggregates.py b/products/engineering_analytics/backend/logic/queries/job_aggregates.py
index eda1d8213c31..b4e4fd54698a 100644
--- a/products/engineering_analytics/backend/logic/queries/job_aggregates.py
+++ b/products/engineering_analytics/backend/logic/queries/job_aggregates.py
@@ -99,7 +99,7 @@ def query_job_aggregates(
# recognisable while its original attempt is still in the scan: a re-run a day or more after the
# original push would otherwise pull the copy inside the window, drop the original below a tight
# floor, and count the copy as an execution. So take the wide floor, same as the cost surfaces.
- cost_source = curated.job_cost_source(created_floor=True)
+ cost_source = curated.job_cost_source(created_floor=True, since=date_from)
if cost_source is None:
return []
@@ -120,7 +120,7 @@ def query_job_aggregates(
template.replace("__COST_SOURCE__", cost_source)
.replace("__COST_RUN_SCOPE__", cost_run_scope_filter_clause(run_scope))
.replace("__RUNS_RUN_SCOPE__", run_scope_filter_clause(run_scope))
- .replace("__RUNS_SOURCE__", curated.run_source())
+ .replace("__RUNS_SOURCE__", curated.run_source(since=date_from))
.replace("__DATE_TO__", date_to_clause)
.replace("__RUNS_DATE_TO__", runs_date_to_clause)
.replace("__BRANCH__", branch_clause)
diff --git a/products/engineering_analytics/backend/logic/queries/master_failures.py b/products/engineering_analytics/backend/logic/queries/master_failures.py
index 81ca76705cc4..476b1e2982b6 100644
--- a/products/engineering_analytics/backend/logic/queries/master_failures.py
+++ b/products/engineering_analytics/backend/logic/queries/master_failures.py
@@ -69,7 +69,9 @@ def query_master_failures(
placeholders["date_to"] = ast.Constant(value=date_to)
runs_response = curated.run(
- _FAILED_RUNS_SELECT.replace("__RUNS_SOURCE__", curated.run_source()).replace("__DATE_TO__", date_to_clause),
+ _FAILED_RUNS_SELECT.replace("__RUNS_SOURCE__", curated.run_source(since=date_from)).replace(
+ "__DATE_TO__", date_to_clause
+ ),
query_type="engineering_analytics.master_failures_runs",
placeholders=placeholders,
)
@@ -83,7 +85,7 @@ def query_master_failures(
# run_id IN (...) scopes the result but not the scan, and the builder's is_rerun_copy window would
# otherwise sort the whole jobs history. The run ids come from a run_started_at-windowed scan, so
# the floor takes the wider re-run slack.
- jobs_source = curated.jobs_source(created_floor=True)
+ jobs_source = curated.jobs_source(created_floor=True, since=date_from)
if jobs_source is not None:
jobs_response = curated.run(
_FAILED_JOBS_SELECT.replace("__JOBS_SOURCE__", jobs_source),
diff --git a/products/engineering_analytics/backend/logic/queries/merge_queue_overview.py b/products/engineering_analytics/backend/logic/queries/merge_queue_overview.py
index 9fbf8e6c4fa0..9f8ad41968bd 100644
--- a/products/engineering_analytics/backend/logic/queries/merge_queue_overview.py
+++ b/products/engineering_analytics/backend/logic/queries/merge_queue_overview.py
@@ -220,7 +220,7 @@ def query_merge_queue_overview(
}
date_to_merged_clause = date_to_filter_clause(date_to, placeholders, column="pr.merged_at")
gate_attempts = gate_attempts_sql(
- runs_source=curated.run_source(started_floor=True),
+ runs_source=curated.run_source(started_floor=True, since=gate_from),
pull_requests_source=curated.pr_source(),
pull_request_filter="""pr.merged_at IS NOT NULL AND (
(pr.merged_at >= {date_from} __DATE_TO_MERGED__ AND r.run_started_at >= {current_gate_from})
diff --git a/products/engineering_analytics/backend/logic/queries/pr_cost.py b/products/engineering_analytics/backend/logic/queries/pr_cost.py
index d24914250ab5..9e7d0a9a09ad 100644
--- a/products/engineering_analytics/backend/logic/queries/pr_cost.py
+++ b/products/engineering_analytics/backend/logic/queries/pr_cost.py
@@ -237,7 +237,7 @@ def query_pr_costs(
A population that can be repo-wide (every PR merged in a window) passes ``run_from``: runs started
before it are left out, which trims the jobs scan for a population of thousands of PRs.
"""
- cost_source = curated.job_cost_source(created_floor=run_from is not None)
+ cost_source = curated.job_cost_source(created_floor=run_from is not None, since=run_from)
if cost_source is None or not pr_numbers:
return {}
placeholders: dict[str, ast.Expr] = {"pr_numbers": ast.Constant(value=pr_numbers)}
@@ -303,7 +303,7 @@ def query_workflow_window_costs(
run-level ``query_workflow_health`` (a job whose run is missing has a NULL run_started_at and is
excluded by the window filter, matching the legacy INNER JOIN).
"""
- cost_source = curated.job_cost_source(created_floor=True)
+ cost_source = curated.job_cost_source(created_floor=True, since=date_from)
if cost_source is None:
return {}
placeholders: dict[str, ast.Expr] = {
@@ -355,7 +355,7 @@ def query_author_workflow_costs(
author→runs link goes through their PR numbers (the one attribution rule, SPEC §6) — the cost
source's normalized pr_number never matches on an unattributed (NULL) run.
"""
- cost_source = curated.job_cost_source(created_floor=True)
+ cost_source = curated.job_cost_source(created_floor=True, since=date_from)
if cost_source is None:
return []
placeholders: dict[str, ast.Expr] = {
@@ -430,7 +430,7 @@ def query_workflow_window_costs_with_prev(
prior implementation), so a workflow present only in the other window doesn't create a phantom
zero-cost entry.
"""
- cost_source = curated.job_cost_source(created_floor=True)
+ cost_source = curated.job_cost_source(created_floor=True, since=prev_from)
if cost_source is None:
return _EMPTY_WINDOW_COSTS
placeholders: dict[str, ast.Expr] = {
@@ -537,7 +537,7 @@ def query_cost_per_merge_series(
there's no cost to divide, so the series is empty (the UI shows the same "sync jobs" state as
the other cost surfaces).
"""
- cost_source = curated.job_cost_source(created_floor=True)
+ cost_source = curated.job_cost_source(created_floor=True, since=date_from)
if cost_source is None:
return []
@@ -630,7 +630,7 @@ def query_workflow_runner_costs(
"""A workflow's CI cost broken down by runner tier over [date_from, date_to] (optional branch and
run scope), highest spend first. Empty when the jobs source isn't synced. Grouped by the rendered
(provider, os, vcpu) tier and mapped to its display badge/label via ``runner_tier_descriptor``."""
- cost_source = curated.job_cost_source(created_floor=True)
+ cost_source = curated.job_cost_source(created_floor=True, since=date_from)
if cost_source is None:
return []
placeholders: dict[str, ast.Expr] = {
diff --git a/products/engineering_analytics/backend/logic/queries/pull_request_timelines.py b/products/engineering_analytics/backend/logic/queries/pull_request_timelines.py
index faae54a7fb9e..e00dac407912 100644
--- a/products/engineering_analytics/backend/logic/queries/pull_request_timelines.py
+++ b/products/engineering_analytics/backend/logic/queries/pull_request_timelines.py
@@ -407,7 +407,7 @@ class PullRequestTimelinesQuery:
self, pr_numbers: list[int], run_from: datetime, default_branch: str
) -> tuple[dict[int, list[RunAttempt]], MasterFailureIndex]:
rows = self._curated.run_paged(
- _RUNS_SELECT.replace("__RUNS_SOURCE__", self._curated.run_source(started_floor=True)),
+ _RUNS_SELECT.replace("__RUNS_SOURCE__", self._curated.run_source(started_floor=True, since=run_from)),
page_key=(("id", 0), ("ci_engine", 11)),
query_type="engineering_analytics.pull_request_timelines_runs",
placeholders=self._runs_placeholders(pr_numbers, run_from),
@@ -505,7 +505,7 @@ class PullRequestTimelinesQuery:
def _query_pushes(self, pr_numbers: list[int], run_from: datetime) -> dict[int, list[Push]]:
sql = push_rows_select(
- runs_source=self._curated.run_source(started_floor=True),
+ runs_source=self._curated.run_source(started_floor=True, since=run_from),
run_filter="pr_number IN {pr_numbers} AND run_started_at >= {run_from}",
)
rows = self._curated.run_paged(
@@ -524,7 +524,7 @@ class PullRequestTimelinesQuery:
gate_from = self._date_from - GATE_RUN_LOOKBACK
rows = self._curated.run_paged(
gate_attempts_sql(
- runs_source=self._curated.run_source(started_floor=True),
+ runs_source=self._curated.run_source(started_floor=True, since=gate_from),
pull_requests_source=self._curated.pr_source(),
pull_request_filter="pr.number IN {pr_numbers}",
decisive_failure_conclusions_sql=DECISIVE_FAILURE_CONCLUSIONS_SQL,
@@ -551,7 +551,7 @@ class PullRequestTimelinesQuery:
return gates
def _query_job_attempts(self, run_ids: list[int], run_from: datetime) -> dict[tuple[str, int], list[_JobAttempt]]:
- source = self._curated.jobs_source(created_floor=True)
+ source = self._curated.jobs_source(created_floor=True, since=run_from)
if source is None or not run_ids:
return {}
rows = self._curated.run_paged(
@@ -583,12 +583,12 @@ class PullRequestTimelinesQuery:
) -> MasterFailureIndex:
# Filtered by workflow, not by job name: the index compares job names without their shard
# suffix, and an exact-name filter would drop the other shards of the same job.
- jobs_source = self._curated.jobs_source(created_floor=True)
+ jobs_source = self._curated.jobs_source(created_floor=True, since=run_from)
if not workflow_names or not default_branch or jobs_source is None:
return MasterFailureIndex([])
rows = self._curated.run_paged(
_MASTER_FAILURES_SELECT.replace("__JOBS_SOURCE__", jobs_source).replace(
- "__RUNS_SOURCE__", self._curated.run_source(started_floor=True)
+ "__RUNS_SOURCE__", self._curated.run_source(started_floor=True, since=run_from)
),
page_key=(("job_id", 3), ("ci_engine", 4)),
query_type="engineering_analytics.pull_request_timelines_master_failures",
diff --git a/products/engineering_analytics/backend/logic/queries/repo_overview.py b/products/engineering_analytics/backend/logic/queries/repo_overview.py
index 45df500aa260..a9b0eea8113d 100644
--- a/products/engineering_analytics/backend/logic/queries/repo_overview.py
+++ b/products/engineering_analytics/backend/logic/queries/repo_overview.py
@@ -115,9 +115,9 @@ def query_default_branch(
if date_to is not None:
placeholders["date_to"] = ast.Constant(value=date_to)
response = curated.run(
- _DEFAULT_BRANCH_SELECT.replace("__RUNS_SOURCE__", curated.run_source(started_floor=True)).replace(
- "__DATE_TO__", date_to_clause
- ),
+ _DEFAULT_BRANCH_SELECT.replace(
+ "__RUNS_SOURCE__", curated.run_source(started_floor=True, since=date_from)
+ ).replace("__DATE_TO__", date_to_clause),
query_type="engineering_analytics.default_branch",
placeholders=placeholders,
)
@@ -206,7 +206,7 @@ def query_success_rate_series(
if date_to is not None:
placeholders["date_to"] = ast.Constant(value=date_to)
sql = (
- _PASS_RATE_SERIES_SELECT.replace("__RUNS_SOURCE__", curated.run_source(started_floor=True))
+ _PASS_RATE_SERIES_SELECT.replace("__RUNS_SOURCE__", curated.run_source(started_floor=True, since=date_from))
.replace("__DATE_TO__", date_to_clause)
.replace("__BUCKET_FN__", bucket_expr(granularity, "run_started_at"))
)
@@ -425,7 +425,7 @@ def _query_run_headlines(
sql = (
_RUNS_SELECT.replace("__CUR__", windows.current)
.replace("__PREV__", windows.previous)
- .replace("__RUNS_SOURCE__", curated.run_source(started_floor=True))
+ .replace("__RUNS_SOURCE__", curated.run_source(started_floor=True, since=prev_from))
.replace("__DATE_TO__", date_to_clause)
)
response = curated.run(sql, query_type="engineering_analytics.repo_overview_runs", placeholders=placeholders)
diff --git a/products/engineering_analytics/backend/logic/queries/repo_run_activity.py b/products/engineering_analytics/backend/logic/queries/repo_run_activity.py
index 8a4402bc7b1e..e2f748cadff7 100644
--- a/products/engineering_analytics/backend/logic/queries/repo_run_activity.py
+++ b/products/engineering_analytics/backend/logic/queries/repo_run_activity.py
@@ -86,7 +86,7 @@ def query_repo_run_activity(
date_to_clause = "AND r.run_started_at <= {date_to}"
placeholders["date_to"] = ast.Constant(value=date_to)
response = curated.run(
- _SELECT.replace("__RUNS_SOURCE__", curated.run_source()).replace("__DATE_TO__", date_to_clause),
+ _SELECT.replace("__RUNS_SOURCE__", curated.run_source(since=date_from)).replace("__DATE_TO__", date_to_clause),
query_type="engineering_analytics.repo_run_activity",
placeholders=placeholders,
)
diff --git a/products/engineering_analytics/backend/logic/queries/timing_candidates.py b/products/engineering_analytics/backend/logic/queries/timing_candidates.py
index 95b2c0295c05..7174ffb857c7 100644
--- a/products/engineering_analytics/backend/logic/queries/timing_candidates.py
+++ b/products/engineering_analytics/backend/logic/queries/timing_candidates.py
@@ -87,7 +87,7 @@ def query_timing_candidates(
identity = "workflow_name = {workflow_name}"
placeholders["workflow_name"] = ast.Constant(value=run.workflow_name)
response = curated.run(
- _SELECT.replace("__RUNS_SOURCE__", curated.run_source(started_floor=True)).replace(
+ _SELECT.replace("__RUNS_SOURCE__", curated.run_source(started_floor=True, since=window_start)).replace(
"__WORKFLOW_IDENTITY__", identity
),
query_type="engineering_analytics.ci_timing_candidates",
diff --git a/products/engineering_analytics/backend/logic/queries/workflow_flakiness.py b/products/engineering_analytics/backend/logic/queries/workflow_flakiness.py
index d8f76f6229bc..35bee9769fef 100644
--- a/products/engineering_analytics/backend/logic/queries/workflow_flakiness.py
+++ b/products/engineering_analytics/backend/logic/queries/workflow_flakiness.py
@@ -77,12 +77,12 @@ def query_workflow_flakiness(
min_failed_duration_seconds: int = NO_OP_JOB_MAX_SECONDS,
workload: Workload = Workload.DEFAULT,
) -> list[FlakyJobRun]:
- jobs_source = curated.jobs_source(created_floor=True)
+ jobs_source = curated.jobs_source(created_floor=True, since=date_from)
if jobs_source is None:
return []
response = curated.run(
_SELECT.replace("__JOBS_SOURCE__", jobs_source).replace(
- "__RUNS_SOURCE__", curated.run_source(started_floor=True)
+ "__RUNS_SOURCE__", curated.run_source(started_floor=True, since=date_from)
),
query_type="engineering_analytics.workflow_flakiness",
workload=workload,
diff --git a/products/engineering_analytics/backend/logic/queries/workflow_health.py b/products/engineering_analytics/backend/logic/queries/workflow_health.py
index fb3c05fd8176..4e5ffd370cda 100644
--- a/products/engineering_analytics/backend/logic/queries/workflow_health.py
+++ b/products/engineering_analytics/backend/logic/queries/workflow_health.py
@@ -31,6 +31,7 @@ from products.engineering_analytics.backend.facade.contracts import (
WorkflowHealthItem,
WorkflowHealthRunScope,
)
+from products.engineering_analytics.backend.logic._shared import _prior_window
from products.engineering_analytics.backend.logic.queries._buckets import (
Granularity,
bucket_expr,
@@ -230,7 +231,7 @@ def query_time_to_green_series(
}
date_to_clause = date_to_filter_clause(date_to, placeholders)
sql = (
- _TIME_TO_GREEN_SELECT.replace("__RUNS_SOURCE__", curated.run_source(started_floor=True))
+ _TIME_TO_GREEN_SELECT.replace("__RUNS_SOURCE__", curated.run_source(started_floor=True, since=date_from))
.replace("__DATE_TO__", date_to_clause)
.replace("__BUCKET_FN__", bucket_expr(granularity, "round_start"))
)
@@ -272,7 +273,7 @@ def query_time_to_green_window(
}
date_to_clause = date_to_filter_clause(date_to, placeholders)
sql = (
- _TIME_TO_GREEN_WINDOW_SELECT.replace("__RUNS_SOURCE__", curated.run_source(started_floor=True))
+ _TIME_TO_GREEN_WINDOW_SELECT.replace("__RUNS_SOURCE__", curated.run_source(started_floor=True, since=prev_from))
.replace("__DATE_TO__", date_to_clause)
.replace("__CUR__", windows.current)
.replace("__PREV__", windows.previous)
@@ -302,7 +303,9 @@ def query_workflow_health(
run_scope_clause = run_scope_filter_clause(run_scope)
workflow_clause = workflow_name_filter_clause(workflow_name, placeholders)
- runs_source = curated.run_source(started_floor=True)
+ prev_from = _prior_window(date_from, date_to).scan_from
+ # One source feeds both the current and the previous window, so it must cover the earlier start.
+ runs_source = curated.run_source(started_floor=True, since=prev_from)
def fill(template: str) -> str:
return (
@@ -314,8 +317,6 @@ def query_workflow_health(
.replace("__BUCKET_FN__", bucket_expr(granularity))
)
- end = date_to or datetime.now(tz=date_from.tzinfo)
- prev_from = date_from - (end - date_from)
with curated.concurrent_reads() as reads:
headline_read = reads.submit(
lambda: curated.run(
diff --git a/products/engineering_analytics/backend/logic/queries/workflow_jobs.py b/products/engineering_analytics/backend/logic/queries/workflow_jobs.py
index 54e23364623c..df61025444bd 100644
--- a/products/engineering_analytics/backend/logic/queries/workflow_jobs.py
+++ b/products/engineering_analytics/backend/logic/queries/workflow_jobs.py
@@ -150,7 +150,7 @@ def query_jobs_by_run(
"""The job list of one attempt of each run, keyed by run id. ``run_attempts`` maps each run id
to the attempt to read, and ``earliest_run_started_at`` is the earliest start among those runs.
``job_name`` keeps one job name. Steps are the widest column, so they are read only on request."""
- jobs_source = curated.jobs_source(created_floor=True, include_steps=include_steps)
+ jobs_source = curated.jobs_source(created_floor=True, since=earliest_run_started_at, include_steps=include_steps)
if jobs_source is None or not run_attempts:
return {}
placeholders: dict[str, ast.Expr] = {
diff --git a/products/engineering_analytics/backend/logic/queries/workflow_run_activity.py b/products/engineering_analytics/backend/logic/queries/workflow_run_activity.py
index afb0ff4e1522..d4ceef0358fb 100644
--- a/products/engineering_analytics/backend/logic/queries/workflow_run_activity.py
+++ b/products/engineering_analytics/backend/logic/queries/workflow_run_activity.py
@@ -81,7 +81,7 @@ def query_workflow_run_activity(
date_to_clause = date_to_filter_clause(date_to, placeholders)
branch_clause = branch_filter_clause(branch, placeholders)
response = curated.run(
- _SELECT.replace("__RUNS_SOURCE__", curated.run_source())
+ _SELECT.replace("__RUNS_SOURCE__", curated.run_source(since=date_from))
.replace("__DATE_TO__", date_to_clause)
.replace("__BRANCH__", branch_clause)
.replace("__RUN_SCOPE__", run_scope_filter_clause(run_scope)),
diff --git a/products/engineering_analytics/backend/logic/views/incremental_floor.py b/products/engineering_analytics/backend/logic/views/incremental_floor.py
index a2a6149fb090..10836add2681 100644
--- a/products/engineering_analytics/backend/logic/views/incremental_floor.py
+++ b/products/engineering_analytics/backend/logic/views/incremental_floor.py
@@ -1,7 +1,10 @@
"""The rows one incremental materialization reads: the rows created at or after the run's window
start. The window start reaches the view as the ``{incremental_window_start}`` placeholder, which
data modeling binds to the key's watermark minus the lookback on an incremental run and to NULL on
-a full build, so the floor then keeps every row.
+a full build. A full build seeds only the rows created in the last ``SEED_LOOKBACK``, because it
+runs as one ClickHouse query and a busy repository's whole job history does not fit in one. Each
+incremental run adds newer rows, and a read that starts before the seed uses the raw tables instead
+(``MaterializedCI.covers``).
The floors compare the raw ISO-8601 ``created_at`` strings with the window start formatted the
same way, which is the one comparison that lets a warehouse scan skip files; the filter data
@@ -11,9 +14,16 @@ workflow took decides whether a GitHub run is a shell. A scan for such rows reac
``JOB_FLOOR_SLACK_ON_RUN_STARTED`` before the window.
"""
-from products.engineering_analytics.backend.logic.queries._workflow_filters import JOB_FLOOR_SLACK_ON_RUN_STARTED
+from products.engineering_analytics.backend.logic._shared import DEFAULT_WINDOW
+from products.engineering_analytics.backend.logic.queries._workflow_filters import (
+ JOB_FLOOR_SLACK_ON_RUN_STARTED,
+ WIDEST_SCAN_FLOOR_SLACK,
+)
WINDOW_START = "{incremental_window_start}"
+# The slack keeps a default window's scan floor inside the seed, so the first build can serve a
+# default read.
+SEED_LOOKBACK = DEFAULT_WINDOW + WIDEST_SCAN_FLOOR_SLACK
def stored_rows(column: str = "created_at") -> str:
@@ -27,8 +37,9 @@ def stored_rows_and_earlier(column: str = "created_at") -> str:
def _floor(*, days_before: int) -> str:
- start = f"toTimeZone({WINDOW_START}, 'UTC')"
+ seed_start = f"toTimeZone(now(), 'UTC') - INTERVAL {SEED_LOOKBACK.days} DAY"
+ start = f"ifNull(toTimeZone({WINDOW_START}, 'UTC'), {seed_start})"
if days_before:
start = f"{start} - INTERVAL {days_before} DAY"
# A fractional timestamp sorts before the same second's trailing Z, so use the second's prefix.
- return f"ifNull(formatDateTime({start}, '%Y-%m-%dT%H:%i:%S'), '')"
+ return f"formatDateTime({start}, '%Y-%m-%dT%H:%i:%S')"
diff --git a/products/engineering_analytics/backend/tests/test_access_control.py b/products/engineering_analytics/backend/tests/test_access_control.py
index b15aff9212e3..8d278acadc75 100644
--- a/products/engineering_analytics/backend/tests/test_access_control.py
+++ b/products/engineering_analytics/backend/tests/test_access_control.py
@@ -39,7 +39,7 @@ from products.engineering_analytics.backend.tests._github_fixtures import (
create_warehouse_table_row,
link_schema,
)
-from products.engineering_analytics.backend.tests.test_materialized_ci import _ready_views
+from products.engineering_analytics.backend.tests.test_materialized_ci import _covered_since, _ready_views
from products.warehouse_sources.backend.facade.models import ExternalDataSource
from products.warehouse_sources.backend.facade.testing import WarehouseAccessControlTestMixin
@@ -272,7 +272,7 @@ class TestEngineeringAnalyticsWarehouseAcl(WarehouseAccessControlTestMixin):
reader = CuratedGitHubSource.for_team(
self.team, user_access_control=UserAccessControl(self.no_access_user, self.team)
)
- source = reader.run_source()
+ source = reader.run_source(since=_covered_since())
if condition == "allowed":
assert ci_rows.RUNS_VIEW_NAME in source
else:
diff --git a/products/engineering_analytics/backend/tests/test_ci_views.py b/products/engineering_analytics/backend/tests/test_ci_views.py
index 50486d780ef1..328a8285e21c 100644
--- a/products/engineering_analytics/backend/tests/test_ci_views.py
+++ b/products/engineering_analytics/backend/tests/test_ci_views.py
@@ -1,7 +1,7 @@
import json
import tempfile
from collections.abc import Callable
-from datetime import UTC, datetime
+from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
from uuid import uuid4
@@ -40,6 +40,7 @@ from products.engineering_analytics.backend.logic.views import (
job_costs,
pr_friction,
)
+from products.engineering_analytics.backend.logic.views.incremental_floor import SEED_LOOKBACK
from products.engineering_analytics.backend.logic.views.source_schema import (
WORKFLOW_JOBS_COLUMNS,
WORKFLOW_RUNS_COLUMNS,
@@ -444,8 +445,8 @@ class TestStoredCIConfig(SimpleTestCase):
class TestStoredCIViews(ClickhouseTestMixin, BaseTest):
@parameterized.expand(
[
- ("runs", ci_rows.runs_select, ci_rows.RUN_FIELDS, 2, {"is_handoff_shell": 0}),
- ("jobs", ci_rows.jobs_select, ci_rows.JOB_FIELDS, 3, {"is_rerun_copy": 0, "is_handoff_shell": 0}),
+ ("runs", ci_rows.runs_select, ci_rows.RUN_FIELDS, [100, 200], {"is_handoff_shell": 0}),
+ ("jobs", ci_rows.jobs_select, ci_rows.JOB_FIELDS, [1, 2, 3], {"is_rerun_copy": 0, "is_handoff_shell": 0}),
]
)
def test_view_runs_with_the_window_start_bound(
@@ -453,10 +454,12 @@ class TestStoredCIViews(ClickhouseTestMixin, BaseTest):
_name: str,
select: Callable[[JobSourceTables], str],
fields: dict[str, Any],
- seeded_rows: int,
+ seeded_ids: list[int],
flags: dict[str, int],
) -> None:
- created_at = "2026-01-01T10:00:00.123Z"
+ recent = datetime.now(UTC).replace(microsecond=0) - timedelta(days=1)
+ created_at = recent.strftime("%Y-%m-%dT%H:%M:%S.123Z")
+ before_seed = (recent - SEED_LOOKBACK).strftime("%Y-%m-%dT%H:%M:%S.123Z")
jobs_table = _create_table(
self,
"github_workflow_jobs",
@@ -465,6 +468,7 @@ class TestStoredCIViews(ClickhouseTestMixin, BaseTest):
{**_job_row(1, run_id=100, name="a"), "created_at": created_at},
{**_job_row(2, run_id=100, name="b"), "created_at": created_at},
{**_job_row(3, run_id=200, name="c"), "created_at": created_at},
+ {**_job_row(4, run_id=300, name="d"), "created_at": before_seed},
],
)
runs_table = _create_table(
@@ -474,6 +478,7 @@ class TestStoredCIViews(ClickhouseTestMixin, BaseTest):
[
{**_run_row(100, head_sha="s100", pr_numbers=[5], head_commit={}), "created_at": created_at},
{**_run_row(200, head_sha="s200", pr_numbers=[], head_commit={}), "created_at": created_at},
+ {**_run_row(300, head_sha="s300", pr_numbers=[], head_commit={}), "created_at": before_seed},
],
)
source = JobSourceTables(
@@ -485,10 +490,11 @@ class TestStoredCIViews(ClickhouseTestMixin, BaseTest):
query = f"SELECT {', '.join(fields)} FROM ({select(source)}) ORDER BY id"
rows_by_window = {}
- for since, expected_rows in (
- (None, seeded_rows),
- (datetime(2026, 1, 1, 10, tzinfo=UTC), seeded_rows),
- (datetime(2026, 1, 1, 11, tzinfo=UTC), 0),
+ # A full build (NULL window start) seeds only the last SEED_LOOKBACK, so the older row stays out.
+ for since, expected_ids in (
+ (None, seeded_ids),
+ (recent, seeded_ids),
+ (recent + timedelta(hours=1), []),
):
response = execute_hogql_query(
query=query,
@@ -497,7 +503,7 @@ class TestStoredCIViews(ClickhouseTestMixin, BaseTest):
placeholders={"incremental_window_start": ast.Constant(value=since)},
)
assert response.columns == list(fields)
- assert len(response.results) == expected_rows, since
+ assert [dict(zip(fields, row))["id"] for row in response.results] == expected_ids, since
rows_by_window[since] = response.results
first = dict(zip(fields, rows_by_window[None][0]))
diff --git a/products/engineering_analytics/backend/tests/test_materialized_ci.py b/products/engineering_analytics/backend/tests/test_materialized_ci.py
index 9f155d5bac44..7663103a0d50 100644
--- a/products/engineering_analytics/backend/tests/test_materialized_ci.py
+++ b/products/engineering_analytics/backend/tests/test_materialized_ci.py
@@ -17,6 +17,7 @@ from posthog.hogql.query import execute_hogql_query
from products.data_modeling.backend.facade.api import (
definition_fingerprint,
get_incremental_config,
+ get_incremental_state,
set_incremental_state,
)
from products.data_modeling.backend.facade.models import (
@@ -27,6 +28,7 @@ from products.data_modeling.backend.facade.models import (
from products.engineering_analytics.backend.facade.contracts import MATERIALIZED_VIEWS_FEATURE_FLAG
from products.engineering_analytics.backend.logic.materialized_ci import MAX_MATERIALIZATION_AGE
from products.engineering_analytics.backend.logic.queries._curated import CuratedGitHubSource
+from products.engineering_analytics.backend.logic.queries._workflow_filters import WIDEST_SCAN_FLOOR_SLACK
from products.engineering_analytics.backend.logic.sources import (
DEPOT_JOB_ATTEMPTS_SCHEMA,
PULL_REQUESTS_SCHEMA,
@@ -34,6 +36,7 @@ from products.engineering_analytics.backend.logic.sources import (
WORKFLOW_RUNS_SCHEMA,
)
from products.engineering_analytics.backend.logic.views import ci_rows
+from products.engineering_analytics.backend.logic.views.incremental_floor import SEED_LOOKBACK
from products.engineering_analytics.backend.logic.views.source_schema import (
DEPOT_JOB_ATTEMPTS_COLUMNS,
PULL_REQUESTS_COLUMNS,
@@ -54,6 +57,10 @@ from products.warehouse_sources.backend.facade.models import DataWarehouseTable,
from products.warehouse_sources.backend.facade.types import DataWarehouseManagedViewSetKind
+def _covered_since() -> datetime:
+ return timezone.now() - timedelta(days=7)
+
+
def _ready_views(
test: BaseTest, *, backing_tables: dict[str, DataWarehouseTable] | None = None
) -> dict[str, DataWarehouseSavedQuery]:
@@ -137,10 +144,11 @@ class TestMaterializedCI(BaseTest):
_ready_views(self)
with patch("posthoganalytics.feature_enabled", return_value=enabled):
reader = self._reader()
- runs = set(get_table_names(parse_select(reader.run_source())))
- jobs = set(get_table_names(parse_select(reader.jobs_source() or "")))
- costs = set(get_table_names(parse_select(reader.job_cost_source() or "")))
- detail = set(get_table_names(parse_select(reader.jobs_source(include_steps=True) or "")))
+ since = _covered_since()
+ runs = set(get_table_names(parse_select(reader.run_source(since=since))))
+ jobs = set(get_table_names(parse_select(reader.jobs_source(since=since) or "")))
+ costs = set(get_table_names(parse_select(reader.job_cost_source(since=since) or "")))
+ detail = set(get_table_names(parse_select(reader.jobs_source(since=since, include_steps=True) or "")))
current = set(get_table_names(parse_select(reader.runs_cte().removeprefix("runs AS "))))
if stored:
assert runs == {ci_rows.RUNS_VIEW_NAME}
@@ -161,12 +169,18 @@ class TestMaterializedCI(BaseTest):
"new_definition",
"missing_stored_column",
"wrong_viewset",
+ "full_build_not_recorded",
]
)
def test_unusable_materializations_fall_back_together(self, condition: str) -> None:
views = _ready_views(self)
jobs = views[ci_rows.JOBS_VIEW_NAME]
- if condition == "missing_pair":
+ if condition == "full_build_not_recorded":
+ fingerprint = get_incremental_state(jobs).definition_fingerprint
+ DataWarehouseSavedQuery.objects.filter(id=jobs.id).update(incremental_state={})
+ jobs.refresh_from_db()
+ set_incremental_state(jobs, watermark=timezone.now(), fingerprint=fingerprint, mode="incremental")
+ elif condition == "missing_pair":
DataWarehouseSavedQuery.objects.filter(id=jobs.id).update(deleted=True)
elif condition == "disabled_view":
DataWarehouseSavedQuery.objects.filter(id=jobs.id).update(is_materialized=False)
@@ -188,8 +202,45 @@ class TestMaterializedCI(BaseTest):
with patch("posthoganalytics.feature_enabled", return_value=True):
reader = self._reader()
- assert "raw_github_workflow_runs" in get_table_names(parse_select(reader.run_source()))
- assert "raw_github_workflow_jobs" in get_table_names(parse_select(reader.jobs_source() or ""))
+ assert "raw_github_workflow_runs" in get_table_names(
+ parse_select(reader.run_source(since=_covered_since()))
+ )
+ assert "raw_github_workflow_jobs" in get_table_names(
+ parse_select(reader.jobs_source(since=_covered_since()) or "")
+ )
+
+ @parameterized.expand(
+ [
+ ("unbounded", None, timedelta(0), False),
+ ("floor_reaches_before_the_seed", WIDEST_SCAN_FLOOR_SLACK - timedelta(seconds=1), timedelta(0), False),
+ ("inside_the_seed", WIDEST_SCAN_FLOOR_SLACK, timedelta(0), True),
+ ("inside_the_older_seed_only", WIDEST_SCAN_FLOOR_SLACK - timedelta(days=10), timedelta(days=10), False),
+ ]
+ )
+ def test_reads_windowed_before_the_stored_coverage_use_raw_tables(
+ self, _: str, offset_from_coverage_start: timedelta | None, jobs_seeded_earlier_by: timedelta, stored: bool
+ ) -> None:
+ views = _ready_views(self)
+ if jobs_seeded_earlier_by:
+ jobs_view = views[ci_rows.JOBS_VIEW_NAME]
+ fingerprint = get_incremental_state(jobs_view).definition_fingerprint
+ with time_machine.travel(timezone.now() - jobs_seeded_earlier_by, tick=False):
+ set_incremental_state(jobs_view, watermark=timezone.now(), fingerprint=fingerprint, mode="full_refresh")
+ # The coverage starts at the later of the two seeds.
+ covers_from = timezone.now() - SEED_LOOKBACK
+ window_start = covers_from + offset_from_coverage_start if offset_from_coverage_start is not None else None
+ with patch("posthoganalytics.feature_enabled", return_value=True):
+ reader = self._reader()
+ runs = set(get_table_names(parse_select(reader.run_source(since=window_start))))
+ jobs = set(get_table_names(parse_select(reader.jobs_source(since=window_start) or "")))
+ costs = set(get_table_names(parse_select(reader.job_cost_source(since=window_start) or "")))
+ if stored:
+ assert runs == {ci_rows.RUNS_VIEW_NAME}
+ assert jobs == {ci_rows.JOBS_VIEW_NAME}
+ assert costs == {ci_rows.RUNS_VIEW_NAME, ci_rows.JOBS_VIEW_NAME}
+ else:
+ assert "raw_github_workflow_runs" in runs
+ assert "raw_github_workflow_jobs" in jobs & costs
@parameterized.expand([DataModelingJob.Status.RUNNING, DataModelingJob.Status.FAILED])
def test_an_unfinished_newer_build_keeps_the_completed_one_serving(self, status: str) -> None:
@@ -198,8 +249,12 @@ class TestMaterializedCI(BaseTest):
with patch("posthoganalytics.feature_enabled", return_value=True):
reader = self._reader()
- assert set(get_table_names(parse_select(reader.run_source()))) == {ci_rows.RUNS_VIEW_NAME}
- assert set(get_table_names(parse_select(reader.jobs_source() or ""))) == {ci_rows.JOBS_VIEW_NAME}
+ assert set(get_table_names(parse_select(reader.run_source(since=_covered_since())))) == {
+ ci_rows.RUNS_VIEW_NAME
+ }
+ assert set(get_table_names(parse_select(reader.jobs_source(since=_covered_since()) or ""))) == {
+ ci_rows.JOBS_VIEW_NAME
+ }
def test_flag_exception_falls_back_to_live_sources(self) -> None:
_ready_views(self)
@@ -210,7 +265,9 @@ class TestMaterializedCI(BaseTest):
return False
with patch("posthoganalytics.feature_enabled", side_effect=flag):
- assert "raw_github_workflow_runs" in get_table_names(parse_select(self._reader().run_source()))
+ assert "raw_github_workflow_runs" in get_table_names(
+ parse_select(self._reader().run_source(since=_covered_since()))
+ )
def test_freshness_and_cache_scope_follow_materialization(self) -> None:
_ready_views(self)
@@ -256,8 +313,10 @@ class TestMaterializedCI(BaseTest):
_ready_views(self)
with patch("posthoganalytics.feature_enabled", return_value=True):
reader = CuratedGitHubSource.for_team(self.team, source_id=str(other.id), repo="example/widgets")
- assert "second_github_workflow_runs" in get_table_names(parse_select(reader.run_source()))
- assert "depot_attempts" in get_table_names(parse_select(reader.jobs_source() or ""))
+ assert "second_github_workflow_runs" in get_table_names(
+ parse_select(reader.run_source(since=_covered_since()))
+ )
+ assert "depot_attempts" in get_table_names(parse_select(reader.jobs_source(since=_covered_since()) or ""))
@time_machine.travel(lambda: datetime.now(UTC), tick=False)
@@ -426,13 +485,14 @@ class TestMaterializedCIParity(_WarehouseMixin):
def read(*, stored: bool = False, floor: datetime | None = None) -> dict[str, list[tuple]]:
reader = CuratedGitHubSource.for_team(self.team, source_id=str(source.id), repo=repository)
+ since = _covered_since()
if stored:
- assert set(get_table_names(parse_select(reader.run_source()))) == {ci_rows.RUNS_VIEW_NAME}
+ assert set(get_table_names(parse_select(reader.run_source(since=since)))) == {ci_rows.RUNS_VIEW_NAME}
floor_value = floor.strftime("%Y-%m-%d") if floor is not None else None
queries = {
- "runs": f"SELECT id, run_attempt, pr_number, ci_engine, stopped_reporting FROM {reader.run_source(started_floor=floor is not None)} ORDER BY ci_engine, id",
- "jobs": f"SELECT id, run_id, run_attempt, is_rerun_copy, ci_engine FROM {reader.jobs_source(created_floor=floor is not None)} ORDER BY ci_engine, id",
- "costs": f"SELECT run_id, job_name, run_attempt, head_branch, estimated_cost_usd, pr_number, ci_engine FROM {reader.job_cost_source(created_floor=floor is not None)} ORDER BY ci_engine, run_id, job_name, run_attempt",
+ "runs": f"SELECT id, run_attempt, pr_number, ci_engine, stopped_reporting FROM {reader.run_source(started_floor=floor is not None, since=since)} ORDER BY ci_engine, id",
+ "jobs": f"SELECT id, run_id, run_attempt, is_rerun_copy, ci_engine FROM {reader.jobs_source(created_floor=floor is not None, since=since)} ORDER BY ci_engine, id",
+ "costs": f"SELECT run_id, job_name, run_attempt, head_branch, estimated_cost_usd, pr_number, ci_engine FROM {reader.job_cost_source(created_floor=floor is not None, since=since)} ORDER BY ci_engine, run_id, job_name, run_attempt",
}
return {
kind: list(