:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(eng-analytics): read materialized ci data (#121466)
Raúl Negrón-Otero05d327b7b218
diff --git a/products/engineering_analytics/SPEC.md b/products/engineering_analytics/SPEC.mdindex e8eeb7ab93c3..c2f997c355cb 100644--- a/products/engineering_analytics/SPEC.md+++ b/products/engineering_analytics/SPEC.md@@ -148,6 +148,8 @@ The per-PR friction view and the stored CI rows are the exceptions, below: they- `is_handoff_shell` is a flag on both stored grains, not a filter. A jobs build resolves the flag from raw run and handoff evidence in the same query, across run history because a new retry job can belong to an old run. A reader filters each grain on its own flag, so independently refreshed runs cannot remove jobs before their replacement jobs arrive. This is the one exception to the §6 rule that a shell is filtered at the substrate: the stored rows are storage under the read layer, and the read that swaps onto them applies the flag, so every consumer still sees the same suppression.- The views name the run's window start as `{incremental_window_start}` (`logic/views/incremental_floor.py`) and floor their primary scans on the raw `created_at` string formatted from it, because a predicate on the parsed `created_at` cannot prune the parquet scan. The duplicate scan and the runs build's handoff scan start seven days earlier. The jobs build resolves suppression across run history so a recent job from an older run still gets its flag. On a full build the placeholder is NULL and the floors keep every row.- Rows carry `source_id` and `repository`, so a product read filters on the one source the caller may use, like the friction view. They spend the team's warehouse compute every 15 minutes, so they exist only for the organizations the friction flag targets. When flag evaluation gives no answer, each existing managed view is preserved individually and no new one is enabled. A team whose GitHub source syncs less often than every 15 minutes cannot be fed at that cadence; data modeling then leaves the views computed at query time.+- Historical aggregates and cost reads use this stored pair through the shared curated layer when the friction flag is enabled. Each view's newest completed build must be at most 30 minutes old and carry the fingerprint of the current definition; a refresh that is running or failed leaves the previous build serving until that limit passes. The reader checks permission on the selected raw sources and both views, then filters on source and repository. Any failed readiness check sends the pair back to raw queries.+- Current CI status, single-run and single-PR detail, engine identity checks, and raw job steps stay live. Their latest-attempt lookup also stays live, so a newer rerun cannot select an older attempt's jobs. Stored reads report conservative source import timestamps from before the builds, rather than later source syncs, and their caches include the materialization versions. The producer's two-day reconciliation bound above still applies to historical aggregates.## 6. Locked decisionsdiff --git a/products/engineering_analytics/backend/facade/warehouse_views.py b/products/engineering_analytics/backend/facade/warehouse_views.pyindex 5c201e7a73a9..103b8681f489 100644--- a/products/engineering_analytics/backend/facade/warehouse_views.py+++ b/products/engineering_analytics/backend/facade/warehouse_views.py@@ -8,17 +8,13 @@ provider-neutral ``ExpectedWarehouseView`` contract.from typing import Any-import posthoganalytics-from posthog.hogql.database.models import FieldOrTablefrom posthog.models.team import Teamfrom products.data_modeling.backend.facade.models import DataWarehouseSavedQuery-from products.engineering_analytics.backend.facade.contracts import (- MATERIALIZED_VIEWS_FEATURE_FLAG,- ExpectedWarehouseView,-)+from products.engineering_analytics.backend.facade.contracts import ExpectedWarehouseView+from products.engineering_analytics.backend.logic.materialized_ci import materialized_views_enabledfrom products.engineering_analytics.backend.logic.views import (ci_failures,ci_job_history,@@ -78,16 +74,7 @@ def _stored_view(def _enabled_materialized_views(team: Team) -> set[str]:names = {pr_friction.VIEW_NAME, ci_rows.RUNS_VIEW_NAME, ci_rows.JOBS_VIEW_NAME}- org_id = str(team.organization_id)- project_id = str(team.id)- enabled = posthoganalytics.feature_enabled(- MATERIALIZED_VIEWS_FEATURE_FLAG,- str(team.uuid),- groups={"organization": org_id, "project": project_id},- group_properties={"organization": {"id": org_id}, "project": {"id": project_id}},- only_evaluate_locally=False,- send_feature_flag_events=False,- )+ enabled = materialized_views_enabled(team)if enabled is None:# No answer (the flag service failed, or the flag does not exist). The sync deletes every view it# does not expect, so keep the views the team has rather than drop a materialized table on an outage.diff --git a/products/engineering_analytics/backend/logic/materialized_ci.py b/products/engineering_analytics/backend/logic/materialized_ci.pynew file mode 100644index 000000000000..3645061d080f--- /dev/null+++ b/products/engineering_analytics/backend/logic/materialized_ci.py@@ -0,0 +1,211 @@+from datetime import datetime+from functools import lru_cache+from typing import TYPE_CHECKING++import structlog+import posthoganalytics++from posthog.hogql.database.s3_table import DataWarehouseTable as HogQLDataWarehouseTable+from posthog.hogql.escape_sql import escape_hogql_string++from posthog.dataclasses import frozen++from products.data_modeling.backend.facade.api import (+ IncrementalConfig,+ definition_fingerprint,+ get_incremental_config,+ get_incremental_state,+ is_materialization_fresh,+)+from products.data_modeling.backend.facade.models import DataModelingJob, DataWarehouseSavedQuery+from products.engineering_analytics.backend.facade.contracts import MATERIALIZED_VIEWS_FEATURE_FLAG, CIDataFreshness+from products.engineering_analytics.backend.logic.sources import GitHubTables, resolve_job_source_tables+from products.engineering_analytics.backend.logic.views import ci_rows, depot_ci, workflow_jobs, workflow_runs+from products.warehouse_sources.backend.facade.api import import_started_before+from products.warehouse_sources.backend.facade.types import DataWarehouseManagedViewSetKind++if TYPE_CHECKING:+ from posthog.hogql.database.database import Database++ from posthog.models.team import Team++logger = structlog.get_logger(__name__)++MAX_MATERIALIZATION_AGE = ci_rows.SYNC_FREQUENCY * 2+++def ci_tables(tables: GitHubTables, depot: depot_ci.DepotJobAttempts | None) -> list[str]:+ """Every raw table a CI read of this repository touches."""+ names = [tables.pull_requests, tables.workflow_runs, tables.workflow_jobs]+ if depot is not None:+ names.append(depot.table)+ return [name for name in names if name]+++def _readable(catalog: "Database", name: str) -> bool:+ return catalog.has_table(name) and not catalog.is_table_access_denied(name)+++@lru_cache(maxsize=32)+def _definition_fingerprint(sql: str, config: IncrementalConfig) -> str | None:+ return definition_fingerprint({"query": sql}, config)+++def materialized_views_enabled(team: "Team") -> bool | None:+ org_id = str(team.organization_id)+ project_id = str(team.id)+ try:+ return posthoganalytics.feature_enabled(+ MATERIALIZED_VIEWS_FEATURE_FLAG,+ str(team.uuid),+ groups={"organization": org_id, "project": project_id},+ group_properties={"organization": {"id": org_id}, "project": {"id": project_id}},+ only_evaluate_locally=False,+ send_feature_flag_events=False,+ )+ except Exception:+ logger.warning("engineering_analytics_materialized_flag_unavailable", team_id=team.id, exc_info=True)+ return None+++@frozen+class MaterializedCI:+ source_id: str+ repository: str+ freshness: CIDataFreshness+ cache_scope: str++ @classmethod+ def resolve(+ cls,+ *,+ team: "Team",+ tables: GitHubTables,+ depot: depot_ci.DepotJobAttempts | None,+ catalog: "Database",+ ) -> "MaterializedCI | None":+ if not tables.workflow_jobs or materialized_views_enabled(team) is not True:+ return None+ sources = resolve_job_source_tables(team)+ selected = next(+ (+ source+ for source in sources+ if source.source_id == tables.source_id and source.repository.casefold() == tables.repository.casefold()+ ),+ None,+ )+ # Stored rows embed the system resolver's Depot selection, which can differ from the caller's.+ if (+ selected is None+ or selected.github_workflow_runs != tables.workflow_runs+ or selected.github_workflow_jobs != tables.workflow_jobs+ or selected.pull_requests != tables.pull_requests+ or selected.depot_job_attempts != depot+ ):+ return None+ raw_tables = ci_tables(tables, depot)+ if not all(_readable(catalog, name) for name in raw_tables):+ return None++ # Read completion and its published table in one snapshot, so publication cannot race readiness.+ # A refresh publishes only on completion, so a running or failed one leaves the previous build readable.+ newest_build = {+ job.saved_query.name: job+ for job in DataModelingJob.objects.filter(+ team_id=team.id,+ status=DataModelingJob.Status.COMPLETED,+ engine=DataModelingJob.Engine.CLICKHOUSE,+ saved_query__team_id=team.id,+ saved_query__origin=DataWarehouseSavedQuery.Origin.MANAGED_VIEWSET,+ saved_query__managed_viewset__kind=DataWarehouseManagedViewSetKind.ENGINEERING_ANALYTICS,+ saved_query__deleted=False,+ saved_query__is_materialized=True,+ saved_query__table__deleted=False,+ saved_query__name__in=(ci_rows.RUNS_VIEW_NAME, ci_rows.JOBS_VIEW_NAME),+ )+ .select_related("saved_query__table")+ .order_by("saved_query__name", "-last_run_at")+ .distinct("saved_query__name")+ if job.saved_query is not None+ }+ expected = ci_rows.build_views(sources)+ materialized_at: dict[str, datetime] = {}+ started_at: dict[str, datetime] = {}+ for name, fields, config, sql in (+ (ci_rows.RUNS_VIEW_NAME, ci_rows.RUN_FIELDS, ci_rows.RUN_INCREMENTAL_CONFIG, expected.runs),+ (ci_rows.JOBS_VIEW_NAME, ci_rows.JOB_FIELDS, ci_rows.JOB_INCREMENTAL_CONFIG, expected.jobs),+ ):+ job = newest_build.get(name)+ view = job.saved_query if job is not None else None+ if (+ job is None+ or view is None+ or view.table is None+ or view.table.team_id != team.id+ or not fields.keys() <= (view.table.columns or {}).keys()+ or view.query != {"kind": "HogQLQuery", "query": sql}+ or view.incremental_config != config+ or not _readable(catalog, name)+ ):+ return None+ table = catalog.get_table(name)+ if (+ not isinstance(table, HogQLDataWarehouseTable)+ or table.saved_query_id != str(view.id)+ or table.table_id != str(view.table_id)+ or table.url != view.table.url_pattern+ or table.queryable_folder != view.table.queryable_folder+ or not fields.keys() <= table.fields.keys()+ ):+ return None+ incremental = get_incremental_config(view)+ fingerprint = _definition_fingerprint(sql, incremental) if incremental is not None else None+ if fingerprint is None or fingerprint != get_incremental_state(view).definition_fingerprint:+ return None+ completed_at = job.last_run_at+ if not is_materialization_fresh(completed_at, int(MAX_MATERIALIZATION_AGE.total_seconds())):+ return None+ materialized_at[name] = completed_at+ started_at[name] = job.created_at++ runs_at = materialized_at[ci_rows.RUNS_VIEW_NAME]+ jobs_at = materialized_at[ci_rows.JOBS_VIEW_NAME]+ # Later source syncs do not make an older stored result more current.+ frontiers = import_started_before(team.id, raw_tables, min(started_at.values()))++ def synced_at(table: str) -> datetime | None:+ names = [table, *([depot.table] if depot else [])]+ times = [frontiers.get(name) for name in names]+ return min(time for time in times if time is not None) if all(time is not None for time in times) else None++ return cls(+ source_id=tables.source_id,+ repository=tables.repository.casefold(),+ freshness=CIDataFreshness(+ runs_synced_at=synced_at(tables.workflow_runs), jobs_synced_at=synced_at(tables.workflow_jobs)+ ),+ cache_scope=f"materialized:{runs_at.isoformat()}:{jobs_at.isoformat()}",+ )++ def _scope(self) -> str:+ return (+ f"source_id = {escape_hogql_string(self.source_id)} AND repository = {escape_hogql_string(self.repository)}"+ )++ def runs(self, *, started_floor: bool = False, include_shells: bool = False) -> str:+ columns = [column for column in ci_rows.RUN_COLUMNS if column != "is_handoff_shell"]+ floor = " AND run_started_at_raw >= {run_started_floor}" if started_floor else ""+ shells = "" if include_shells else " AND NOT is_handoff_shell"+ return f"""SELECT {", ".join(columns)}, {workflow_runs.STOPPED_REPORTING_SQL} AS stopped_reporting+ FROM {ci_rows.RUNS_VIEW_NAME} WHERE {self._scope()}{shells}{floor}"""++ def jobs(self, *, created_floor: bool = False, with_cost: bool = False) -> str:+ columns = (+ [column for column in ci_rows.JOB_COLUMNS if column != "is_handoff_shell"]+ if with_cost+ else workflow_jobs.COLUMNS+ )+ floor = " AND created_at_raw >= {job_created_floor}" if created_floor else ""+ return f"""SELECT {", ".join(columns)} FROM {ci_rows.JOBS_VIEW_NAME}+ WHERE {self._scope()} AND NOT is_handoff_shell{floor}"""diff --git a/products/engineering_analytics/backend/logic/queries/_curated.py b/products/engineering_analytics/backend/logic/queries/_curated.pyindex c8eded9228e3..8ebf2609fcc7 100644--- a/products/engineering_analytics/backend/logic/queries/_curated.py+++ b/products/engineering_analytics/backend/logic/queries/_curated.py@@ -46,6 +46,7 @@ from posthog.hogql_queries.utils.parallel import run_in_parallel_threadsfrom posthog.models.team import Teamfrom products.engineering_analytics.backend.facade.contracts import CIDataFreshness, QueryWorkLimitExceededError+from products.engineering_analytics.backend.logic.materialized_ci import MaterializedCI, ci_tablesfrom products.engineering_analytics.backend.logic.queries._workflow_filters import DECISIVE_FAILURE_CONCLUSIONS_SQLfrom products.engineering_analytics.backend.logic.sources import (GitHubTables,@@ -195,6 +196,8 @@ class CuratedGitHubSource:self._depot_job_attempts_table: depot_ci.DepotJobAttempts | None = Noneself._depot_job_attempts_resolved = Falseself._database: Database | None = None+ self._materialized_ci: MaterializedCI | None = None+ self._materialized_ci_resolved = False# Guards the query budget, the catalog and the lazily resolved sources, which concurrent reads share.self._lock = threading.Lock()@@ -240,10 +243,12 @@ class CuratedGitHubSource:"""Curated pull-requests ``SELECT``, parenthesised for use as a subquery."""return f"({pull_requests.build_query(self._tables.pull_requests)})"- def run_source(self, *, started_floor: bool = False) -> str:+ def run_source(self, *, started_floor: bool = False, live: bool = False) -> str:"""Curated workflow-runs ``SELECT``, parenthesised for use as a subquery. ``started_floor``adds the raw-string scan floor — callers must register {run_started_floor} (see- run_started_floor_constant)."""+ run_started_floor_constant). ``live`` preserves current-status reads and raw step lookups."""+ if not live and (stored := self._stored_ci()):+ return f"({stored.runs(started_floor=started_floor)})"query = workflow_runs.build_query(self._runs_table(),pull_requests_table=self._tables.pull_requests,@@ -261,6 +266,8 @@ class CuratedGitHubSource:``include_steps`` appends the raw ``steps`` JSON column, for a read bounded to one run."""if not self._tables.workflow_jobs:return None+ if not include_steps and (stored := self._stored_ci()):+ return f"({stored.jobs(created_floor=created_floor)})"query = workflow_jobs.build_query(self._jobs_table(self._tables.workflow_jobs), created_floor=created_floor, include_steps=include_steps)@@ -292,7 +299,9 @@ class CuratedGitHubSource:without it would serve an answer built from Depot rows to a caller who has no access to them."""depot = self._depot_job_attempts()- return f"{self._tables.workflow_runs}|{self._tables.workflow_jobs or ''}|{depot.table if depot else ''}"+ scope = f"{self._tables.workflow_runs}|{self._tables.workflow_jobs or ''}|{depot.table if depot else ''}"+ stored = self._stored_ci()+ return f"{scope}|{stored.cache_scope}" if stored else scopedef may_read_ci_tables(self) -> bool:"""Whether this reader's catalog grants every table a CI read uses.@@ -300,19 +309,31 @@ class CuratedGitHubSource:A cached CI answer skips the query that would have applied the table permissions, so a caller mustask this before it serves one."""- depot = self._depot_job_attempts()- tables = [self._tables.pull_requests, self._tables.workflow_runs, self._tables.workflow_jobs]- if depot is not None:- tables.append(depot.table)catalog = self._catalog()- return all(catalog.has_table(table) and not catalog.is_table_access_denied(table) for table in tables if table)+ return all(+ catalog.has_table(table) and not catalog.is_table_access_denied(table)+ for table in ci_tables(self._tables, self._depot_job_attempts())+ )def ci_data_freshness(self) -> CIDataFreshness:"""When the runs and jobs this handle reads were last synced from their sources."""+ if stored := self._stored_ci():+ return stored.freshnessreturn resolve_ci_data_freshness(team=self._team, tables=self._tables, user_access_control=self._user_access_control)+ def _stored_ci(self) -> MaterializedCI | None:+ catalog = self._catalog()+ depot = self._depot_job_attempts()+ with self._lock:+ if not self._materialized_ci_resolved:+ self._materialized_ci = MaterializedCI.resolve(+ team=self._team, tables=self._tables, depot=depot, catalog=catalog+ )+ self._materialized_ci_resolved = True+ return self._materialized_ci+def _jobs_table(self, workflow_jobs_table: str) -> workflow_jobs.JobsTable:return depot_ci.with_depot_jobs(workflow_jobs_table, self._depot_job_attempts(), self._tables.workflow_runs)@@ -472,6 +493,12 @@ class CuratedGitHubSource:"""if not self._tables.workflow_jobs:return None+ if stored := self._stored_ci():+ query = job_costs.build_attributed_query(+ costed_jobs=stored.jobs(created_floor=created_floor, with_cost=True),+ runs=stored.runs(include_shells=True),+ )+ return f"({query})"query = job_costs.build_query(jobs_table=self._jobs_table(self._tables.workflow_jobs),runs_table=self._runs_table(),@@ -485,7 +512,7 @@ class CuratedGitHubSource:ClickHouse inlines a CTE at every reference, so each extra reader of ``runs`` scans andparses the whole runs source again."""- return f"runs AS {self.run_source()}"+ return f"runs AS {self.run_source(live=True)}"def _pr_scope_cte(self, pr_scope_where: str) -> str:"""CTE: the number and head SHA of PRs matching ``pr_scope_where`` (a predicate overdiff --git a/products/engineering_analytics/backend/logic/queries/ci_failure_logs.py b/products/engineering_analytics/backend/logic/queries/ci_failure_logs.pyindex a9c4fcab502d..330f9ae7af1d 100644--- a/products/engineering_analytics/backend/logic/queries/ci_failure_logs.py+++ b/products/engineering_analytics/backend/logic/queries/ci_failure_logs.py@@ -101,7 +101,7 @@ def _group_jobs(rows: list[tuple]) -> list[CIJobFailureLog]:def _legacy_run_ids(*, curated: CuratedGitHubSource, run_ids: list[int]) -> list[str]:# Old records did not stamp an engine. Keep them only for IDs with one engine in the source.response = curated.run(- f"SELECT id FROM {curated.run_source()} AS r WHERE id IN {{run_ids}} "+ f"SELECT id FROM {curated.run_source(live=True)} AS r WHERE id IN {{run_ids}} ""GROUP BY id HAVING uniq(ci_engine) = 1 LIMIT 1000000",query_type="engineering_analytics.failure_logs_legacy_identity",placeholders={"run_ids": ast.Constant(value=run_ids)},diff --git a/products/engineering_analytics/backend/logic/queries/current_branch_health.py b/products/engineering_analytics/backend/logic/queries/current_branch_health.pyindex be8170aca8e2..18de9804dcb2 100644--- a/products/engineering_analytics/backend/logic/queries/current_branch_health.py+++ b/products/engineering_analytics/backend/logic/queries/current_branch_health.py@@ -29,7 +29,7 @@ def query_current_branch_health(*, curated: CuratedGitHubSource, date_from: datetime, branch: str) -> CurrentBranchHealth:response = curated.run(- _SELECT.replace("__RUNS_SOURCE__", curated.run_source()),+ _SELECT.replace("__RUNS_SOURCE__", curated.run_source(live=True)),query_type="engineering_analytics.current_branch_health",placeholders={"date_from": ast.Constant(value=date_from),diff --git a/products/engineering_analytics/backend/logic/queries/pr_lifecycle.py b/products/engineering_analytics/backend/logic/queries/pr_lifecycle.pyindex 66b83725d002..f7de04938424 100644--- a/products/engineering_analytics/backend/logic/queries/pr_lifecycle.py+++ b/products/engineering_analytics/backend/logic/queries/pr_lifecycle.py@@ -149,7 +149,7 @@ def query_pr_lifecycle(runs = (curated.run(- _RUNS.replace("__RUNS_SOURCE__", curated.run_source()),+ _RUNS.replace("__RUNS_SOURCE__", curated.run_source(live=True)),query_type="engineering_analytics.pr_lifecycle.runs",placeholders={"head_sha": ast.Constant(value=head_sha)},)diff --git a/products/engineering_analytics/backend/logic/queries/pr_runs.py b/products/engineering_analytics/backend/logic/queries/pr_runs.pyindex 35b7ef78caff..1bbd93dd6437 100644--- a/products/engineering_analytics/backend/logic/queries/pr_runs.py+++ b/products/engineering_analytics/backend/logic/queries/pr_runs.py@@ -39,7 +39,7 @@ def query_pr_runs(repo_name: str,) -> list[WorkflowRunDetail]:response = curated.run(- _SELECT.replace("__RUNS_SOURCE__", curated.run_source()),+ _SELECT.replace("__RUNS_SOURCE__", curated.run_source(live=True)),query_type="engineering_analytics.pr_runs",placeholders={"pr_number": ast.Constant(value=pr_number),diff --git a/products/engineering_analytics/backend/logic/queries/workflow_jobs.py b/products/engineering_analytics/backend/logic/queries/workflow_jobs.pyindex 21832868c495..54e23364623c 100644--- a/products/engineering_analytics/backend/logic/queries/workflow_jobs.py+++ b/products/engineering_analytics/backend/logic/queries/workflow_jobs.py@@ -178,7 +178,7 @@ def query_jobs_by_run(def _latest_run_attempt(*, curated: CuratedGitHubSource, run_id: int, ci_engine: CIEngine | None) -> int | None:response = curated.run(- _LATEST_ATTEMPT_SELECT.replace("__RUNS_SOURCE__", curated.run_source()),+ _LATEST_ATTEMPT_SELECT.replace("__RUNS_SOURCE__", curated.run_source(live=True)),query_type="engineering_analytics.workflow_jobs_latest_attempt",placeholders={"run_id": ast.Constant(value=run_id)},)diff --git a/products/engineering_analytics/backend/logic/queries/workflow_run.py b/products/engineering_analytics/backend/logic/queries/workflow_run.pyindex 8d8c1a9c7499..cb689e0173e4 100644--- a/products/engineering_analytics/backend/logic/queries/workflow_run.py+++ b/products/engineering_analytics/backend/logic/queries/workflow_run.py@@ -26,7 +26,7 @@ def query_workflow_run(*, curated: CuratedGitHubSource, run_id: int, ci_engine: CIEngine | None = None) -> WorkflowRunDetail | None:response = curated.run(- _SELECT.replace("__RUNS_SOURCE__", curated.run_source()),+ _SELECT.replace("__RUNS_SOURCE__", curated.run_source(live=True)),query_type="engineering_analytics.workflow_run",placeholders={"run_id": ast.Constant(value=run_id),diff --git a/products/engineering_analytics/backend/logic/queries/workflow_run_list.py b/products/engineering_analytics/backend/logic/queries/workflow_run_list.pyindex 3ca225f8c458..768ebdc38a42 100644--- a/products/engineering_analytics/backend/logic/queries/workflow_run_list.py+++ b/products/engineering_analytics/backend/logic/queries/workflow_run_list.py@@ -53,7 +53,7 @@ def query_workflow_run_list(date_to_clause = date_to_filter_clause(date_to, placeholders)branch_clause = branch_filter_clause(branch, placeholders)response = curated.run(- _SELECT.replace("__RUNS_SOURCE__", curated.run_source())+ _SELECT.replace("__RUNS_SOURCE__", curated.run_source(live=True)).replace("__DATE_TO__", date_to_clause).replace("__BRANCH__", branch_clause).replace("__RUN_SCOPE__", run_scope_filter_clause(run_scope)),diff --git a/products/engineering_analytics/backend/tests/test_access_control.py b/products/engineering_analytics/backend/tests/test_access_control.pyindex dd653bb70b0f..b15aff9212e3 100644--- a/products/engineering_analytics/backend/tests/test_access_control.py+++ b/products/engineering_analytics/backend/tests/test_access_control.py@@ -18,6 +18,7 @@ from typing import Anyimport pytestfrom unittest import mock+from parameterized import parameterizedfrom rest_framework import statusfrom posthog.hogql.database.database import Database@@ -30,6 +31,7 @@ from products.engineering_analytics.backend.logic.sources import (WORKFLOW_RUNS_SCHEMA,resolve_github_tables,)+from products.engineering_analytics.backend.logic.views import ci_rowsfrom products.engineering_analytics.backend.tests._github_fixtures import (GITHUB_SOURCE_PREFIX,connect_github_source_without_data,@@ -37,6 +39,7 @@ from products.engineering_analytics.backend.tests._github_fixtures import (create_warehouse_table_row,link_schema,)+from products.engineering_analytics.backend.tests.test_materialized_ci import _ready_viewsfrom products.warehouse_sources.backend.facade.models import ExternalDataSourcefrom products.warehouse_sources.backend.facade.testing import WarehouseAccessControlTestMixin@@ -249,3 +252,29 @@ class TestEngineeringAnalyticsWarehouseAcl(WarehouseAccessControlTestMixin):assert database.has_table(tables.pull_requests)assert database.has_table(tables.workflow_runs)assert tables.workflow_jobs is not None and database.has_table(tables.workflow_jobs)++ @parameterized.expand(["allowed", "raw_table_denied", "stored_view_denied"])+ def test_materialized_reads_preserve_raw_and_view_permissions(self, condition: str) -> None:+ views = _ready_views(self)+ if condition != "allowed":+ self._create_access_control(+ self.no_access_user,+ resource="warehouse_table" if condition == "raw_table_denied" else "warehouse_view",+ resource_id=str(self.tables[WORKFLOW_RUNS_SCHEMA].id)+ if condition == "raw_table_denied"+ else str(views[ci_rows.RUNS_VIEW_NAME].id),+ access_level="none",+ )+ with (+ mock.patch("posthoganalytics.feature_enabled", return_value=True),+ mock.patch(_FLAG, return_value=True),+ ):+ reader = CuratedGitHubSource.for_team(+ self.team, user_access_control=UserAccessControl(self.no_access_user, self.team)+ )+ source = reader.run_source()+ if condition == "allowed":+ assert ci_rows.RUNS_VIEW_NAME in source+ else:+ assert ci_rows.RUNS_VIEW_NAME not in source+ assert reader.may_read_ci_tables() is (condition != "raw_table_denied")diff --git a/products/engineering_analytics/backend/tests/test_materialized_ci.py b/products/engineering_analytics/backend/tests/test_materialized_ci.pynew file mode 100644index 000000000000..9f155d5bac44--- /dev/null+++ b/products/engineering_analytics/backend/tests/test_materialized_ci.py@@ -0,0 +1,523 @@+from datetime import UTC, datetime, timedelta+from typing import Any++import time_machine+from posthog.test.base import BaseTest+from unittest.mock import patch++from django.utils import timezone++from parameterized import parameterized++from posthog.hogql import ast+from posthog.hogql.metadata import get_table_names+from posthog.hogql.parser import parse_select+from posthog.hogql.query import execute_hogql_query++from products.data_modeling.backend.facade.api import (+ definition_fingerprint,+ get_incremental_config,+ set_incremental_state,+)+from products.data_modeling.backend.facade.models import (+ DataModelingJob,+ DataWarehouseManagedViewSet,+ DataWarehouseSavedQuery,+)+from products.engineering_analytics.backend.facade.contracts import MATERIALIZED_VIEWS_FEATURE_FLAG+from products.engineering_analytics.backend.logic.materialized_ci import MAX_MATERIALIZATION_AGE+from products.engineering_analytics.backend.logic.queries._curated import CuratedGitHubSource+from products.engineering_analytics.backend.logic.sources import (+ DEPOT_JOB_ATTEMPTS_SCHEMA,+ PULL_REQUESTS_SCHEMA,+ WORKFLOW_JOBS_SCHEMA,+ WORKFLOW_RUNS_SCHEMA,+)+from products.engineering_analytics.backend.logic.views import ci_rows+from products.engineering_analytics.backend.logic.views.source_schema import (+ DEPOT_JOB_ATTEMPTS_COLUMNS,+ PULL_REQUESTS_COLUMNS,+ WORKFLOW_JOBS_COLUMNS,+ WORKFLOW_RUNS_COLUMNS,+)+from products.engineering_analytics.backend.tests._github_fixtures import (+ _depot_attempt_row,+ _pr_row,+ _run_row,+ create_depot_source,+ create_github_source,+ create_warehouse_table_row,+ link_schema,+)+from products.engineering_analytics.backend.tests._logic_helpers import _ago_with_duration, _job_row, _WarehouseMixin+from products.warehouse_sources.backend.facade.models import DataWarehouseTable, ExternalDataJob, ExternalDataSchema+from products.warehouse_sources.backend.facade.types import DataWarehouseManagedViewSetKind+++def _ready_views(+ test: BaseTest, *, backing_tables: dict[str, DataWarehouseTable] | None = None+) -> dict[str, DataWarehouseSavedQuery]:+ queries = ci_rows.build_team_views(test.team)+ assert queries is not None+ for schema in ExternalDataSchema.objects.filter(team=test.team, deleted=False, should_sync=True):+ imported = ExternalDataJob.objects.create(+ team=test.team,+ pipeline=schema.source,+ schema=schema,+ status=ExternalDataJob.Status.COMPLETED,+ finished_at=timezone.now() - timedelta(minutes=11),+ )+ ExternalDataJob.objects.filter(id=imported.id).update(created_at=timezone.now() - timedelta(minutes=12))+ viewset = DataWarehouseManagedViewSet.objects.create(+ team=test.team, kind=DataWarehouseManagedViewSetKind.ENGINEERING_ANALYTICS+ )+ views = {}+ for name, query, fields, config in (+ (ci_rows.RUNS_VIEW_NAME, queries.runs, ci_rows.RUN_FIELDS, ci_rows.RUN_INCREMENTAL_CONFIG),+ (ci_rows.JOBS_VIEW_NAME, queries.jobs, ci_rows.JOB_FIELDS, ci_rows.JOB_INCREMENTAL_CONFIG),+ ):+ columns = DataWarehouseManagedViewSet._get_columns_from_fields(fields)+ table = (+ backing_tables[name]+ if backing_tables is not None+ else create_warehouse_table_row(test.team, name=f"backing_{name}")+ )+ table.columns = columns+ table.save(update_fields=["columns"])+ view = DataWarehouseSavedQuery.objects.create(+ team=test.team,+ name=name,+ query={"kind": "HogQLQuery", "query": query},+ columns=columns,+ table=table,+ is_materialized=True,+ managed_viewset=viewset,+ origin=DataWarehouseSavedQuery.Origin.MANAGED_VIEWSET,+ incremental_config=config,+ )+ incremental = get_incremental_config(view)+ assert incremental is not None+ fingerprint = definition_fingerprint(view.query, incremental)+ assert fingerprint is not None+ set_incremental_state(view, watermark=timezone.now(), fingerprint=fingerprint, mode="full_refresh")+ job = DataModelingJob.objects.create(+ team=test.team,+ saved_query=view,+ status=DataModelingJob.Status.COMPLETED,+ last_run_at=timezone.now() - timedelta(minutes=5),+ )+ DataModelingJob.objects.filter(id=job.id).update(created_at=timezone.now() - timedelta(minutes=6))+ views[name] = view+ return views+++@time_machine.travel("2026-10-21T12:00:00Z", tick=False)+class TestMaterializedCI(BaseTest):+ def setUp(self) -> None:+ super().setUp()+ self.source = create_github_source(self.team, repository="example/widgets")+ for endpoint, columns in (+ (PULL_REQUESTS_SCHEMA, PULL_REQUESTS_COLUMNS),+ (WORKFLOW_RUNS_SCHEMA, WORKFLOW_RUNS_COLUMNS),+ (WORKFLOW_JOBS_SCHEMA, WORKFLOW_JOBS_COLUMNS),+ ):+ table = create_warehouse_table_row(self.team, name=f"raw_github_{endpoint}", source=self.source)+ table.columns = columns+ table.save(update_fields=["columns"])+ schema = link_schema(self.team, self.source, name=endpoint, table=table)+ ExternalDataSchema.objects.filter(id=schema.id).update(last_synced_at=timezone.now())++ def _reader(self) -> CuratedGitHubSource:+ return CuratedGitHubSource.for_team(self.team, source_id=str(self.source.id), repo="example/widgets")++ @parameterized.expand([("enabled", True, True), ("disabled", False, False), ("unavailable", None, False)])+ def test_shared_sources_choose_ready_materializations_only_when_enabled(+ self, _: str, enabled: bool | None, stored: bool+ ) -> None:+ _ready_views(self)+ with patch("posthoganalytics.feature_enabled", return_value=enabled):+ reader = self._reader()+ runs = set(get_table_names(parse_select(reader.run_source())))+ jobs = set(get_table_names(parse_select(reader.jobs_source() or "")))+ costs = set(get_table_names(parse_select(reader.job_cost_source() or "")))+ detail = set(get_table_names(parse_select(reader.jobs_source(include_steps=True) or "")))+ current = set(get_table_names(parse_select(reader.runs_cte().removeprefix("runs AS "))))+ if stored:+ assert runs == {ci_rows.RUNS_VIEW_NAME}+ assert jobs == {ci_rows.JOBS_VIEW_NAME}+ assert costs == {ci_rows.RUNS_VIEW_NAME, ci_rows.JOBS_VIEW_NAME}+ else:+ assert "raw_github_workflow_runs" in runs+ assert "raw_github_workflow_jobs" in jobs & costs+ assert "raw_github_workflow_jobs" in detail+ assert "raw_github_workflow_runs" in current++ @parameterized.expand(+ [+ "missing_pair",+ "disabled_view",+ "stale",+ "definition_not_built",+ "new_definition",+ "missing_stored_column",+ "wrong_viewset",+ ]+ )+ def test_unusable_materializations_fall_back_together(self, condition: str) -> None:+ views = _ready_views(self)+ jobs = views[ci_rows.JOBS_VIEW_NAME]+ if condition == "missing_pair":+ DataWarehouseSavedQuery.objects.filter(id=jobs.id).update(deleted=True)+ elif condition == "disabled_view":+ DataWarehouseSavedQuery.objects.filter(id=jobs.id).update(is_materialized=False)+ elif condition == "stale":+ DataModelingJob.objects.filter(saved_query=jobs).update(+ last_run_at=timezone.now() - MAX_MATERIALIZATION_AGE - timedelta(seconds=1)+ )+ elif condition == "definition_not_built":+ DataWarehouseSavedQuery.objects.filter(id=jobs.id).update(incremental_state={})+ elif condition == "new_definition":+ DataWarehouseSavedQuery.objects.filter(id=jobs.id).update(query={"kind": "HogQLQuery", "query": "SELECT 1"})+ elif condition == "missing_stored_column":+ assert jobs.table is not None+ assert jobs.table.columns is not None+ jobs.table.columns.pop("native_job_id")+ jobs.table.save(update_fields=["columns"])+ elif condition == "wrong_viewset":+ DataWarehouseSavedQuery.objects.filter(id=jobs.id).update(managed_viewset=None)++ with patch("posthoganalytics.feature_enabled", return_value=True):+ reader = self._reader()+ assert "raw_github_workflow_runs" in get_table_names(parse_select(reader.run_source()))+ assert "raw_github_workflow_jobs" in get_table_names(parse_select(reader.jobs_source() or ""))++ @parameterized.expand([DataModelingJob.Status.RUNNING, DataModelingJob.Status.FAILED])+ def test_an_unfinished_newer_build_keeps_the_completed_one_serving(self, status: str) -> None:+ views = _ready_views(self)+ DataModelingJob.objects.create(team=self.team, saved_query=views[ci_rows.JOBS_VIEW_NAME], status=status)++ with patch("posthoganalytics.feature_enabled", return_value=True):+ reader = self._reader()+ assert set(get_table_names(parse_select(reader.run_source()))) == {ci_rows.RUNS_VIEW_NAME}+ assert set(get_table_names(parse_select(reader.jobs_source() or ""))) == {ci_rows.JOBS_VIEW_NAME}++ def test_flag_exception_falls_back_to_live_sources(self) -> None:+ _ready_views(self)++ def flag(key: str, *args: object, **kwargs: object) -> bool:+ if key == MATERIALIZED_VIEWS_FEATURE_FLAG:+ raise RuntimeError("flag unavailable")+ return False++ with patch("posthoganalytics.feature_enabled", side_effect=flag):+ assert "raw_github_workflow_runs" in get_table_names(parse_select(self._reader().run_source()))++ def test_freshness_and_cache_scope_follow_materialization(self) -> None:+ _ready_views(self)+ consumed_at = timezone.now() - timedelta(minutes=10)+ for schema in ExternalDataSchema.objects.filter(source=self.source):+ old = ExternalDataJob.objects.create(+ team=self.team,+ pipeline=self.source,+ schema=schema,+ status=ExternalDataJob.Status.COMPLETED,+ created_at=consumed_at,+ finished_at=consumed_at + timedelta(minutes=1),+ )+ ExternalDataJob.objects.filter(id=old.id).update(created_at=consumed_at)+ recent = ExternalDataJob.objects.create(+ team=self.team,+ pipeline=self.source,+ schema=schema,+ status=ExternalDataJob.Status.COMPLETED,+ created_at=timezone.now() - timedelta(minutes=2),+ finished_at=timezone.now() - timedelta(minutes=1),+ )+ ExternalDataJob.objects.filter(id=recent.id).update(created_at=timezone.now() - timedelta(minutes=2))+ with patch("posthoganalytics.feature_enabled", return_value=True):+ stored = self._reader()+ freshness = stored.ci_data_freshness()+ assert freshness.runs_synced_at == consumed_at+ assert freshness.jobs_synced_at == consumed_at+ stored_scope = stored.ci_read_scope()+ with patch("posthoganalytics.feature_enabled", return_value=False):+ assert self._reader().ci_read_scope() != stored_scope++ def test_a_second_github_source_keeps_its_live_depot_rows(self) -> None:+ depot = create_depot_source(self.team, repository="example/widgets")+ table = create_warehouse_table_row(self.team, name="depot_attempts", source=depot)+ table.columns = DEPOT_JOB_ATTEMPTS_COLUMNS+ table.save(update_fields=["columns"])+ link_schema(self.team, depot, name=DEPOT_JOB_ATTEMPTS_SCHEMA, table=table)+ other = create_github_source(self.team, prefix="second_", repository="example/widgets")+ for endpoint in (PULL_REQUESTS_SCHEMA, WORKFLOW_RUNS_SCHEMA, WORKFLOW_JOBS_SCHEMA):+ table = create_warehouse_table_row(self.team, name=f"second_github_{endpoint}", source=other)+ link_schema(self.team, other, name=endpoint, table=table)+ _ready_views(self)+ with patch("posthoganalytics.feature_enabled", return_value=True):+ reader = CuratedGitHubSource.for_team(self.team, source_id=str(other.id), repo="example/widgets")+ assert "second_github_workflow_runs" in get_table_names(parse_select(reader.run_source()))+ assert "depot_attempts" in get_table_names(parse_select(reader.jobs_source() or ""))+++@time_machine.travel(lambda: datetime.now(UTC), tick=False)+class TestMaterializedCIParity(_WarehouseMixin):+ def test_stored_and_live_reads_agree_across_sources_retries_and_handoffs(self) -> None:+ repository = "example/widgets"+ source = create_github_source(self.team, repository=repository)+ started, completed = _ago_with_duration(1, 120)+ self._create_table(+ "github_pull_requests",+ PULL_REQUESTS_COLUMNS,+ [_pr_row(10, "developer", "open", 0, started, full_name=repository, head_ref="main")],+ source=source,+ )+ self._create_table(+ "github_workflow_runs",+ WORKFLOW_RUNS_COLUMNS,+ [+ _run_row(+ 100,+ "CI",+ "retry-head",+ "completed",+ "failure",+ started,+ completed,+ full_name=repository,+ run_attempt=2,+ pr_number=10,+ ),+ {+ **_run_row(+ 101,+ "Backend CI",+ "handoff-head",+ "completed",+ "success",+ started,+ completed,+ full_name=repository,+ pr_number=10,+ ),+ "created_at": (timezone.now() - timedelta(days=10)).strftime("%Y-%m-%dT%H:%M:%SZ"),+ },+ ],+ source=source,+ )+ jobs = [+ _job_row(501, 100, "tests", "failure", started=started, completed=completed),+ _job_row(502, 100, "tests", "failure", run_attempt=2, started=started, completed=completed),+ _job_row(503, 999, "orphan", "success", started=started, completed=completed),+ _job_row(601, 101, "Hand off backend tests to Depot CI", "success", started=started, completed=completed),+ _job_row(602, 101, "Django Tests Pass", "success", started=started, completed=completed),+ ]+ self._create_table("github_workflow_jobs", WORKFLOW_JOBS_COLUMNS, jobs, source=source)+ depot = create_depot_source(self.team, prefix="depot_", repository=repository)+ attempt = _depot_attempt_row(+ repo=repository,+ ref="refs/pull/10/merge",+ head_sha="handoff-head",+ sha="handoff-head",+ workflow_name="Backend CI on Depot",+ workflow_status="finished",+ workflow_created_at=started,+ workflow_started_at=started,+ workflow_finished_at=completed,+ job_key="ci-backend.yml:tests",+ job_display_name="Depot tests",+ attempt_status="finished",+ attempt_started_at=started,+ attempt_finished_at=completed,+ )+ wait = {+ **attempt,+ "job_id": "2222222222",+ "attempt_id": "3333333333",+ "job_key": "ci-backend.yml:wait-for-handoff",+ "job_display_name": "Wait for handoff",+ }+ self._create_table(+ "depot_job_attempts",+ DEPOT_JOB_ATTEMPTS_COLUMNS,+ [+ attempt,+ wait,+ {+ **wait,+ "workflow_id": "4444444444",+ "run_id": "5555555555",+ "job_id": "6666666666",+ "attempt_id": "7777777777",+ },+ ],+ source=depot,+ prefix="depot_",+ schema_name=DEPOT_JOB_ATTEMPTS_SCHEMA,+ )+ other = create_github_source(self.team, prefix="other_", repository=repository)+ for endpoint, columns, rows in (+ (+ PULL_REQUESTS_SCHEMA,+ PULL_REQUESTS_COLUMNS,+ [_pr_row(10, "other-developer", "open", 0, started, full_name=repository)],+ ),+ (+ WORKFLOW_RUNS_SCHEMA,+ WORKFLOW_RUNS_COLUMNS,+ [+ _run_row(+ 100,+ "other-workflow",+ "other-head",+ "completed",+ "success",+ started,+ completed,+ full_name=repository,+ pr_number=10,+ )+ ],+ ),+ (+ WORKFLOW_JOBS_SCHEMA,+ WORKFLOW_JOBS_COLUMNS,+ [_job_row(501, 100, "other-tests", "success", started=started, completed=completed)],+ ),+ ):+ self._create_table(f"github_{endpoint}", columns, rows, source=other, prefix="other_")+ for endpoint, columns, rows in (+ (+ PULL_REQUESTS_SCHEMA,+ PULL_REQUESTS_COLUMNS,+ [_pr_row(20, "another-developer", "open", 0, started, full_name="example/another")],+ ),+ (+ WORKFLOW_RUNS_SCHEMA,+ WORKFLOW_RUNS_COLUMNS,+ [+ _run_row(+ 200,+ "another-workflow",+ "another-head",+ "completed",+ "success",+ started,+ completed,+ full_name="example/another",+ pr_number=20,+ )+ ],+ ),+ (+ WORKFLOW_JOBS_SCHEMA,+ WORKFLOW_JOBS_COLUMNS,+ [_job_row(701, 200, "another-tests", "success", started=started, completed=completed)],+ ),+ ):+ self._create_table(+ f"another_{endpoint}",+ columns,+ rows,+ source=source,+ prefix="another_",+ schema_name=f"example/another.{endpoint}",+ )++ def read(*, stored: bool = False, floor: datetime | None = None) -> dict[str, list[tuple]]:+ reader = CuratedGitHubSource.for_team(self.team, source_id=str(source.id), repo=repository)+ if stored:+ assert set(get_table_names(parse_select(reader.run_source()))) == {ci_rows.RUNS_VIEW_NAME}+ floor_value = floor.strftime("%Y-%m-%d") if floor is not None else None+ queries = {+ "runs": f"SELECT id, run_attempt, pr_number, ci_engine, stopped_reporting FROM {reader.run_source(started_floor=floor is not None)} ORDER BY ci_engine, id",+ "jobs": f"SELECT id, run_id, run_attempt, is_rerun_copy, ci_engine FROM {reader.jobs_source(created_floor=floor is not None)} ORDER BY ci_engine, id",+ "costs": f"SELECT run_id, job_name, run_attempt, head_branch, estimated_cost_usd, pr_number, ci_engine FROM {reader.job_cost_source(created_floor=floor is not None)} ORDER BY ci_engine, run_id, job_name, run_attempt",+ }+ return {+ kind: list(+ reader.run(+ sql,+ query_type="engineering_analytics.test_stored_ci",+ placeholders={+ "run_started_floor": ast.Constant(value=floor_value),+ "job_created_floor": ast.Constant(value=floor_value),+ },+ ).results+ or []+ )+ for kind, sql in queries.items()+ }++ with patch("posthoganalytics.feature_enabled", return_value=False):+ live = read()+ future = timezone.now() + timedelta(days=1)+ assert read(floor=future) == {"runs": [], "jobs": [], "costs": []}+ assert len(live["runs"]) == 2+ assert len(live["jobs"]) == 5+ assert any(row[3] for row in live["jobs"])++ queries = ci_rows.build_team_views(self.team)+ assert queries is not None+ backing_tables = {}+ stored_run_rows: list[dict[str, Any]] = []+ for name, sql, fields in (+ (ci_rows.RUNS_VIEW_NAME, queries.runs, ci_rows.RUN_FIELDS),+ (ci_rows.JOBS_VIEW_NAME, queries.jobs, ci_rows.JOB_FIELDS),+ ):+ response = execute_hogql_query(+ sql,+ team=self.team,+ query_type="engineering_analytics.test_build_stored_ci",+ placeholders={"incremental_window_start": ast.Constant(value=None)},+ bypass_warehouse_access_control=True,+ )+ if name == ci_rows.JOBS_VIEW_NAME:+ windowed = execute_hogql_query(+ sql,+ team=self.team,+ query_type="engineering_analytics.test_build_stored_ci",+ placeholders={"incremental_window_start": ast.Constant(value=timezone.now() - timedelta(days=2))},+ bypass_warehouse_access_control=True,+ )+ assert sorted(windowed.results or []) == sorted(response.results or [])+ stored_rows: list[dict[str, Any]] = [+ {+ str(column): value.astimezone(UTC).strftime("%Y-%m-%d %H:%M:%S")+ if isinstance(value, datetime)+ else value+ for column, value in zip(response.columns or [], row, strict=True)+ }+ for row in response.results or []+ ]+ if name == ci_rows.RUNS_VIEW_NAME:+ stored_run_rows = stored_rows+ self._create_table(+ name,+ DataWarehouseManagedViewSet._get_columns_from_fields(fields),+ stored_rows,+ source=source,+ prefix="stored_",+ )+ backing_tables[name] = DataWarehouseTable.objects.get(team=self.team, name=f"stored_{name}")+ _ready_views(self, backing_tables=backing_tables)++ with patch("posthoganalytics.feature_enabled", return_value=True):+ observed = read(stored=True)+ assert observed == live, repr({"stored": observed, "live": live})+ assert read(stored=True, floor=timezone.now() - timedelta(days=2)) == live+ assert read(stored=True, floor=future) == {"runs": [], "jobs": [], "costs": []}++ self._create_table(+ "skewed_ci_runs",+ DataWarehouseManagedViewSet._get_columns_from_fields(ci_rows.RUN_FIELDS),+ [{**row, "is_handoff_shell": 1} if row["id"] == 100 else row for row in stored_run_rows],+ source=source,+ prefix="stored_",+ )+ skewed_runs = DataWarehouseTable.objects.get(team=self.team, name="stored_skewed_ci_runs")+ DataWarehouseSavedQuery.objects.filter(team=self.team, name=ci_rows.RUNS_VIEW_NAME).update(table=skewed_runs)+ with patch("posthoganalytics.feature_enabled", return_value=True):+ skewed = read(stored=True)+ assert skewed["jobs"] == live["jobs"]+ assert skewed["costs"] == live["costs"]diff --git a/products/warehouse_sources/backend/facade/api.py b/products/warehouse_sources/backend/facade/api.pyindex f38f537ecd1c..1b9d3efa1dfc 100644--- a/products/warehouse_sources/backend/facade/api.py+++ b/products/warehouse_sources/backend/facade/api.py@@ -20,6 +20,7 @@ exception is ``soft_delete_tables``: consumers may not iterate the model to callfrom collections import defaultdictfrom collections.abc import Collection+from datetime import datetimefrom typing import TYPE_CHECKINGfrom uuid import UUID@@ -104,6 +105,7 @@ __all__ = ["all_queryable_table_columns","list_tables_for_source","list_jobs_for_source",+ "import_started_before","list_column_statistics","saved_queries_feeding_consumers","schema_needs_webhook",@@ -806,6 +808,30 @@ def list_jobs_for_source(return [_to_job(j) for j in qs]+def import_started_before(+ team_id: int, table_names: Collection[str], finished_by: datetime+) -> dict[str, datetime | None]:+ """Start time of the newest import that finished by `finished_by`, per synced table name. None when there is none.++ Ordered on `created_at` so each probe walks `idx_extdatajob_latest_run` and stops at the first match.+ """+ consumed = (+ _ExternalDataJob.objects.filter(+ team_id=team_id,+ pipeline_id=OuterRef("source_id"),+ schema_id=OuterRef("pk"),+ status=_ExternalDataJob.Status.COMPLETED,+ finished_at__lte=finished_by,+ )+ .order_by("-created_at")+ .values("created_at")[:1]+ )+ schemas = _ExternalDataSchema.objects.filter(+ team_id=team_id, table__name__in=table_names, deleted=False, should_sync=True+ ).annotate(consumed_started_at=Subquery(consumed))+ return dict(schemas.values_list("table__name", "consumed_started_at"))++def list_column_statistics(team_id: int) -> list[contracts.ColumnStatistics]:"""Every column profile for a team, for describing warehouse tables in a query schema."""qs = _WarehouseColumnStatistics.objects.for_team(team_id).values_list(