Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(eng-analytics): store ci runs and jobs as incremental managed views (#118657)

Raúl Negrón-Otero11ee385db479
diff --git a/products/engineering_analytics/README.md b/products/engineering_analytics/README.md
index 26a6bd5fd6b2..fd09286aeafa 100644
--- a/products/engineering_analytics/README.md
+++ b/products/engineering_analytics/README.md
@@ -77,7 +77,7 @@ graph LR
CUR --> EP
EP --> MCP["MCP tools<br/>enabled subset in mcp/tools.yaml"]
EP --> UI["UI scenes"]
- CUR --> WV["managed warehouse views<br/>job_costs / ci_job_history / ci_failures / pr_friction<br/>for insights + SQL"]
+ CUR --> WV["managed warehouse views<br/>job_costs / ci_job_history / ci_failures / pr_friction<br/>ci_runs / ci_jobs (stored)<br/>for insights + SQL"]
```
- Job-level CI: per-job duration, queue time, runner tier, and dollar cost ride `workflow_jobs` (webhook stream plus bounded backfill).
diff --git a/products/engineering_analytics/SPEC.md b/products/engineering_analytics/SPEC.md
index 952ddaa315f6..e8eeb7ab93c3 100644
--- a/products/engineering_analytics/SPEC.md
+++ b/products/engineering_analytics/SPEC.md
@@ -39,7 +39,7 @@ graph TB
subgraph "Surface (thin)"
Endpoints["named typed DRF endpoints<br/>@extend_schema → OpenAPI → MCP tools + UI client"]
- WV["managed warehouse views<br/>job_costs / ci_job_history / ci_failures / pr_friction"]
+ WV["managed warehouse views<br/>job_costs / ci_job_history / ci_failures / pr_friction<br/>ci_runs / ci_jobs (stored)"]
end
subgraph "Curated read layer (domain rules defined ONCE)"
@@ -104,7 +104,7 @@ Per-team managed views (`DataWarehouseSavedQuery`, kind `engineering_analytics`)
The team's managed viewset row is its opt-in. The `enable` endpoint creates it when a user opens the product with a GitHub source connected, and no background job creates it: a team with a GitHub source it connected for another product gets no views. Every background job of this product (the post-load view sync, the daily test census, the GitHub and Depot job-log sweeps) runs only for enrolled teams, filtered in the database query through `logic/enrollment.py`. CI Signals has its own opt-in on top.
One gate for the three per-job views: an enrolled team gets them only when a GitHub source has **both** `workflow_runs` and `workflow_jobs` synced, so they appear together or not at all.
They are non-materialized: the rendered SQL is persisted per team and re-synced on every runs/jobs load and every Depot job-attempts load, so a builder change reaches active teams within one sync cycle.
-The per-PR friction view is the exception, below: it also needs the `pull_requests` snapshot, and it is materialized.
+The per-PR friction view and the stored CI rows are the exceptions, below: they are materialized, and the friction view also needs the `pull_requests` snapshot.
#### `engineering_analytics_job_costs`
@@ -141,6 +141,14 @@ The per-PR friction view is the exception, below: it also needs the `pull_reques
- A materialized view spends the team's warehouse compute, so it exists only for enrolled teams in organizations the `engineering-analytics-friction` flag targets. The flag is evaluated per organization, because the view sync runs with no user. When the flag service gives no answer, the sync keeps the view the team already has.
- The view unions every GitHub source of the team, and each row carries its `source_id`. A product read filters on the one source the caller may use. A query on the materialized view itself checks access to the view, not to each source in it, like any materialized managed view.
+#### `engineering_analytics_ci_runs` and `engineering_analytics_ci_jobs`
+
+- The stored CI rows: the output of the runs builder and of the jobs builder with the cost columns, as two materialized views that data modeling updates in place every 15 minutes (`logic/views/ci_rows.py`). A run upserts the rows created since two days before the newest stored row, keyed on the parsed `created_at`. Runs use `(source_id, ci_engine, id)` as their unique key, so a retry replaces the previous snapshot. Jobs use `(source_id, ci_engine, native_job_id, run_attempt)`, so a Depot job retry replaces its carried-over copy even though the execution ID changes. A row that settles later than two days after its creation is read again only by a full build, which a definition change triggers. Until then its stored snapshot keeps the earlier conclusion and duration in every historical aggregate and cost total. A rerun of a run created more than two days earlier is stored as a new attempt without rewriting the run row, and a rerun whose first attempt is more than nine days before the window (the two-day lookback plus the seven-day duplicate scan) is counted as an execution because the scan no longer sees the copy it re-lists.
+- Two grains, never joined rows. A re-run mutates the run row and adds job rows with new ids, so a stored joined row would go stale. A read joins them on `(ci_engine, run_id)`, as `job_costs` does.
+- `is_handoff_shell` is a flag on both stored grains, not a filter. A jobs build resolves the flag from raw run and handoff evidence in the same query, across run history because a new retry job can belong to an old run. A reader filters each grain on its own flag, so independently refreshed runs cannot remove jobs before their replacement jobs arrive. This is the one exception to the §6 rule that a shell is filtered at the substrate: the stored rows are storage under the read layer, and the read that swaps onto them applies the flag, so every consumer still sees the same suppression.
+- The views name the run's window start as `{incremental_window_start}` (`logic/views/incremental_floor.py`) and floor their primary scans on the raw `created_at` string formatted from it, because a predicate on the parsed `created_at` cannot prune the parquet scan. The duplicate scan and the runs build's handoff scan start seven days earlier. The jobs build resolves suppression across run history so a recent job from an older run still gets its flag. On a full build the placeholder is NULL and the floors keep every row.
+- Rows carry `source_id` and `repository`, so a product read filters on the one source the caller may use, like the friction view. They spend the team's warehouse compute every 15 minutes, so they exist only for the organizations the friction flag targets. When flag evaluation gives no answer, each existing managed view is preserved individually and no new one is enabled. A team whose GitHub source syncs less often than every 15 minutes cannot be fed at that cadence; data modeling then leaves the views computed at query time.
+
## 6. Locked decisions
Engineering-specific decisions. Product-level decisions live in README → Locked decisions. If you want to change one, do it in a separate PR with a written reason.
diff --git a/products/engineering_analytics/backend/facade/contracts.py b/products/engineering_analytics/backend/facade/contracts.py
index 646f512b522d..42c7b5797b6d 100644
--- a/products/engineering_analytics/backend/facade/contracts.py
+++ b/products/engineering_analytics/backend/facade/contracts.py
@@ -22,7 +22,8 @@ intentionally absent until the warehouse data that backs them lands.
"""
from dataclasses import field
-from datetime import date, datetime
+from datetime import date, datetime, timedelta
+from typing import Any
from pydantic.dataclasses import dataclass
@@ -30,6 +31,8 @@ from posthog.hogql.database.models import FieldOrTable
from posthog.enums import LabeledStrEnum
+from products.data_modeling.backend.facade.managed_viewset_hooks import DEFAULT_MANAGED_VIEW_SYNC_FREQUENCY
+
class CIEngine(LabeledStrEnum):
GITHUB_ACTIONS = "github_actions", "GitHub Actions"
@@ -58,7 +61,7 @@ class GitHubSourceNotConnectedError(Exception):
ENGINEERING_ANALYTICS_FEATURE_FLAG = "engineering-analytics"
# Evaluated per organization, not per person: the view sync runs with no user, and a materialized view
# spends the team's warehouse compute, so no team gets one without opting in.
-FRICTION_VIEW_FEATURE_FLAG = "engineering-analytics-friction"
+MATERIALIZED_VIEWS_FEATURE_FLAG = "engineering-analytics-friction"
# The default labels would equal the choices of data_warehouse's backfill lifecycle state. Two classes with
@@ -304,6 +307,8 @@ class ExpectedWarehouseView:
query: str
fields: dict[str, FieldOrTable]
materialized: bool = False
+ sync_frequency: timedelta = DEFAULT_MANAGED_VIEW_SYNC_FREQUENCY
+ incremental_config: dict[str, Any] | None = None
@dataclass(frozen=True)
diff --git a/products/engineering_analytics/backend/facade/warehouse_views.py b/products/engineering_analytics/backend/facade/warehouse_views.py
index 288dd0b1650e..5c201e7a73a9 100644
--- a/products/engineering_analytics/backend/facade/warehouse_views.py
+++ b/products/engineering_analytics/backend/facade/warehouse_views.py
@@ -1,38 +1,60 @@
"""The warehouse views this product exposes, for data_modeling's managed-viewset sync.
data_modeling calls ``get_expected_warehouse_views(team)`` and adapts the returned frozen
-contracts into its own ``ExpectedView`` rows (as non-materialized saved queries). Kept behind the
+contracts into its own ``ExpectedView`` rows. Kept behind the
facade so data_modeling never imports this product's read layer directly — it depends only on the
provider-neutral ``ExpectedWarehouseView`` contract.
"""
+from typing import Any
+
import posthoganalytics
+from posthog.hogql.database.models import FieldOrTable
+
from posthog.models.team import Team
from products.data_modeling.backend.facade.models import DataWarehouseSavedQuery
-from products.engineering_analytics.backend.facade.contracts import FRICTION_VIEW_FEATURE_FLAG, ExpectedWarehouseView
-from products.engineering_analytics.backend.logic.views import ci_failures, ci_job_history, job_costs, pr_friction
+from products.engineering_analytics.backend.facade.contracts import (
+ MATERIALIZED_VIEWS_FEATURE_FLAG,
+ ExpectedWarehouseView,
+)
+from products.engineering_analytics.backend.logic.views import (
+ ci_failures,
+ ci_job_history,
+ ci_rows,
+ job_costs,
+ pr_friction,
+)
from products.warehouse_sources.backend.facade.types import DataWarehouseManagedViewSetKind
def get_expected_warehouse_views(team: Team) -> list[ExpectedWarehouseView]:
"""The team's exposed warehouse views. Empty when the team has no GitHub source with both the
- workflow_runs and workflow_jobs endpoints synced (so no view is created for such teams).
-
- The three per-job views share that same qualifying-source gate, so they appear together or not at all:
- per-job cost, per-job-attempt history with commit attribution, and fingerprinted CI failure lines.
- The per-PR friction view also needs the pull-request snapshot, and is materialized because replaying
- every pull request's timeline is too heavy to run on each read. It exists only for organizations the
- friction flag targets.
+ workflow_runs and workflow_jobs endpoints synced, so they appear together or not at all. The
+ materialized views spend the team's warehouse compute, so they exist only for organizations the
+ flag targets.
"""
views: list[ExpectedWarehouseView] = []
for module in (job_costs, ci_job_history, ci_failures):
query = module.build_team_view(team)
if query is not None:
views.append(ExpectedWarehouseView(name=module.VIEW_NAME, query=query, fields=module.FIELDS))
- friction_query = pr_friction.build_team_view(team) if _friction_view_enabled(team) else None
- if friction_query is not None:
+ if not views:
+ return views
+ enabled_views = _enabled_materialized_views(team)
+ if enabled_views.intersection((ci_rows.RUNS_VIEW_NAME, ci_rows.JOBS_VIEW_NAME)) and (
+ stored := ci_rows.build_team_views(team)
+ ):
+ if ci_rows.RUNS_VIEW_NAME in enabled_views:
+ views.append(
+ _stored_view(ci_rows.RUNS_VIEW_NAME, stored.runs, ci_rows.RUN_FIELDS, ci_rows.RUN_INCREMENTAL_CONFIG)
+ )
+ if ci_rows.JOBS_VIEW_NAME in enabled_views:
+ views.append(
+ _stored_view(ci_rows.JOBS_VIEW_NAME, stored.jobs, ci_rows.JOB_FIELDS, ci_rows.JOB_INCREMENTAL_CONFIG)
+ )
+ if pr_friction.VIEW_NAME in enabled_views and (friction_query := pr_friction.build_team_view(team)) is not None:
views.append(
ExpectedWarehouseView(
name=pr_friction.VIEW_NAME, query=friction_query, fields=pr_friction.FIELDS, materialized=True
@@ -41,11 +63,25 @@ def get_expected_warehouse_views(team: Team) -> list[ExpectedWarehouseView]:
return views
-def _friction_view_enabled(team: Team) -> bool:
+def _stored_view(
+ name: str, query: str, fields: dict[str, FieldOrTable], incremental_config: dict[str, Any]
+) -> ExpectedWarehouseView:
+ return ExpectedWarehouseView(
+ name=name,
+ query=query,
+ fields=fields,
+ materialized=True,
+ sync_frequency=ci_rows.SYNC_FREQUENCY,
+ incremental_config=incremental_config,
+ )
+
+
+def _enabled_materialized_views(team: Team) -> set[str]:
+ names = {pr_friction.VIEW_NAME, ci_rows.RUNS_VIEW_NAME, ci_rows.JOBS_VIEW_NAME}
org_id = str(team.organization_id)
project_id = str(team.id)
enabled = posthoganalytics.feature_enabled(
- FRICTION_VIEW_FEATURE_FLAG,
+ MATERIALIZED_VIEWS_FEATURE_FLAG,
str(team.uuid),
groups={"organization": org_id, "project": project_id},
group_properties={"organization": {"id": org_id}, "project": {"id": project_id}},
@@ -54,15 +90,15 @@ def _friction_view_enabled(team: Team) -> bool:
)
if enabled is None:
# No answer (the flag service failed, or the flag does not exist). The sync deletes every view it
- # does not expect, so keep the view the team has rather than drop a materialized table on an outage.
- return (
+ # does not expect, so keep the views the team has rather than drop a materialized table on an outage.
+ return set(
# Only this product's managed view counts: a user's own saved query can carry the same name.
DataWarehouseSavedQuery.objects.filter(
team_id=team.id,
- name=pr_friction.VIEW_NAME,
+ name__in=names,
managed_viewset__kind=DataWarehouseManagedViewSetKind.ENGINEERING_ANALYTICS,
)
.exclude(deleted=True)
- .exists()
+ .values_list("name", flat=True)
)
- return enabled
+ return names if enabled else set()
diff --git a/products/engineering_analytics/backend/logic/queries/_curated.py b/products/engineering_analytics/backend/logic/queries/_curated.py
index 57c70475f1ce..c8eded9228e3 100644
--- a/products/engineering_analytics/backend/logic/queries/_curated.py
+++ b/products/engineering_analytics/backend/logic/queries/_curated.py
@@ -455,9 +455,9 @@ class CuratedGitHubSource:
"""
def job_cost_source(self, *, created_floor: bool = False) -> str | None:
- """Per-job cost ``SELECT`` subquery — the same view body ``engineering_analytics_job_costs``
- exposes, but with the endpoint-only run pass-through columns (``run_started_at`` /
- ``run_head_branch``). None when the jobs table isn't synced, exactly like ``jobs_source``.
+ """Per-job cost ``SELECT`` subquery: the rows ``engineering_analytics_job_costs`` exposes, with
+ the pass-through columns the public view leaves out (``run_started_at`` and ``run_head_branch``
+ among them). None when the jobs table isn't synced, exactly like ``jobs_source``.
This is the single cost-computation path: ``provider`` / ``os`` / ``vcpu`` / ``billable_seconds``
/ ``estimated_cost_usd`` are rendered from ``logic.cost`` in ClickHouse, so every endpoint cost
@@ -475,7 +475,6 @@ class CuratedGitHubSource:
query = job_costs.build_query(
jobs_table=self._jobs_table(self._tables.workflow_jobs),
runs_table=self._runs_table(),
- include_run_columns=True,
created_floor=created_floor,
)
return f"({query})"
diff --git a/products/engineering_analytics/backend/logic/queries/_workflow_filters.py b/products/engineering_analytics/backend/logic/queries/_workflow_filters.py
index e7c7ac0a20e5..161662efd7a7 100644
--- a/products/engineering_analytics/backend/logic/queries/_workflow_filters.py
+++ b/products/engineering_analytics/backend/logic/queries/_workflow_filters.py
@@ -119,7 +119,7 @@ def run_started_floor_constant(window_start: datetime) -> ast.Constant:
# late re-run would turn the copy back into an execution. Queries that keep copies (red/green reads)
# can use the tight floor.
_JOB_FLOOR_SLACK_ON_JOB_CREATED = timedelta(days=1)
-_JOB_FLOOR_SLACK_ON_RUN_STARTED = timedelta(days=7)
+JOB_FLOOR_SLACK_ON_RUN_STARTED = timedelta(days=7)
def _date_floor(window_start: datetime, slack: timedelta) -> ast.Constant:
@@ -138,8 +138,8 @@ def run_windowed_job_created_floor_constant(window_start: datetime) -> ast.Const
"""Raw-string scan floor for the jobs builder's {job_created_floor} placeholder, for a query that
windows the RUN's start (every cost surface does). Wider than the job-created floor because a
re-run's earlier attempts were created before its run_started_at — see
- ``_JOB_FLOOR_SLACK_ON_RUN_STARTED``."""
- return _date_floor(window_start, _JOB_FLOOR_SLACK_ON_RUN_STARTED)
+ ``JOB_FLOOR_SLACK_ON_RUN_STARTED``."""
+ return _date_floor(window_start, JOB_FLOOR_SLACK_ON_RUN_STARTED)
def branch_filter_clause(
diff --git a/products/engineering_analytics/backend/logic/sources.py b/products/engineering_analytics/backend/logic/sources.py
index f5c8497b2400..825a6d8c2910 100644
--- a/products/engineering_analytics/backend/logic/sources.py
+++ b/products/engineering_analytics/backend/logic/sources.py
@@ -238,6 +238,7 @@ class JobSourceTables:
issue_events: str | None = None
reviews: str | None = None
source_id: str = ""
+ repository: str = ""
depot_job_attempts: depot_ci.DepotJobAttempts | None = None
workflow_runs_optional_columns: frozenset[str] = frozenset()
@@ -283,6 +284,7 @@ def resolve_job_source_tables(team: Team) -> list[JobSourceTables]:
issue_events=tables.get(ISSUE_EVENTS_SCHEMA),
reviews=tables.get(REVIEWS_SCHEMA),
source_id=str(source.id),
+ repository=repository,
workflow_runs_optional_columns=repo_tables.workflow_runs_optional_columns,
)
entries.append((repository, entry))
diff --git a/products/engineering_analytics/backend/logic/views/ci_rows.py b/products/engineering_analytics/backend/logic/views/ci_rows.py
new file mode 100644
index 000000000000..1707e2296857
--- /dev/null
+++ b/products/engineering_analytics/backend/logic/views/ci_rows.py
@@ -0,0 +1,179 @@
+"""The stored CI rows: two materialized managed views, one per grain, that data modeling updates
+in place every 15 minutes. SPEC.md section 5 has the design.
+"""
+
+from collections.abc import Sequence
+from datetime import timedelta
+from typing import TYPE_CHECKING, Any
+
+from posthog.hogql.database.models import (
+ BooleanDatabaseField,
+ DateTimeDatabaseField,
+ FieldOrTable,
+ IntegerDatabaseField,
+ StringDatabaseField,
+)
+from posthog.hogql.escape_sql import escape_hogql_string
+
+from posthog.dataclasses import frozen
+
+from products.engineering_analytics.backend.logic.sources import JobSourceTables, resolve_job_source_tables
+from products.engineering_analytics.backend.logic.views import depot_ci, job_costs, workflow_jobs, workflow_runs
+
+if TYPE_CHECKING:
+ from posthog.models.team import Team
+
+RUNS_VIEW_NAME = "engineering_analytics_ci_runs"
+JOBS_VIEW_NAME = "engineering_analytics_ci_jobs"
+
+SYNC_FREQUENCY = timedelta(minutes=15)
+
+# Almost every run settles within a day of its creation, so two days of lookback re-read the few
+# that settle later without re-reading much. A run that settles later than that is read again only
+# by a full build.
+LOOKBACK = timedelta(days=2)
+
+
+_INCREMENTAL_CONFIG: dict[str, Any] = {
+ "enabled": True,
+ "incremental_key": "created_at",
+ "lookback_seconds": int(LOOKBACK.total_seconds()),
+}
+
+# A rerun replaces the run snapshot. Depot's stable job ID also replaces a carried-over copy
+# when that job executes again with a new attempt ID.
+RUN_INCREMENTAL_CONFIG = {**_INCREMENTAL_CONFIG, "unique_key": ["source_id", "ci_engine", "id"]}
+JOB_INCREMENTAL_CONFIG = {
+ **_INCREMENTAL_CONFIG,
+ "unique_key": ["source_id", "ci_engine", "native_job_id", "run_attempt"],
+}
+
+_SOURCE_FIELDS: dict[str, FieldOrTable] = {
+ "source_id": StringDatabaseField(name="source_id"),
+ "repository": StringDatabaseField(name="repository"),
+}
+
+RUN_FIELDS: dict[str, FieldOrTable] = {
+ **_SOURCE_FIELDS,
+ "id": IntegerDatabaseField(name="id"),
+ "workflow_name": StringDatabaseField(name="workflow_name"),
+ "head_sha": StringDatabaseField(name="head_sha", nullable=True),
+ "head_branch": StringDatabaseField(name="head_branch", nullable=True),
+ "status": StringDatabaseField(name="status", nullable=True),
+ "conclusion": StringDatabaseField(name="conclusion", nullable=True),
+ "run_started_at": DateTimeDatabaseField(name="run_started_at", nullable=True),
+ "updated_at": DateTimeDatabaseField(name="updated_at", nullable=True),
+ "created_at": DateTimeDatabaseField(name="created_at", nullable=True),
+ "run_attempt": IntegerDatabaseField(name="run_attempt", nullable=True),
+ "is_merge_queue": BooleanDatabaseField(name="is_merge_queue"),
+ "pr_number": IntegerDatabaseField(name="pr_number", nullable=True),
+ "commit_pr_number": IntegerDatabaseField(name="commit_pr_number", nullable=True),
+ "duration_seconds": IntegerDatabaseField(name="duration_seconds", nullable=True),
+ "repo_owner": StringDatabaseField(name="repo_owner", nullable=True),
+ "repo_name": StringDatabaseField(name="repo_name", nullable=True),
+ "ci_engine": StringDatabaseField(name="ci_engine"),
+ "native_run_id": StringDatabaseField(name="native_run_id", nullable=True),
+ "native_workflow_run_id": StringDatabaseField(name="native_workflow_run_id", nullable=True),
+ "workflow_id": IntegerDatabaseField(name="workflow_id", nullable=True),
+ "event": StringDatabaseField(name="event", nullable=True),
+ "created_at_raw": StringDatabaseField(name="created_at_raw"),
+ "run_started_at_raw": StringDatabaseField(name="run_started_at_raw"),
+ "is_handoff_shell": BooleanDatabaseField(name="is_handoff_shell"),
+}
+
+JOB_FIELDS: dict[str, FieldOrTable] = {
+ **_SOURCE_FIELDS,
+ "id": IntegerDatabaseField(name="id"),
+ "run_id": IntegerDatabaseField(name="run_id"),
+ "run_attempt": IntegerDatabaseField(name="run_attempt", nullable=True),
+ "name": StringDatabaseField(name="name"),
+ "workflow_name": StringDatabaseField(name="workflow_name"),
+ "head_sha": StringDatabaseField(name="head_sha"),
+ "head_branch": StringDatabaseField(name="head_branch"),
+ "status": StringDatabaseField(name="status"),
+ "conclusion": StringDatabaseField(name="conclusion", nullable=True),
+ "labels": StringDatabaseField(name="labels"),
+ "runner_name": StringDatabaseField(name="runner_name"),
+ "created_at": DateTimeDatabaseField(name="created_at", nullable=True),
+ "created_at_raw": StringDatabaseField(name="created_at_raw"),
+ "started_at": DateTimeDatabaseField(name="started_at", nullable=True),
+ "completed_at": DateTimeDatabaseField(name="completed_at", nullable=True),
+ "duration_seconds": IntegerDatabaseField(name="duration_seconds", nullable=True),
+ "queue_seconds": IntegerDatabaseField(name="queue_seconds", nullable=True),
+ "provisioning_seconds": IntegerDatabaseField(name="provisioning_seconds", nullable=True),
+ "is_rerun_copy": BooleanDatabaseField(name="is_rerun_copy"),
+ "ci_engine": StringDatabaseField(name="ci_engine"),
+ "native_run_id": StringDatabaseField(name="native_run_id", nullable=True),
+ "native_workflow_run_id": StringDatabaseField(name="native_workflow_run_id", nullable=True),
+ "native_job_id": StringDatabaseField(name="native_job_id", nullable=True),
+ "native_attempt_id": StringDatabaseField(name="native_attempt_id", nullable=True),
+ **{column: job_costs.FIELDS[column] for column in job_costs.COST_COLUMNS},
+ "is_handoff_shell": BooleanDatabaseField(name="is_handoff_shell"),
+}
+
+RUN_COLUMNS = tuple(column for column in RUN_FIELDS if column not in _SOURCE_FIELDS)
+JOB_COLUMNS = tuple(column for column in JOB_FIELDS if column not in _SOURCE_FIELDS)
+assert JOB_COLUMNS == (*workflow_jobs.COLUMNS, *job_costs.COST_COLUMNS, "is_handoff_shell")
+
+
+def _stored_rows(source: JobSourceTables, columns: tuple[str, ...], rows: str) -> str:
+ # GitHub names are case-insensitive, and a source can store them in either case.
+ return f"""
+ SELECT
+ {escape_hogql_string(source.source_id)} AS source_id,
+ {escape_hogql_string(source.repository.casefold())} AS repository,
+ {", ".join(columns)}
+ FROM ({rows})
+ """
+
+
+def _stored_runs(source: JobSourceTables, *, full_history: bool = False) -> str:
+ return depot_ci.stored_runs(
+ source.github_workflow_runs,
+ source.depot_job_attempts,
+ source.pull_requests,
+ source.github_workflow_jobs,
+ optional_columns=source.workflow_runs_optional_columns,
+ full_history=full_history,
+ )
+
+
+def runs_select(source: JobSourceTables) -> str:
+ rows = workflow_runs.build_query(
+ _stored_runs(source), pull_requests_table=source.pull_requests, passthrough=depot_ci.STORED_RUN_COLUMNS
+ )
+ return _stored_rows(source, RUN_COLUMNS, rows)
+
+
+def jobs_select(source: JobSourceTables) -> str:
+ jobs = depot_ci.stored_jobs(source.github_workflow_jobs, source.depot_job_attempts)
+ rows = job_costs.build_costed_jobs_query(workflow_jobs.build_query(jobs, compact_duplicates=False))
+ if source.depot_job_attempts is not None:
+ # A new retry job can belong to an old run, so the suppression lookup must cover run history.
+ runs = _stored_runs(source, full_history=True)
+ shell = f"(ci_engine, run_id) IN (SELECT ci_engine, id FROM {runs} WHERE is_handoff_shell)"
+ else:
+ shell = "0"
+ rows = f"SELECT j.*, {shell} AS is_handoff_shell FROM ({rows}) AS j"
+ return _stored_rows(source, JOB_COLUMNS, rows)
+
+
+@frozen
+class CIViewQueries:
+ runs: str
+ jobs: str
+
+
+def build_views(sources: Sequence[JobSourceTables]) -> CIViewQueries:
+ """The runs view and the jobs view over these sources. The reader compares a stored definition
+ against this output, so it is the one place that assembles either view."""
+ return CIViewQueries(
+ runs="\nUNION ALL\n".join(runs_select(source) for source in sources),
+ jobs="\nUNION ALL\n".join(jobs_select(source) for source in sources),
+ )
+
+
+def build_team_views(team: "Team") -> CIViewQueries | None:
+ """Both views over every qualifying source of the team, or None when there is none."""
+ sources = resolve_job_source_tables(team)
+ return build_views(sources) if sources else None
diff --git a/products/engineering_analytics/backend/logic/views/depot_ci.py b/products/engineering_analytics/backend/logic/views/depot_ci.py
index 113d9821b996..96958fd1d436 100644
--- a/products/engineering_analytics/backend/logic/views/depot_ci.py
+++ b/products/engineering_analytics/backend/logic/views/depot_ci.py
@@ -29,7 +29,7 @@ from products.engineering_analytics.backend.logic.queries._workflow_filters impo
DECISIVE_FAILURE_CONCLUSIONS_SQL,
SUCCESSFUL_RUN_CONDITION,
)
-from products.engineering_analytics.backend.logic.views import workflow_jobs, workflow_runs
+from products.engineering_analytics.backend.logic.views import incremental_floor, workflow_jobs, workflow_runs
from products.engineering_analytics.backend.logic.views.source_schema import (
OPTIONAL_WORKFLOW_RUNS_COLUMNS,
WORKFLOW_JOBS_COLUMNS,
@@ -136,7 +136,7 @@ def build_billed_seconds_query(depot: DepotJobAttempts) -> str:
"""
-def _attempts(depot: DepotJobAttempts, pull_requests_table: str | None) -> str:
+def _attempts(depot: DepotJobAttempts, pull_requests_table: str | None, where: str = "1") -> str:
# Depot reports no branch, so a PR run takes its head branch from the PR snapshot, and branch
# filters then match it like a GitHub run of the same PR.
pr_number = "ifNull(toInt(extract(a.ref, '^refs/pull/([0-9]+)/')), 0)"
@@ -187,10 +187,11 @@ def _attempts(depot: DepotJobAttempts, pull_requests_table: str | None) -> str:
{branch_join}
WHERE {_of_repository(depot)}
AND {_is_id("a.run_id")} AND {_is_id("a.workflow_id")} AND {_is_id("a.attempt_id")}
+ AND ({where})
)"""
-def _runs(attempts: str) -> str:
+def _runs(attempts: str, extra_columns: str = "") -> str:
return f"""
SELECT
github_run_id AS id,
@@ -215,13 +216,15 @@ def _runs(attempts: str) -> str:
{OPTIONAL_WORKFLOW_RUNS_COLUMNS["event"]} AS event,
'depot_ci' AS ci_engine,
any(native_run_id) AS native_run_id,
- any(native_workflow_run_id) AS native_workflow_run_id
+ any(native_workflow_run_id) AS native_workflow_run_id{extra_columns}
FROM {attempts}
GROUP BY github_run_id
"""
-def _jobs(attempts: str) -> str:
+def _jobs(attempts: str, run_attempts: str | None = None) -> str:
+ """``run_attempts`` holds every attempt of the runs in ``attempts``. It defaults to ``attempts``."""
+ run_attempts = run_attempts or attempts
# GitHub lists every job of a run under each run attempt. A job that a run attempt did not re-run keeps
# the timestamps of the attempt that ran, and the jobs builder flags that row as a copy. Depot numbers
# attempts per job, so each job's last attempt is listed again under every later attempt of its run.
@@ -254,17 +257,17 @@ def _jobs(attempts: str) -> str:
SELECT jobs.depot_job_id AS depot_job_id, jobs.job_attempt AS job_attempt, runs.run_attempt AS run_attempt
FROM (
SELECT github_run_id, depot_job_id, max(attempt) AS job_attempt
- FROM {attempts}
+ FROM {run_attempts}
GROUP BY github_run_id, depot_job_id
) AS jobs
INNER JOIN (
- SELECT github_run_id, max(attempt) AS run_attempt FROM {attempts} GROUP BY github_run_id
+ SELECT github_run_id, max(attempt) AS run_attempt FROM {run_attempts} GROUP BY github_run_id
) AS runs ON jobs.github_run_id = runs.github_run_id
) AS last ON a.depot_job_id = last.depot_job_id
"""
-def _handoff_workflows(depot: DepotJobAttempts) -> str:
+def _handoff_workflows(depot: DepotJobAttempts, where: str = "1") -> str:
# Depot lists no attempt for a skipped job, so a workflow that declined the hand-off holds the wait job alone.
is_wait = f"endsWith(ifNull(job_key, ''), '{_DEPOT_WAIT_JOB_KEY_SUFFIX}')"
return f"""(
@@ -275,37 +278,57 @@ def _handoff_workflows(depot: DepotJobAttempts) -> str:
any(workflow_status) AS workflow_status,
min(parseDateTimeBestEffort(workflow_created_at)) AS created_at,
countIf(NOT {is_wait}) > 0 AS took_handoff
- FROM {_attempts(depot, pull_requests_table=None)}
+ FROM {_attempts(depot, pull_requests_table=None, where=where)}
GROUP BY github_run_id
HAVING countIf({is_wait}) > 0
)"""
+def _declined_handoffs(handoffs: str) -> str:
+ """The Depot workflows that waited for a hand-off that never came, and passed."""
+ return f"SELECT github_run_id FROM {handoffs} WHERE NOT took_handoff AND workflow_status = 'finished'"
+
+
def _executed_attempts(depot: DepotJobAttempts, handoffs: str, pull_requests_table: str | None) -> str:
return f"""(
SELECT * FROM {_attempts(depot, pull_requests_table)}
- WHERE github_run_id NOT IN (
- SELECT github_run_id FROM {handoffs} WHERE NOT took_handoff AND workflow_status = 'finished'
- )
+ WHERE github_run_id NOT IN ({_declined_handoffs(handoffs)})
)"""
-def _github_shells(jobs_table: str, runs_table: str, handoffs: str) -> str:
+@frozen
+class _ShellScan:
+ """Predicates over the raw GitHub rows that bound the scans of ``_github_shells``."""
+
+ runs: str
+ jobs: str
+ handoff_jobs: str
+
+
+def _whole_history_shell_scan(jobs_table: str, handoffs: str) -> _ShellScan:
# No hand-off job predates Depot's first hand-off, so the day before it floors the hand-off and relay jobs.
floor = f"(SELECT toString(subtractDays(toDate(min(created_at)), 1)) FROM {handoffs})"
- handed_off = f"name = '{_GITHUB_HANDOFF_JOB}' AND conclusion = 'success' AND created_at >= {floor}"
+ handoff_jobs = f"created_at >= {floor}"
# A run can recover weeks after its failed attempt, so the failure check has no date floor. Every attempt
# keeps the id of its run, so an id bound cannot cut an attempt. Run ids grow with time, so the first run
# that handed off is a bound a scan can skip files on.
- first_run = f"(SELECT min(run_id) FROM {jobs_table} WHERE {handed_off})"
+ first_run = f"""(
+ SELECT min(run_id) FROM {jobs_table}
+ WHERE name = '{_GITHUB_HANDOFF_JOB}' AND conclusion = 'success' AND {handoff_jobs}
+ )"""
+ return _ShellScan(runs=f"id >= {first_run}", jobs=f"run_id >= {first_run}", handoff_jobs=handoff_jobs)
+
+
+def _github_shells(jobs_table: str, runs_table: str, handoffs: str, scan: _ShellScan) -> str:
+ handed_off = f"name = '{_GITHUB_HANDOFF_JOB}' AND conclusion = 'success' AND {scan.handoff_jobs}"
relays = f"""
SELECT run_id
FROM {jobs_table}
- WHERE run_id >= {first_run}
+ WHERE {scan.jobs}
GROUP BY run_id
HAVING countIf({handed_off}) > 0
AND argMaxIf(
- ifNull(conclusion, ''), tuple(run_attempt, id), name = '{_GITHUB_RELAY_JOB}' AND created_at >= {floor}
+ ifNull(conclusion, ''), tuple(run_attempt, id), name = '{_GITHUB_RELAY_JOB}' AND {scan.handoff_jobs}
) = 'success'
AND countIf(conclusion IN ({DECISIVE_FAILURE_CONCLUSIONS_SQL})) = 0
"""
@@ -314,7 +337,7 @@ def _github_shells(jobs_table: str, runs_table: str, handoffs: str) -> str:
return f"""
SELECT id
FROM {runs_table}
- WHERE id >= {first_run} AND {SUCCESSFUL_RUN_CONDITION} AND id IN ({relays})
+ WHERE {scan.runs} AND {SUCCESSFUL_RUN_CONDITION} AND id IN ({relays})
AND (head_sha, {workflow_runs.ASSOCIATION_PR_NUMBER}) IN (
SELECT head_sha, pr_number FROM {handoffs}
WHERE took_handoff AND pr_number > 0 AND workflow_status = 'finished'
@@ -324,7 +347,9 @@ def _github_shells(jobs_table: str, runs_table: str, handoffs: str) -> str:
# UNION ALL matches columns by position, so the GitHub selects name them in the contract order the
# Depot side follows.
-def _github_runs(table: str, where: str = "1", optional_columns: frozenset[str] = frozenset()) -> str:
+def _github_runs(
+ table: str, where: str = "1", *, optional_columns: frozenset[str] = frozenset(), extra_columns: str = ""
+) -> str:
columns = [
column
if column not in OPTIONAL_WORKFLOW_RUNS_COLUMNS or column in optional_columns
@@ -332,7 +357,7 @@ def _github_runs(table: str, where: str = "1", optional_columns: frozenset[str]
for column in WORKFLOW_RUNS_COLUMNS
]
return f"""SELECT {", ".join(columns)}, 'github_actions' AS ci_engine,
- toString(id) AS native_run_id, toString(id) AS native_workflow_run_id
+ toString(id) AS native_run_id, toString(id) AS native_workflow_run_id{extra_columns}
FROM {table} WHERE {where}"""
@@ -360,9 +385,99 @@ def with_depot_runs(
if depot is None:
return f"({_github_runs(runs_table, optional_columns=optional_columns)})"
handoffs = _handoff_workflows(depot)
- where = f"id NOT IN ({_github_shells(jobs_table, runs_table, handoffs)})" if jobs_table else "1"
+ where = "1"
+ if jobs_table:
+ scan = _whole_history_shell_scan(jobs_table, handoffs)
+ where = f"id NOT IN ({_github_shells(jobs_table, runs_table, handoffs, scan)})"
depot_runs = _runs(_executed_attempts(depot, handoffs, pull_requests_table))
- return f"({_github_runs(runs_table, where, optional_columns)} UNION ALL {depot_runs})"
+ return f"({_github_runs(runs_table, where, optional_columns=optional_columns)} UNION ALL {depot_runs})"
+
+
+STORED_RUN_COLUMNS = ("created_at_raw", "run_started_at_raw", "is_handoff_shell")
+
+
+def stored_runs(
+ runs_table: str,
+ depot: DepotJobAttempts | None,
+ pull_requests_table: str | None,
+ jobs_table: str,
+ *,
+ optional_columns: frozenset[str] = frozenset(),
+ full_history: bool = False,
+) -> str:
+ """The runs of both engines with ``STORED_RUN_COLUMNS``, normally bounded by the stored window.
+
+ A hand-off shell stays and is flagged, where ``with_depot_runs`` drops it. A stored job row
+ cannot tell a dropped run from a run that never synced, and it needs that to leave the jobs of a
+ shell out. ``full_history`` lets a new job resolve the suppression of an older run.
+ """
+ floor = "1" if full_history else incremental_floor.stored_rows()
+ if depot is None:
+ github_runs = _github_runs(
+ runs_table,
+ floor,
+ optional_columns=optional_columns,
+ extra_columns=_stored_run_columns("0"),
+ )
+ return f"({github_runs})"
+ handoffs = _handoff_workflows(
+ depot, "1" if full_history else incremental_floor.stored_rows_and_earlier("a.workflow_created_at")
+ )
+ # A job is created after its run, so the jobs of the runs in the window start at the window too.
+ scan = (
+ _whole_history_shell_scan(jobs_table, handoffs)
+ if full_history
+ else _ShellScan(runs=floor, jobs=incremental_floor.stored_rows(), handoff_jobs="1")
+ )
+ github_runs = _github_runs(
+ runs_table,
+ floor,
+ optional_columns=optional_columns,
+ extra_columns=_stored_run_columns(f"id IN ({_github_shells(jobs_table, runs_table, handoffs, scan)})"),
+ )
+ depot_runs = _runs(
+ _attempts(
+ depot, pull_requests_table, "1" if full_history else incremental_floor.stored_rows("a.workflow_created_at")
+ ),
+ f""",
+ ifNull(any(workflow_created_at), '') AS created_at_raw,
+ ifNull(any(workflow_started_at), '') AS run_started_at_raw,
+ github_run_id IN ({_declined_handoffs(handoffs)}) AS is_handoff_shell""",
+ )
+ return f"({github_runs} UNION ALL {depot_runs})"
+
+
+def _stored_run_columns(is_handoff_shell: str) -> str:
+ # The stored columns are plain strings, so a run that has not started stores ''.
+ return (
+ ", ifNull(created_at, '') AS created_at_raw, ifNull(run_started_at, '') AS run_started_at_raw"
+ f", {is_handoff_shell} AS is_handoff_shell"
+ )
+
+
+def stored_jobs(jobs_table: str, depot: DepotJobAttempts | None) -> workflow_jobs.JobsTable:
+ """The job attempts of both engines created at or after the stored window.
+
+ The jobs of a hand-off shell stay so the stored builder can flag them in the same snapshot. The
+ source of the duplicate scan starts earlier, because the first listing of a job row can be older
+ than its re-listed copy.
+ """
+ rows = _github_jobs(jobs_table, incremental_floor.stored_rows())
+ duplicates = _github_jobs(jobs_table, incremental_floor.stored_rows_and_earlier())
+ if depot is None:
+ return workflow_jobs.JobsTable(rows=f"({rows})", duplicates=f"({duplicates})")
+ # Depot lists the copies of an attempt next to it, so its rows hold their own duplicates. The highest
+ # attempt of the run sets the number of copies, and another job can reach it on an earlier day.
+ depot_jobs = _jobs(
+ _attempts(depot, pull_requests_table=None, where=incremental_floor.stored_rows("a.attempt_started_at")),
+ run_attempts=_attempts(
+ depot, pull_requests_table=None, where=incremental_floor.stored_rows_and_earlier("a.attempt_started_at")
+ ),
+ )
+ return workflow_jobs.JobsTable(
+ rows=f"({rows} UNION ALL {depot_jobs})",
+ duplicates=f"({duplicates} UNION ALL {depot_jobs})",
+ )
def with_depot_jobs(jobs_table: str, depot: DepotJobAttempts | None, runs_table: str) -> workflow_jobs.JobsTable:
@@ -377,7 +492,8 @@ def with_depot_jobs(jobs_table: str, depot: DepotJobAttempts | None, runs_table:
if depot is None:
return workflow_jobs.JobsTable.of(f"({_github_jobs(jobs_table)})")
handoffs = _handoff_workflows(depot)
- where = f"run_id NOT IN ({_github_shells(jobs_table, runs_table, handoffs)})"
+ scan = _whole_history_shell_scan(jobs_table, handoffs)
+ where = f"run_id NOT IN ({_github_shells(jobs_table, runs_table, handoffs, scan)})"
depot_jobs = _jobs(_executed_attempts(depot, handoffs, pull_requests_table=None))
every_depot_job = _jobs(_attempts(depot, pull_requests_table=None))
return workflow_jobs.JobsTable(
diff --git a/products/engineering_analytics/backend/logic/views/incremental_floor.py b/products/engineering_analytics/backend/logic/views/incremental_floor.py
new file mode 100644
index 000000000000..a2a6149fb090
--- /dev/null
+++ b/products/engineering_analytics/backend/logic/views/incremental_floor.py
@@ -0,0 +1,34 @@
+"""The rows one incremental materialization reads: the rows created at or after the run's window
+start. The window start reaches the view as the ``{incremental_window_start}`` placeholder, which
+data modeling binds to the key's watermark minus the lookback on an incremental run and to NULL on
+a full build, so the floor then keeps every row.
+
+The floors compare the raw ISO-8601 ``created_at`` strings with the window start formatted the
+same way, which is the one comparison that lets a warehouse scan skip files; the filter data
+modeling injects on the parsed ``created_at`` cannot. Another row can change what a stored row
+says: an earlier attempt shows that a job row is a re-listed copy, and the hand-off a Depot
+workflow took decides whether a GitHub run is a shell. A scan for such rows reaches
+``JOB_FLOOR_SLACK_ON_RUN_STARTED`` before the window.
+"""
+
+from products.engineering_analytics.backend.logic.queries._workflow_filters import JOB_FLOOR_SLACK_ON_RUN_STARTED
+
+WINDOW_START = "{incremental_window_start}"
+
+
+def stored_rows(column: str = "created_at") -> str:
+ """The rows the run stores."""
+ return f"{column} >= {_floor(days_before=0)}"
+
+
+def stored_rows_and_earlier(column: str = "created_at") -> str:
+ """The rows the run stores and the rows up to the slack before them."""
+ return f"{column} >= {_floor(days_before=JOB_FLOOR_SLACK_ON_RUN_STARTED.days)}"
+
+
+def _floor(*, days_before: int) -> str:
+ start = f"toTimeZone({WINDOW_START}, 'UTC')"
+ if days_before:
+ start = f"{start} - INTERVAL {days_before} DAY"
+ # A fractional timestamp sorts before the same second's trailing Z, so use the second's prefix.
+ return f"ifNull(formatDateTime({start}, '%Y-%m-%dT%H:%i:%S'), '')"
diff --git a/products/engineering_analytics/backend/logic/views/job_costs.py b/products/engineering_analytics/backend/logic/views/job_costs.py
index ad7d77caf216..1603d2b95297 100644
--- a/products/engineering_analytics/backend/logic/views/job_costs.py
+++ b/products/engineering_analytics/backend/logic/views/job_costs.py
@@ -37,6 +37,7 @@ from typing import TYPE_CHECKING
from posthog.hogql.database.models import (
BooleanDatabaseField,
+ DatabaseField,
DateTimeDatabaseField,
FieldOrTable,
FloatDatabaseField,
@@ -123,36 +124,21 @@ FIELDS: dict[str, FieldOrTable] = {
}
-# The two endpoint-only run pass-through columns — the run's start time and the *run's* head branch
-# (distinct from the job's ``head_branch``), used only by the product's endpoint cost queries to
-# window and branch-filter on run attributes. One source of truth: the innermost join layer renders
-# "<expr> AS <alias>" and every outer layer re-projects the bare aliases, so a new pass-through is
-# added in exactly one place. Deliberately kept out of the public view (``build_team_view`` uses the
-# default): ``run_head_branch`` would duplicate ``head_branch`` for the exposed grain, and the view
-# already carries ``created_at`` for time filtering.
-_RUN_PASSTHROUGH: tuple[tuple[str, str], ...] = (
- ("run_started_at", "r.run_started_at"),
- ("run_head_branch", "r.head_branch"),
+_PASSTHROUGH: tuple[tuple[str, DatabaseField], ...] = (
+ # The public view omits the run's start time and branch: ``run_head_branch`` would duplicate
+ # ``head_branch`` for the exposed grain, and the view already carries ``created_at`` for time filtering.
+ ("r.run_started_at", DateTimeDatabaseField(name="run_started_at", nullable=True)),
+ ("r.head_branch", StringDatabaseField(name="run_head_branch", nullable=True)),
)
+BUILDER_FIELDS: dict[str, FieldOrTable] = {**FIELDS, **{field.name: field for _, field in _PASSTHROUGH}}
-def _run_passthrough_defs() -> str:
- """ "<expr> AS <alias>" for each run pass-through — the innermost join layer that first reads them."""
- return "".join(f",\n {expr} AS {alias}" for alias, expr in _RUN_PASSTHROUGH)
+_PASSTHROUGH_DEFS = "".join(f",\n {expr} AS {field.name}" for expr, field in _PASSTHROUGH)
+COST_COLUMNS = ("provider", "os", "vcpu", "multiplier", "billable_seconds", "estimated_cost_usd")
-def _run_passthrough_aliases() -> str:
- """Bare aliases for each run pass-through — re-projected by every layer above the join."""
- return "".join(f",\n {alias}" for alias, _ in _RUN_PASSTHROUGH)
-
-def build_query(
- *,
- jobs_table: workflow_jobs.JobsTable,
- runs_table: str,
- include_run_columns: bool = False,
- created_floor: bool = False,
-) -> str:
+def build_query(*, jobs_table: workflow_jobs.JobsTable, runs_table: str, created_floor: bool = False) -> str:
"""The per-job cost SELECT for one GitHub source: curated jobs LEFT JOIN curated runs.
Grain is one row per job attempt (a retry appears once per attempt — correct for cost). The
@@ -160,15 +146,7 @@ def build_query(
unjoined run (no ``r`` row) leaves only the attribution columns (``repo_owner`` / ``repo_name`` /
``pr_number``) NULL.
- Layered so each per-row classification step is computed once: the join layer parses ``labels_arr``
- and derives ``billed_seconds`` (an internal column — the exposed ``billable_seconds`` is that clock
- gated on the row being billable);
- the label layer picks ``depot_label`` / ``hosted_label`` from it (one ``arrayFilter`` scan each);
- the tier layer derives ``provider`` / ``os`` / ``vcpu`` from those two cheap columns; the final
- layer derives ``multiplier`` / ``billable_seconds`` / ``estimated_cost_usd``.
-
- ``include_run_columns`` threads the ``_RUN_PASSTHROUGH`` run columns through every layer — used
- only by the endpoint cost queries; the public view omits them.
+ The result has the ``BUILDER_FIELDS`` columns. The public view selects its ``FIELDS`` from them.
``created_floor`` threads the jobs builder's raw-string scan floor (its ``{job_created_floor}``
placeholder, which the caller must register) down to the jobs scan. Every windowed cost query
@@ -179,136 +157,110 @@ def build_query(
``created_at`` bound with a coarse ``created_at_raw`` floor, which is the predicate the scan can
prune on and the reason the view exposes that column.
"""
- jobs = workflow_jobs.build_query(jobs_table, created_floor=created_floor)
- runs = workflow_runs.build_query(runs_table)
+ costed_jobs = build_costed_jobs_query(workflow_jobs.build_query(jobs_table, created_floor=created_floor))
+ return build_attributed_query(costed_jobs=costed_jobs, runs=workflow_runs.build_query(runs_table))
+
+def build_costed_jobs_query(jobs: str) -> str:
+ """The rows of the jobs builder with ``COST_COLUMNS``. ``jobs`` is a query with the columns of
+ ``workflow_jobs.COLUMNS``.
+
+ The cost of a job reads no column of its run, so a stored job row can carry it.
+
+ Layered so each per-row classification step is computed once: the innermost layer parses
+ ``labels_arr`` and derives ``billed_seconds`` (an internal column — the exposed ``billable_seconds``
+ is that clock gated on the row being billable);
+ the label layer picks ``depot_label`` / ``hosted_label`` from it (one ``arrayFilter`` scan each);
+ the tier layer derives ``provider`` / ``os`` / ``vcpu`` from those two cheap columns; the final
+ layer derives ``multiplier`` / ``billable_seconds`` / ``estimated_cost_usd``.
+ """
+ job_columns = ", ".join(workflow_jobs.COLUMNS)
# labels is already ifNull'd to '[]' by the jobs builder; JSONExtract to Array(String) yields
# [] for any non-array/invalid JSON, matching cost._parse_labels' empty-on-bad-input behavior.
labels_array = "JSONExtract(labels, 'Array(String)')"
- billed_seconds = render_billed_elapsed_seconds("j.duration_seconds", "j.provisioning_seconds")
-
- inner_run_columns = _run_passthrough_defs() if include_run_columns else ""
- run_columns = _run_passthrough_aliases() if include_run_columns else ""
-
+ billed_seconds = render_billed_elapsed_seconds("duration_seconds", "provisioning_seconds")
return f"""
SELECT
- repo_owner,
- repo_name,
- pr_number,
- workflow_name,
- job_name,
- run_id,
- run_attempt,
- head_branch,
- status,
- conclusion,
- runner_name,
- created_at,
- started_at,
- completed_at,
- queue_seconds,
- duration_seconds,
+ {job_columns},
provider,
os,
vcpu,
{render_multiplier("vcpu")} AS multiplier,
{render_billable_seconds("provider", "os", "is_rerun_copy", "billed_seconds")} AS billable_seconds,
- {render_estimated_cost_usd("provider", "os", "vcpu", "is_rerun_copy", "billed_seconds")} AS estimated_cost_usd,
- is_merge_queue,
- is_rerun_copy,
- created_at_raw,
- ci_engine, native_run_id, native_workflow_run_id, native_job_id, native_attempt_id{run_columns}
+ {render_estimated_cost_usd("provider", "os", "vcpu", "is_rerun_copy", "billed_seconds")} AS estimated_cost_usd
FROM (
SELECT
- repo_owner,
- repo_name,
- pr_number,
- workflow_name,
- job_name,
- run_id,
- run_attempt,
- head_branch,
- status,
- conclusion,
- runner_name,
- created_at,
- started_at,
- completed_at,
- queue_seconds,
- duration_seconds,
+ {job_columns},
billed_seconds,
- is_merge_queue,
- is_rerun_copy,
- created_at_raw,
- ci_engine, native_run_id, native_workflow_run_id, native_job_id, native_attempt_id,
{render_provider("depot_label", "hosted_label")} AS provider,
{render_os("depot_label", "hosted_label")} AS os,
- {render_vcpu("depot_label", "hosted_label")} AS vcpu{run_columns}
+ {render_vcpu("depot_label", "hosted_label")} AS vcpu
FROM (
SELECT
- repo_owner,
- repo_name,
- pr_number,
- workflow_name,
- job_name,
- run_id,
- run_attempt,
- head_branch,
- status,
- conclusion,
- runner_name,
- created_at,
- started_at,
- completed_at,
- queue_seconds,
- duration_seconds,
+ {job_columns},
billed_seconds,
- is_merge_queue,
- is_rerun_copy,
- created_at_raw,
- ci_engine, native_run_id, native_workflow_run_id, native_job_id, native_attempt_id,
{render_depot_label("labels_arr")} AS depot_label,
- {render_hosted_label("labels_arr")} AS hosted_label{run_columns}
+ {render_hosted_label("labels_arr")} AS hosted_label
FROM (
SELECT
- r.repo_owner AS repo_owner,
- r.repo_name AS repo_name,
- -- An unattributed run surfaces pr_number 0 in the runs builder; normalize both
- -- that and the LEFT-JOIN NULL to NULL so a missing PR is never read as PR #0.
- nullIf(r.pr_number, 0) AS pr_number,
- j.workflow_name AS workflow_name,
- j.name AS job_name,
- j.run_id AS run_id,
- j.run_attempt AS run_attempt,
- {workflow_jobs.branch("j", "r")} AS head_branch,
- j.status AS status,
- j.conclusion AS conclusion,
- j.runner_name AS runner_name,
- j.created_at AS created_at,
- j.started_at AS started_at,
- j.completed_at AS completed_at,
- j.queue_seconds AS queue_seconds,
- j.duration_seconds AS duration_seconds,
+ {job_columns},
-- What Depot actually bills: the wall-clock minus the runner boot GitHub
-- stamps into it. duration_seconds stays the full window for queue/duration UX.
{billed_seconds} AS billed_seconds,
- r.is_merge_queue AS is_merge_queue,
- j.is_rerun_copy AS is_rerun_copy,
- j.created_at_raw AS created_at_raw,
- j.ci_engine AS ci_engine,
- j.native_run_id AS native_run_id,
- j.native_workflow_run_id AS native_workflow_run_id,
- j.native_job_id AS native_job_id,
- j.native_attempt_id AS native_attempt_id,
- {labels_array} AS labels_arr{inner_run_columns}
- FROM ({jobs}) AS j
- LEFT JOIN ({runs}) AS r ON j.run_id = r.id AND j.ci_engine = r.ci_engine
+ {labels_array} AS labels_arr
+ FROM ({jobs})
)
)
)
"""
+def build_attributed_query(*, costed_jobs: str, runs: str) -> str:
+ """Costed job rows with the attribution of their runs: the ``BUILDER_FIELDS`` columns.
+
+ ``costed_jobs`` has the columns of ``build_costed_jobs_query`` and ``runs`` the columns of the runs
+ builder. The join reads the run row as it is now, so a re-run that moves the start of a run moves
+ every job of that run with it.
+ """
+ return f"""
+ SELECT
+ r.repo_owner AS repo_owner,
+ r.repo_name AS repo_name,
+ -- An unattributed run surfaces pr_number 0 in the runs builder; normalize both
+ -- that and the LEFT-JOIN NULL to NULL so a missing PR is never read as PR #0.
+ nullIf(r.pr_number, 0) AS pr_number,
+ j.workflow_name AS workflow_name,
+ j.name AS job_name,
+ j.run_id AS run_id,
+ j.run_attempt AS run_attempt,
+ {workflow_jobs.branch("j", "r")} AS head_branch,
+ j.status AS status,
+ j.conclusion AS conclusion,
+ j.runner_name AS runner_name,
+ j.created_at AS created_at,
+ j.started_at AS started_at,
+ j.completed_at AS completed_at,
+ j.queue_seconds AS queue_seconds,
+ j.duration_seconds AS duration_seconds,
+ j.provider AS provider,
+ j.os AS os,
+ j.vcpu AS vcpu,
+ j.multiplier AS multiplier,
+ j.billable_seconds AS billable_seconds,
+ j.estimated_cost_usd AS estimated_cost_usd,
+ r.is_merge_queue AS is_merge_queue,
+ j.is_rerun_copy AS is_rerun_copy,
+ j.created_at_raw AS created_at_raw,
+ j.ci_engine AS ci_engine,
+ j.native_run_id AS native_run_id,
+ j.native_workflow_run_id AS native_workflow_run_id,
+ j.native_job_id AS native_job_id,
+ j.native_attempt_id AS native_attempt_id{_PASSTHROUGH_DEFS}
+ FROM ({costed_jobs}) AS j
+ LEFT JOIN ({runs}) AS r ON j.run_id = r.id AND j.ci_engine = r.ci_engine
+ """
+
+
def build_team_view(team: "Team") -> str | None:
"""The full view body for a team: every GitHub source with both runs and jobs synced, unioned.
@@ -318,5 +270,7 @@ def build_team_view(team: "Team") -> str | None:
sources = resolve_job_source_tables(team)
if not sources:
return None
- selects = [build_query(jobs_table=source.jobs_source, runs_table=source.runs_source) for source in sources]
- return "\nUNION ALL\n".join(selects)
+ return "\nUNION ALL\n".join(
+ f"SELECT {', '.join(FIELDS)} FROM ({build_query(jobs_table=source.jobs_source, runs_table=source.runs_source)})"
+ for source in sources
+ )
diff --git a/products/engineering_analytics/backend/logic/views/workflow_jobs.py b/products/engineering_analytics/backend/logic/views/workflow_jobs.py
index 7bc25df12fc2..df805d1c8338 100644
--- a/products/engineering_analytics/backend/logic/views/workflow_jobs.py
+++ b/products/engineering_analytics/backend/logic/views/workflow_jobs.py
@@ -90,7 +90,37 @@ class JobsTable:
return cls(rows=table, duplicates=table)
-def build_query(table: JobsTable, *, created_floor: bool = False, include_steps: bool = False) -> str:
+COLUMNS = (
+ "id",
+ "run_id",
+ "run_attempt",
+ "name",
+ "workflow_name",
+ "head_sha",
+ "head_branch",
+ "status",
+ "conclusion",
+ "labels",
+ "runner_name",
+ "created_at",
+ "created_at_raw",
+ "started_at",
+ "completed_at",
+ "duration_seconds",
+ "queue_seconds",
+ "provisioning_seconds",
+ "is_rerun_copy",
+ "ci_engine",
+ "native_run_id",
+ "native_workflow_run_id",
+ "native_job_id",
+ "native_attempt_id",
+)
+
+
+def build_query(
+ table: JobsTable, *, created_floor: bool = False, include_steps: bool = False, compact_duplicates: bool = True
+) -> str:
"""``include_steps`` appends the raw ``steps`` JSON as the last column. It is the widest column of
the table, so only a read bounded to one run asks for it."""
@@ -105,6 +135,11 @@ def build_query(table: JobsTable, *, created_floor: bool = False, include_steps:
duplicates_source = floored(table.duplicates)
steps_column = ",\n job.steps AS steps" if include_steps else ""
parsed_steps_column = ",\n ifNull(toString(steps), '[]') AS steps" if include_steps else ""
+ # The HAVING saves memory on unbounded live reads. Bounded materializations retain singleton
+ # groups too, so the lookup has no group-removing construct that incremental validation rejects.
+ duplicate_filter = (
+ "HAVING count() > 1 AND started_key IS NOT NULL AND completed_key IS NOT NULL" if compact_duplicates else ""
+ )
return f"""
SELECT
job.id AS id,
@@ -149,11 +184,11 @@ def build_query(table: JobsTable, *, created_floor: bool = False, include_steps:
ci_engine, native_run_id, native_workflow_run_id, native_job_id, native_attempt_id,
run_id,
run_attempt,
- name,
+ ifNull(name, '') AS name,
ifNull(workflow_name, '') AS workflow_name,
ifNull(head_sha, '') AS head_sha,
ifNull(head_branch, '') AS head_branch,
- status,
+ ifNull(status, '') AS status,
conclusion,
ifNull(labels, '[]') AS labels,
ifNull(runner_name, '') AS runner_name,
@@ -181,7 +216,7 @@ def build_query(table: JobsTable, *, created_floor: bool = False, include_steps:
min(run_attempt) AS first_attempt
FROM {duplicates_source}
GROUP BY ci_engine, run_id, name, started_key, completed_key
- HAVING count() > 1 AND started_key IS NOT NULL AND completed_key IS NOT NULL
+ {duplicate_filter}
) AS dupes
ON job.run_id = dupes.run_id AND job.ci_engine = dupes.ci_engine
AND job.name = dupes.name
diff --git a/products/engineering_analytics/backend/logic/views/workflow_runs.py b/products/engineering_analytics/backend/logic/views/workflow_runs.py
index 2450b1b16bfc..196bf4f716a3 100644
--- a/products/engineering_analytics/backend/logic/views/workflow_runs.py
+++ b/products/engineering_analytics/backend/logic/views/workflow_runs.py
@@ -84,6 +84,8 @@ Every query module embeds this ``SELECT`` as a subquery (see ``_curated``);
nothing registers it as a global HogQL view.
"""
+from collections.abc import Sequence
+
from products.engineering_analytics.backend.logic.merge_queue import source_pr_number_expr
# The source PR of a merge-queue gate branch, corroborated against the run's actor so a
@@ -137,7 +139,15 @@ def _merged_pr_index(pull_requests_table: str) -> str:
"""
-def build_query(table_name: str, *, pull_requests_table: str | None = None, started_floor: bool = False) -> str:
+def build_query(
+ table_name: str,
+ *,
+ pull_requests_table: str | None = None,
+ started_floor: bool = False,
+ passthrough: Sequence[str] = (),
+) -> str:
+ """``passthrough`` names columns of ``table_name`` that the result carries unchanged."""
+ extra_columns = "".join(f", {column}" for column in passthrough)
# The raw floor must live in its OWN innermost SELECT, not the parsing SELECT below: that SELECT
# aliases parseDateTimeBestEffort(run_started_at) AS run_started_at, and ClickHouse alias resolution
# would make a WHERE there compare the parsed DateTime against the string. Keep it on the raw column.
@@ -175,14 +185,14 @@ def build_query(table_name: str, *, pull_requests_table: str | None = None, star
arrayElement(repo_parts, 2) AS repo_name,
ci_engine, native_run_id, native_workflow_run_id,
workflow_id,
- event
+ event{extra_columns}
FROM (
SELECT
id,
name AS workflow_name,
ci_engine, native_run_id, native_workflow_run_id,
workflow_id,
- event,
+ event{extra_columns},
head_sha,
head_branch,
status,
diff --git a/products/engineering_analytics/backend/tests/test_ci_views.py b/products/engineering_analytics/backend/tests/test_ci_views.py
index cbb23af4c2bc..50486d780ef1 100644
--- a/products/engineering_analytics/backend/tests/test_ci_views.py
+++ b/products/engineering_analytics/backend/tests/test_ci_views.py
@@ -1,18 +1,28 @@
import json
import tempfile
+from collections.abc import Callable
+from datetime import UTC, datetime
from pathlib import Path
from typing import Any
+from uuid import uuid4
from posthog.test.base import BaseTest, ClickhouseTestMixin
from unittest.mock import patch
+from django.test import SimpleTestCase
+
import pandas as pd
+import pyarrow as pa
+from deltalake import DeltaTable, write_deltalake
from parameterized import parameterized
+from posthog.hogql import ast
from posthog.hogql.query import execute_hogql_query
from posthog.clickhouse.client import sync_execute
+from posthog.temporal.data_modeling.activities.incremental_write import UniqueKeyTracker, upsert_batch
+from products.data_modeling.backend.facade.api import check_incremental_eligibility, get_incremental_config
from products.data_modeling.backend.facade.models import DataWarehouseManagedViewSet, DataWarehouseSavedQuery
from products.engineering_analytics.backend.facade.warehouse_views import get_expected_warehouse_views
from products.engineering_analytics.backend.logic.job_logs.constants import CI_LOGS_SERVICE_NAME
@@ -20,10 +30,12 @@ from products.engineering_analytics.backend.logic.sources import (
PULL_REQUESTS_SCHEMA,
WORKFLOW_JOBS_SCHEMA,
WORKFLOW_RUNS_SCHEMA,
+ JobSourceTables,
)
from products.engineering_analytics.backend.logic.views import (
ci_failures,
ci_job_history,
+ ci_rows,
depot_ci,
job_costs,
pr_friction,
@@ -98,31 +110,33 @@ def _run_row(
}
+def _create_table(test: BaseTest, base_name: str, columns: dict, rows: list[dict[str, Any]]) -> str:
+ df = pd.DataFrame(rows, columns=list(columns.keys()))
+ tmp = tempfile.NamedTemporaryFile(mode="w", suffix=".csv", delete=False)
+ df.to_csv(tmp.name, index=False)
+ tmp.close()
+ test.addCleanup(Path(tmp.name).unlink, missing_ok=True)
+ with seeding_object_storage(test):
+ table, _source, _credential, _df, cleanup = create_data_warehouse_table_from_csv(
+ csv_path=Path(tmp.name),
+ table_name=base_name,
+ table_columns=columns,
+ test_bucket=TEST_BUCKET,
+ team=test.team,
+ source_prefix=GITHUB_SOURCE_PREFIX,
+ )
+ test.addCleanup(cleanup)
+ return table.name
+
+
class TestCIJobHistoryView(ClickhouseTestMixin, BaseTest):
"""The generated ci_job_history SQL must expose exactly the FIELDS columns and correctly derive
commit attribution — the substrate the green/red boundary analysis reads. Skips when object
storage is unreachable so the suite still runs without the dev stack."""
- def _create_table(self, base_name: str, columns: dict, rows: list[dict[str, Any]]) -> str:
- df = pd.DataFrame(rows, columns=list(columns.keys()))
- tmp = tempfile.NamedTemporaryFile(mode="w", suffix=".csv", delete=False)
- df.to_csv(tmp.name, index=False)
- tmp.close()
- self.addCleanup(Path(tmp.name).unlink, missing_ok=True)
- with seeding_object_storage(self):
- table, _source, _credential, _df, cleanup = create_data_warehouse_table_from_csv(
- csv_path=Path(tmp.name),
- table_name=base_name,
- table_columns=columns,
- test_bucket=TEST_BUCKET,
- team=self.team,
- source_prefix=GITHUB_SOURCE_PREFIX,
- )
- self.addCleanup(cleanup)
- return table.name
-
def test_exposes_fields_and_derives_commit_attribution(self) -> None:
- jobs_table = self._create_table(
+ jobs_table = _create_table(
+ self,
"github_workflow_jobs",
WORKFLOW_JOBS_COLUMNS,
[
@@ -136,7 +150,8 @@ class TestCIJobHistoryView(ClickhouseTestMixin, BaseTest):
_job_row(5, run_id=400, name="job-e"),
],
)
- runs_table = self._create_table(
+ runs_table = _create_table(
+ self,
"github_workflow_runs",
WORKFLOW_RUNS_COLUMNS,
[
@@ -246,10 +261,11 @@ class TestCIJobHistoryView(ClickhouseTestMixin, BaseTest):
def test_union_all_of_two_sources_parses_and_stacks(self) -> None:
# build_team_view stitches per-source SELECTs with UNION ALL; the columns must line up across
# the boundary. Unioning one source's SELECT with itself proves the shape agrees and stacks.
- jobs_table = self._create_table(
- "github_workflow_jobs", WORKFLOW_JOBS_COLUMNS, [_job_row(1, run_id=100, name="job-a")]
+ jobs_table = _create_table(
+ self, "github_workflow_jobs", WORKFLOW_JOBS_COLUMNS, [_job_row(1, run_id=100, name="job-a")]
)
- runs_table = self._create_table(
+ runs_table = _create_table(
+ self,
"github_workflow_runs",
WORKFLOW_RUNS_COLUMNS,
[_run_row(100, head_sha="s", pr_numbers=[1], head_commit={"message": "m"})],
@@ -354,6 +370,142 @@ class TestCIFailuresView(ClickhouseTestMixin, BaseTest):
assert no_detail[field_index["fingerprint"]] == "posthog/api/test_baz.py::test_qux | "
+class TestStoredCIConfig(SimpleTestCase):
+ @parameterized.expand(
+ [
+ ("runs", ci_rows.RUN_INCREMENTAL_CONFIG, "github_actions", 1, 1, 2, False),
+ ("github_jobs", ci_rows.JOB_INCREMENTAL_CONFIG, "github_actions", 1, 2, 2, True),
+ ("depot_jobs", ci_rows.JOB_INCREMENTAL_CONFIG, "depot_ci", 2, 2, 2, False),
+ ]
+ )
+ def test_retry_upserts_preserve_each_source_and_replace_only_the_same_logical_row(
+ self,
+ _name: str,
+ raw_config: dict[str, Any],
+ engine: str,
+ previous_attempt: int,
+ next_id: int,
+ next_attempt: int,
+ keeps_previous: bool,
+ ) -> None:
+ config = get_incremental_config(DataWarehouseSavedQuery(incremental_config=raw_config))
+ assert config is not None
+ original = {
+ "source_id": "first",
+ "ci_engine": engine,
+ "id": 1,
+ "native_job_id": "job-1",
+ "run_attempt": previous_attempt,
+ }
+ initial = pa.RecordBatch.from_pylist([original, {**original, "source_id": "second"}])
+ UniqueKeyTracker(config.unique_key).check(initial)
+ updated = {
+ **original,
+ "id": next_id,
+ "run_attempt": next_attempt,
+ "native_job_id": f"job-{next_id}" if engine == "github_actions" else "job-1",
+ }
+ with tempfile.TemporaryDirectory() as directory:
+ write_deltalake(directory, pa.Table.from_batches([initial]))
+ upsert_batch(directory, {}, pa.RecordBatch.from_pylist([updated]), config.unique_key, commit_metadata={})
+ rows = DeltaTable(directory).to_pyarrow_table().to_pylist()
+
+ expected = {("second", 1, previous_attempt), ("first", next_id, next_attempt)}
+ if keeps_previous:
+ expected.add(("first", 1, previous_attempt))
+ assert {(row["source_id"], row["id"], row["run_attempt"]) for row in rows} == expected
+
+ @parameterized.expand(
+ [
+ ("runs", ci_rows.runs_select, ci_rows.RUN_INCREMENTAL_CONFIG, False),
+ ("jobs", ci_rows.jobs_select, ci_rows.JOB_INCREMENTAL_CONFIG, False),
+ ("runs_with_depot", ci_rows.runs_select, ci_rows.RUN_INCREMENTAL_CONFIG, True),
+ ("jobs_with_depot", ci_rows.jobs_select, ci_rows.JOB_INCREMENTAL_CONFIG, True),
+ ]
+ )
+ def test_stored_views_are_eligible_for_their_incremental_config(
+ self, _name: str, select: Callable[[JobSourceTables], str], raw_config: dict[str, Any], with_depot: bool
+ ) -> None:
+ source = JobSourceTables(
+ github_workflow_jobs="github_workflow_jobs",
+ github_workflow_runs="github_workflow_runs",
+ source_id="source",
+ repository=_REPO,
+ depot_job_attempts=depot_ci.DepotJobAttempts(table="depot_job_attempts", repository=_REPO)
+ if with_depot
+ else None,
+ )
+ config = get_incremental_config(DataWarehouseSavedQuery(incremental_config=raw_config))
+ assert config is not None
+ result = check_incremental_eligibility(select(source), config)
+ assert result.eligible, result.blockers
+
+
+class TestStoredCIViews(ClickhouseTestMixin, BaseTest):
+ @parameterized.expand(
+ [
+ ("runs", ci_rows.runs_select, ci_rows.RUN_FIELDS, 2, {"is_handoff_shell": 0}),
+ ("jobs", ci_rows.jobs_select, ci_rows.JOB_FIELDS, 3, {"is_rerun_copy": 0, "is_handoff_shell": 0}),
+ ]
+ )
+ def test_view_runs_with_the_window_start_bound(
+ self,
+ _name: str,
+ select: Callable[[JobSourceTables], str],
+ fields: dict[str, Any],
+ seeded_rows: int,
+ flags: dict[str, int],
+ ) -> None:
+ created_at = "2026-01-01T10:00:00.123Z"
+ jobs_table = _create_table(
+ self,
+ "github_workflow_jobs",
+ WORKFLOW_JOBS_COLUMNS,
+ [
+ {**_job_row(1, run_id=100, name="a"), "created_at": created_at},
+ {**_job_row(2, run_id=100, name="b"), "created_at": created_at},
+ {**_job_row(3, run_id=200, name="c"), "created_at": created_at},
+ ],
+ )
+ runs_table = _create_table(
+ self,
+ "github_workflow_runs",
+ WORKFLOW_RUNS_COLUMNS,
+ [
+ {**_run_row(100, head_sha="s100", pr_numbers=[5], head_commit={}), "created_at": created_at},
+ {**_run_row(200, head_sha="s200", pr_numbers=[], head_commit={}), "created_at": created_at},
+ ],
+ )
+ source = JobSourceTables(
+ github_workflow_jobs=jobs_table,
+ github_workflow_runs=runs_table,
+ source_id=str(uuid4()),
+ repository=_REPO,
+ )
+ query = f"SELECT {', '.join(fields)} FROM ({select(source)}) ORDER BY id"
+
+ rows_by_window = {}
+ for since, expected_rows in (
+ (None, seeded_rows),
+ (datetime(2026, 1, 1, 10, tzinfo=UTC), seeded_rows),
+ (datetime(2026, 1, 1, 11, tzinfo=UTC), 0),
+ ):
+ response = execute_hogql_query(
+ query=query,
+ team=self.team,
+ query_type="engineering_analytics.test",
+ placeholders={"incremental_window_start": ast.Constant(value=since)},
+ )
+ assert response.columns == list(fields)
+ assert len(response.results) == expected_rows, since
+ rows_by_window[since] = response.results
+
+ first = dict(zip(fields, rows_by_window[None][0]))
+ assert first["repository"] == _REPO.casefold()
+ assert first["created_at_raw"] == created_at
+ assert {flag: first[flag] for flag in flags} == flags
+
+
class TestExpectedWarehouseViews(BaseTest):
"""The facade must expose all three views together for a qualifying GitHub source, and nothing
for a team without one — so a consumer sees a coherent set, never a partial one."""
@@ -387,40 +539,54 @@ class TestExpectedWarehouseViews(BaseTest):
def test_no_source_exposes_no_views(self) -> None:
assert get_expected_warehouse_views(self.team) == []
- def test_qualifying_source_exposes_all_three_views(self) -> None:
+ def test_qualifying_source_exposes_every_view(self) -> None:
self._qualifying_source()
- names = {view.name for view in get_expected_warehouse_views(self.team)}
- assert names == {job_costs.VIEW_NAME, ci_job_history.VIEW_NAME, ci_failures.VIEW_NAME}
+ with patch("posthoganalytics.feature_enabled", return_value=True):
+ views = {view.name: view for view in get_expected_warehouse_views(self.team)}
+ assert set(views) == {
+ job_costs.VIEW_NAME,
+ ci_job_history.VIEW_NAME,
+ ci_failures.VIEW_NAME,
+ ci_rows.RUNS_VIEW_NAME,
+ ci_rows.JOBS_VIEW_NAME,
+ }
+ for name, config in (
+ (ci_rows.RUNS_VIEW_NAME, ci_rows.RUN_INCREMENTAL_CONFIG),
+ (ci_rows.JOBS_VIEW_NAME, ci_rows.JOB_INCREMENTAL_CONFIG),
+ ):
+ assert views[name].sync_frequency == ci_rows.SYNC_FREQUENCY
+ assert views[name].incremental_config == config
@parameterized.expand(
[
- ("flag_on", True, None, True),
- ("flag_off", False, "managed", False),
- # No answer from the flag service keeps what the team has, so an outage never drops the table.
- ("no_answer_keeps_the_view", None, "managed", True),
- ("no_answer_ignores_a_user_query_of_that_name", None, "user", False),
- ("no_answer_adds_no_view", None, None, False),
+ ("flag_on", True, None, False, (ci_rows.RUNS_VIEW_NAME, ci_rows.JOBS_VIEW_NAME, pr_friction.VIEW_NAME)),
+ ("flag_off", False, pr_friction.VIEW_NAME, True, ()),
+ ("no_answer_keeps_friction_only", None, pr_friction.VIEW_NAME, True, (pr_friction.VIEW_NAME,)),
+ ("no_answer_keeps_runs_only", None, ci_rows.RUNS_VIEW_NAME, True, (ci_rows.RUNS_VIEW_NAME,)),
+ ("no_answer_keeps_jobs_only", None, ci_rows.JOBS_VIEW_NAME, True, (ci_rows.JOBS_VIEW_NAME,)),
+ ("no_answer_ignores_a_user_query_of_that_name", None, pr_friction.VIEW_NAME, False, ()),
+ ("no_answer_adds_no_view", None, None, False, ()),
]
)
- def test_friction_view_needs_the_pull_request_snapshot_and_the_flag(
- self, _name: str, flag: bool | None, existing_view: str | None, expected: bool
+ def test_materialized_views_preserve_only_existing_managed_views_when_the_flag_has_no_answer(
+ self, _name: str, flag: bool | None, existing_name: str | None, managed: bool, expected: tuple[str, ...]
) -> None:
self._qualifying_source(with_pull_requests=True)
- if existing_view:
+ if existing_name:
viewset = (
DataWarehouseManagedViewSet.objects.create(
team=self.team, kind=DataWarehouseManagedViewSetKind.ENGINEERING_ANALYTICS
)
- if existing_view == "managed"
+ if managed
else None
)
DataWarehouseSavedQuery.objects.create(
team=self.team,
- name=pr_friction.VIEW_NAME,
+ name=existing_name,
query={"kind": "HogQLQuery", "query": "SELECT 1"},
managed_viewset=viewset,
)
with patch("posthoganalytics.feature_enabled", return_value=flag):
materialized = {view.name: view.materialized for view in get_expected_warehouse_views(self.team)}
per_job = {job_costs.VIEW_NAME: False, ci_job_history.VIEW_NAME: False, ci_failures.VIEW_NAME: False}
- assert materialized == ({**per_job, pr_friction.VIEW_NAME: True} if expected else per_job)
+ assert materialized == {**per_job, **dict.fromkeys(expected, True)}
diff --git a/products/engineering_analytics/backend/tests/test_job_costs.py b/products/engineering_analytics/backend/tests/test_job_costs.py
index 2cf41d39815d..794e908522af 100644
--- a/products/engineering_analytics/backend/tests/test_job_costs.py
+++ b/products/engineering_analytics/backend/tests/test_job_costs.py
@@ -162,10 +162,9 @@ class TestJobCostsViewParity(ClickhouseTestMixin, BaseTest):
self.addCleanup(cleanup)
return table.name
- def test_exposed_view_columns_match_the_field_contract(self) -> None:
- # The view body projects its column list through four nested SELECTs. Appending to FIELDS but
- # missing a layer yields a query that still runs and silently drops the column from the
- # exposed view, so assert the two agree — the same guard ci_job_history has.
+ def test_builder_columns_match_the_field_contract(self) -> None:
+ # The exposed view selects FIELDS by name from the builder's rows, so a column the builder
+ # drops must fail here and not in a team's saved query.
jobs_table = self._create_table("github_workflow_jobs", WORKFLOW_JOBS_COLUMNS, [_job_row(0, *_MATRIX[0][1:])])
runs_table = self._create_table(
"github_workflow_runs", WORKFLOW_RUNS_COLUMNS, [dict.fromkeys(WORKFLOW_RUNS_COLUMNS)]
@@ -174,7 +173,7 @@ class TestJobCostsViewParity(ClickhouseTestMixin, BaseTest):
columns = execute_hogql_query(
query=f"SELECT * FROM ({query})", team=self.team, query_type="engineering_analytics.test"
).columns
- assert columns == list(job_costs.FIELDS)
+ assert columns == list(job_costs.BUILDER_FIELDS)
def test_view_matches_python_cost_model(self) -> None:
jobs_table = self._create_table(
diff --git a/products/engineering_analytics/backend/tests/test_logic_sources.py b/products/engineering_analytics/backend/tests/test_logic_sources.py
index 088850589f99..301b49cd250a 100644
--- a/products/engineering_analytics/backend/tests/test_logic_sources.py
+++ b/products/engineering_analytics/backend/tests/test_logic_sources.py
@@ -578,12 +578,14 @@ class TestMultiRepoGitHubResolution(BaseTest):
github_workflow_runs="costgithub_posthog_posthog_workflow_runs",
pull_requests=None,
source_id=str(source.id),
+ repository="posthog/posthog",
),
JobSourceTables(
github_workflow_jobs="prsgithub_posthog_posthog_workflow_jobs",
github_workflow_runs="prsgithub_posthog_posthog_workflow_runs",
pull_requests="prsgithub_posthog_posthog_pull_requests",
source_id=str(with_prs.id),
+ repository="posthog/posthog",
depot_job_attempts=DepotJobAttempts(table="cidepot_job_attempts", repository="posthog/posthog"),
),
JobSourceTables(
@@ -591,6 +593,7 @@ class TestMultiRepoGitHubResolution(BaseTest):
github_workflow_runs="costgithub_posthog_posthog_com_workflow_runs",
pull_requests=None,
source_id=str(source.id),
+ repository="posthog/posthog.com",
),
}
diff --git a/products/engineering_analytics/backend/warehouse_view_provider.py b/products/engineering_analytics/backend/warehouse_view_provider.py
index d3f023184dfd..d6eae98b5ac0 100644
--- a/products/engineering_analytics/backend/warehouse_view_provider.py
+++ b/products/engineering_analytics/backend/warehouse_view_provider.py
@@ -18,6 +18,13 @@ def get_provided_views(team: Team) -> list[ProvidedView]:
schedule / managed DAG.
"""
return [
- ProvidedView(name=view.name, query=view.query, fields=view.fields, materialized=view.materialized)
+ ProvidedView(
+ name=view.name,
+ query=view.query,
+ fields=view.fields,
+ materialized=view.materialized,
+ sync_frequency=view.sync_frequency,
+ incremental_config=view.incremental_config,
+ )
for view in get_expected_warehouse_views(team)
]