Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(engineering-analytics): gate background work on enrollment (#129263)

Raúl Negrón-Oteroec16edf0a085
diff --git a/products/engineering_analytics/README.md b/products/engineering_analytics/README.md
index 7b768e94e520..26a6bd5fd6b2 100644
--- a/products/engineering_analytics/README.md
+++ b/products/engineering_analytics/README.md
@@ -83,7 +83,7 @@ graph LR
- Job-level CI: per-job duration, queue time, runner tier, and dollar cost ride `workflow_jobs` (webhook stream plus bounded backfill).
- One write surface: test quarantine. The `quarantine/request` endpoint files a tracking issue plus a PR against the repo's checked-in `.test_quarantine.json` through the team's GitHub App; it is reachable via the API and MCP only, with no in-app UI. Everything else is read-only.
- The Tests tab is the Trunk quarantine debt scoreboard: every test Trunk currently quarantines, attributed to the team that owns its file in the repository today, aged against a TTL, and rolled up per team. A quarantine masks a failure without fixing it, so the board is the standing work queue of masked tests. The flaky-test queue itself (failures plus same-commit recovery evidence from the pytest and Jest suites, counted per CI run, ranked by blast radius) remains an endpoint and MCP tool; the weekly Slack report requests each runner separately and shows up to 10 pytest tests followed by up to 10 Jest tests; it is informational, so suppressed tests stay listed with their quarantine state. With the `FLAKY_REPORT_TEAM_CHANNELS` repository variable set to `true`, each owning team also gets its rows in its own notifications channel, linked from the report thread; a week without Trunk state keeps those rows in the thread. Logs-based rescue counts and links remain pytest-only.
-- Access control: per-user warehouse RBAC at the source resolver, `engineering_analytics:read` scopes on tools, feature-flag gated.
+- Access control: per-user warehouse RBAC at the source resolver, `engineering_analytics:read` scopes on tools, feature-flag gated. Background work (view sync, the test census, CI failure log collection) runs only for teams that enabled the product, which happens when a user opens it with a GitHub source connected.
## The goal: CI Signals for PostHog Desktop
diff --git a/products/engineering_analytics/SPEC.md b/products/engineering_analytics/SPEC.md
index da7c82251647..952ddaa315f6 100644
--- a/products/engineering_analytics/SPEC.md
+++ b/products/engineering_analytics/SPEC.md
@@ -101,7 +101,8 @@ The endpoint catalog is `presentation/views.py`; the agent-facing descriptions l
### Exposed warehouse views
Per-team managed views (`DataWarehouseSavedQuery`, kind `engineering_analytics`) expose the curated CI substrate to insights, subscriptions, other products, and `execute-sql`: the only surface where the read layer is reachable as data rather than through the named endpoints.
-One gate for the three per-job views: a team gets them only when a GitHub source has **both** `workflow_runs` and `workflow_jobs` synced, so they appear together or not at all.
+The team's managed viewset row is its opt-in. The `enable` endpoint creates it when a user opens the product with a GitHub source connected, and no background job creates it: a team with a GitHub source it connected for another product gets no views. Every background job of this product (the post-load view sync, the daily test census, the GitHub and Depot job-log sweeps) runs only for enrolled teams, filtered in the database query through `logic/enrollment.py`. CI Signals has its own opt-in on top.
+One gate for the three per-job views: an enrolled team gets them only when a GitHub source has **both** `workflow_runs` and `workflow_jobs` synced, so they appear together or not at all.
They are non-materialized: the rendered SQL is persisted per team and re-synced on every runs/jobs load and every Depot job-attempts load, so a builder change reaches active teams within one sync cycle.
The per-PR friction view is the exception, below: it also needs the `pull_requests` snapshot, and it is materialized.
@@ -137,7 +138,7 @@ The per-PR friction view is the exception, below: it also needs the `pull_reques
- The red and running time is the PR timeline replay (§6) written as set-based HogQL, so one query covers every pull request in the window. `tests/test_pr_friction.py` replays seeded pull requests through both and asserts they agree, the way the cost view is held to the Python cost model.
- It counts friction and does not score it. Curves and weights are applied when the view is read, so a weight change needs no rematerialization.
- Materialized on the managed-view schedule, because the replay is too heavy to run on every read. Each run replaces the table, so it only holds the pull requests inside the window.
-- A materialized view spends the team's warehouse compute, so it exists only for organizations the `engineering-analytics-friction` flag targets. The flag is evaluated per organization, because the view sync runs with no user. When the flag service gives no answer, the sync keeps the view the team already has.
+- A materialized view spends the team's warehouse compute, so it exists only for enrolled teams in organizations the `engineering-analytics-friction` flag targets. The flag is evaluated per organization, because the view sync runs with no user. When the flag service gives no answer, the sync keeps the view the team already has.
- The view unions every GitHub source of the team, and each row carries its `source_id`. A product read filters on the one source the caller may use. A query on the materialized view itself checks access to the view, not to each source in it, like any materialized managed view.
## 6. Locked decisions
diff --git a/products/engineering_analytics/backend/facade/api.py b/products/engineering_analytics/backend/facade/api.py
index de45d036f5c8..587ad5864b50 100644
--- a/products/engineering_analytics/backend/facade/api.py
+++ b/products/engineering_analytics/backend/facade/api.py
@@ -101,6 +101,11 @@ def _authorized_source(
)
+def enable(*, team: Team) -> None:
+ """Opt the team in: create its managed viewset and sync the views once. Idempotent."""
+ logic.enroll(team)
+
+
def get_ci_signals_config(*, team: Team, user_access_control: "UserAccessControl | None" = None) -> CISignalsConfig:
return logic.get_ci_signals_config(team=team, user_access_control=user_access_control)
diff --git a/products/engineering_analytics/backend/logic/__init__.py b/products/engineering_analytics/backend/logic/__init__.py
index 05e7d71b663e..012462a996b7 100644
--- a/products/engineering_analytics/backend/logic/__init__.py
+++ b/products/engineering_analytics/backend/logic/__init__.py
@@ -32,6 +32,7 @@ from products.engineering_analytics.backend.logic.delivery_scope import (
)
from products.engineering_analytics.backend.logic.depot_ci_usage import build_depot_ci_usage as build_depot_ci_usage
from products.engineering_analytics.backend.logic.dora import build_dora_overview as build_dora_overview
+from products.engineering_analytics.backend.logic.enrollment import enroll as enroll
from products.engineering_analytics.backend.logic.friction import (
build_author_friction as build_author_friction,
build_author_friction_detail as build_author_friction_detail,
diff --git a/products/engineering_analytics/backend/logic/enrollment.py b/products/engineering_analytics/backend/logic/enrollment.py
new file mode 100644
index 000000000000..69621901dcdd
--- /dev/null
+++ b/products/engineering_analytics/backend/logic/enrollment.py
@@ -0,0 +1,43 @@
+"""Which teams opted into engineering analytics: the ones with the product's managed viewset row.
+
+A user action creates the row; no background job does. See SPEC.md, "Exposed warehouse views".
+"""
+
+from django.db.models import QuerySet
+
+from posthog.models.team import Team
+
+from products.data_modeling.backend.facade.models import DataWarehouseManagedViewSet
+from products.warehouse_sources.backend.facade.models import ExternalDataSource
+from products.warehouse_sources.backend.facade.types import DataWarehouseManagedViewSetKind, ExternalDataSourceType
+
+_KIND = DataWarehouseManagedViewSetKind.ENGINEERING_ANALYTICS
+_enrollments = DataWarehouseManagedViewSet.objects.filter(kind=_KIND)
+
+
+def enrolled_sources(source_type: ExternalDataSourceType) -> QuerySet[ExternalDataSource]:
+ """Every enrolled team's non-deleted sources of ``source_type``, with the team: the root of each
+ cross-team sweep."""
+ return (
+ ExternalDataSource.objects.filter(
+ source_type=source_type, team_id__in=_enrollments.values_list("team_id", flat=True)
+ )
+ .exclude(deleted=True)
+ .select_related("team")
+ )
+
+
+def get_enrollment(team: Team) -> DataWarehouseManagedViewSet | None:
+ return _enrollments.filter(team=team).first()
+
+
+def enroll(team: Team) -> None:
+ """Create the opt-in row if missing and sync the views once, on creation.
+
+ The post-load hook re-syncs an enrolled team's views after every relevant warehouse load, so a
+ repeat call does nothing: the UI calls this on every product open, and a sync here would hold
+ the team's sync lock and reach Temporal inside a web request each time.
+ """
+ managed_viewset, created = DataWarehouseManagedViewSet.objects.get_or_create(team=team, kind=_KIND)
+ if created:
+ managed_viewset.sync_views()
diff --git a/products/engineering_analytics/backend/logic/job_logs/coordinator.py b/products/engineering_analytics/backend/logic/job_logs/coordinator.py
index 46faf3c38556..cff99fe4884d 100644
--- a/products/engineering_analytics/backend/logic/job_logs/coordinator.py
+++ b/products/engineering_analytics/backend/logic/job_logs/coordinator.py
@@ -15,7 +15,6 @@ once the Logs endpoint is deployed, regardless of deploy order.
import re
import json
import dataclasses
-from collections.abc import Iterator
from datetime import timedelta
from typing import Any
@@ -39,6 +38,7 @@ from posthog.sync import database_sync_to_async
from posthog.temporal.common.base import PostHogWorkflow
from products.engineering_analytics.backend.facade.contracts import GitHubSourceNotConnectedError
+from products.engineering_analytics.backend.logic.enrollment import enrolled_sources
from products.engineering_analytics.backend.logic.job_logs.activity import (
FetchDepotJobLogInputs,
FetchDepotJobLogWorkflow,
@@ -121,12 +121,6 @@ def _query_jobs_with_diagnostics(team: Team, prefix: str, cutoff_iso: str, repo:
return [dict(zip(response.columns or [], row)) for row in response.results]
-def _live_sources(source_type: ExternalDataSourceType) -> Iterator[ExternalDataSource]:
- """Every team's non-deleted sources of ``source_type``, with the team, for the cross-team sweep."""
- sources = ExternalDataSource.objects.filter(source_type=source_type).exclude(deleted=True).select_related("team")
- return sources.iterator()
-
-
def _github_source_params(job_inputs: dict[str, Any] | None) -> tuple[int, str] | None:
"""``(integration_id, repo)`` from a GitHub source's ``job_inputs``, or None if unusable.
@@ -164,7 +158,7 @@ def _discover_jobs_with_diagnostics(cutoff_iso: str) -> list[dict[str, Any]]:
found: list[dict[str, Any]] = []
eligible_sources = 0
skipped_sources = 0
- for source in _live_sources(ExternalDataSourceType.GITHUB):
+ for source in enrolled_sources(ExternalDataSourceType.GITHUB).iterator():
try:
inputs = SourceInputsStore.load(source)
except UnreadableSourceInputsError:
@@ -371,7 +365,7 @@ def _discover_failed_depot_attempts(inputs: DepotDiscoveryInputs) -> DepotDiscov
return DepotDiscovery(attempts=[], cursors={})
attempts: list[FetchDepotJobLogInputs] = []
cursors: dict[str, DepotAttemptCursor] = {}
- for source in _live_sources(ExternalDataSourceType.DEPOT):
+ for source in enrolled_sources(ExternalDataSourceType.DEPOT).iterator():
source_id = str(source.id)
cursor = inputs.cursors.get(source_id)
limit = MAX_DISCOVERED_JOBS - len(attempts)
diff --git a/products/engineering_analytics/backend/logic/job_logs/test/test_coordinator.py b/products/engineering_analytics/backend/logic/job_logs/test/test_coordinator.py
index 7a1a5c288dcf..0947b79d18c4 100644
--- a/products/engineering_analytics/backend/logic/job_logs/test/test_coordinator.py
+++ b/products/engineering_analytics/backend/logic/job_logs/test/test_coordinator.py
@@ -33,6 +33,7 @@ from products.engineering_analytics.backend.tests._github_fixtures import (
create_github_source,
create_github_warehouse_table,
create_warehouse_table_row,
+ enroll_team,
link_schema,
)
from products.warehouse_sources.backend.facade.models import DataWarehouseTable, ExternalDataSource
@@ -84,6 +85,7 @@ class TestDiscoverFailedDepotAttempts(ClickhouseTestMixin, BaseTest):
# Only recent failed attempts are fetched, keyed by the integer ids the curated views and CI
# traces carry. A push run's id has no integer form, so its logs could never join and it is
# skipped rather than emitted under a wrong run_id.
+ enroll_team(self.team)
source = ExternalDataSource.objects.create(
team=self.team,
source_id="src-depot",
diff --git a/products/engineering_analytics/backend/presentation/views/__init__.py b/products/engineering_analytics/backend/presentation/views/__init__.py
index b1c5b6de8cc2..77c9f909b09f 100644
--- a/products/engineering_analytics/backend/presentation/views/__init__.py
+++ b/products/engineering_analytics/backend/presentation/views/__init__.py
@@ -14,6 +14,7 @@ from products.engineering_analytics.backend.presentation.views._base import (
from products.engineering_analytics.backend.presentation.views.ci_signals import CISignalsConfigMixin
from products.engineering_analytics.backend.presentation.views.delivery import DeliveryActionsMixin
from products.engineering_analytics.backend.presentation.views.dora import DoraActionsMixin
+from products.engineering_analytics.backend.presentation.views.enrollment import EnrollmentMixin
from products.engineering_analytics.backend.presentation.views.friction import FrictionActionsMixin
from products.engineering_analytics.backend.presentation.views.pull_requests import PullRequestActionsMixin
from products.engineering_analytics.backend.presentation.views.sources import SourcesMixin
@@ -25,6 +26,7 @@ from products.engineering_analytics.backend.presentation.views.workflows import
@extend_schema(tags=[ENGINEERING_ANALYTICS_TAG])
class EngineeringAnalyticsViewSet(
SourcesMixin,
+ EnrollmentMixin,
CISignalsConfigMixin,
PullRequestActionsMixin,
DeliveryActionsMixin,
@@ -51,6 +53,7 @@ class EngineeringAnalyticsViewSet(
*FrictionActionsMixin.READ_ACTIONS,
]
scope_object_write_actions: list[str] = [
+ *EnrollmentMixin.WRITE_ACTIONS,
*CISignalsConfigMixin.WRITE_ACTIONS,
*SuiteHealthActionsMixin.WRITE_ACTIONS,
]
diff --git a/products/engineering_analytics/backend/presentation/views/enrollment.py b/products/engineering_analytics/backend/presentation/views/enrollment.py
new file mode 100644
index 000000000000..4c01083ab3b6
--- /dev/null
+++ b/products/engineering_analytics/backend/presentation/views/enrollment.py
@@ -0,0 +1,27 @@
+from drf_spectacular.utils import extend_schema
+from rest_framework import status
+from rest_framework.decorators import action
+from rest_framework.request import Request
+from rest_framework.response import Response
+
+from products.engineering_analytics.backend.facade import api
+from products.engineering_analytics.backend.presentation.views._base import EngineeringAnalyticsViewSetBase
+
+
+class EnrollmentMixin(EngineeringAnalyticsViewSetBase):
+ WRITE_ACTIONS = ["enable"]
+
+ @extend_schema(
+ operation_id="engineering_analytics_enable",
+ request=None,
+ responses={204: None},
+ description=(
+ "Enable engineering analytics for this project. Creates the project's managed warehouse views "
+ "and opts it into the background work (view sync after each GitHub or Depot load, the daily "
+ "test census, CI failure log collection). Safe to call again: an enabled project stays enabled."
+ ),
+ )
+ @action(detail=False, methods=["post"], pagination_class=None)
+ def enable(self, request: Request, **kwargs) -> Response:
+ api.enable(team=self.team)
+ return Response(status=status.HTTP_204_NO_CONTENT)
diff --git a/products/engineering_analytics/backend/tasks/tasks.py b/products/engineering_analytics/backend/tasks/tasks.py
index c65af45af5e7..82bb88811864 100644
--- a/products/engineering_analytics/backend/tasks/tasks.py
+++ b/products/engineering_analytics/backend/tasks/tasks.py
@@ -10,23 +10,19 @@ from posthog.models.team import Team
from posthog.scoping_audit import skip_team_scope_audit
from products.engineering_analytics.backend.logic.census import collect_repo_census, emit_census_events
+from products.engineering_analytics.backend.logic.enrollment import enrolled_sources
from products.engineering_analytics.backend.logic.sources import list_github_sources
-from products.warehouse_sources.backend.facade.models import ExternalDataSource
from products.warehouse_sources.backend.facade.types import ExternalDataSourceType
logger = structlog.get_logger(__name__)
@shared_task(ignore_result=True)
-# Fan-out across every team with a GitHub source: cross-team by design, and the model still
-# uses a plain manager, so there is no unscoped() to declare it with.
+# Fan-out across every enrolled team with a GitHub source: cross-team by design, and the model
+# still uses a plain manager, so there is no unscoped() to declare it with.
@skip_team_scope_audit
def emit_test_ownership_census() -> None:
- team_ids = (
- ExternalDataSource.objects.filter(source_type=ExternalDataSourceType.GITHUB, deleted=False)
- .values_list("team_id", flat=True)
- .distinct()
- )
+ team_ids = enrolled_sources(ExternalDataSourceType.GITHUB).values_list("team_id", flat=True).distinct()
for team_id in team_ids:
emit_team_test_census.delay(team_id=team_id)
diff --git a/products/engineering_analytics/backend/tests/_github_fixtures.py b/products/engineering_analytics/backend/tests/_github_fixtures.py
index f8912d8032d0..3a0e449beb7f 100644
--- a/products/engineering_analytics/backend/tests/_github_fixtures.py
+++ b/products/engineering_analytics/backend/tests/_github_fixtures.py
@@ -16,6 +16,7 @@ import pandas as pd
from posthog.models.team import Team
+from products.data_modeling.backend.facade.models import DataWarehouseManagedViewSet
from products.engineering_analytics.backend.logic.sources import (
ISSUE_EVENTS_SCHEMA,
PULL_REQUESTS_SCHEMA,
@@ -29,7 +30,12 @@ from products.engineering_analytics.backend.logic.views.source_schema import (
)
from products.warehouse_sources.backend.facade.models import DataWarehouseTable, ExternalDataSchema, ExternalDataSource
from products.warehouse_sources.backend.facade.testing import create_data_warehouse_table_from_csv
-from products.warehouse_sources.backend.facade.types import ExternalDataSourceType
+from products.warehouse_sources.backend.facade.types import DataWarehouseManagedViewSetKind, ExternalDataSourceType
+
+
+def enroll_team(team: Team) -> None:
+ DataWarehouseManagedViewSet.objects.create(team=team, kind=DataWarehouseManagedViewSetKind.ENGINEERING_ANALYTICS)
+
TEST_BUCKET = "test_storage_bucket-posthog.products.engineering_analytics.github_fixtures"
diff --git a/products/engineering_analytics/backend/tests/test_census.py b/products/engineering_analytics/backend/tests/test_census.py
index 6d7b0f236056..f25ffbd9ca5a 100644
--- a/products/engineering_analytics/backend/tests/test_census.py
+++ b/products/engineering_analytics/backend/tests/test_census.py
@@ -2,11 +2,16 @@ import io
import tarfile
from types import SimpleNamespace
+from posthog.test.base import BaseTest
from unittest.mock import patch
from owners_yaml import TeamTestCensus
+from posthog.models.team import Team
+
from products.engineering_analytics.backend.logic.census import CENSUS_EVENT, collect_repo_census, emit_census_events
+from products.engineering_analytics.backend.tasks.tasks import emit_test_ownership_census
+from products.engineering_analytics.backend.tests._github_fixtures import create_github_source, enroll_team
def _tarball(files: dict[str, bytes]) -> bytes:
@@ -82,3 +87,16 @@ def test_emit_census_events_captures_the_contract_layer_4_reads() -> None:
},
}
]
+
+
+class TestCensusFanOut(BaseTest):
+ @patch("products.engineering_analytics.backend.tasks.tasks.emit_team_test_census.delay")
+ def test_fans_out_only_to_enrolled_teams(self, mock_delay) -> None:
+ unenrolled = Team.objects.create(organization=self.organization, name="unenrolled")
+ create_github_source(self.team)
+ create_github_source(unenrolled)
+ enroll_team(self.team)
+
+ emit_test_ownership_census()
+
+ assert [call.kwargs["team_id"] for call in mock_delay.call_args_list] == [self.team.id]
diff --git a/products/engineering_analytics/backend/tests/test_enrollment.py b/products/engineering_analytics/backend/tests/test_enrollment.py
new file mode 100644
index 000000000000..f15f10a86d52
--- /dev/null
+++ b/products/engineering_analytics/backend/tests/test_enrollment.py
@@ -0,0 +1,26 @@
+from posthog.test.base import BaseTest
+
+from parameterized import parameterized
+
+from posthog.models.team import Team
+
+from products.engineering_analytics.backend.logic.enrollment import enrolled_sources
+from products.engineering_analytics.backend.tests._github_fixtures import (
+ create_depot_source,
+ create_github_source,
+ enroll_team,
+)
+from products.warehouse_sources.backend.facade.types import ExternalDataSourceType
+
+
+class TestEnrolledSources(BaseTest):
+ @parameterized.expand([("github", ExternalDataSourceType.GITHUB), ("depot", ExternalDataSourceType.DEPOT)])
+ def test_lists_only_enrolled_teams(self, _name: str, source_type: ExternalDataSourceType) -> None:
+ # Every cross-team sweep starts here, so an unenrolled team's rows must never appear.
+ unenrolled = Team.objects.create(organization=self.organization, name="unenrolled")
+ create = create_github_source if source_type == ExternalDataSourceType.GITHUB else create_depot_source
+ enrolled_source = create(self.team)
+ create(unenrolled)
+ enroll_team(self.team)
+
+ assert [source.id for source in enrolled_sources(source_type)] == [enrolled_source.id]
diff --git a/products/engineering_analytics/backend/tests/test_presentation.py b/products/engineering_analytics/backend/tests/test_presentation.py
index a6c2f59cfdf4..e455c7ceabd2 100644
--- a/products/engineering_analytics/backend/tests/test_presentation.py
+++ b/products/engineering_analytics/backend/tests/test_presentation.py
@@ -11,6 +11,7 @@ import requests
from parameterized import parameterized
from rest_framework import status
+from products.data_modeling.backend.facade.models import DataWarehouseManagedViewSet
from products.engineering_analytics.backend.facade import contracts
from products.engineering_analytics.backend.logic.ci_signals_config import (
AUTHORIZED_SOURCES_CONFIG_KEY,
@@ -25,6 +26,7 @@ from products.engineering_analytics.backend.tests._github_fixtures import (
)
from products.signals.backend.models import SignalSourceConfig
from products.warehouse_sources.backend.facade.contracts import GitHubSourceCredential
+from products.warehouse_sources.backend.facade.types import DataWarehouseManagedViewSetKind
class TestScopeEnrollment(SimpleTestCase):
@@ -800,11 +802,27 @@ class TestEngineeringAnalyticsAPI(APIBaseTest):
assert response.status_code in (status.HTTP_401_UNAUTHORIZED, status.HTTP_403_FORBIDDEN)
- def test_requires_rollout_feature_flag(self) -> None:
- # The whole viewset is gated on the engineering-analytics rollout flag (which the
- # conftest fixture enables); with the flag off, every endpoint must 403.
+ @parameterized.expand([("sources", "get"), ("enable", "post")])
+ def test_requires_rollout_feature_flag(self, action: str, method: str) -> None:
+ # The conftest fixture enables the rollout flag; with it off, every endpoint must 403.
with mock.patch("posthoganalytics.feature_enabled", return_value=False):
- response = self.client.get(self._url("sources"))
+ response = getattr(self.client, method)(self._url(action))
assert response.status_code == status.HTTP_403_FORBIDDEN
assert "engineering-analytics" in response.json()["detail"]
+
+ def test_enable_is_idempotent(self) -> None:
+ with mock.patch.object(DataWarehouseManagedViewSet, "sync_views") as mock_sync:
+ first = self.client.post(self._url("enable"))
+ second = self.client.post(self._url("enable"))
+
+ assert first.status_code == status.HTTP_204_NO_CONTENT
+ assert second.status_code == status.HTTP_204_NO_CONTENT
+ # The views sync once, on creation; the post-load hook keeps them current afterwards.
+ mock_sync.assert_called_once()
+ assert (
+ DataWarehouseManagedViewSet.objects.filter(
+ team=self.team, kind=DataWarehouseManagedViewSetKind.ENGINEERING_ANALYTICS
+ ).count()
+ == 1
+ )
diff --git a/products/engineering_analytics/backend/tests/test_warehouse_view_sync.py b/products/engineering_analytics/backend/tests/test_warehouse_view_sync.py
index 47d5c94be148..2da4961f3c40 100644
--- a/products/engineering_analytics/backend/tests/test_warehouse_view_sync.py
+++ b/products/engineering_analytics/backend/tests/test_warehouse_view_sync.py
@@ -11,7 +11,7 @@ from products.engineering_analytics.backend.logic.sources import (
WORKFLOW_JOBS_SCHEMA,
WORKFLOW_RUNS_SCHEMA,
)
-from products.engineering_analytics.backend.tests._github_fixtures import create_depot_source
+from products.engineering_analytics.backend.tests._github_fixtures import create_depot_source, enroll_team
from products.engineering_analytics.backend.warehouse_view_sync import sync_engineering_analytics_views
from products.warehouse_sources.backend.facade.models import DataWarehouseTable, ExternalDataSchema, ExternalDataSource
from products.warehouse_sources.backend.facade.types import DataWarehouseManagedViewSetKind, ExternalDataSourceType
@@ -79,11 +79,10 @@ class TestSyncEngineeringAnalyticsViews(BaseTest):
assert not self._has_viewset()
@patch.object(DataWarehouseManagedViewSet, "sync_views")
- def test_noop_when_jobs_endpoint_not_synced(self, mock_sync) -> None:
- # A GitHub source with only workflow_runs synced has no view to expose yet — don't even
- # create the viewset row until both endpoints exist.
- source = self._github_source()
- schema = self._schema(source, WORKFLOW_RUNS_SCHEMA, self._table(f"{PREFIX}github_workflow_runs", source))
+ def test_noop_for_unenrolled_team(self, mock_sync) -> None:
+ # A team that connected GitHub for another product must not get this product's views.
+ source = self._qualifying_source()
+ schema = ExternalDataSchema.objects.get(source=source, name=WORKFLOW_JOBS_SCHEMA)
sync_engineering_analytics_views(schema, source)
@@ -92,9 +91,10 @@ class TestSyncEngineeringAnalyticsViews(BaseTest):
@parameterized.expand([("github_jobs", False), ("depot_attempts", True)])
@patch.object(DataWarehouseManagedViewSet, "sync_views")
- def test_creates_viewset_and_syncs_for_qualifying_source(
+ def test_syncs_enrolled_team_after_qualifying_load(
self, _name: str, from_depot: bool, mock_sync: MagicMock
) -> None:
+ enroll_team(self.team)
source = self._qualifying_source()
schema = ExternalDataSchema.objects.get(source=source, name=WORKFLOW_JOBS_SCHEMA)
if from_depot:
@@ -112,6 +112,7 @@ class TestSyncEngineeringAnalyticsViews(BaseTest):
@patch.object(DataWarehouseManagedViewSet, "sync_views")
def test_transient_db_error_is_not_captured(self, _name, error_cls, mock_sync, mock_capture) -> None:
mock_sync.side_effect = error_cls("server closed the connection")
+ enroll_team(self.team)
source = self._qualifying_source()
schema = ExternalDataSchema.objects.get(source=source, name=WORKFLOW_JOBS_SCHEMA)
@@ -124,6 +125,7 @@ class TestSyncEngineeringAnalyticsViews(BaseTest):
def test_unexpected_error_is_captured(self, mock_sync, mock_capture) -> None:
error = ValueError("something actually broke")
mock_sync.side_effect = error
+ enroll_team(self.team)
source = self._qualifying_source()
schema = ExternalDataSchema.objects.get(source=source, name=WORKFLOW_JOBS_SCHEMA)
diff --git a/products/engineering_analytics/backend/warehouse_view_sync.py b/products/engineering_analytics/backend/warehouse_view_sync.py
index 812148c6de32..a6c456c6aecf 100644
--- a/products/engineering_analytics/backend/warehouse_view_sync.py
+++ b/products/engineering_analytics/backend/warehouse_view_sync.py
@@ -15,14 +15,13 @@ import structlog
from posthog.exceptions_capture import capture_exception
-from products.data_modeling.backend.facade.models import DataWarehouseManagedViewSet
-from products.engineering_analytics.backend.facade.warehouse_views import get_expected_warehouse_views
+from products.engineering_analytics.backend.logic.enrollment import get_enrollment
from products.engineering_analytics.backend.logic.sources import (
DEPOT_JOB_ATTEMPTS_SCHEMA,
WORKFLOW_JOBS_SCHEMA,
WORKFLOW_RUNS_SCHEMA,
)
-from products.warehouse_sources.backend.facade.types import DataWarehouseManagedViewSetKind, ExternalDataSourceType
+from products.warehouse_sources.backend.facade.types import ExternalDataSourceType
if TYPE_CHECKING:
from products.warehouse_sources.backend.facade.models import ExternalDataSchema, ExternalDataSource
@@ -40,30 +39,23 @@ _RELEVANT_SCHEMAS: dict[str, tuple[str, ...]] = {
def sync_engineering_analytics_views(schema: ExternalDataSchema, source: ExternalDataSource) -> None:
"""Called after validate_schema_and_update_table links a DataWarehouseTable to the schema.
- Returns early unless a GitHub runs/jobs endpoint just loaded. In the steady state the team's
- viewset already exists, so it re-syncs directly (``sync_views`` recomputes the expected views).
- The expected-views guard runs only on first provisioning: a team whose jobs source isn't synced
- yet has no view to expose, so the viewset row isn't created until both endpoints exist.
+ Returns early unless a GitHub runs/jobs or Depot attempts endpoint just loaded, and only
+ re-syncs a team that opted in (``sync_views`` recomputes the expected views). The opt-in row
+ is never created here: a team connects a GitHub source for many reasons, and this product's
+ views must not appear in its warehouse until a user enables the product.
"""
try:
if schema.name not in _RELEVANT_SCHEMAS.get(source.source_type, ()):
return
- managed_viewset = DataWarehouseManagedViewSet.objects.filter(
- team=schema.team,
- kind=DataWarehouseManagedViewSetKind.ENGINEERING_ANALYTICS,
- ).first()
-
- # Only on first provisioning: computing the expected views is the expensive part, so once the
- # viewset exists we skip this guard and re-sync directly (one compute inside sync_views). A
- # team whose jobs source isn't synced yet has no view to expose — don't create the viewset row.
+ managed_viewset = get_enrollment(schema.team)
if managed_viewset is None:
- if not get_expected_warehouse_views(schema.team):
- return
- managed_viewset, _ = DataWarehouseManagedViewSet.objects.get_or_create(
- team=schema.team,
- kind=DataWarehouseManagedViewSetKind.ENGINEERING_ANALYTICS,
+ logger.info(
+ "sync_engineering_analytics_views_skipped_not_enrolled",
+ team_id=schema.team_id,
+ source_id=str(source.id),
)
+ return
logger.info(
"sync_engineering_analytics_views_starting",
diff --git a/products/engineering_analytics/frontend/emptyState/engineeringAnalyticsSetupLogic.test.ts b/products/engineering_analytics/frontend/emptyState/engineeringAnalyticsSetupLogic.test.ts
index 3eedd0d4f598..2ab70cdb7022 100644
--- a/products/engineering_analytics/frontend/emptyState/engineeringAnalyticsSetupLogic.test.ts
+++ b/products/engineering_analytics/frontend/emptyState/engineeringAnalyticsSetupLogic.test.ts
@@ -10,25 +10,39 @@ import { engineeringAnalyticsSetupLogic } from './engineeringAnalyticsSetupLogic
describe('engineeringAnalyticsSetupLogic', () => {
let sourcesSpy: jest.SpyInstance
+ let enableSpy: jest.SpyInstance
beforeEach(() => {
sourcesSpy = jest.spyOn(generatedApi, 'engineeringAnalyticsSources')
+ enableSpy = jest.spyOn(generatedApi, 'engineeringAnalyticsEnable').mockResolvedValue()
initKeaTests()
})
afterEach(() => {
sourcesSpy.mockRestore()
+ enableSpy.mockRestore()
})
// An unsynced source still counts: the scene explains the sync, the empty state does not.
+ // A connected source is also the opt-in, so only that path enables the project.
it.each([
- [[], 'needs-setup'],
- [[{ id: 's1', repo: 'acme/web', prefix: '', synced: false }], 'has-data'],
- ])('pushes sources %j as status %s', async (sources, expected) => {
+ [[], 'needs-setup', 0],
+ [[{ id: 's1', repo: 'acme/web', prefix: '', synced: false }], 'has-data', 1],
+ ])('pushes sources %j as status %s and enables the project %i times', async (sources, expected, enableCalls) => {
sourcesSpy.mockResolvedValue(sources)
const logic = engineeringAnalyticsSetupLogic()
logic.mount()
await expectLogic(logic).toFinishAllListeners()
expect(productSetupStatusLogic({ productKey: ProductKey.ENGINEERING_ANALYTICS }).values.status).toBe(expected)
+ expect(enableSpy).toHaveBeenCalledTimes(enableCalls)
+ })
+
+ it('still shows the product when the viewer cannot enable it', async () => {
+ sourcesSpy.mockResolvedValue([{ id: 's1', repo: 'acme/web', prefix: '', synced: true }])
+ enableSpy.mockRejectedValue(new Error('403'))
+ const logic = engineeringAnalyticsSetupLogic()
+ logic.mount()
+ await expectLogic(logic).toFinishAllListeners()
+ expect(productSetupStatusLogic({ productKey: ProductKey.ENGINEERING_ANALYTICS }).values.status).toBe('has-data')
})
})
diff --git a/products/engineering_analytics/frontend/emptyState/engineeringAnalyticsSetupLogic.ts b/products/engineering_analytics/frontend/emptyState/engineeringAnalyticsSetupLogic.ts
index c046288866b9..032c2dc6a799 100644
--- a/products/engineering_analytics/frontend/emptyState/engineeringAnalyticsSetupLogic.ts
+++ b/products/engineering_analytics/frontend/emptyState/engineeringAnalyticsSetupLogic.ts
@@ -3,12 +3,16 @@ import { projectLogic } from 'scenes/projectLogic'
import { ProductKey } from '~/queries/schema/schema-general'
-import { engineeringAnalyticsSources } from '../generated/api'
+import { engineeringAnalyticsEnable, engineeringAnalyticsSources } from '../generated/api'
/**
* Setup detection for the engineering analytics empty state. The product reads pull
* requests and workflow runs from a GitHub warehouse source, so a project is set up once
* it has one connected, synced or not: the scene explains an unsynced source itself.
+ *
+ * Opening the product with a source connected is the opt-in that lets background work run for
+ * the project. Enabling needs editor access, so a viewer's failure is ignored and the next editor
+ * enables it.
*/
export const engineeringAnalyticsSetupLogic = createSetupDetectionLogic({
productKey: ProductKey.ENGINEERING_ANALYTICS,
@@ -18,6 +22,10 @@ export const engineeringAnalyticsSetupLogic = createSetupDetectionLogic({
detect: async () => {
const projectId = String(projectLogic.findMounted()?.values.currentProjectId)
const sources = await engineeringAnalyticsSources(projectId)
- return sources.length > 0 ? 'has-data' : 'needs-setup'
+ if (sources.length === 0) {
+ return 'needs-setup'
+ }
+ void engineeringAnalyticsEnable(projectId).catch(() => undefined)
+ return 'has-data'
},
})
diff --git a/products/engineering_analytics/frontend/generated/api.ts b/products/engineering_analytics/frontend/generated/api.ts
index fb52d3a08767..98340e0d9254 100644
--- a/products/engineering_analytics/frontend/generated/api.ts
+++ b/products/engineering_analytics/frontend/generated/api.ts
@@ -630,6 +630,20 @@ export const engineeringAnalyticsDora = async (
})
}
+export const getEngineeringAnalyticsEnableUrl = (projectId: string) => {
+ return `/api/projects/${projectId}/engineering_analytics/enable/`
+}
+
+/**
+ * Enable engineering analytics for this project. Creates the project's managed warehouse views and opts it into the background work (view sync after each GitHub or Depot load, the daily test census, CI failure log collection). Safe to call again: an enabled project stays enabled.
+ */
+export const engineeringAnalyticsEnable = async (projectId: string, options?: RequestInit): Promise<void> => {
+ return apiMutator<void>(getEngineeringAnalyticsEnableUrl(projectId), {
+ ...options,
+ method: 'POST',
+ })
+}
+
export const getEngineeringAnalyticsFlakyTestsUrl = (
projectId: string,
params?: EngineeringAnalyticsFlakyTestsParams
diff --git a/products/engineering_analytics/mcp/tools.yaml b/products/engineering_analytics/mcp/tools.yaml
index 555dbd7e9f51..a3e3843c8528 100644
--- a/products/engineering_analytics/mcp/tools.yaml
+++ b/products/engineering_analytics/mcp/tools.yaml
@@ -94,6 +94,10 @@ tools:
the PR-scoped lead-time figures to that team's authors: deploy counts stay repo-wide. deploy_data_available
is false when the deployments endpoints aren't synced on the GitHub source; the fix is enabling them, not
re-querying.
+ engineering-analytics-enable:
+ operation: engineering_analytics_enable
+ enabled: false
+ disabled_reason: The in-app setup state enables the product; an agent should not opt a project into background work.
engineering-analytics-flaky-tests:
operation: engineering_analytics_flaky_tests
enabled: true
diff --git a/products/model_crossing_uses_baseline.txt b/products/model_crossing_uses_baseline.txt
index 6bb55fa7882e..2b81c7430eb1 100644
--- a/products/model_crossing_uses_baseline.txt
+++ b/products/model_crossing_uses_baseline.txt
@@ -895,10 +895,9 @@ warehouse_sources.ExternalDataSource products.data_warehouse.backend.management.
warehouse_sources.ExternalDataSource products.data_warehouse.backend.models.revenue_analytics_config other(keyword) 1
warehouse_sources.ExternalDataSource products.data_warehouse.backend.models.revenue_analytics_config passed-as-class(OneToOneField) 1
warehouse_sources.ExternalDataSource products.data_warehouse.backend.presentation.views.data_warehouse instance-many(filter) 1
-warehouse_sources.ExternalDataSource products.engineering_analytics.backend.logic.job_logs.coordinator instance-many(select_related) 1
+warehouse_sources.ExternalDataSource products.engineering_analytics.backend.logic.enrollment instance-many(select_related) 1
warehouse_sources.ExternalDataSource products.engineering_analytics.backend.logic.sources instance-many(order_by) 1
warehouse_sources.ExternalDataSource products.engineering_analytics.backend.management.commands.seed_engineering_analytics instance-single 1
-warehouse_sources.ExternalDataSource products.engineering_analytics.backend.tasks.tasks instance-many(distinct) 1
warehouse_sources.ExternalDataSource products.managed_warehouse.backend.logic.connection instance-many(exclude) 1
warehouse_sources.ExternalDataSource products.managed_warehouse.backend.logic.connection instance-many(filter) 1
warehouse_sources.ExternalDataSource products.marketing_analytics.backend.demo.health write(create) 1