:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 4 missing tables to the Dbt source (#119829)
Tom Owersf9ac7407033a
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 838b9fc7ceaa..f03420aa3c0d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -3589,10 +3589,10 @@ Diffed against: <https://raw.githubusercontent.com/dbt-labs/dbt-cloud-openapi-sp- [x] `GET /api/v2/accounts/{account_id}/runs/?include_related=["run_steps"] (and /api/v2/accounts/{account_id}/steps/{id}/)` — per-step timings, commands and status inside the runs we already sync — where run duration actually goes (high)- [x] `GET /api/v2/accounts/{account_id}/runs/{run_id}/artifacts/ and /artifacts/{remainder}` — run_results.json and manifest.json per run, giving model-level execution results and node metadata (high)- [x] `GET /api/v3/accounts/{account_id}/audit-logs/` — who changed jobs, environments and permissions, with timestamps (medium)-- [ ] `GET /api/v2/accounts/{account_id}/repositories/ and /api/v3/accounts/{account_id}/projects/{project_id}/repositories/` — lookup resolving the repository and branch behind each project we already sync (medium)-- [ ] `GET /api/v3/accounts/{account_id}/connections/ and /api/v3/accounts/{account_id}/projects/{project_id}/connections/` — lookup resolving the warehouse connection each environment points at (medium)-- [ ] `GET /api/v3/accounts/{account_id}/groups/ and /api/v3/accounts/{account_id}/group-permissions/{group_id}/` — group membership and permission assignment for the users we already sync (medium)-- [ ] `GET /api/v2/accounts/{account_id}/licenses/ and /api/v3/accounts/{account_id}/license-maps/` — seat type per user for license utilization reporting (low)+- [x] `GET /api/v2/accounts/{account_id}/repositories/ and /api/v3/accounts/{account_id}/projects/{project_id}/repositories/` — lookup resolving the repository and branch behind each project we already sync (medium) — added as `repositories` from the v3 per-project route (fan-out over `projects`); the account-level v2 route is deprecated+- [x] `GET /api/v3/accounts/{account_id}/connections/ and /api/v3/accounts/{account_id}/projects/{project_id}/connections/` — lookup resolving the warehouse connection each environment points at (medium) — added as `connections` from the account-level route, which already covers every project's connections+- [x] `GET /api/v3/accounts/{account_id}/groups/ and /api/v3/accounts/{account_id}/group-permissions/{group_id}/` — group membership and permission assignment for the users we already sync (medium) — added as `groups`, which carries each group's `group_permissions` inline; `group-permissions/{group_id}/` is POST-only (assigns permissions), so it is not a table+- [x] `GET /api/v2/accounts/{account_id}/licenses/ and /api/v3/accounts/{account_id}/license-maps/` — seat type per user for license utilization reporting (low) — added as `license_allocations`, one row per license type with seat totals and usage; per-user seat types are already in the `licenses` column of `users`. `license-maps` skipped: it is SSO-group-to-license configurationNote: Diffed against the vendor's own OpenAPI specs (openapi-v2.yaml, openapi-v3.yaml in dbt-labs/dbt-cloud-openapi-spec — the files the docs site renders through Stoplight). The Admin API is well covered; the real gap is the separate Discovery API, whose object list (Models, Tests, Sources, Snapshots, Seeds, Exposures, Tags, Packages, Owners, Model historical runs, Lineage, Job) I read from the schema navigation on https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-schema-environment-applied-models. Excluded as config: environment variables, notifications, service tokens, IP restrictions, SCIM, OAuth configurations, webhooks, credentials, extended attributes.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/canonical_descriptions.pyindex 531d2661d1a2..2e87aa1633fa 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/canonical_descriptions.py@@ -145,6 +145,70 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"service": "The dbt service that emitted the event.",},},+ "repositories": {+ "description": "The git repository connected to a dbt project.",+ "docs_url": "https://docs.getdbt.com/dbt-cloud/api-v3",+ "columns": {+ "id": "Unique identifier for the repository.",+ "account_id": "The account the repository belongs to.",+ "project_id": "The project the repository is connected to.",+ "full_name": "Full name of the repository on the git provider, such as org/repo.",+ "remote_backend": "The git provider: github, gitlab, ado (Azure DevOps), managed or manual_config.",+ "git_clone_strategy": "How dbt clones the repository, such as deploy_key, github_app or git_token.",+ "web_url": "Browser URL of the repository on the git provider.",+ "state": "1 means the repository is active, 2 means it is deleted.",+ "created_at": "When the repository was connected.",+ "updated_at": "When the repository was last updated.",+ },+ },+ "connections": {+ "description": "A data platform connection in the account, which environments use to reach the warehouse.",+ "docs_url": "https://docs.getdbt.com/dbt-cloud/api-v3",+ "columns": {+ "id": "Unique identifier for the connection.",+ "account_id": "The account the connection belongs to.",+ "name": "Name of the connection.",+ "adapter_version": "The warehouse and the version of its adapter, such as snowflake_v0.",+ "is_ssh_tunnel_enabled": "Whether the connection goes through an SSH tunnel.",+ "private_link_endpoint_id": "The private link endpoint the connection uses, if any.",+ "oauth_configuration_id": "The OAuth configuration the connection uses, if any.",+ "environment__count": "Number of environments that use the connection.",+ "config": "Always null in list responses.",+ "created_at": "When the connection was created.",+ "updated_at": "When the connection was last updated.",+ },+ },+ "groups": {+ "description": "A permission group in the account, granting permission sets on projects to its members.",+ "docs_url": "https://docs.getdbt.com/dbt-cloud/api-v3",+ "columns": {+ "id": "Unique identifier for the group.",+ "account_id": "The account the group belongs to.",+ "name": "Name of the group.",+ "state": "1 means the group is active, 2 means it is deleted.",+ "assign_by_default": "Whether new users join the group automatically.",+ "sso_mapping_groups": "SSO group names that map users into this group.",+ "group_permissions": "Permission sets the group grants, each scoped to one project or to all projects.",+ "member_count": "Number of users in the group.",+ "scim_managed": "Whether SCIM provisioning manages the group.",+ "created_at": "When the group was created.",+ "updated_at": "When the group was last updated.",+ },+ },+ "license_allocations": {+ "description": "Seat totals and usage for one license type in the account.",+ "docs_url": "https://docs.getdbt.com/dbt-cloud/api-v2",+ "columns": {+ "license_type": "The license type, such as developer, read_only, analyst or it.",+ "total": "Number of seats of this type the account has.",+ "in_use": "Number of seats of this type in use.",+ "assigned": "Number of seats of this type assigned to users.",+ "available": "Number of seats of this type still available.",+ "limited": "Whether the account has a limit on seats of this type.",+ "over_limit": "Whether the account uses more seats of this type than its limit.",+ "enterprise_plan": "Whether the account is on an Enterprise plan.",+ },+ },"models": {"description": "Current state of a model in a deployment environment, as reported by the Discovery API.","docs_url": "https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-schema-environment-applied-models",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/dbt.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/dbt.pyindex 7d941c82b3dc..dab1e6bdd66d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/dbt.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/dbt.py@@ -320,9 +320,10 @@ def get_endpoint_permissions(if config.discovery is not None:results[endpoint] = discovery_errorcontinue- # A run fan-out reads the runs list plus a per-run route; the list is the permission that- # decides whether the table can sync at all.- probe_path = config.path if config.path is not None else DBT_ENDPOINTS["runs"].path+ # A fan-out reads its parent list plus a per-parent route; the parent list is the permission+ # that decides whether the table can sync at all.+ parent = "projects" if config.project_fanout is not None else "runs"+ probe_path = config.path if config.path is not None else DBT_ENDPOINTS[parent].pathassert probe_path is not Noneurl = f"{base_url}/api{probe_path.format(account_id=account_id)}?limit=1"try:@@ -417,6 +418,8 @@ def _iter_offset_pages(data = _fetch_page(session, url, headers, logger)rows = data.get("data") or []+ if isinstance(rows, dict):+ rows = [rows]if not rows:return@@ -526,6 +529,65 @@ def _iter_run_fanout_rows(resumable_source_manager.save_state(DbtResumeConfig(parent_offset=page.next_offset))+def _keyed_rows(rows: list[dict[str, Any]], key_field: str) -> list[dict[str, Any]]:+ """Turn objects keyed by a type name ({"developer": {...}, ...}) into one row per key."""+ return [{key_field: key, **value} for row in rows for key, value in row.items() if isinstance(value, dict)]+++def _project_ids(+ session: requests.Session,+ base_url: str,+ account_id: str,+ headers: dict[str, str],+ logger: FilteringBoundLogger,+) -> list[int]:+ path = DBT_ENDPOINTS["projects"].path+ assert path is not None++ project_ids: list[int] = []+ for page in _iter_offset_pages(+ session, base_url, path.format(account_id=account_id), {"limit": DBT_PAGE_LIMIT}, headers, logger+ ):+ project_ids.extend(row["id"] for row in page.rows if row.get("id") is not None)+ return project_ids+++def _iter_project_fanout_rows(+ session: requests.Session,+ base_url: str,+ account_id: str,+ config: DbtEndpointConfig,+ headers: dict[str, str],+ logger: FilteringBoundLogger,+ resumable_source_manager: ResumableSourceManager[DbtResumeConfig],+ resume: DbtResumeConfig | None,+) -> Iterator[Any]:+ fanout = config.project_fanout+ assert fanout is not None++ project_ids = _project_ids(session, base_url, account_id, headers, logger)+ start_index = resume.parent_offset if resume is not None else 0++ for index in range(start_index, len(project_ids)):+ start_offset = resume.offset if resume is not None and index == start_index else 0+ for page in _iter_offset_pages(+ session,+ base_url,+ fanout.path.format(account_id=account_id, project_id=project_ids[index]),+ {"limit": DBT_PAGE_LIMIT},+ headers,+ logger,+ start_offset=start_offset,+ ):+ if page.has_more:+ resumable_source_manager.save_state(DbtResumeConfig(offset=page.next_offset, parent_offset=index))+ elif index + 1 < len(project_ids):+ resumable_source_manager.save_state(DbtResumeConfig(parent_offset=index + 1))+ yield page.rows+ # Most projects have one repository at most, so many walks return no rows to yield.+ resumable_source_manager.safe_point()++def _deployment_environment_ids(session: requests.Session,base_url: str,@@ -756,6 +818,19 @@ def get_rows()return+ if config.project_fanout is not None:+ yield from _iter_project_fanout_rows(+ session,+ base_url,+ account_id,+ config,+ headers,+ logger,+ resumable_source_manager,+ resume,+ )+ return+if config.run_fanout is not None:yield from _iter_run_fanout_rows(session,@@ -792,6 +867,8 @@ def get_rows(yield keptif len(kept) < len(page.rows):break+ elif config.keyed_row_field is not None:+ yield _keyed_rows(page.rows, config.keyed_row_field)else:yield page.rowsdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/settings.pyindex 5d7093bdab26..021b518edd50 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/settings.py@@ -79,6 +79,14 @@ class DbtRunFanoutConfig:string_row_field: Optional[str] = None+@frozen+class DbtProjectFanoutConfig:+ """A list resource scoped to a project, walked once per project in the account."""++ # Path template under {base_url}/api, with {account_id} and {project_id} placeholders.+ path: str++@frozenclass DbtDiscoveryConfig:"""An endpoint served by the Discovery API (GraphQL) instead of the Admin API."""@@ -95,8 +103,8 @@ class DbtDiscoveryConfig:class DbtEndpointConfig:name: str# Path template under {base_url}/api, with an {account_id} placeholder. The API version is part- # of the path because coverage is split: jobs/runs live on v2, the rest on v3. None for- # endpoints that are not a top-level Admin API list (run fan-outs and Discovery API tables).+ # of the path because coverage is split: jobs/runs/licenses live on v2, the rest on v3. None+ # for endpoints that are not a top-level Admin API list (fan-outs and Discovery API tables).path: Optional[str] = Noneincremental_fields: list[IncrementalField] = field(default_factory=list)default_incremental_field: Optional[str] = None@@ -112,6 +120,11 @@ class DbtEndpointConfig:incremental_lookback: Optional[timedelta] = None# Set when rows come from a child resource fetched once per run instead of a list endpoint.run_fanout: Optional[DbtRunFanoutConfig] = None+ # Set when rows come from a list resource fetched once per project instead of an account list.+ project_fanout: Optional[DbtProjectFanoutConfig] = None+ # Set when the response holds one object keyed by a type name rather than one row per item;+ # names the column each key lands in, and the key's object becomes that row.+ keyed_row_field: Optional[str] = None# Set when rows come from the Discovery API instead of the Admin API.discovery: Optional[DbtDiscoveryConfig] = Nonedescription: Optional[str] = None@@ -209,6 +222,32 @@ DBT_ENDPOINTS: dict[str, DbtEndpointConfig] = {should_sync_default=False,description="Account audit events: who changed jobs, environments and permissions, and when. Requires an Enterprise plan",),+ "repositories": DbtEndpointConfig(+ name="repositories",+ # The account-level v2 list is deprecated; v3 only lists repositories per project.+ project_fanout=DbtProjectFanoutConfig(path="/v3/accounts/{account_id}/projects/{project_id}/repositories/"),+ description="The git repository behind each project, with its provider and clone strategy. Costs one extra request per project",+ ),+ "connections": DbtEndpointConfig(+ name="connections",+ path="/v3/accounts/{account_id}/connections/",+ description="Data platform connections in the account, with the adapter each one uses",+ ),+ "groups": DbtEndpointConfig(+ name="groups",+ path="/v3/accounts/{account_id}/groups/",+ # Like users, listing groups needs account-level permissions many read-only tokens lack.+ should_sync_default=False,+ description="Permission groups in the account, with their member count, SSO mappings and the permission sets they grant",+ ),+ "license_allocations": DbtEndpointConfig(+ name="license_allocations",+ path="/v2/accounts/{account_id}/licenses/",+ keyed_row_field="license_type",+ primary_keys=["license_type"],+ should_sync_default=False,+ description="Seat totals and usage for each license type in the account. Requires a token that can read the account's licenses",+ ),"models": DbtEndpointConfig(name="models",discovery=DbtDiscoveryConfig(query=MODELS_QUERY, applied_field="models"),diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/tests/test_dbt.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/tests/test_dbt.pyindex e16efa4a8319..41c3e7936454 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/tests/test_dbt.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/tests/test_dbt.py@@ -166,10 +166,11 @@ class TestGetEndpointPermissions:region="us",custom_base_url=None,team_id=1,- endpoints=["projects"],+ endpoints=["projects", "repositories"],)- assert result["projects"] is None+ assert result == {"projects": None, "repositories": None}+ # repositories has no account-level path; its probe is the projects list it fans out over.assert {request.path for request in network.requests_log} == {"/api/v3/accounts/12345/projects/"}def test_discovery_endpoints_report_one_shared_probe_result(self):@@ -261,6 +262,24 @@ class TestGetRows:# newest-first walk: incremental syncs don't re-crawl history.assert result.params("offset") == ["0", "100"]+ @pytest.mark.parametrize(+ "data",+ [+ [{"developer": {"total": 5, "in_use": 3}, "read_only": {"total": 10, "in_use": 0}, "note": None}],+ {"developer": {"total": 5, "in_use": 3}, "read_only": {"total": 10, "in_use": 0}, "note": None},+ ],+ )+ def test_license_allocations_become_one_row_per_license_type(self, data):+ # The spec types `data` as a list, but the allocation is one object per account, so accept both.+ result = _driver().run("license_allocations", [ScriptedResponse(json={"data": data})])++ assert result.raised is None+ assert result.rows == [+ {"license_type": "developer", "total": 5, "in_use": 3},+ {"license_type": "read_only", "total": 10, "in_use": 0},+ ]+ assert result.paths == ["/api/v2/accounts/12345/licenses/"]+def test_custom_base_url_is_used(self):result = _driver(custom_base_url="https://ab123.us1.dbt.com/").run("projects", [_page([])])@@ -444,3 +463,52 @@ class TestDiscoverySync:"model.a.b","model.a.c",]+++class TestProjectFanout:+ def test_walks_every_project_and_pages_within_each(self):+ result = _driver().run(+ "repositories",+ route(+ {+ "/accounts/12345/projects/": [_page([{"id": 1}, {"id": 2}, {"id": 3}])],+ "/projects/1/repositories/": [+ _page([{"id": 10, "project_id": 1}], count=100, total_count=101),+ _page([{"id": 11, "project_id": 1}], count=1, total_count=101),+ ],+ "/projects/2/repositories/": [_page([])],+ "/projects/3/repositories/": [_page([{"id": 30, "project_id": 3}])],+ }+ ),+ )++ assert result.raised is None+ assert [row["id"] for row in result.rows] == [10, 11, 30]+ assert result.paths == [+ "/api/v3/accounts/12345/projects/",+ "/api/v3/accounts/12345/projects/1/repositories/",+ "/api/v3/accounts/12345/projects/1/repositories/",+ "/api/v3/accounts/12345/projects/2/repositories/",+ "/api/v3/accounts/12345/projects/3/repositories/",+ ]+ assert result.saved_states == [+ DbtResumeConfig(offset=100, parent_offset=0),+ DbtResumeConfig(parent_offset=1),+ ]++ def test_resumes_inside_the_saved_project(self):+ result = _driver().run(+ "repositories",+ route(+ {+ "/accounts/12345/projects/": [_page([{"id": 1}, {"id": 2}])],+ "/projects/2/repositories/": [_page([{"id": 21, "project_id": 2}], count=1, total_count=101)],+ }+ ),+ resume_state=DbtResumeConfig(offset=100, parent_offset=1),+ )++ assert result.raised is None+ assert result.rows == [{"id": 21, "project_id": 2}]+ assert result.paths == ["/api/v3/accounts/12345/projects/", "/api/v3/accounts/12345/projects/2/repositories/"]+ assert result.requests[1].param("offset") == "100"