Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 4 missing tables to the Dbt source (#119829)

Tom Owersf9ac7407033a
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 838b9fc7ceaa..f03420aa3c0d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -3589,10 +3589,10 @@ Diffed against: <https://raw.githubusercontent.com/dbt-labs/dbt-cloud-openapi-sp
- [x] `GET /api/v2/accounts/{account_id}/runs/?include_related=["run_steps"] (and /api/v2/accounts/{account_id}/steps/{id}/)` — per-step timings, commands and status inside the runs we already sync — where run duration actually goes (high)
- [x] `GET /api/v2/accounts/{account_id}/runs/{run_id}/artifacts/ and /artifacts/{remainder}` — run_results.json and manifest.json per run, giving model-level execution results and node metadata (high)
- [x] `GET /api/v3/accounts/{account_id}/audit-logs/` — who changed jobs, environments and permissions, with timestamps (medium)
-- [ ] `GET /api/v2/accounts/{account_id}/repositories/ and /api/v3/accounts/{account_id}/projects/{project_id}/repositories/` — lookup resolving the repository and branch behind each project we already sync (medium)
-- [ ] `GET /api/v3/accounts/{account_id}/connections/ and /api/v3/accounts/{account_id}/projects/{project_id}/connections/` — lookup resolving the warehouse connection each environment points at (medium)
-- [ ] `GET /api/v3/accounts/{account_id}/groups/ and /api/v3/accounts/{account_id}/group-permissions/{group_id}/` — group membership and permission assignment for the users we already sync (medium)
-- [ ] `GET /api/v2/accounts/{account_id}/licenses/ and /api/v3/accounts/{account_id}/license-maps/` — seat type per user for license utilization reporting (low)
+- [x] `GET /api/v2/accounts/{account_id}/repositories/ and /api/v3/accounts/{account_id}/projects/{project_id}/repositories/` — lookup resolving the repository and branch behind each project we already sync (medium) — added as `repositories` from the v3 per-project route (fan-out over `projects`); the account-level v2 route is deprecated
+- [x] `GET /api/v3/accounts/{account_id}/connections/ and /api/v3/accounts/{account_id}/projects/{project_id}/connections/` — lookup resolving the warehouse connection each environment points at (medium) — added as `connections` from the account-level route, which already covers every project's connections
+- [x] `GET /api/v3/accounts/{account_id}/groups/ and /api/v3/accounts/{account_id}/group-permissions/{group_id}/` — group membership and permission assignment for the users we already sync (medium) — added as `groups`, which carries each group's `group_permissions` inline; `group-permissions/{group_id}/` is POST-only (assigns permissions), so it is not a table
+- [x] `GET /api/v2/accounts/{account_id}/licenses/ and /api/v3/accounts/{account_id}/license-maps/` — seat type per user for license utilization reporting (low) — added as `license_allocations`, one row per license type with seat totals and usage; per-user seat types are already in the `licenses` column of `users`. `license-maps` skipped: it is SSO-group-to-license configuration
Note: Diffed against the vendor's own OpenAPI specs (openapi-v2.yaml, openapi-v3.yaml in dbt-labs/dbt-cloud-openapi-spec — the files the docs site renders through Stoplight). The Admin API is well covered; the real gap is the separate Discovery API, whose object list (Models, Tests, Sources, Snapshots, Seeds, Exposures, Tags, Packages, Owners, Model historical runs, Lineage, Job) I read from the schema navigation on https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-schema-environment-applied-models. Excluded as config: environment variables, notifications, service tokens, IP restrictions, SCIM, OAuth configurations, webhooks, credentials, extended attributes.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/canonical_descriptions.py
index 531d2661d1a2..2e87aa1633fa 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/canonical_descriptions.py
@@ -145,6 +145,70 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"service": "The dbt service that emitted the event.",
},
},
+ "repositories": {
+ "description": "The git repository connected to a dbt project.",
+ "docs_url": "https://docs.getdbt.com/dbt-cloud/api-v3",
+ "columns": {
+ "id": "Unique identifier for the repository.",
+ "account_id": "The account the repository belongs to.",
+ "project_id": "The project the repository is connected to.",
+ "full_name": "Full name of the repository on the git provider, such as org/repo.",
+ "remote_backend": "The git provider: github, gitlab, ado (Azure DevOps), managed or manual_config.",
+ "git_clone_strategy": "How dbt clones the repository, such as deploy_key, github_app or git_token.",
+ "web_url": "Browser URL of the repository on the git provider.",
+ "state": "1 means the repository is active, 2 means it is deleted.",
+ "created_at": "When the repository was connected.",
+ "updated_at": "When the repository was last updated.",
+ },
+ },
+ "connections": {
+ "description": "A data platform connection in the account, which environments use to reach the warehouse.",
+ "docs_url": "https://docs.getdbt.com/dbt-cloud/api-v3",
+ "columns": {
+ "id": "Unique identifier for the connection.",
+ "account_id": "The account the connection belongs to.",
+ "name": "Name of the connection.",
+ "adapter_version": "The warehouse and the version of its adapter, such as snowflake_v0.",
+ "is_ssh_tunnel_enabled": "Whether the connection goes through an SSH tunnel.",
+ "private_link_endpoint_id": "The private link endpoint the connection uses, if any.",
+ "oauth_configuration_id": "The OAuth configuration the connection uses, if any.",
+ "environment__count": "Number of environments that use the connection.",
+ "config": "Always null in list responses.",
+ "created_at": "When the connection was created.",
+ "updated_at": "When the connection was last updated.",
+ },
+ },
+ "groups": {
+ "description": "A permission group in the account, granting permission sets on projects to its members.",
+ "docs_url": "https://docs.getdbt.com/dbt-cloud/api-v3",
+ "columns": {
+ "id": "Unique identifier for the group.",
+ "account_id": "The account the group belongs to.",
+ "name": "Name of the group.",
+ "state": "1 means the group is active, 2 means it is deleted.",
+ "assign_by_default": "Whether new users join the group automatically.",
+ "sso_mapping_groups": "SSO group names that map users into this group.",
+ "group_permissions": "Permission sets the group grants, each scoped to one project or to all projects.",
+ "member_count": "Number of users in the group.",
+ "scim_managed": "Whether SCIM provisioning manages the group.",
+ "created_at": "When the group was created.",
+ "updated_at": "When the group was last updated.",
+ },
+ },
+ "license_allocations": {
+ "description": "Seat totals and usage for one license type in the account.",
+ "docs_url": "https://docs.getdbt.com/dbt-cloud/api-v2",
+ "columns": {
+ "license_type": "The license type, such as developer, read_only, analyst or it.",
+ "total": "Number of seats of this type the account has.",
+ "in_use": "Number of seats of this type in use.",
+ "assigned": "Number of seats of this type assigned to users.",
+ "available": "Number of seats of this type still available.",
+ "limited": "Whether the account has a limit on seats of this type.",
+ "over_limit": "Whether the account uses more seats of this type than its limit.",
+ "enterprise_plan": "Whether the account is on an Enterprise plan.",
+ },
+ },
"models": {
"description": "Current state of a model in a deployment environment, as reported by the Discovery API.",
"docs_url": "https://docs.getdbt.com/docs/dbt-cloud-apis/discovery-schema-environment-applied-models",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/dbt.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/dbt.py
index 7d941c82b3dc..dab1e6bdd66d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/dbt.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/dbt.py
@@ -320,9 +320,10 @@ def get_endpoint_permissions(
if config.discovery is not None:
results[endpoint] = discovery_error
continue
- # A run fan-out reads the runs list plus a per-run route; the list is the permission that
- # decides whether the table can sync at all.
- probe_path = config.path if config.path is not None else DBT_ENDPOINTS["runs"].path
+ # A fan-out reads its parent list plus a per-parent route; the parent list is the permission
+ # that decides whether the table can sync at all.
+ parent = "projects" if config.project_fanout is not None else "runs"
+ probe_path = config.path if config.path is not None else DBT_ENDPOINTS[parent].path
assert probe_path is not None
url = f"{base_url}/api{probe_path.format(account_id=account_id)}?limit=1"
try:
@@ -417,6 +418,8 @@ def _iter_offset_pages(
data = _fetch_page(session, url, headers, logger)
rows = data.get("data") or []
+ if isinstance(rows, dict):
+ rows = [rows]
if not rows:
return
@@ -526,6 +529,65 @@ def _iter_run_fanout_rows(
resumable_source_manager.save_state(DbtResumeConfig(parent_offset=page.next_offset))
+def _keyed_rows(rows: list[dict[str, Any]], key_field: str) -> list[dict[str, Any]]:
+ """Turn objects keyed by a type name ({"developer": {...}, ...}) into one row per key."""
+ return [{key_field: key, **value} for row in rows for key, value in row.items() if isinstance(value, dict)]
+
+
+def _project_ids(
+ session: requests.Session,
+ base_url: str,
+ account_id: str,
+ headers: dict[str, str],
+ logger: FilteringBoundLogger,
+) -> list[int]:
+ path = DBT_ENDPOINTS["projects"].path
+ assert path is not None
+
+ project_ids: list[int] = []
+ for page in _iter_offset_pages(
+ session, base_url, path.format(account_id=account_id), {"limit": DBT_PAGE_LIMIT}, headers, logger
+ ):
+ project_ids.extend(row["id"] for row in page.rows if row.get("id") is not None)
+ return project_ids
+
+
+def _iter_project_fanout_rows(
+ session: requests.Session,
+ base_url: str,
+ account_id: str,
+ config: DbtEndpointConfig,
+ headers: dict[str, str],
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[DbtResumeConfig],
+ resume: DbtResumeConfig | None,
+) -> Iterator[Any]:
+ fanout = config.project_fanout
+ assert fanout is not None
+
+ project_ids = _project_ids(session, base_url, account_id, headers, logger)
+ start_index = resume.parent_offset if resume is not None else 0
+
+ for index in range(start_index, len(project_ids)):
+ start_offset = resume.offset if resume is not None and index == start_index else 0
+ for page in _iter_offset_pages(
+ session,
+ base_url,
+ fanout.path.format(account_id=account_id, project_id=project_ids[index]),
+ {"limit": DBT_PAGE_LIMIT},
+ headers,
+ logger,
+ start_offset=start_offset,
+ ):
+ if page.has_more:
+ resumable_source_manager.save_state(DbtResumeConfig(offset=page.next_offset, parent_offset=index))
+ elif index + 1 < len(project_ids):
+ resumable_source_manager.save_state(DbtResumeConfig(parent_offset=index + 1))
+ yield page.rows
+ # Most projects have one repository at most, so many walks return no rows to yield.
+ resumable_source_manager.safe_point()
+
+
def _deployment_environment_ids(
session: requests.Session,
base_url: str,
@@ -756,6 +818,19 @@ def get_rows(
)
return
+ if config.project_fanout is not None:
+ yield from _iter_project_fanout_rows(
+ session,
+ base_url,
+ account_id,
+ config,
+ headers,
+ logger,
+ resumable_source_manager,
+ resume,
+ )
+ return
+
if config.run_fanout is not None:
yield from _iter_run_fanout_rows(
session,
@@ -792,6 +867,8 @@ def get_rows(
yield kept
if len(kept) < len(page.rows):
break
+ elif config.keyed_row_field is not None:
+ yield _keyed_rows(page.rows, config.keyed_row_field)
else:
yield page.rows
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/settings.py
index 5d7093bdab26..021b518edd50 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/settings.py
@@ -79,6 +79,14 @@ class DbtRunFanoutConfig:
string_row_field: Optional[str] = None
+@frozen
+class DbtProjectFanoutConfig:
+ """A list resource scoped to a project, walked once per project in the account."""
+
+ # Path template under {base_url}/api, with {account_id} and {project_id} placeholders.
+ path: str
+
+
@frozen
class DbtDiscoveryConfig:
"""An endpoint served by the Discovery API (GraphQL) instead of the Admin API."""
@@ -95,8 +103,8 @@ class DbtDiscoveryConfig:
class DbtEndpointConfig:
name: str
# Path template under {base_url}/api, with an {account_id} placeholder. The API version is part
- # of the path because coverage is split: jobs/runs live on v2, the rest on v3. None for
- # endpoints that are not a top-level Admin API list (run fan-outs and Discovery API tables).
+ # of the path because coverage is split: jobs/runs/licenses live on v2, the rest on v3. None
+ # for endpoints that are not a top-level Admin API list (fan-outs and Discovery API tables).
path: Optional[str] = None
incremental_fields: list[IncrementalField] = field(default_factory=list)
default_incremental_field: Optional[str] = None
@@ -112,6 +120,11 @@ class DbtEndpointConfig:
incremental_lookback: Optional[timedelta] = None
# Set when rows come from a child resource fetched once per run instead of a list endpoint.
run_fanout: Optional[DbtRunFanoutConfig] = None
+ # Set when rows come from a list resource fetched once per project instead of an account list.
+ project_fanout: Optional[DbtProjectFanoutConfig] = None
+ # Set when the response holds one object keyed by a type name rather than one row per item;
+ # names the column each key lands in, and the key's object becomes that row.
+ keyed_row_field: Optional[str] = None
# Set when rows come from the Discovery API instead of the Admin API.
discovery: Optional[DbtDiscoveryConfig] = None
description: Optional[str] = None
@@ -209,6 +222,32 @@ DBT_ENDPOINTS: dict[str, DbtEndpointConfig] = {
should_sync_default=False,
description="Account audit events: who changed jobs, environments and permissions, and when. Requires an Enterprise plan",
),
+ "repositories": DbtEndpointConfig(
+ name="repositories",
+ # The account-level v2 list is deprecated; v3 only lists repositories per project.
+ project_fanout=DbtProjectFanoutConfig(path="/v3/accounts/{account_id}/projects/{project_id}/repositories/"),
+ description="The git repository behind each project, with its provider and clone strategy. Costs one extra request per project",
+ ),
+ "connections": DbtEndpointConfig(
+ name="connections",
+ path="/v3/accounts/{account_id}/connections/",
+ description="Data platform connections in the account, with the adapter each one uses",
+ ),
+ "groups": DbtEndpointConfig(
+ name="groups",
+ path="/v3/accounts/{account_id}/groups/",
+ # Like users, listing groups needs account-level permissions many read-only tokens lack.
+ should_sync_default=False,
+ description="Permission groups in the account, with their member count, SSO mappings and the permission sets they grant",
+ ),
+ "license_allocations": DbtEndpointConfig(
+ name="license_allocations",
+ path="/v2/accounts/{account_id}/licenses/",
+ keyed_row_field="license_type",
+ primary_keys=["license_type"],
+ should_sync_default=False,
+ description="Seat totals and usage for each license type in the account. Requires a token that can read the account's licenses",
+ ),
"models": DbtEndpointConfig(
name="models",
discovery=DbtDiscoveryConfig(query=MODELS_QUERY, applied_field="models"),
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/tests/test_dbt.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/tests/test_dbt.py
index e16efa4a8319..41c3e7936454 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/tests/test_dbt.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dbt/tests/test_dbt.py
@@ -166,10 +166,11 @@ class TestGetEndpointPermissions:
region="us",
custom_base_url=None,
team_id=1,
- endpoints=["projects"],
+ endpoints=["projects", "repositories"],
)
- assert result["projects"] is None
+ assert result == {"projects": None, "repositories": None}
+ # repositories has no account-level path; its probe is the projects list it fans out over.
assert {request.path for request in network.requests_log} == {"/api/v3/accounts/12345/projects/"}
def test_discovery_endpoints_report_one_shared_probe_result(self):
@@ -261,6 +262,24 @@ class TestGetRows:
# newest-first walk: incremental syncs don't re-crawl history.
assert result.params("offset") == ["0", "100"]
+ @pytest.mark.parametrize(
+ "data",
+ [
+ [{"developer": {"total": 5, "in_use": 3}, "read_only": {"total": 10, "in_use": 0}, "note": None}],
+ {"developer": {"total": 5, "in_use": 3}, "read_only": {"total": 10, "in_use": 0}, "note": None},
+ ],
+ )
+ def test_license_allocations_become_one_row_per_license_type(self, data):
+ # The spec types `data` as a list, but the allocation is one object per account, so accept both.
+ result = _driver().run("license_allocations", [ScriptedResponse(json={"data": data})])
+
+ assert result.raised is None
+ assert result.rows == [
+ {"license_type": "developer", "total": 5, "in_use": 3},
+ {"license_type": "read_only", "total": 10, "in_use": 0},
+ ]
+ assert result.paths == ["/api/v2/accounts/12345/licenses/"]
+
def test_custom_base_url_is_used(self):
result = _driver(custom_base_url="https://ab123.us1.dbt.com/").run("projects", [_page([])])
@@ -444,3 +463,52 @@ class TestDiscoverySync:
"model.a.b",
"model.a.c",
]
+
+
+class TestProjectFanout:
+ def test_walks_every_project_and_pages_within_each(self):
+ result = _driver().run(
+ "repositories",
+ route(
+ {
+ "/accounts/12345/projects/": [_page([{"id": 1}, {"id": 2}, {"id": 3}])],
+ "/projects/1/repositories/": [
+ _page([{"id": 10, "project_id": 1}], count=100, total_count=101),
+ _page([{"id": 11, "project_id": 1}], count=1, total_count=101),
+ ],
+ "/projects/2/repositories/": [_page([])],
+ "/projects/3/repositories/": [_page([{"id": 30, "project_id": 3}])],
+ }
+ ),
+ )
+
+ assert result.raised is None
+ assert [row["id"] for row in result.rows] == [10, 11, 30]
+ assert result.paths == [
+ "/api/v3/accounts/12345/projects/",
+ "/api/v3/accounts/12345/projects/1/repositories/",
+ "/api/v3/accounts/12345/projects/1/repositories/",
+ "/api/v3/accounts/12345/projects/2/repositories/",
+ "/api/v3/accounts/12345/projects/3/repositories/",
+ ]
+ assert result.saved_states == [
+ DbtResumeConfig(offset=100, parent_offset=0),
+ DbtResumeConfig(parent_offset=1),
+ ]
+
+ def test_resumes_inside_the_saved_project(self):
+ result = _driver().run(
+ "repositories",
+ route(
+ {
+ "/accounts/12345/projects/": [_page([{"id": 1}, {"id": 2}])],
+ "/projects/2/repositories/": [_page([{"id": 21, "project_id": 2}], count=1, total_count=101)],
+ }
+ ),
+ resume_state=DbtResumeConfig(offset=100, parent_offset=1),
+ )
+
+ assert result.raised is None
+ assert result.rows == [{"id": 21, "project_id": 2}]
+ assert result.paths == ["/api/v3/accounts/12345/projects/", "/api/v3/accounts/12345/projects/2/repositories/"]
+ assert result.requests[1].param("offset") == "100"