Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 7 missing tables to the DigitalOcean source (#120734)

Tom Owers27d986eca8fd
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index f09010069957..563fc3d3ffa0 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -3790,7 +3790,7 @@ Audit logs are not: v3 places them at `/v3/enterprise/audit-logs` and `/v3/enter
## DigitalOcean — gaps
-Today (25): `actions`, `apps`, `billing_history`, `database_backups`, `database_events`, `databases`, `domain_records`, `domains`, `droplets`, `images`, `invoice_items`, `invoice_summaries`, `invoices`, `kubernetes_clusters`, `load_balancers`, `project_resources`, `projects`, `regions`, `reserved_ips`, `sizes`, `snapshots`, `ssh_keys`, `tags`, `volumes`, `vpcs`
+Today (32): `actions`, `app_deployments`, `app_events`, `apps`, `billing_history`, `database_backups`, `database_events`, `databases`, `domain_records`, `domains`, `droplets`, `images`, `invoice_items`, `invoice_summaries`, `invoices`, `kubernetes_clusters`, `kubernetes_node_pools`, `load_balancers`, `project_resources`, `projects`, `regions`, `registries`, `registry_repositories`, `reserved_ips`, `sizes`, `snapshots`, `ssh_keys`, `tags`, `uptime_check_states`, `uptime_checks`, `volumes`, `vpcs`
Diffed against: <https://api-engineering.nyc3.cdn.digitaloceanspaces.com/spec-ci/DigitalOcean-public.v2.yaml>
@@ -3799,13 +3799,13 @@ Diffed against: <https://api-engineering.nyc3.cdn.digitaloceanspaces.com/spec-ci
- [x] `/v2/regions` — lookup resolving the region slug carried by droplets, databases, load balancers and volumes (high)
- [x] `/v2/projects/{project_id}/resources (and /v2/projects/default/resources)` — the join table mapping every synced resource URN to a project — projects are synced but the membership is not (high) — added as `project_resources` (`/v2/projects/default/resources` is an alias for the project flagged `is_default`, which the fan-out over `/v2/projects` already visits)
- [x] `/v2/actions (and /v2/droplets/{droplet_id}/actions)` — account-wide action history: creates, resizes, power cycles with status and timing — the state-transition log for infrastructure (high)
-- [ ] `/v2/apps/{app_id}/deployments (+ /v2/apps/{app_id}/events)` — App Platform deployment history and phase transitions; apps are synced but not their deploy activity (medium)
-- [ ] `/v2/kubernetes/clusters/{cluster_id}/node_pools` — node pool sizing per cluster, needed to explain Kubernetes cost and capacity (medium)
+- [x] `/v2/apps/{app_id}/deployments (+ /v2/apps/{app_id}/events)` — App Platform deployment history and phase transitions; apps are synced but not their deploy activity (medium) — added as `app_deployments` and `app_events`
+- [x] `/v2/kubernetes/clusters/{cluster_id}/node_pools` — node pool sizing per cluster, needed to explain Kubernetes cost and capacity (medium) — added as `kubernetes_node_pools`
- [ ] `/v2/tags/{tag_id}/resources` — resolves tags (already synced as names) to the resources they are applied to (medium) — not buildable: the spec exposes only POST and DELETE on this path, so tag membership cannot be read back. `/v2/tags` carries per-resource-type counts and a `last_tagged_uri`, which is the closest available data and is already synced.
- [x] `/v2/domains/{domain_name}/records` — DNS records under the domains already synced (medium) — added as `domain_records`
- [x] `/v2/databases/{database_cluster_uuid}/backups and /v2/databases/{database_cluster_uuid}/events` — backup inventory and cluster event history for managed databases (medium) — added as `database_backups` and `database_events`
-- [ ] `/v2/uptime/checks (+ /checks/{check_id}/state)` — uptime check inventory and current state for availability reporting (low)
-- [ ] `/v2/registry/{registry_name}/repositories (+ /tags, /digests)` — container registry repository and tag inventory with sizes (low)
+- [x] `/v2/uptime/checks (+ /checks/{check_id}/state)` — uptime check inventory and current state for availability reporting (low) — added as `uptime_checks` and `uptime_check_states`
+- [x] `/v2/registry/{registry_name}/repositories (+ /tags, /digests)` — container registry repository and tag inventory with sizes (low) — added as `registries` and `registry_repositories`, from the non-deprecated `/v2/registries` and `/v2/registries/{registry_name}/repositoriesV2` paths. Per-tag and per-digest tables are not added: they need a two-level fan-out, and repository names that contain `/` must be sent URL-encoded. The repository rows already carry `tag_count`, `manifest_count` and the sizes of the latest manifest.
Note: The public spec has 447 paths, but a large share is GenAI, monitoring metric time series, and mutation-only endpoints. Firewalls, certificates, CDN endpoints and monitoring alert policies were treated as config and excluded.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/canonical_descriptions.py
index 49745d88dd9c..5fa7d95b48df 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/canonical_descriptions.py
@@ -325,4 +325,96 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"create_time": "Time the event was generated (ISO 8601).",
},
},
+ "app_deployments": {
+ "description": "Deployment history of each App Platform app, with the phase each deployment reached.",
+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Apps",
+ "columns": {
+ "app_id": "ID of the app the deployment belongs to, taken from the parent app.",
+ "id": "ID of the deployment.",
+ "cause": "What caused this deployment to be created.",
+ "cloned_from": "ID of a previous deployment that this deployment was cloned from.",
+ "phase": "Phase of the deployment (e.g. PENDING_BUILD, BUILDING, DEPLOYING, ACTIVE, SUPERSEDED, ERROR, CANCELED).",
+ "phase_last_updated_at": "Time when the deployment phase was last updated (ISO 8601).",
+ "progress": "Counts of pending, running, successful and failed deployment steps.",
+ "spec": "App specification used for this deployment, with environment variables and credentials removed.",
+ "tier_slug": "Pricing tier slug of the deployment.",
+ "created_at": "Time when the deployment was created (ISO 8601).",
+ "updated_at": "Time when the deployment was last updated (ISO 8601).",
+ },
+ },
+ "app_events": {
+ "description": "Deployment and autoscaling events of each App Platform app.",
+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Apps",
+ "columns": {
+ "app_id": "ID of the app the event belongs to, taken from the parent app.",
+ "id": "ID of the event (UUID).",
+ "type": "Type of the event (UNKNOWN, DEPLOYMENT or AUTOSCALING).",
+ "created_at": "Time when the event was created (ISO 8601).",
+ "deployment_id": "ID of the deployment associated with the event.",
+ "deployment": "The deployment for a deployment event, with environment variables and credentials removed.",
+ "autoscaling": "Autoscaling details. Only present for autoscaling events.",
+ },
+ },
+ "kubernetes_node_pools": {
+ "description": "Node pools of each Kubernetes cluster, with their Droplet size, node count and autoscaling bounds.",
+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Kubernetes",
+ "columns": {
+ "kubernetes_cluster_id": "ID of the cluster the node pool belongs to, taken from the parent cluster.",
+ "id": "Unique ID of the node pool.",
+ "name": "Human-readable name of the node pool.",
+ "size": "Slug of the Droplet size used for the worker nodes.",
+ "count": "Number of Droplet instances in the node pool.",
+ "auto_scale": "Whether autoscaling is enabled for the node pool.",
+ "min_nodes": "Minimum number of nodes the pool can scale to (0 when autoscaling is off).",
+ "max_nodes": "Maximum number of nodes the pool can scale to (0 when autoscaling is off).",
+ "nodes": "Worker nodes in the pool, with their status and Droplet ID.",
+ "tags": "Tags applied to the node pool.",
+ "labels": "Key/value labels applied to all nodes in the pool.",
+ "taints": "Taints applied to all nodes in the pool.",
+ },
+ },
+ "uptime_checks": {
+ "description": "Uptime checks that monitor the availability of a URL or host.",
+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Uptime",
+ "columns": {
+ "id": "Unique ID of the check.",
+ "name": "Display name of the check.",
+ "type": "Type of health check (ping, http or https).",
+ "target": "Endpoint the check monitors.",
+ "regions": "Regions the check runs from.",
+ "enabled": "Whether the check is enabled.",
+ },
+ },
+ "uptime_check_states": {
+ "description": "Current per-region status and the most recent outage of each uptime check.",
+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Uptime",
+ "columns": {
+ "check_id": "ID of the uptime check, taken from the parent check.",
+ "regions": "Map of region to its status, last status change time and 30-day uptime percentage.",
+ "previous_outage": "Most recent outage, with its region, start, end and duration in seconds.",
+ },
+ },
+ "registries": {
+ "description": "Container registries in the account.",
+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Container-Registries",
+ "columns": {
+ "name": "Globally unique name of the registry.",
+ "created_at": "Time when the registry was created (ISO 8601).",
+ "region": "Slug of the region where the registry data is stored.",
+ "storage_usage_bytes": "Storage used by the registry, in bytes.",
+ "storage_usage_bytes_updated_at": "Time when the storage usage was last calculated (ISO 8601).",
+ "subscription": "Subscription tier of the registry.",
+ },
+ },
+ "registry_repositories": {
+ "description": "Repositories in each container registry, with tag and manifest counts.",
+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Container-Registries",
+ "columns": {
+ "registry_name": "Name of the registry the repository belongs to.",
+ "name": "Name of the repository.",
+ "tag_count": "Number of tags in the repository.",
+ "manifest_count": "Number of manifests in the repository.",
+ "latest_manifest": "Most recently updated manifest, with its digest, tags and compressed and uncompressed sizes.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/settings.py
index 2dbf2fe16028..ec234c0f845d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/settings.py
@@ -344,6 +344,112 @@ DIGITALOCEAN_ENDPOINTS: dict[str, DigitalOceanEndpointConfig] = {
parent_params={"per_page": PAGE_SIZE},
),
),
+ # Deployment history per App Platform app, with the phase each deployment reached. Every
+ # deployment carries a copy of the app spec, so the same secrets as `apps` are stripped, plus
+ # the image pull credentials that the deployment spec can hold.
+ "app_deployments": DigitalOceanEndpointConfig(
+ name="app_deployments",
+ path="/v2/apps/{app_id}/deployments",
+ data_selector="deployments",
+ primary_keys=["app_id", "id"],
+ partition_key="created_at",
+ sensitive_fields=frozenset({"envs", "log_destinations", "registry_credentials"}),
+ fanout=DependentEndpointConfig(
+ parent_name="apps",
+ resolve_param="app_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "app_id"},
+ parent_params={"per_page": PAGE_SIZE},
+ child_params={"per_page": PAGE_SIZE},
+ ),
+ ),
+ # Deployment and autoscaling events per app. A deployment event embeds the full deployment,
+ # spec included, so it is stripped the same way as `app_deployments`.
+ "app_events": DigitalOceanEndpointConfig(
+ name="app_events",
+ path="/v2/apps/{app_id}/events",
+ data_selector="events",
+ primary_keys=["app_id", "id"],
+ partition_key="created_at",
+ sensitive_fields=frozenset({"envs", "log_destinations", "registry_credentials"}),
+ fanout=DependentEndpointConfig(
+ parent_name="apps",
+ resolve_param="app_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "app_id"},
+ parent_params={"per_page": PAGE_SIZE},
+ child_params={"per_page": PAGE_SIZE},
+ ),
+ ),
+ # Node pool sizing and autoscaling bounds per Kubernetes cluster. The endpoint takes no page
+ # params and returns every pool of the cluster in one response.
+ "kubernetes_node_pools": DigitalOceanEndpointConfig(
+ name="kubernetes_node_pools",
+ path="/v2/kubernetes/clusters/{cluster_id}/node_pools",
+ data_selector="node_pools",
+ primary_keys=["kubernetes_cluster_id", "id"],
+ fanout=DependentEndpointConfig(
+ parent_name="kubernetes_clusters",
+ resolve_param="cluster_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "kubernetes_cluster_id"},
+ parent_params={"per_page": PAGE_SIZE},
+ ),
+ ),
+ "uptime_checks": DigitalOceanEndpointConfig(
+ name="uptime_checks",
+ path="/v2/uptime/checks",
+ data_selector="checks",
+ ),
+ # Current per-region status and the last outage of each uptime check. The endpoint returns
+ # one object under `state` rather than a list, and the object carries no check id.
+ "uptime_check_states": DigitalOceanEndpointConfig(
+ name="uptime_check_states",
+ path="/v2/uptime/checks/{check_id}/state",
+ data_selector="state",
+ primary_keys=["check_id"],
+ paginated=False,
+ fanout=DependentEndpointConfig(
+ parent_name="uptime_checks",
+ resolve_param="check_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "check_id"},
+ parent_params={"per_page": PAGE_SIZE},
+ ),
+ ),
+ # `/v2/registries` replaces the deprecated single-registry `/v2/registry` endpoints, which
+ # stop working once an account has more than one registry.
+ "registries": DigitalOceanEndpointConfig(
+ name="registries",
+ path="/v2/registries",
+ data_selector="registries",
+ primary_keys=["name"],
+ partition_key="created_at",
+ ),
+ # Repositories per container registry, with tag and manifest counts and the size of the
+ # latest manifest.
+ "registry_repositories": DigitalOceanEndpointConfig(
+ name="registry_repositories",
+ path="/v2/registries/{registry_name}/repositoriesV2",
+ data_selector="repositories",
+ # Repository names are only unique within their registry.
+ primary_keys=["registry_name", "name"],
+ fanout=DependentEndpointConfig(
+ parent_name="registries",
+ resolve_param="registry_name",
+ resolve_field="name",
+ include_from_parent=["name"],
+ # Lands on the row's own `registry_name`, which holds the same value, and keeps the
+ # repository's `name` column intact.
+ parent_field_renames={"name": "registry_name"},
+ parent_params={"per_page": PAGE_SIZE},
+ child_params={"per_page": PAGE_SIZE},
+ ),
+ ),
}
ENDPOINTS = tuple(DIGITALOCEAN_ENDPOINTS.keys())
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/tests/test_digitalocean.py b/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/tests/test_digitalocean.py
index 77a8c9f31e8c..6ffaa8ba7592 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/tests/test_digitalocean.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/tests/test_digitalocean.py
@@ -6,6 +6,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.tes
ScriptedResponse,
SourceDriver,
always,
+ route,
scripted_network,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.digitalocean.digitalocean import (
@@ -85,6 +86,8 @@ class TestDigitalOceanSensitiveFields:
pytest.param("invoice_items", True, id="resource_level_spend_in_the_response"),
pytest.param("database_backups", True, id="secrets_in_the_fanout_parent_response"),
pytest.param("database_events", True, id="secrets_in_the_other_fanout_parent_response"),
+ pytest.param("app_deployments", True, id="app_specs_in_the_parent_and_child_response"),
+ pytest.param("app_events", True, id="app_specs_in_the_parent_and_embedded_deployments"),
pytest.param("droplets", False, id="nothing_to_withhold"),
],
)
@@ -97,7 +100,7 @@ class TestDigitalOceanSensitiveFields:
# tracked client's default (capture on).
result = _driver().run(
endpoint,
- [ScriptedResponse(json={"databases": [], "droplets": [], "invoices": []})],
+ [ScriptedResponse(json={"apps": [], "databases": [], "droplets": [], "invoices": []})],
)
assert result.raised is None
@@ -107,6 +110,105 @@ class TestDigitalOceanSensitiveFields:
assert result.session_options == [{"redact_values": ("dop_v1_token",)}]
+class TestDigitalOceanFanOut:
+ @pytest.mark.parametrize(
+ "endpoint,script,expected_paths,expected_rows",
+ [
+ pytest.param(
+ "app_deployments",
+ {
+ "/v2/apps": [ScriptedResponse(json={"apps": [{"id": "app-1"}]})],
+ "/v2/apps/app-1/deployments": [
+ ScriptedResponse(
+ json={
+ "deployments": [
+ {
+ "id": "dep-2",
+ "phase": "ACTIVE",
+ "spec": {
+ "envs": [{"key": "TOKEN", "value": "leak-me"}],
+ "services": [
+ {
+ "name": "api",
+ "image": {"registry_credentials": "user:leak-me"},
+ "log_destinations": [{"logtail": {"token": "leak-me"}}],
+ }
+ ],
+ },
+ }
+ ],
+ "links": {
+ "pages": {
+ "next": "https://api.digitalocean.com/v2/apps/app-1/deployments?page=2&per_page=200"
+ }
+ },
+ }
+ ),
+ ScriptedResponse(json={"deployments": [{"id": "dep-1", "phase": "SUPERSEDED"}], "links": {}}),
+ ],
+ },
+ ["/v2/apps", "/v2/apps/app-1/deployments", "/v2/apps/app-1/deployments"],
+ [
+ {
+ "id": "dep-2",
+ "phase": "ACTIVE",
+ "spec": {"services": [{"name": "api", "image": {}}]},
+ "app_id": "app-1",
+ },
+ {"id": "dep-1", "phase": "SUPERSEDED", "app_id": "app-1"},
+ ],
+ id="child_pages_walked_and_nested_app_secrets_stripped",
+ ),
+ pytest.param(
+ "uptime_check_states",
+ {
+ "/v2/uptime/checks": [ScriptedResponse(json={"checks": [{"id": "chk-1"}, {"id": "chk-2"}]})],
+ "/v2/uptime/checks/chk-1/state": [
+ ScriptedResponse(json={"state": {"regions": {"us_east": {"status": "UP"}}}})
+ ],
+ "/v2/uptime/checks/chk-2/state": [
+ ScriptedResponse(
+ json={"state": {"regions": {"eu_west": {"status": "DOWN"}}, "previous_outage": None}}
+ )
+ ],
+ },
+ ["/v2/uptime/checks", "/v2/uptime/checks/chk-1/state", "/v2/uptime/checks/chk-2/state"],
+ [
+ {"regions": {"us_east": {"status": "UP"}}, "check_id": "chk-1"},
+ {"regions": {"eu_west": {"status": "DOWN"}}, "previous_outage": None, "check_id": "chk-2"},
+ ],
+ id="single_state_object_becomes_one_row_per_check",
+ ),
+ pytest.param(
+ "registry_repositories",
+ {
+ "/v2/registries": [ScriptedResponse(json={"registries": [{"name": "reg-a"}]})],
+ "/v2/registries/reg-a/repositoriesV2": [
+ ScriptedResponse(
+ json={"repositories": [{"registry_name": "reg-a", "name": "web", "tag_count": 3}]}
+ )
+ ],
+ },
+ ["/v2/registries", "/v2/registries/reg-a/repositoriesV2"],
+ [{"registry_name": "reg-a", "name": "web", "tag_count": 3}],
+ id="parent_registry_name_does_not_replace_repository_name",
+ ),
+ ],
+ )
+ def test_child_rows_carry_their_parent_id(
+ self,
+ endpoint: str,
+ script: dict[str, list[ScriptedResponse]],
+ expected_paths: list[str],
+ expected_rows: list[dict[str, Any]],
+ ) -> None:
+ result = _driver().run(endpoint, route(script))
+
+ assert result.raised is None
+ assert result.paths == expected_paths
+ assert result.rows == expected_rows
+
+
class TestDigitalOceanValidateCredentials:
@pytest.mark.parametrize(
"status_code,expected",