:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 7 missing tables to the DigitalOcean source (#120734)
Tom Owers27d986eca8fd
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex f09010069957..563fc3d3ffa0 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -3790,7 +3790,7 @@ Audit logs are not: v3 places them at `/v3/enterprise/audit-logs` and `/v3/enter## DigitalOcean — gaps-Today (25): `actions`, `apps`, `billing_history`, `database_backups`, `database_events`, `databases`, `domain_records`, `domains`, `droplets`, `images`, `invoice_items`, `invoice_summaries`, `invoices`, `kubernetes_clusters`, `load_balancers`, `project_resources`, `projects`, `regions`, `reserved_ips`, `sizes`, `snapshots`, `ssh_keys`, `tags`, `volumes`, `vpcs`+Today (32): `actions`, `app_deployments`, `app_events`, `apps`, `billing_history`, `database_backups`, `database_events`, `databases`, `domain_records`, `domains`, `droplets`, `images`, `invoice_items`, `invoice_summaries`, `invoices`, `kubernetes_clusters`, `kubernetes_node_pools`, `load_balancers`, `project_resources`, `projects`, `regions`, `registries`, `registry_repositories`, `reserved_ips`, `sizes`, `snapshots`, `ssh_keys`, `tags`, `uptime_check_states`, `uptime_checks`, `volumes`, `vpcs`Diffed against: <https://api-engineering.nyc3.cdn.digitaloceanspaces.com/spec-ci/DigitalOcean-public.v2.yaml>@@ -3799,13 +3799,13 @@ Diffed against: <https://api-engineering.nyc3.cdn.digitaloceanspaces.com/spec-ci- [x] `/v2/regions` — lookup resolving the region slug carried by droplets, databases, load balancers and volumes (high)- [x] `/v2/projects/{project_id}/resources (and /v2/projects/default/resources)` — the join table mapping every synced resource URN to a project — projects are synced but the membership is not (high) — added as `project_resources` (`/v2/projects/default/resources` is an alias for the project flagged `is_default`, which the fan-out over `/v2/projects` already visits)- [x] `/v2/actions (and /v2/droplets/{droplet_id}/actions)` — account-wide action history: creates, resizes, power cycles with status and timing — the state-transition log for infrastructure (high)-- [ ] `/v2/apps/{app_id}/deployments (+ /v2/apps/{app_id}/events)` — App Platform deployment history and phase transitions; apps are synced but not their deploy activity (medium)-- [ ] `/v2/kubernetes/clusters/{cluster_id}/node_pools` — node pool sizing per cluster, needed to explain Kubernetes cost and capacity (medium)+- [x] `/v2/apps/{app_id}/deployments (+ /v2/apps/{app_id}/events)` — App Platform deployment history and phase transitions; apps are synced but not their deploy activity (medium) — added as `app_deployments` and `app_events`+- [x] `/v2/kubernetes/clusters/{cluster_id}/node_pools` — node pool sizing per cluster, needed to explain Kubernetes cost and capacity (medium) — added as `kubernetes_node_pools`- [ ] `/v2/tags/{tag_id}/resources` — resolves tags (already synced as names) to the resources they are applied to (medium) — not buildable: the spec exposes only POST and DELETE on this path, so tag membership cannot be read back. `/v2/tags` carries per-resource-type counts and a `last_tagged_uri`, which is the closest available data and is already synced.- [x] `/v2/domains/{domain_name}/records` — DNS records under the domains already synced (medium) — added as `domain_records`- [x] `/v2/databases/{database_cluster_uuid}/backups and /v2/databases/{database_cluster_uuid}/events` — backup inventory and cluster event history for managed databases (medium) — added as `database_backups` and `database_events`-- [ ] `/v2/uptime/checks (+ /checks/{check_id}/state)` — uptime check inventory and current state for availability reporting (low)-- [ ] `/v2/registry/{registry_name}/repositories (+ /tags, /digests)` — container registry repository and tag inventory with sizes (low)+- [x] `/v2/uptime/checks (+ /checks/{check_id}/state)` — uptime check inventory and current state for availability reporting (low) — added as `uptime_checks` and `uptime_check_states`+- [x] `/v2/registry/{registry_name}/repositories (+ /tags, /digests)` — container registry repository and tag inventory with sizes (low) — added as `registries` and `registry_repositories`, from the non-deprecated `/v2/registries` and `/v2/registries/{registry_name}/repositoriesV2` paths. Per-tag and per-digest tables are not added: they need a two-level fan-out, and repository names that contain `/` must be sent URL-encoded. The repository rows already carry `tag_count`, `manifest_count` and the sizes of the latest manifest.Note: The public spec has 447 paths, but a large share is GenAI, monitoring metric time series, and mutation-only endpoints. Firewalls, certificates, CDN endpoints and monitoring alert policies were treated as config and excluded.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/canonical_descriptions.pyindex 49745d88dd9c..5fa7d95b48df 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/canonical_descriptions.py@@ -325,4 +325,96 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"create_time": "Time the event was generated (ISO 8601).",},},+ "app_deployments": {+ "description": "Deployment history of each App Platform app, with the phase each deployment reached.",+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Apps",+ "columns": {+ "app_id": "ID of the app the deployment belongs to, taken from the parent app.",+ "id": "ID of the deployment.",+ "cause": "What caused this deployment to be created.",+ "cloned_from": "ID of a previous deployment that this deployment was cloned from.",+ "phase": "Phase of the deployment (e.g. PENDING_BUILD, BUILDING, DEPLOYING, ACTIVE, SUPERSEDED, ERROR, CANCELED).",+ "phase_last_updated_at": "Time when the deployment phase was last updated (ISO 8601).",+ "progress": "Counts of pending, running, successful and failed deployment steps.",+ "spec": "App specification used for this deployment, with environment variables and credentials removed.",+ "tier_slug": "Pricing tier slug of the deployment.",+ "created_at": "Time when the deployment was created (ISO 8601).",+ "updated_at": "Time when the deployment was last updated (ISO 8601).",+ },+ },+ "app_events": {+ "description": "Deployment and autoscaling events of each App Platform app.",+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Apps",+ "columns": {+ "app_id": "ID of the app the event belongs to, taken from the parent app.",+ "id": "ID of the event (UUID).",+ "type": "Type of the event (UNKNOWN, DEPLOYMENT or AUTOSCALING).",+ "created_at": "Time when the event was created (ISO 8601).",+ "deployment_id": "ID of the deployment associated with the event.",+ "deployment": "The deployment for a deployment event, with environment variables and credentials removed.",+ "autoscaling": "Autoscaling details. Only present for autoscaling events.",+ },+ },+ "kubernetes_node_pools": {+ "description": "Node pools of each Kubernetes cluster, with their Droplet size, node count and autoscaling bounds.",+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Kubernetes",+ "columns": {+ "kubernetes_cluster_id": "ID of the cluster the node pool belongs to, taken from the parent cluster.",+ "id": "Unique ID of the node pool.",+ "name": "Human-readable name of the node pool.",+ "size": "Slug of the Droplet size used for the worker nodes.",+ "count": "Number of Droplet instances in the node pool.",+ "auto_scale": "Whether autoscaling is enabled for the node pool.",+ "min_nodes": "Minimum number of nodes the pool can scale to (0 when autoscaling is off).",+ "max_nodes": "Maximum number of nodes the pool can scale to (0 when autoscaling is off).",+ "nodes": "Worker nodes in the pool, with their status and Droplet ID.",+ "tags": "Tags applied to the node pool.",+ "labels": "Key/value labels applied to all nodes in the pool.",+ "taints": "Taints applied to all nodes in the pool.",+ },+ },+ "uptime_checks": {+ "description": "Uptime checks that monitor the availability of a URL or host.",+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Uptime",+ "columns": {+ "id": "Unique ID of the check.",+ "name": "Display name of the check.",+ "type": "Type of health check (ping, http or https).",+ "target": "Endpoint the check monitors.",+ "regions": "Regions the check runs from.",+ "enabled": "Whether the check is enabled.",+ },+ },+ "uptime_check_states": {+ "description": "Current per-region status and the most recent outage of each uptime check.",+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Uptime",+ "columns": {+ "check_id": "ID of the uptime check, taken from the parent check.",+ "regions": "Map of region to its status, last status change time and 30-day uptime percentage.",+ "previous_outage": "Most recent outage, with its region, start, end and duration in seconds.",+ },+ },+ "registries": {+ "description": "Container registries in the account.",+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Container-Registries",+ "columns": {+ "name": "Globally unique name of the registry.",+ "created_at": "Time when the registry was created (ISO 8601).",+ "region": "Slug of the region where the registry data is stored.",+ "storage_usage_bytes": "Storage used by the registry, in bytes.",+ "storage_usage_bytes_updated_at": "Time when the storage usage was last calculated (ISO 8601).",+ "subscription": "Subscription tier of the registry.",+ },+ },+ "registry_repositories": {+ "description": "Repositories in each container registry, with tag and manifest counts.",+ "docs_url": "https://docs.digitalocean.com/reference/api/api-reference/#tag/Container-Registries",+ "columns": {+ "registry_name": "Name of the registry the repository belongs to.",+ "name": "Name of the repository.",+ "tag_count": "Number of tags in the repository.",+ "manifest_count": "Number of manifests in the repository.",+ "latest_manifest": "Most recently updated manifest, with its digest, tags and compressed and uncompressed sizes.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/settings.pyindex 2dbf2fe16028..ec234c0f845d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/settings.py@@ -344,6 +344,112 @@ DIGITALOCEAN_ENDPOINTS: dict[str, DigitalOceanEndpointConfig] = {parent_params={"per_page": PAGE_SIZE},),),+ # Deployment history per App Platform app, with the phase each deployment reached. Every+ # deployment carries a copy of the app spec, so the same secrets as `apps` are stripped, plus+ # the image pull credentials that the deployment spec can hold.+ "app_deployments": DigitalOceanEndpointConfig(+ name="app_deployments",+ path="/v2/apps/{app_id}/deployments",+ data_selector="deployments",+ primary_keys=["app_id", "id"],+ partition_key="created_at",+ sensitive_fields=frozenset({"envs", "log_destinations", "registry_credentials"}),+ fanout=DependentEndpointConfig(+ parent_name="apps",+ resolve_param="app_id",+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": "app_id"},+ parent_params={"per_page": PAGE_SIZE},+ child_params={"per_page": PAGE_SIZE},+ ),+ ),+ # Deployment and autoscaling events per app. A deployment event embeds the full deployment,+ # spec included, so it is stripped the same way as `app_deployments`.+ "app_events": DigitalOceanEndpointConfig(+ name="app_events",+ path="/v2/apps/{app_id}/events",+ data_selector="events",+ primary_keys=["app_id", "id"],+ partition_key="created_at",+ sensitive_fields=frozenset({"envs", "log_destinations", "registry_credentials"}),+ fanout=DependentEndpointConfig(+ parent_name="apps",+ resolve_param="app_id",+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": "app_id"},+ parent_params={"per_page": PAGE_SIZE},+ child_params={"per_page": PAGE_SIZE},+ ),+ ),+ # Node pool sizing and autoscaling bounds per Kubernetes cluster. The endpoint takes no page+ # params and returns every pool of the cluster in one response.+ "kubernetes_node_pools": DigitalOceanEndpointConfig(+ name="kubernetes_node_pools",+ path="/v2/kubernetes/clusters/{cluster_id}/node_pools",+ data_selector="node_pools",+ primary_keys=["kubernetes_cluster_id", "id"],+ fanout=DependentEndpointConfig(+ parent_name="kubernetes_clusters",+ resolve_param="cluster_id",+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": "kubernetes_cluster_id"},+ parent_params={"per_page": PAGE_SIZE},+ ),+ ),+ "uptime_checks": DigitalOceanEndpointConfig(+ name="uptime_checks",+ path="/v2/uptime/checks",+ data_selector="checks",+ ),+ # Current per-region status and the last outage of each uptime check. The endpoint returns+ # one object under `state` rather than a list, and the object carries no check id.+ "uptime_check_states": DigitalOceanEndpointConfig(+ name="uptime_check_states",+ path="/v2/uptime/checks/{check_id}/state",+ data_selector="state",+ primary_keys=["check_id"],+ paginated=False,+ fanout=DependentEndpointConfig(+ parent_name="uptime_checks",+ resolve_param="check_id",+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": "check_id"},+ parent_params={"per_page": PAGE_SIZE},+ ),+ ),+ # `/v2/registries` replaces the deprecated single-registry `/v2/registry` endpoints, which+ # stop working once an account has more than one registry.+ "registries": DigitalOceanEndpointConfig(+ name="registries",+ path="/v2/registries",+ data_selector="registries",+ primary_keys=["name"],+ partition_key="created_at",+ ),+ # Repositories per container registry, with tag and manifest counts and the size of the+ # latest manifest.+ "registry_repositories": DigitalOceanEndpointConfig(+ name="registry_repositories",+ path="/v2/registries/{registry_name}/repositoriesV2",+ data_selector="repositories",+ # Repository names are only unique within their registry.+ primary_keys=["registry_name", "name"],+ fanout=DependentEndpointConfig(+ parent_name="registries",+ resolve_param="registry_name",+ resolve_field="name",+ include_from_parent=["name"],+ # Lands on the row's own `registry_name`, which holds the same value, and keeps the+ # repository's `name` column intact.+ parent_field_renames={"name": "registry_name"},+ parent_params={"per_page": PAGE_SIZE},+ child_params={"per_page": PAGE_SIZE},+ ),+ ),}ENDPOINTS = tuple(DIGITALOCEAN_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/tests/test_digitalocean.py b/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/tests/test_digitalocean.pyindex 77a8c9f31e8c..6ffaa8ba7592 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/tests/test_digitalocean.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/digitalocean/tests/test_digitalocean.py@@ -6,6 +6,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.tesScriptedResponse,SourceDriver,always,+ route,scripted_network,)from products.warehouse_sources.backend.temporal.data_imports.sources.digitalocean.digitalocean import (@@ -85,6 +86,8 @@ class TestDigitalOceanSensitiveFields:pytest.param("invoice_items", True, id="resource_level_spend_in_the_response"),pytest.param("database_backups", True, id="secrets_in_the_fanout_parent_response"),pytest.param("database_events", True, id="secrets_in_the_other_fanout_parent_response"),+ pytest.param("app_deployments", True, id="app_specs_in_the_parent_and_child_response"),+ pytest.param("app_events", True, id="app_specs_in_the_parent_and_embedded_deployments"),pytest.param("droplets", False, id="nothing_to_withhold"),],)@@ -97,7 +100,7 @@ class TestDigitalOceanSensitiveFields:# tracked client's default (capture on).result = _driver().run(endpoint,- [ScriptedResponse(json={"databases": [], "droplets": [], "invoices": []})],+ [ScriptedResponse(json={"apps": [], "databases": [], "droplets": [], "invoices": []})],)assert result.raised is None@@ -107,6 +110,105 @@ class TestDigitalOceanSensitiveFields:assert result.session_options == [{"redact_values": ("dop_v1_token",)}]+class TestDigitalOceanFanOut:+ @pytest.mark.parametrize(+ "endpoint,script,expected_paths,expected_rows",+ [+ pytest.param(+ "app_deployments",+ {+ "/v2/apps": [ScriptedResponse(json={"apps": [{"id": "app-1"}]})],+ "/v2/apps/app-1/deployments": [+ ScriptedResponse(+ json={+ "deployments": [+ {+ "id": "dep-2",+ "phase": "ACTIVE",+ "spec": {+ "envs": [{"key": "TOKEN", "value": "leak-me"}],+ "services": [+ {+ "name": "api",+ "image": {"registry_credentials": "user:leak-me"},+ "log_destinations": [{"logtail": {"token": "leak-me"}}],+ }+ ],+ },+ }+ ],+ "links": {+ "pages": {+ "next": "https://api.digitalocean.com/v2/apps/app-1/deployments?page=2&per_page=200"+ }+ },+ }+ ),+ ScriptedResponse(json={"deployments": [{"id": "dep-1", "phase": "SUPERSEDED"}], "links": {}}),+ ],+ },+ ["/v2/apps", "/v2/apps/app-1/deployments", "/v2/apps/app-1/deployments"],+ [+ {+ "id": "dep-2",+ "phase": "ACTIVE",+ "spec": {"services": [{"name": "api", "image": {}}]},+ "app_id": "app-1",+ },+ {"id": "dep-1", "phase": "SUPERSEDED", "app_id": "app-1"},+ ],+ id="child_pages_walked_and_nested_app_secrets_stripped",+ ),+ pytest.param(+ "uptime_check_states",+ {+ "/v2/uptime/checks": [ScriptedResponse(json={"checks": [{"id": "chk-1"}, {"id": "chk-2"}]})],+ "/v2/uptime/checks/chk-1/state": [+ ScriptedResponse(json={"state": {"regions": {"us_east": {"status": "UP"}}}})+ ],+ "/v2/uptime/checks/chk-2/state": [+ ScriptedResponse(+ json={"state": {"regions": {"eu_west": {"status": "DOWN"}}, "previous_outage": None}}+ )+ ],+ },+ ["/v2/uptime/checks", "/v2/uptime/checks/chk-1/state", "/v2/uptime/checks/chk-2/state"],+ [+ {"regions": {"us_east": {"status": "UP"}}, "check_id": "chk-1"},+ {"regions": {"eu_west": {"status": "DOWN"}}, "previous_outage": None, "check_id": "chk-2"},+ ],+ id="single_state_object_becomes_one_row_per_check",+ ),+ pytest.param(+ "registry_repositories",+ {+ "/v2/registries": [ScriptedResponse(json={"registries": [{"name": "reg-a"}]})],+ "/v2/registries/reg-a/repositoriesV2": [+ ScriptedResponse(+ json={"repositories": [{"registry_name": "reg-a", "name": "web", "tag_count": 3}]}+ )+ ],+ },+ ["/v2/registries", "/v2/registries/reg-a/repositoriesV2"],+ [{"registry_name": "reg-a", "name": "web", "tag_count": 3}],+ id="parent_registry_name_does_not_replace_repository_name",+ ),+ ],+ )+ def test_child_rows_carry_their_parent_id(+ self,+ endpoint: str,+ script: dict[str, list[ScriptedResponse]],+ expected_paths: list[str],+ expected_rows: list[dict[str, Any]],+ ) -> None:+ result = _driver().run(endpoint, route(script))++ assert result.raised is None+ assert result.paths == expected_paths+ assert result.rows == expected_rows++class TestDigitalOceanValidateCredentials:@pytest.mark.parametrize("status_code,expected",