:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse_sources): add four missing doit api tables (#120217)
Tom Owers16e0824d1b71
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex f9ca631fff31..b954dea9319d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -3925,20 +3925,20 @@ Note: Extracted every GET path from the official Python SDK operation index and## DoIt — **thin**-Today: report tables are discovered from the user's account at sync time, alongside the fixed `Budgets`, `Insights`, `Anomalies`, and `Invoices` tables.+Today: report tables are discovered from the user's account at sync time, alongside the fixed `Budgets`, `Insights`, `InsightResourceResults`, `Anomalies`, `Invoices`, `Assets`, `Annotations`, and `SupportTickets` tables.Diffed against: <https://developer.doit.com/openapi/doit-api.yaml>-- [ ] `Assets (GET /billing/v1/assets)` — identify billing accounts and licensed subscriptions referenced in saved cloud cost reports (high)+- [x] `Assets (GET /billing/v1/assets)` — identify billing accounts and licensed subscriptions referenced in saved cloud cost reports (high)- [x] `Invoices (GET /billing/v1/invoices)` — reconcile cloud spending with actual invoices, outstanding balances, and payment status (high)- [x] `Anomalies (GET /anomalies/v1)` — measure unexpected cloud spending, severity, and anomaly costs (high)- [x] `Budgets (GET /analytics/v1/budgets)` — compare budget limits with current utilization, forecasts, and risk status (high)- [x] `Insights (GET /insights/v1/results)` — track optimization opportunities, estimated savings, and operational risks (high)-- [ ] `Insight resource results (GET /insights/v1/results/source/{sourceID}/insight/{insightKey}/resource-results)` — identify cloud resources responsible for savings opportunities and risks (medium)+- [x] `Insight resource results (GET /insights/v1/results/source/{sourceID}/insight/{insightKey}/resource-results)` — identify cloud resources responsible for savings opportunities and risks (medium)- [ ] `Commitments (GET /analytics/v1/commitment-manager)` — compare contractual cloud commitments with current attainment and forecast spending (medium)- [ ] `Allocations (GET /analytics/v1/allocations)` — explain how cloud spending is assigned to teams, products, and cost centers (medium)-- [ ] `Annotations (GET /analytics/v1/annotations)` — relate spending changes to recorded operational events and report annotations (medium)-- [ ] `Support requests (GET /support/v1/tickets)` — analyze support demand, resolution times, and ticket status alongside cloud spending (medium)+- [x] `Annotations (GET /analytics/v1/annotations)` — relate spending changes to recorded operational events and report annotations (medium)+- [x] `Support requests (GET /support/v1/tickets)` — analyze support demand, resolution times, and ticket status alongside cloud spending (medium)Note: Parsed every GET operation in the official OpenAPI document and checked selected billing and analytics schemas. Source code creates one table per accessible saved report, using its normalized name; no static names exist without credentials. The source lists reports for discovery and imports their result rows, so today remains empty. Insight resources need parent fan-out; IAM, themes, auth, write operations, and partner provisioning were excluded.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/canonical_descriptions.pyindex 6d9813322e4a..7f0fa5dc90cb 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/canonical_descriptions.py@@ -84,4 +84,68 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"currency": "Currency of the amounts.",},},+ "Assets": {+ "description": "A DoiT asset: a cloud billing account or licensed subscription, such as a Google Workspace or Microsoft 365 subscription.",+ "docs_url": "https://developer.doit.com/openapi/doit-api.yaml",+ "columns": {+ "id": "Unique identifier of the asset.",+ "name": "Readable label for the asset, usually the product name with the domain or account ID.",+ "type": "Asset type, for example `g-suite`, `office-365`, `google-cloud`, or `amazon-web-services`.",+ "quantity": "Number of licenses or seats assigned to the asset.",+ "create_time": "Time the asset was created, in milliseconds since the epoch.",+ "url": "Link to the asset details page in the DoiT console.",+ },+ },+ "Annotations": {+ "description": "A DoiT annotation: a note attached to a date in cloud cost reports, for example a deployment or a pricing change.",+ "docs_url": "https://developer.doit.com/openapi/doit-api.yaml",+ "columns": {+ "id": "Unique identifier of the annotation.",+ "content": "Text of the annotation.",+ "timestamp": "Date the annotation refers to.",+ "reports": "IDs of the reports the annotation is attached to.",+ "labels": "Labels on the annotation, each with an `id` and a `name`.",+ "create_time": "Time the annotation was created.",+ "update_time": "Time the annotation was last updated.",+ },+ },+ "SupportTickets": {+ "description": "A support request opened with DoiT, with its severity, status, and the product it concerns.",+ "docs_url": "https://developer.doit.com/openapi/doit-api.yaml",+ "columns": {+ "id": "Ticket ID.",+ "subject": "Subject of the ticket.",+ "requester": "Email address of the person who opened the ticket.",+ "severity": "Severity of the ticket.",+ "status": "Status of the ticket.",+ "platform": "Platform the ticket is about.",+ "product": "Product the ticket is about.",+ "is_public": "Whether the ticket is public.",+ "tags": "Tags on the ticket.",+ "create_time": "Time the ticket was created, in milliseconds since the epoch.",+ "update_time": "Time the ticket was last updated, in milliseconds since the epoch.",+ "url_ui": "Link to the ticket in the DoiT console.",+ },+ },+ "InsightResourceResults": {+ "description": "A cloud resource affected by a DoiT insight, with the savings or risk the insight finds for that resource.",+ "docs_url": "https://developer.doit.com/openapi/doit-api.yaml",+ "columns": {+ "insight_source": "Source of the insight the result belongs to. Joins to `source` in the insights table.",+ "insight_key": "Key of the insight the result belongs to. Joins to `key` in the insights table.",+ "resource_id": "Cloud resource identifier.",+ "resource_type": "Kind of resource, for example `instance` for an EC2 resource.",+ "cloud_provider": "Cloud provider of the resource.",+ "account": "Cloud account or project ID that contains the resource.",+ "location": "Region or zone of the resource.",+ "resolved": "Whether the issue on this resource is resolved.",+ "severity": "Severity of the finding.",+ "external_id": "ID of the resource in the provider's system.",+ "external_url": "URL of the issue in the provider's system.",+ "result_type": "Kind of result, which sets the fields that `result` contains.",+ "result": "Result data, for example the daily saving amount or the security risk counts.",+ "metadata": "Additional information about the resource result.",+ "enhancement": "DoiT-managed metadata on the result, such as tags and priority.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/doit.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/doit.pyindex 335dec2618f5..23e2384d502c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/doit.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/doit.py@@ -1,7 +1,8 @@import hashlibimport datetime-from typing import Any, Optional-from urllib.parse import urlencode+from collections.abc import Callable+from typing import Any, Optional, cast+from urllib.parse import quote, urlencodeimport pyarrow as paimport structlog@@ -20,8 +21,16 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resRESTAPIConfig,rest_api_resource,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (+ build_chained_resource,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (+ ClientConfig,+ Endpoint,+ EndpointResource,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse-from products.warehouse_sources.backend.temporal.data_imports.sources.doit.settings import DOIT_ENDPOINTS+from products.warehouse_sources.backend.temporal.data_imports.sources.doit.settings import DOIT_ENDPOINTS, DoItEndpointfrom products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.doit import DoItSourceConfigfrom products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType@@ -253,40 +262,92 @@ def doit_source(return SourceResponse(name=report_name, items=lambda: get_rows(report_id), primary_keys=["id"])+def _endpoint_resource(name: str, settings: DoItEndpoint) -> EndpointResource:+ return {+ "name": name,+ "table_name": name.lower(),+ "write_disposition": "replace",+ "endpoint": {+ "path": settings.path,+ "params": dict(settings.params),+ "data_selector": settings.data_selector,+ "data_selector_required": True,+ "paginator": {+ "type": "cursor",+ "cursor_path": settings.cursor_path,+ "cursor_param": "pageToken",+ "param_location": "query",+ "raise_on_repeated_cursor": True,+ },+ },+ "table_format": "delta",+ }+++def _path_segment_field(field: str) -> str:+ return f"{field}_path_segment"+++def _add_path_segments(fields: list[str]) -> Callable[[dict[str, Any]], dict[str, Any]]:+ # The fan-out binds parent values into the path with no escaping, and DoiT insight keys are free+ # strings that can hold `/`, `?` or `#`. The raw values still go into the child columns.+ def _mapper(row: dict[str, Any]) -> dict[str, Any]:+ for field in fields:+ if row.get(field) is not None:+ row[_path_segment_field(field)] = quote(str(row[field]), safe="")+ return row++ return _mapper++def doit_endpoint_source(config: DoItSourceConfig, endpoint: str, team_id: int, job_id: str) -> SourceResponse:settings = DOIT_ENDPOINTS[endpoint]- rest_config: RESTAPIConfig = {- "client": {- "base_url": DOIT_BASE_URL,- "auth": {"type": "bearer", "token": config.api_key},+ client_config: ClientConfig = {+ "base_url": DOIT_BASE_URL,+ "auth": {"type": "bearer", "token": config.api_key},+ }++ if settings.parent is None:+ rest_config: RESTAPIConfig = {+ "client": client_config,+ "resource_defaults": {},+ "resources": [_endpoint_resource(endpoint, settings)],+ }+ resource = rest_api_resource(rest_config, team_id, job_id, None)+ return SourceResponse(+ name=endpoint,+ items=lambda: resource,+ primary_keys=settings.primary_keys,+ column_hints=resource.column_hints,+ )++ parent = settings.parent+ child = _endpoint_resource(endpoint, settings)+ child["include_from_parent"] = list(parent.include_fields)+ child_endpoint = cast(Endpoint, child["endpoint"])+ child_endpoint["params"] = {+ **(child_endpoint.get("params") or {}),+ **{+ param: {"type": "resolve", "resource": parent.name, "field": _path_segment_field(field)}+ for param, field in parent.resolve_fields.items()},- "resource_defaults": {"write_disposition": "replace"},- "resources": [- {- "name": endpoint,- "table_name": endpoint.lower(),- "endpoint": {- "path": settings.path,- "params": dict(settings.params),- "data_selector": settings.data_selector,- "data_selector_required": True,- "paginator": {- "type": "cursor",- "cursor_path": settings.cursor_path,- "cursor_param": "pageToken",- "param_location": "query",- "raise_on_repeated_cursor": True,- },- },- "table_format": "delta",- }- ],}- resource = rest_api_resource(rest_config, team_id, job_id, None)+ # An insight that DoiT drops between the insights listing and this fetch must not sink the run.+ child_endpoint["response_actions"] = [{"status_code": 404, "action": "ignore"}]++ parent_resource = _endpoint_resource(parent.name, DOIT_ENDPOINTS[parent.name])+ parent_resource["data_map"] = _add_path_segments(list(parent.resolve_fields.values()))return SourceResponse(name=endpoint,- items=lambda: resource,+ items=lambda: build_chained_resource(+ resources=[parent_resource, child],+ child_name=endpoint,+ parent_name=parent.name,+ parent_field_renames=parent.include_fields,+ client_config=client_config,+ team_id=team_id,+ job_id=job_id,+ ),primary_keys=settings.primary_keys,- column_hints=resource.column_hints,)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/settings.pyindex 6da8e5407fa4..cadceb0ff157 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/settings.py@@ -3,6 +3,16 @@ from typing import Anyfrom posthog.dataclasses import frozen+@frozen+class DoItParent:+ # Key of the parent endpoint in `DOIT_ENDPOINTS`.+ name: str+ # Path param -> parent row field bound into it.+ resolve_fields: dict[str, str]+ # Parent row field -> child column it is copied into.+ include_fields: dict[str, str]++@frozenclass DoItEndpoint:path: str@@ -12,12 +22,14 @@ class DoItEndpoint:cursor_path: strprimary_keys: list[str]params: dict[str, Any]+ parent: DoItParent | None = None# Fixed list endpoints synced alongside the account's reports. The names are PascalCase so they can# never equal a report schema name, which is always a normalized lowercase identifier.-# All of them are full refresh: budgets, insights, anomalies, and invoices change in place-# (utilization, status, balance), and DoiT has no updated-since filter to pick those changes up.+# All of them are full refresh: their rows change in place (utilization, status, balance, ticket+# status), and DoiT has no updated-since filter to pick those changes up. Support tickets only filter+# on creation time, which would leave the status of older tickets stale.DOIT_ENDPOINTS: dict[str, DoItEndpoint] = {"Budgets": DoItEndpoint(path="/analytics/v1/budgets",@@ -51,4 +63,39 @@ DOIT_ENDPOINTS: dict[str, DoItEndpoint] = {primary_keys=["platform", "id"],params={"maxResults": 50},),+ "Assets": DoItEndpoint(+ path="/billing/v1/assets",+ data_selector="assets",+ cursor_path="pageToken",+ primary_keys=["id"],+ params={"maxResults": 50},+ ),+ "Annotations": DoItEndpoint(+ path="/analytics/v1/annotations",+ data_selector="annotations",+ cursor_path="pageToken",+ primary_keys=["id"],+ # A fixed order keeps page boundaries stable if annotations are added during the sync.+ params={"maxResults": 50, "sortBy": "timeCreated", "sortOrder": "asc"},+ ),+ "SupportTickets": DoItEndpoint(+ path="/support/v1/tickets",+ data_selector="tickets",+ cursor_path="pageToken",+ primary_keys=["id"],+ params={"maxResults": 100},+ ),+ "InsightResourceResults": DoItEndpoint(+ path="/insights/v1/results/source/{sourceID}/insight/{insightKey}/resource-results",+ data_selector="resourceResults",+ cursor_path="pageToken",+ # A resource can carry a result for several insights, so the insight is part of the key.+ primary_keys=["insight_source", "insight_key", "resourceId"],+ params={"maxResults": 1000},+ parent=DoItParent(+ name="Insights",+ resolve_fields={"sourceID": "source", "insightKey": "key"},+ include_fields={"source": "insight_source", "key": "insight_key"},+ ),+ ),}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit.pyindex 781ec792aabc..fd31b2a83084 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit.py@@ -103,6 +103,9 @@ class TestDoitEndpointSource:("Insights", "/insights/v1/results", "results", lambda token: {"pagination": {"pageToken": token}}),("Anomalies", "/anomalies/v1", "anomalies", lambda token: {"pageToken": token}),("Invoices", "/billing/v1/invoices", "invoices", lambda token: {"pageToken": token}),+ ("Assets", "/billing/v1/assets", "assets", lambda token: {"pageToken": token}),+ ("Annotations", "/analytics/v1/annotations", "annotations", lambda token: {"pageToken": token}),+ ("SupportTickets", "/support/v1/tickets", "tickets", lambda token: {"pageToken": token}),],)def test_follows_the_page_token_to_the_last_page(self, endpoint, path, data_key, next_page) -> None:@@ -120,6 +123,34 @@ class TestDoitEndpointSource:assert [row["id"] for row in result.rows] == ["1", "2"]assert result.requests[0].headers["authorization"] == "Bearer key"+ def test_insight_resource_results_fan_out_over_every_insight(self) -> None:+ resource_results = "/insights/v1/results/source/{}/insight/{}/resource-results"+ result = SourceDriver(DoItSource(), CONFIG).run(+ "InsightResourceResults",+ [+ ScriptedResponse(+ json={"results": [{"source": "doit", "key": "idle/vms #2"}, {"source": "aws", "key": "gone"}]}+ ),+ ScriptedResponse(json={"resourceResults": [{"resourceId": "vm-1"}], "pageToken": "tok-2"}),+ ScriptedResponse(json={"resourceResults": [{"resourceId": "vm-2"}]}),+ # An insight removed after the listing is skipped, not fatal.+ ScriptedResponse(status=404, json={"error": "not found"}),+ ],+ )++ assert result.raised is None+ assert result.paths == [+ "/insights/v1/results",+ resource_results.format("doit", "idle%2Fvms%20%232"),+ resource_results.format("doit", "idle%2Fvms%20%232"),+ resource_results.format("aws", "gone"),+ ]+ assert result.params("pageToken") == [None, None, "tok-2", None]+ assert [(row["insight_source"], row["insight_key"], row["resourceId"]) for row in result.rows] == [+ ("doit", "idle/vms #2", "vm-1"),+ ("doit", "idle/vms #2", "vm-2"),+ ]+def test_forbidden_table_fails_as_non_retryable(self) -> None:result = SourceDriver(DoItSource(), CONFIG).run("Invoices", always(ScriptedResponse(status=403, json={"error": "forbidden"}))diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit_source.pyindex 8846c9d58936..4364152011ab 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit_source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit_source.py@@ -49,6 +49,10 @@ class TestDoItSource:("Insights", None, None),("Anomalies", None, None),("Invoices", None, None),+ ("Assets", None, None),+ ("Annotations", None, None),+ ("SupportTickets", None, None),+ ("InsightResourceResults", None, None),]def test_get_schemas_lets_a_report_keep_a_name_a_fixed_table_would_take(self):@@ -59,7 +63,16 @@ class TestDoItSource:):schemas = self.source.get_schemas(CONFIG, team_id=1)- assert [s.name for s in schemas] == ["invoices", "Budgets", "Insights", "Anomalies"]+ assert [s.name for s in schemas] == [+ "invoices",+ "Budgets",+ "Insights",+ "Anomalies",+ "Assets",+ "Annotations",+ "SupportTickets",+ "InsightResourceResults",+ ]def test_source_for_pipeline_fetches_the_report_id_from_schema_metadata(self):inputs = MagicMock(