Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse_sources): add four missing doit api tables (#120217)

Tom Owers16e0824d1b71
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index f9ca631fff31..b954dea9319d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -3925,20 +3925,20 @@ Note: Extracted every GET path from the official Python SDK operation index and
## DoIt — **thin**
-Today: report tables are discovered from the user's account at sync time, alongside the fixed `Budgets`, `Insights`, `Anomalies`, and `Invoices` tables.
+Today: report tables are discovered from the user's account at sync time, alongside the fixed `Budgets`, `Insights`, `InsightResourceResults`, `Anomalies`, `Invoices`, `Assets`, `Annotations`, and `SupportTickets` tables.
Diffed against: <https://developer.doit.com/openapi/doit-api.yaml>
-- [ ] `Assets (GET /billing/v1/assets)` — identify billing accounts and licensed subscriptions referenced in saved cloud cost reports (high)
+- [x] `Assets (GET /billing/v1/assets)` — identify billing accounts and licensed subscriptions referenced in saved cloud cost reports (high)
- [x] `Invoices (GET /billing/v1/invoices)` — reconcile cloud spending with actual invoices, outstanding balances, and payment status (high)
- [x] `Anomalies (GET /anomalies/v1)` — measure unexpected cloud spending, severity, and anomaly costs (high)
- [x] `Budgets (GET /analytics/v1/budgets)` — compare budget limits with current utilization, forecasts, and risk status (high)
- [x] `Insights (GET /insights/v1/results)` — track optimization opportunities, estimated savings, and operational risks (high)
-- [ ] `Insight resource results (GET /insights/v1/results/source/{sourceID}/insight/{insightKey}/resource-results)` — identify cloud resources responsible for savings opportunities and risks (medium)
+- [x] `Insight resource results (GET /insights/v1/results/source/{sourceID}/insight/{insightKey}/resource-results)` — identify cloud resources responsible for savings opportunities and risks (medium)
- [ ] `Commitments (GET /analytics/v1/commitment-manager)` — compare contractual cloud commitments with current attainment and forecast spending (medium)
- [ ] `Allocations (GET /analytics/v1/allocations)` — explain how cloud spending is assigned to teams, products, and cost centers (medium)
-- [ ] `Annotations (GET /analytics/v1/annotations)` — relate spending changes to recorded operational events and report annotations (medium)
-- [ ] `Support requests (GET /support/v1/tickets)` — analyze support demand, resolution times, and ticket status alongside cloud spending (medium)
+- [x] `Annotations (GET /analytics/v1/annotations)` — relate spending changes to recorded operational events and report annotations (medium)
+- [x] `Support requests (GET /support/v1/tickets)` — analyze support demand, resolution times, and ticket status alongside cloud spending (medium)
Note: Parsed every GET operation in the official OpenAPI document and checked selected billing and analytics schemas. Source code creates one table per accessible saved report, using its normalized name; no static names exist without credentials. The source lists reports for discovery and imports their result rows, so today remains empty. Insight resources need parent fan-out; IAM, themes, auth, write operations, and partner provisioning were excluded.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/canonical_descriptions.py
index 6d9813322e4a..7f0fa5dc90cb 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/canonical_descriptions.py
@@ -84,4 +84,68 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"currency": "Currency of the amounts.",
},
},
+ "Assets": {
+ "description": "A DoiT asset: a cloud billing account or licensed subscription, such as a Google Workspace or Microsoft 365 subscription.",
+ "docs_url": "https://developer.doit.com/openapi/doit-api.yaml",
+ "columns": {
+ "id": "Unique identifier of the asset.",
+ "name": "Readable label for the asset, usually the product name with the domain or account ID.",
+ "type": "Asset type, for example `g-suite`, `office-365`, `google-cloud`, or `amazon-web-services`.",
+ "quantity": "Number of licenses or seats assigned to the asset.",
+ "create_time": "Time the asset was created, in milliseconds since the epoch.",
+ "url": "Link to the asset details page in the DoiT console.",
+ },
+ },
+ "Annotations": {
+ "description": "A DoiT annotation: a note attached to a date in cloud cost reports, for example a deployment or a pricing change.",
+ "docs_url": "https://developer.doit.com/openapi/doit-api.yaml",
+ "columns": {
+ "id": "Unique identifier of the annotation.",
+ "content": "Text of the annotation.",
+ "timestamp": "Date the annotation refers to.",
+ "reports": "IDs of the reports the annotation is attached to.",
+ "labels": "Labels on the annotation, each with an `id` and a `name`.",
+ "create_time": "Time the annotation was created.",
+ "update_time": "Time the annotation was last updated.",
+ },
+ },
+ "SupportTickets": {
+ "description": "A support request opened with DoiT, with its severity, status, and the product it concerns.",
+ "docs_url": "https://developer.doit.com/openapi/doit-api.yaml",
+ "columns": {
+ "id": "Ticket ID.",
+ "subject": "Subject of the ticket.",
+ "requester": "Email address of the person who opened the ticket.",
+ "severity": "Severity of the ticket.",
+ "status": "Status of the ticket.",
+ "platform": "Platform the ticket is about.",
+ "product": "Product the ticket is about.",
+ "is_public": "Whether the ticket is public.",
+ "tags": "Tags on the ticket.",
+ "create_time": "Time the ticket was created, in milliseconds since the epoch.",
+ "update_time": "Time the ticket was last updated, in milliseconds since the epoch.",
+ "url_ui": "Link to the ticket in the DoiT console.",
+ },
+ },
+ "InsightResourceResults": {
+ "description": "A cloud resource affected by a DoiT insight, with the savings or risk the insight finds for that resource.",
+ "docs_url": "https://developer.doit.com/openapi/doit-api.yaml",
+ "columns": {
+ "insight_source": "Source of the insight the result belongs to. Joins to `source` in the insights table.",
+ "insight_key": "Key of the insight the result belongs to. Joins to `key` in the insights table.",
+ "resource_id": "Cloud resource identifier.",
+ "resource_type": "Kind of resource, for example `instance` for an EC2 resource.",
+ "cloud_provider": "Cloud provider of the resource.",
+ "account": "Cloud account or project ID that contains the resource.",
+ "location": "Region or zone of the resource.",
+ "resolved": "Whether the issue on this resource is resolved.",
+ "severity": "Severity of the finding.",
+ "external_id": "ID of the resource in the provider's system.",
+ "external_url": "URL of the issue in the provider's system.",
+ "result_type": "Kind of result, which sets the fields that `result` contains.",
+ "result": "Result data, for example the daily saving amount or the security risk counts.",
+ "metadata": "Additional information about the resource result.",
+ "enhancement": "DoiT-managed metadata on the result, such as tags and priority.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/doit.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/doit.py
index 335dec2618f5..23e2384d502c 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/doit.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/doit.py
@@ -1,7 +1,8 @@
import hashlib
import datetime
-from typing import Any, Optional
-from urllib.parse import urlencode
+from collections.abc import Callable
+from typing import Any, Optional, cast
+from urllib.parse import quote, urlencode
import pyarrow as pa
import structlog
@@ -20,8 +21,16 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
RESTAPIConfig,
rest_api_resource,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
+ build_chained_resource,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (
+ ClientConfig,
+ Endpoint,
+ EndpointResource,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
-from products.warehouse_sources.backend.temporal.data_imports.sources.doit.settings import DOIT_ENDPOINTS
+from products.warehouse_sources.backend.temporal.data_imports.sources.doit.settings import DOIT_ENDPOINTS, DoItEndpoint
from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.doit import DoItSourceConfig
from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType
@@ -253,40 +262,92 @@ def doit_source(
return SourceResponse(name=report_name, items=lambda: get_rows(report_id), primary_keys=["id"])
+def _endpoint_resource(name: str, settings: DoItEndpoint) -> EndpointResource:
+ return {
+ "name": name,
+ "table_name": name.lower(),
+ "write_disposition": "replace",
+ "endpoint": {
+ "path": settings.path,
+ "params": dict(settings.params),
+ "data_selector": settings.data_selector,
+ "data_selector_required": True,
+ "paginator": {
+ "type": "cursor",
+ "cursor_path": settings.cursor_path,
+ "cursor_param": "pageToken",
+ "param_location": "query",
+ "raise_on_repeated_cursor": True,
+ },
+ },
+ "table_format": "delta",
+ }
+
+
+def _path_segment_field(field: str) -> str:
+ return f"{field}_path_segment"
+
+
+def _add_path_segments(fields: list[str]) -> Callable[[dict[str, Any]], dict[str, Any]]:
+ # The fan-out binds parent values into the path with no escaping, and DoiT insight keys are free
+ # strings that can hold `/`, `?` or `#`. The raw values still go into the child columns.
+ def _mapper(row: dict[str, Any]) -> dict[str, Any]:
+ for field in fields:
+ if row.get(field) is not None:
+ row[_path_segment_field(field)] = quote(str(row[field]), safe="")
+ return row
+
+ return _mapper
+
+
def doit_endpoint_source(config: DoItSourceConfig, endpoint: str, team_id: int, job_id: str) -> SourceResponse:
settings = DOIT_ENDPOINTS[endpoint]
- rest_config: RESTAPIConfig = {
- "client": {
- "base_url": DOIT_BASE_URL,
- "auth": {"type": "bearer", "token": config.api_key},
+ client_config: ClientConfig = {
+ "base_url": DOIT_BASE_URL,
+ "auth": {"type": "bearer", "token": config.api_key},
+ }
+
+ if settings.parent is None:
+ rest_config: RESTAPIConfig = {
+ "client": client_config,
+ "resource_defaults": {},
+ "resources": [_endpoint_resource(endpoint, settings)],
+ }
+ resource = rest_api_resource(rest_config, team_id, job_id, None)
+ return SourceResponse(
+ name=endpoint,
+ items=lambda: resource,
+ primary_keys=settings.primary_keys,
+ column_hints=resource.column_hints,
+ )
+
+ parent = settings.parent
+ child = _endpoint_resource(endpoint, settings)
+ child["include_from_parent"] = list(parent.include_fields)
+ child_endpoint = cast(Endpoint, child["endpoint"])
+ child_endpoint["params"] = {
+ **(child_endpoint.get("params") or {}),
+ **{
+ param: {"type": "resolve", "resource": parent.name, "field": _path_segment_field(field)}
+ for param, field in parent.resolve_fields.items()
},
- "resource_defaults": {"write_disposition": "replace"},
- "resources": [
- {
- "name": endpoint,
- "table_name": endpoint.lower(),
- "endpoint": {
- "path": settings.path,
- "params": dict(settings.params),
- "data_selector": settings.data_selector,
- "data_selector_required": True,
- "paginator": {
- "type": "cursor",
- "cursor_path": settings.cursor_path,
- "cursor_param": "pageToken",
- "param_location": "query",
- "raise_on_repeated_cursor": True,
- },
- },
- "table_format": "delta",
- }
- ],
}
- resource = rest_api_resource(rest_config, team_id, job_id, None)
+ # An insight that DoiT drops between the insights listing and this fetch must not sink the run.
+ child_endpoint["response_actions"] = [{"status_code": 404, "action": "ignore"}]
+
+ parent_resource = _endpoint_resource(parent.name, DOIT_ENDPOINTS[parent.name])
+ parent_resource["data_map"] = _add_path_segments(list(parent.resolve_fields.values()))
return SourceResponse(
name=endpoint,
- items=lambda: resource,
+ items=lambda: build_chained_resource(
+ resources=[parent_resource, child],
+ child_name=endpoint,
+ parent_name=parent.name,
+ parent_field_renames=parent.include_fields,
+ client_config=client_config,
+ team_id=team_id,
+ job_id=job_id,
+ ),
primary_keys=settings.primary_keys,
- column_hints=resource.column_hints,
)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/settings.py
index 6da8e5407fa4..cadceb0ff157 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/settings.py
@@ -3,6 +3,16 @@ from typing import Any
from posthog.dataclasses import frozen
+@frozen
+class DoItParent:
+ # Key of the parent endpoint in `DOIT_ENDPOINTS`.
+ name: str
+ # Path param -> parent row field bound into it.
+ resolve_fields: dict[str, str]
+ # Parent row field -> child column it is copied into.
+ include_fields: dict[str, str]
+
+
@frozen
class DoItEndpoint:
path: str
@@ -12,12 +22,14 @@ class DoItEndpoint:
cursor_path: str
primary_keys: list[str]
params: dict[str, Any]
+ parent: DoItParent | None = None
# Fixed list endpoints synced alongside the account's reports. The names are PascalCase so they can
# never equal a report schema name, which is always a normalized lowercase identifier.
-# All of them are full refresh: budgets, insights, anomalies, and invoices change in place
-# (utilization, status, balance), and DoiT has no updated-since filter to pick those changes up.
+# All of them are full refresh: their rows change in place (utilization, status, balance, ticket
+# status), and DoiT has no updated-since filter to pick those changes up. Support tickets only filter
+# on creation time, which would leave the status of older tickets stale.
DOIT_ENDPOINTS: dict[str, DoItEndpoint] = {
"Budgets": DoItEndpoint(
path="/analytics/v1/budgets",
@@ -51,4 +63,39 @@ DOIT_ENDPOINTS: dict[str, DoItEndpoint] = {
primary_keys=["platform", "id"],
params={"maxResults": 50},
),
+ "Assets": DoItEndpoint(
+ path="/billing/v1/assets",
+ data_selector="assets",
+ cursor_path="pageToken",
+ primary_keys=["id"],
+ params={"maxResults": 50},
+ ),
+ "Annotations": DoItEndpoint(
+ path="/analytics/v1/annotations",
+ data_selector="annotations",
+ cursor_path="pageToken",
+ primary_keys=["id"],
+ # A fixed order keeps page boundaries stable if annotations are added during the sync.
+ params={"maxResults": 50, "sortBy": "timeCreated", "sortOrder": "asc"},
+ ),
+ "SupportTickets": DoItEndpoint(
+ path="/support/v1/tickets",
+ data_selector="tickets",
+ cursor_path="pageToken",
+ primary_keys=["id"],
+ params={"maxResults": 100},
+ ),
+ "InsightResourceResults": DoItEndpoint(
+ path="/insights/v1/results/source/{sourceID}/insight/{insightKey}/resource-results",
+ data_selector="resourceResults",
+ cursor_path="pageToken",
+ # A resource can carry a result for several insights, so the insight is part of the key.
+ primary_keys=["insight_source", "insight_key", "resourceId"],
+ params={"maxResults": 1000},
+ parent=DoItParent(
+ name="Insights",
+ resolve_fields={"sourceID": "source", "insightKey": "key"},
+ include_fields={"source": "insight_source", "key": "insight_key"},
+ ),
+ ),
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit.py
index 781ec792aabc..fd31b2a83084 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit.py
@@ -103,6 +103,9 @@ class TestDoitEndpointSource:
("Insights", "/insights/v1/results", "results", lambda token: {"pagination": {"pageToken": token}}),
("Anomalies", "/anomalies/v1", "anomalies", lambda token: {"pageToken": token}),
("Invoices", "/billing/v1/invoices", "invoices", lambda token: {"pageToken": token}),
+ ("Assets", "/billing/v1/assets", "assets", lambda token: {"pageToken": token}),
+ ("Annotations", "/analytics/v1/annotations", "annotations", lambda token: {"pageToken": token}),
+ ("SupportTickets", "/support/v1/tickets", "tickets", lambda token: {"pageToken": token}),
],
)
def test_follows_the_page_token_to_the_last_page(self, endpoint, path, data_key, next_page) -> None:
@@ -120,6 +123,34 @@ class TestDoitEndpointSource:
assert [row["id"] for row in result.rows] == ["1", "2"]
assert result.requests[0].headers["authorization"] == "Bearer key"
+ def test_insight_resource_results_fan_out_over_every_insight(self) -> None:
+ resource_results = "/insights/v1/results/source/{}/insight/{}/resource-results"
+ result = SourceDriver(DoItSource(), CONFIG).run(
+ "InsightResourceResults",
+ [
+ ScriptedResponse(
+ json={"results": [{"source": "doit", "key": "idle/vms #2"}, {"source": "aws", "key": "gone"}]}
+ ),
+ ScriptedResponse(json={"resourceResults": [{"resourceId": "vm-1"}], "pageToken": "tok-2"}),
+ ScriptedResponse(json={"resourceResults": [{"resourceId": "vm-2"}]}),
+ # An insight removed after the listing is skipped, not fatal.
+ ScriptedResponse(status=404, json={"error": "not found"}),
+ ],
+ )
+
+ assert result.raised is None
+ assert result.paths == [
+ "/insights/v1/results",
+ resource_results.format("doit", "idle%2Fvms%20%232"),
+ resource_results.format("doit", "idle%2Fvms%20%232"),
+ resource_results.format("aws", "gone"),
+ ]
+ assert result.params("pageToken") == [None, None, "tok-2", None]
+ assert [(row["insight_source"], row["insight_key"], row["resourceId"]) for row in result.rows] == [
+ ("doit", "idle/vms #2", "vm-1"),
+ ("doit", "idle/vms #2", "vm-2"),
+ ]
+
def test_forbidden_table_fails_as_non_retryable(self) -> None:
result = SourceDriver(DoItSource(), CONFIG).run(
"Invoices", always(ScriptedResponse(status=403, json={"error": "forbidden"}))
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit_source.py
index 8846c9d58936..4364152011ab 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit_source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit_source.py
@@ -49,6 +49,10 @@ class TestDoItSource:
("Insights", None, None),
("Anomalies", None, None),
("Invoices", None, None),
+ ("Assets", None, None),
+ ("Annotations", None, None),
+ ("SupportTickets", None, None),
+ ("InsightResourceResults", None, None),
]
def test_get_schemas_lets_a_report_keep_a_name_a_fixed_table_would_take(self):
@@ -59,7 +63,16 @@ class TestDoItSource:
):
schemas = self.source.get_schemas(CONFIG, team_id=1)
- assert [s.name for s in schemas] == ["invoices", "Budgets", "Insights", "Anomalies"]
+ assert [s.name for s in schemas] == [
+ "invoices",
+ "Budgets",
+ "Insights",
+ "Anomalies",
+ "Assets",
+ "Annotations",
+ "SupportTickets",
+ "InsightResourceResults",
+ ]
def test_source_for_pipeline_fetches_the_report_id_from_schema_metadata(self):
inputs = MagicMock(