:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse): add budgets, insights, anomalies and invoices tables to doit (#119549)
Tom Owers670fb8c9be35
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 52f78455754c..fb4bdf2623f8 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -3925,15 +3925,15 @@ Note: Extracted every GET path from the official Python SDK operation index and## DoIt — **thin**-Today: tables are discovered from the user's account at sync time, so there is no fixed list.+Today: report tables are discovered from the user's account at sync time, alongside the fixed `Budgets`, `Insights`, `Anomalies`, and `Invoices` tables.Diffed against: <https://developer.doit.com/openapi/doit-api.yaml>- [ ] `Assets (GET /billing/v1/assets)` — identify billing accounts and licensed subscriptions referenced in saved cloud cost reports (high)-- [ ] `Invoices (GET /billing/v1/invoices)` — reconcile cloud spending with actual invoices, outstanding balances, and payment status (high)-- [ ] `Anomalies (GET /anomalies/v1)` — measure unexpected cloud spending, severity, and anomaly costs (high)-- [ ] `Budgets (GET /analytics/v1/budgets)` — compare budget limits with current utilization, forecasts, and risk status (high)-- [ ] `Insights (GET /insights/v1/results)` — track optimization opportunities, estimated savings, and operational risks (high)+- [x] `Invoices (GET /billing/v1/invoices)` — reconcile cloud spending with actual invoices, outstanding balances, and payment status (high)+- [x] `Anomalies (GET /anomalies/v1)` — measure unexpected cloud spending, severity, and anomaly costs (high)+- [x] `Budgets (GET /analytics/v1/budgets)` — compare budget limits with current utilization, forecasts, and risk status (high)+- [x] `Insights (GET /insights/v1/results)` — track optimization opportunities, estimated savings, and operational risks (high)- [ ] `Insight resource results (GET /insights/v1/results/source/{sourceID}/insight/{insightKey}/resource-results)` — identify cloud resources responsible for savings opportunities and risks (medium)- [ ] `Commitments (GET /analytics/v1/commitment-manager)` — compare contractual cloud commitments with current attainment and forecast spending (medium)- [ ] `Allocations (GET /analytics/v1/allocations)` — explain how cloud spending is assigned to teams, products, and cost centers (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/SOURCES.md b/products/warehouse_sources/backend/temporal/data_imports/sources/SOURCES.mdindex 6f09fefac244..84d65fb422b3 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/SOURCES.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/SOURCES.md@@ -297,7 +297,7 @@ the row lists both.| docuseal | HTTP | requests | ✅ || docusign | HTTP | requests | ✅ || dodopayments | HTTP | requests + `rest_source.RESTClient` | ✅ |-| doit | HTTP | requests | ✅ |+| doit | HTTP | requests + `rest_source.RESTClient` | ✅ || donorbox | HTTP | requests | ✅ || doppler | HTTP | requests | ✅ || dovetail | HTTP | requests + `rest_source.RESTClient` | ✅ |diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/canonical_descriptions.pynew file mode 100644index 000000000000..6d9813322e4a--- /dev/null+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/canonical_descriptions.py@@ -0,0 +1,87 @@+"""Canonical, documentation-sourced descriptions for the fixed DoIt tables.++Sourced from the official DoiT API reference (https://developer.doit.com/openapi/doit-api.yaml).+Keyed by the names in `settings.py` `DOIT_ENDPOINTS`. Report tables have account-specific columns,+so they have no entry here and fall back to LLM enrichment.+"""++from products.warehouse_sources.backend.temporal.data_imports.sources.common.canonical_descriptions import (+ CanonicalDescriptions,+)++CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {+ "Budgets": {+ "description": (+ "A DoiT budget: a spend limit for a scope of cloud costs over a period, with its current "+ "utilization, forecast, and risk status."+ ),+ "docs_url": "https://developer.doit.com/reference/listbudgets",+ "columns": {+ "id": "Unique identifier for the budget.",+ "budget_name": "Name of the budget.",+ "amount": "Budget limit for the period, in the budget currency.",+ "currency": "Currency of the budget amount.",+ "current_utilization": "Spend so far in the current budget period.",+ "forecasted_utilization_date": "Forecasted time the budget is reached, in milliseconds since the epoch.",+ "risk_status": "Server-computed risk classification: `atRisk`, `onTrack`, or `unknown`.",+ "scopes": "The filters that define which costs count toward the budget.",+ "time_interval": "Length of each budget period.",+ "start_period": "Start of the budget, in milliseconds since the epoch.",+ "end_period": "End of the budget, in milliseconds since the epoch.",+ "create_time": "Time the budget was created, in milliseconds since the epoch.",+ "update_time": "Time the budget was last changed, in milliseconds since the epoch.",+ },+ },+ "Insights": {+ "description": (+ "A DoiT insight: an optimization opportunity or risk found in the cloud environment, with "+ "aggregate savings and risk counts across its affected resources."+ ),+ "docs_url": "https://developer.doit.com/openapi/doit-api.yaml",+ "columns": {+ "source": "The source that generated the insight.",+ "key": "Key that identifies the insight within its source.",+ "title": "Display title of the insight.",+ "short_description": "Brief summary of the insight.",+ "display_status": "Display status, for example `actionable`, `optimized`, or `dismissed`.",+ "cloud_provider": "Cloud provider of the affected resources.",+ "categories": "Categories the insight belongs to, for example `FinOps` or `Security`.",+ "summary": "Aggregate summary of risks and savings across all resource results for the insight.",+ "last_updated": "Time of the last update to the insight.",+ },+ },+ "Anomalies": {+ "description": (+ "A cloud cost anomaly DoiT detected: spend above the expected normal range for a service or SKU."+ ),+ "docs_url": "https://developer.doit.com/reference/listanomalies",+ "columns": {+ "id": "Unique identifier for the anomaly.",+ "platform": "Cloud provider name.",+ "billing_account": "Billing account ID.",+ "service_name": "Service the anomaly was detected on.",+ "severity_level": "Severity level: `information`, `warning`, or `critical`.",+ "cost_of_anomaly": "Excess cost above the expected normal cost.",+ "actual_cost": "Observed cost during the anomaly.",+ "expected_max_cost": "Maximum cost within the expected normal range.",+ "start_time": "Usage start time of the anomaly, in milliseconds since the epoch.",+ "end_time": "End of the anomaly, in milliseconds since the epoch.",+ "status": "Whether the anomaly is `active` or `inactive`.",+ "acknowledged": "Whether a user has acknowledged the anomaly.",+ },+ },+ "Invoices": {+ "description": "A current or historical DoiT invoice, with its total, outstanding balance, and payment status.",+ "docs_url": "https://developer.doit.com/reference/listinvoices",+ "columns": {+ "id": "Invoice number.",+ "platform": "Platform the invoice is for, for example `google-cloud` or `amazon-web-services`.",+ "invoice_date": "Time the invoice was issued, in milliseconds since the epoch.",+ "due_date": "Last day to pay the invoice, in milliseconds since the epoch.",+ "status": "Invoice status: `OPEN`, `PAST DUE`, or `PAID`.",+ "total_amount": "Total invoiced amount.",+ "balance_amount": "Invoice balance still to be paid.",+ "currency": "Currency of the amounts.",+ },+ },+}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/doit.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/doit.pyindex fbe0e3cabbab..335dec2618f5 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/doit.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/doit.py@@ -16,7 +16,12 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.httDEFAULT_RETRY,make_tracked_session,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source import (+ RESTAPIConfig,+ rest_api_resource,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse+from products.warehouse_sources.backend.temporal.data_imports.sources.doit.settings import DOIT_ENDPOINTSfrom products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.doit import DoItSourceConfigfrom products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType@@ -40,7 +45,8 @@ REPORT_TIMEOUT_SECONDS = (DOIT_CONNECT_TIMEOUT_SECONDS, 300)# Key under a schema's persisted `schema_metadata` holding the DoIt report id.REPORT_ID_METADATA_KEY = "report_id"-DOIT_REPORTS_URL = "https://api.doit.com/analytics/v1/reports"+DOIT_BASE_URL = "https://api.doit.com"+DOIT_REPORTS_URL = f"{DOIT_BASE_URL}/analytics/v1/reports"# The listing endpoint paginates at 50 reports per page by default, signalling more pages via a# `pageToken` in the response body. The cap is a safety valve against a server that keeps returning@@ -245,3 +251,42 @@ def doit_source(yield table_from_iterator((append_primary_key(dict(zip(column_names, row))) for row in rows), arrow_schema)return SourceResponse(name=report_name, items=lambda: get_rows(report_id), primary_keys=["id"])+++def doit_endpoint_source(config: DoItSourceConfig, endpoint: str, team_id: int, job_id: str) -> SourceResponse:+ settings = DOIT_ENDPOINTS[endpoint]+ rest_config: RESTAPIConfig = {+ "client": {+ "base_url": DOIT_BASE_URL,+ "auth": {"type": "bearer", "token": config.api_key},+ },+ "resource_defaults": {"write_disposition": "replace"},+ "resources": [+ {+ "name": endpoint,+ "table_name": endpoint.lower(),+ "endpoint": {+ "path": settings.path,+ "params": dict(settings.params),+ "data_selector": settings.data_selector,+ "data_selector_required": True,+ "paginator": {+ "type": "cursor",+ "cursor_path": settings.cursor_path,+ "cursor_param": "pageToken",+ "param_location": "query",+ "raise_on_repeated_cursor": True,+ },+ },+ "table_format": "delta",+ }+ ],+ }+ resource = rest_api_resource(rest_config, team_id, job_id, None)++ return SourceResponse(+ name=endpoint,+ items=lambda: resource,+ primary_keys=settings.primary_keys,+ column_hints=resource.column_hints,+ )diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/settings.pynew file mode 100644index 000000000000..6da8e5407fa4--- /dev/null+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/settings.py@@ -0,0 +1,54 @@+from typing import Any++from posthog.dataclasses import frozen+++@frozen+class DoItEndpoint:+ path: str+ # Key of the row array in the response envelope.+ data_selector: str+ # Where the next-page token sits in the response body.+ cursor_path: str+ primary_keys: list[str]+ params: dict[str, Any]+++# Fixed list endpoints synced alongside the account's reports. The names are PascalCase so they can+# never equal a report schema name, which is always a normalized lowercase identifier.+# All of them are full refresh: budgets, insights, anomalies, and invoices change in place+# (utilization, status, balance), and DoiT has no updated-since filter to pick those changes up.+DOIT_ENDPOINTS: dict[str, DoItEndpoint] = {+ "Budgets": DoItEndpoint(+ path="/analytics/v1/budgets",+ data_selector="budgets",+ cursor_path="pageToken",+ primary_keys=["id"],+ params={"maxResults": 50},+ ),+ "Insights": DoItEndpoint(+ path="/insights/v1/results",+ data_selector="results",+ cursor_path="pagination.pageToken",+ # The resource-results path addresses an insight by source and key, so a key alone is not+ # documented as unique.+ primary_keys=["source", "key"],+ params={"maxResults": 500},+ ),+ "Anomalies": DoItEndpoint(+ path="/anomalies/v1",+ data_selector="anomalies",+ cursor_path="pageToken",+ primary_keys=["id"],+ # Without `maxResults` DoiT returns every anomaly in a single response.+ params={"maxResults": 50},+ ),+ "Invoices": DoItEndpoint(+ path="/billing/v1/invoices",+ data_selector="invoices",+ cursor_path="pageToken",+ # The id is the invoice number, which each billing platform issues on its own.+ primary_keys=["platform", "id"],+ params={"maxResults": 50},+ ),+}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/source.pyindex 7b94df439602..c4b094f8dfe3 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/source.py@@ -7,17 +7,23 @@ from products.warehouse_sources.backend.facade.source_config import (SourceFieldInputConfig,SourceFieldInputConfigType,)+from products.warehouse_sources.backend.temporal.data_imports.naming_convention import NamingConventionfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.base import FieldType, SimpleSource+from products.warehouse_sources.backend.temporal.data_imports.sources.common.canonical_descriptions import (+ CanonicalDescriptions,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistryfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import SourceSchemafrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceInputs, SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.doit.doit import (DOIT_INCREMENTAL_FIELDS,REPORT_ID_METADATA_KEY,+ doit_endpoint_source,doit_list_reports,doit_source,resolve_report_id,)+from products.warehouse_sources.backend.temporal.data_imports.sources.doit.settings import DOIT_ENDPOINTSfrom products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.doit import DoItSourceConfigfrom products.warehouse_sources.backend.types import ExternalDataSourceType@@ -57,6 +63,15 @@ class DoItSource(SimpleSource[DoItSourceConfig]):for report in reports]+ # A report and a fixed table with the same normalized name would land in the same warehouse+ # table. The report was there first, so it keeps the name.+ report_names = {report.name for report in reports}+ schemas.extend(+ SourceSchema(name=endpoint, supports_incremental=False, supports_append=False, incremental_fields=[])+ for endpoint in DOIT_ENDPOINTS+ if NamingConvention.normalize_identifier(endpoint) not in report_names+ )+if names is not None:names_set = set(names)schemas = [s for s in schemas if s.name in names_set]@@ -77,8 +92,18 @@ class DoItSource(SimpleSource[DoItSourceConfig]):# DoIt answers 403 with this text for a key it no longer accepts. Only a new key fixes it,# so a retry cannot succeed."invalid token: missing expiration": bad_key,+ # The fixed tables go through the shared REST client, which raises `requests` HTTP errors.+ "401 Client Error: Unauthorized for url: https://api.doit.com": bad_key,+ "403 Client Error: Forbidden for url: https://api.doit.com": "DoIt refused access to this table. Check that your API key is still valid and that its user has permission to read this data in DoIt, then reconnect or deselect the table.",}+ def get_canonical_descriptions(self) -> CanonicalDescriptions:+ from products.warehouse_sources.backend.temporal.data_imports.sources.doit.canonical_descriptions import (+ CANONICAL_DESCRIPTIONS,+ )++ return CANONICAL_DESCRIPTIONS+def get_retryable_errors(self) -> set[str]:# `DOIT_RETRY` already retries a 429 at the transport level; this is DoIt's own account-wide# concurrency quota (multiple schemas of one source can query reports at once), which can@@ -89,6 +114,9 @@ class DoItSource(SimpleSource[DoItSourceConfig]):return {"exhausted; please wait for previous requests to complete"}def source_for_pipeline(self, config: DoItSourceConfig, inputs: SourceInputs) -> SourceResponse:+ if inputs.schema_name in DOIT_ENDPOINTS:+ return doit_endpoint_source(config, inputs.schema_name, inputs.team_id, inputs.job_id)+report_id = resolve_report_id(config, inputs.schema_name, inputs.schema_metadata, logger=inputs.logger)return doit_source(config,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit.pyindex 7b610e71d69a..781ec792aabc 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit.py@@ -4,8 +4,10 @@ from typing import Any, castimport pytestfrom unittest.mock import MagicMock, patch+from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import error_message_matchesfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.testing import (ScriptedResponse,+ SourceDriver,always,scripted_network,)@@ -16,6 +18,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.doit.doitdoit_source,resolve_report_id,)+from products.warehouse_sources.backend.temporal.data_imports.sources.doit.source import DoItSourcefrom products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.doit import DoItSourceConfig_MODULE = "products.warehouse_sources.backend.temporal.data_imports.sources.doit.doit"@@ -90,3 +93,37 @@ class TestRequestTimeouts:list(cast(Iterable[Any], source.items()))assert mock_session.return_value.get.call_args.kwargs["timeout"] == REPORT_TIMEOUT_SECONDS+++class TestDoitEndpointSource:+ @pytest.mark.parametrize(+ "endpoint,path,data_key,next_page",+ [+ ("Budgets", "/analytics/v1/budgets", "budgets", lambda token: {"pageToken": token}),+ ("Insights", "/insights/v1/results", "results", lambda token: {"pagination": {"pageToken": token}}),+ ("Anomalies", "/anomalies/v1", "anomalies", lambda token: {"pageToken": token}),+ ("Invoices", "/billing/v1/invoices", "invoices", lambda token: {"pageToken": token}),+ ],+ )+ def test_follows_the_page_token_to_the_last_page(self, endpoint, path, data_key, next_page) -> None:+ result = SourceDriver(DoItSource(), CONFIG).run(+ endpoint,+ [+ ScriptedResponse(json={data_key: [{"id": "1"}], **next_page("tok-2")}),+ ScriptedResponse(json={data_key: [{"id": "2"}]}),+ ],+ )++ assert result.raised is None+ assert result.paths == [path, path]+ assert result.params("pageToken") == [None, "tok-2"]+ assert [row["id"] for row in result.rows] == ["1", "2"]+ assert result.requests[0].headers["authorization"] == "Bearer key"++ def test_forbidden_table_fails_as_non_retryable(self) -> None:+ result = SourceDriver(DoItSource(), CONFIG).run(+ "Invoices", always(ScriptedResponse(status=403, json={"error": "forbidden"}))+ )++ assert result.raised is not None+ assert error_message_matches(str(result.raised), DoItSource().get_non_retryable_errors().keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit_source.pyindex 25977aa87034..8846c9d58936 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit_source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/doit/tests/test_doit_source.py@@ -44,9 +44,23 @@ class TestDoItSource:schemas = self.source.get_schemas(CONFIG, team_id=1)assert [(s.name, s.label, s.schema_metadata) for s in schemas] == [- ("cost_by_product", "Cost by product", {"report_id": "r1"})+ ("cost_by_product", "Cost by product", {"report_id": "r1"}),+ ("Budgets", None, None),+ ("Insights", None, None),+ ("Anomalies", None, None),+ ("Invoices", None, None),]+ def test_get_schemas_lets_a_report_keep_a_name_a_fixed_table_would_take(self):+ # Both would normalize to the same warehouse table, and the report's table already exists.+ with patch(+ f"{_SOURCE_MODULE}.doit_list_reports",+ return_value=[DoItReport(id="r1", name="invoices", report_name="Invoices")],+ ):+ schemas = self.source.get_schemas(CONFIG, team_id=1)++ assert [s.name for s in schemas] == ["invoices", "Budgets", "Insights", "Anomalies"]+def test_source_for_pipeline_fetches_the_report_id_from_schema_metadata(self):inputs = MagicMock(spec=SourceInputs,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txtindex ef25f449b902..4c9d094d4443 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tests/source_contract_baseline.txt@@ -220,7 +220,7 @@ DigitalOcean fail:stall-budgetDingConnect fail:stall-budgetDiscourse fail:safe-pointDixa fail:rate-limit-budget,safe-point,stall-budget-DoIt not-checkable:schemas+DoIt fail:stall-budgetDockerhub fail:rate-limit-budget,safe-point,stall-budgetDocuseal fail:stall-budgetDocusign not-checkable:no-request