Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 3 missing tables to the Wasabi source (#125276)

Tom Owersb54ee5825977
diff --git a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt b/posthog/test/repo_invariants/dataclass_frozen_baseline.txt
index 8491e938bfc3..c547dd5dec09 100644
--- a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt
+++ b/posthog/test/repo_invariants/dataclass_frozen_baseline.txt
@@ -1135,7 +1135,6 @@
1 products/warehouse_sources/backend/temporal/data_imports/sources/vercel/settings.py
1 products/warehouse_sources/backend/temporal/data_imports/sources/vercel/vercel.py
1 products/warehouse_sources/backend/temporal/data_imports/sources/vultr/settings.py
-1 products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/settings.py
1 products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/wasabi.py
1 products/warehouse_sources/backend/temporal/data_imports/sources/watchmode/watchmode.py
1 products/warehouse_sources/backend/temporal/data_imports/sources/webflow/settings.py
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 10c32d2cc83a..dfb8c7bf6af3 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -13086,9 +13086,9 @@ Today (4): `accounts`, `bucket_utilizations`, `sub_account_invoices`, `utilizati
Diffed against: <https://docs.wasabi.com/llms.txt>
-- [ ] `GET /v1/accounts/{AcctNum}/invoices/{SubInvoiceNum}/regional` — per-region cost and usage breakdown of the sub-invoices we already sync; Wasabi explicitly recommends this over the flat detail endpoint (high)
-- [ ] `GET /v1/accounts/{AcctNum}/invoices/{SubInvoiceNum}` — sub-invoice line detail (unit cost, resource type); being deprecated in favor of the regional variant but still the only flat line-item view (medium)
-- [ ] `GET /v1/accounts/{AcctNum}/air-jobs` — Wasabi AiR job history per sub-account, an activity table with no equivalent today (low)
+- [x] `GET /v1/accounts/{AcctNum}/invoices/{SubInvoiceNum}/regional` — per-region cost and usage breakdown of the sub-invoices we already sync; Wasabi explicitly recommends this over the flat detail endpoint (high)
+- [x] `GET /v1/accounts/{AcctNum}/invoices/{SubInvoiceNum}` — sub-invoice line detail (unit cost, resource type); being deprecated in favor of the regional variant but still the only flat line-item view (medium)
+- [x] `GET /v1/accounts/{AcctNum}/air-jobs` — Wasabi AiR job history per sub-account, an activity table with no equivalent today (low)
Note: Coverage is close to complete for the Wasabi Account Control API (partner.wasabisys.com). I verified from the endpoint docs that /v1/utilizations and /v1/utilizations/buckets already return BOTH control and sub-account rows, so /v1/control-account/utilizations, /v1/control-account/bucket-utilizations, /v1/accounts/{acct}/utilizations and /v1/utilizations/status are all redundant with what is already synced and I deliberately excluded them. Wasabi also ships two other API families on different hosts and credentials — the WACM Connect API (api.wacm.wasabisys.com: control accounts, channel accounts, members, standalone accounts, /v1/invoices, /v1/usages) and the Stats API (/v1/standalone/utilizations) — which would be separate sources, not gaps in this one. Wasabi publishes a docs llms.txt plus per-page .md, which is the fastest way to enumerate.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/canonical_descriptions.py
index 12e1a527e820..db705fa8497d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/canonical_descriptions.py
@@ -28,6 +28,23 @@ _UTILIZATION_METRIC_COLUMNS = {
"DeleteBytes": "Bytes deleted in the period.",
}
+_SUB_INVOICE_ITEM_COLUMNS = {
+ "SubInvoiceItemNum": "Unique identifier of the sub-invoice line item.",
+ "SubInvoiceNum": "Identifier of the sub-invoice the line item belongs to.",
+ "Type": "Resource type the line item charges for.",
+ "DisplayName": "Display name of the line item.",
+ "Description": "Description of the line item.",
+ "Qty": "Billed quantity of the resource.",
+ "UnitCost": "Cost per unit of the resource.",
+ "Total": "Total amount of the line item.",
+ "Currency": "Currency of the line item amounts.",
+ "AcctNum": "Account number of the sub-account the sub-invoice belongs to.",
+ "InvoiceNum": "Identifier of the Control Account invoice the sub-invoice rolls up into.",
+ "CreateTime": "When the sub-invoice was created.",
+ "PeriodStart": "Start of the billing period the sub-invoice covers.",
+ "PeriodEnd": "End of the billing period the sub-invoice covers.",
+}
+
CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"accounts": {
"description": "Sub-accounts associated with the Wasabi Control Account, with summary profile information.",
@@ -82,4 +99,25 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"Status": "Payment status of the sub-invoice.",
},
},
+ "sub_account_invoice_items": {
+ "description": "Line items of each sub-invoice, such as storage and egress charges. Where a sub-account has buckets in regions with different prices, the unit cost is the weighted average across those regions.",
+ "docs_url": "https://docs.wasabi.com/apidocs/get-details-for-a-sub-invoice",
+ "columns": _SUB_INVOICE_ITEM_COLUMNS,
+ },
+ "sub_account_invoice_regional_items": {
+ "description": "Line items of each sub-invoice, broken down by Wasabi region.",
+ "docs_url": "https://docs.wasabi.com/apidocs/get-details-and-regional-usage-for-a-sub-invoice",
+ "columns": _SUB_INVOICE_ITEM_COLUMNS,
+ },
+ "air_jobs": {
+ "description": "Finished Wasabi AiR jobs for each sub-account. Jobs that are still in progress are not listed.",
+ "docs_url": "https://docs.wasabi.com/apidocs/get-air-jobs-for-a-sub-account",
+ "columns": {
+ "JobId": "Unique identifier of the AiR job.",
+ "AcctNum": "Account number of the sub-account that ran the job.",
+ "MediaRuntimeTotalSeconds": "Total runtime of the processed media, in seconds.",
+ "CreatedAt": "When the job was created.",
+ "FinishedAt": "When the job finished.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/settings.py
index 41934db971f4..2afa025bb942 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/settings.py
@@ -17,15 +17,29 @@ UTILIZATION_WINDOW_DAYS = 30
# (the merge on primary keys dedupes the overlap).
UTILIZATION_LOOKBACK_SECONDS = 3 * 24 * 60 * 60
+# AiR jobs are listed only once finished, and Wasabi doesn't say whether from/to filter on the
+# start or the finish time, so re-read a trailing window to pick up jobs that ran across days.
+AIR_JOBS_LOOKBACK_SECONDS = 3 * 24 * 60 * 60
+
+# Wasabi documents no maximum page size for AiR jobs.
+AIR_JOBS_PAGE_SIZE = 100
+
START_TIME_INCREMENTAL: IncrementalField = incremental_field("StartTime", IncrementalFieldType.DateTime)
+FINISHED_AT_INCREMENTAL: IncrementalField = incremental_field("FinishedAt", IncrementalFieldType.DateTime)
ACCOUNTS = "accounts"
UTILIZATIONS = "utilizations"
BUCKET_UTILIZATIONS = "bucket_utilizations"
SUB_ACCOUNT_INVOICES = "sub_account_invoices"
+SUB_ACCOUNT_INVOICE_ITEMS = "sub_account_invoice_items"
+SUB_ACCOUNT_INVOICE_REGIONAL_ITEMS = "sub_account_invoice_regional_items"
+AIR_JOBS = "air_jobs"
+
+# Sub-invoice line items carry only SubInvoiceNum, so the account and billing period come from the parent sub-invoice.
+SUB_INVOICE_PARENT_FIELDS = ["AcctNum", "InvoiceNum", "CreateTime", "PeriodStart", "PeriodEnd"]
-@dataclass
+@dataclass(frozen=True)
class WasabiEndpointConfig:
name: str
path: str
@@ -34,8 +48,13 @@ class WasabiEndpointConfig:
incremental_fields: list[IncrementalField] = field(default_factory=list)
# Endpoints that accept from/to date filters are walked in ascending date windows.
date_windowed: bool = False
- # Fan-out endpoints iterate this parent endpoint and resolve {acct_num} per parent row.
+ # Fan-out endpoints iterate this parent endpoint and resolve each path param per parent row.
parent: str | None = None
+ resolve_fields: dict[str, str] = field(default_factory=lambda: {"acct_num": "AcctNum"})
+ include_from_parent: list[str] = field(default_factory=list)
+ data_selector: str = "$"
+ # Endpoints paged by a 0-based pageNumber query param.
+ page_size: int | None = None
WASABI_ENDPOINTS: dict[str, WasabiEndpointConfig] = {
@@ -72,6 +91,37 @@ WASABI_ENDPOINTS: dict[str, WasabiEndpointConfig] = {
partition_key="CreateTime",
parent=ACCOUNTS,
),
+ SUB_ACCOUNT_INVOICE_ITEMS: WasabiEndpointConfig(
+ name=SUB_ACCOUNT_INVOICE_ITEMS,
+ path="/v1/accounts/{acct_num}/invoices/{sub_invoice_num}",
+ primary_keys=["AcctNum", "SubInvoiceNum", "SubInvoiceItemNum"],
+ partition_key="CreateTime",
+ parent=SUB_ACCOUNT_INVOICES,
+ resolve_fields={"acct_num": "AcctNum", "sub_invoice_num": "SubInvoiceNum"},
+ include_from_parent=SUB_INVOICE_PARENT_FIELDS,
+ data_selector="SubInvoiceItems",
+ ),
+ SUB_ACCOUNT_INVOICE_REGIONAL_ITEMS: WasabiEndpointConfig(
+ name=SUB_ACCOUNT_INVOICE_REGIONAL_ITEMS,
+ path="/v1/accounts/{acct_num}/invoices/{sub_invoice_num}/regional",
+ primary_keys=["AcctNum", "SubInvoiceNum", "SubInvoiceItemNum"],
+ partition_key="CreateTime",
+ parent=SUB_ACCOUNT_INVOICES,
+ resolve_fields={"acct_num": "AcctNum", "sub_invoice_num": "SubInvoiceNum"},
+ include_from_parent=SUB_INVOICE_PARENT_FIELDS,
+ data_selector="SubInvoiceItems",
+ ),
+ AIR_JOBS: WasabiEndpointConfig(
+ name=AIR_JOBS,
+ path="/v1/accounts/{acct_num}/air-jobs",
+ primary_keys=["AcctNum", "JobId"],
+ partition_key="CreatedAt",
+ incremental_fields=[FINISHED_AT_INCREMENTAL],
+ parent=ACCOUNTS,
+ # Wasabi documents the body as an array of {Pagination, Records} pages; also accept a bare object.
+ data_selector="$..Records[*]",
+ page_size=AIR_JOBS_PAGE_SIZE,
+ ),
}
ENDPOINTS = tuple(WASABI_ENDPOINTS)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/source.py
index 6361b7342e57..ac80b3c8bbb5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/source.py
@@ -21,6 +21,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.sch
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceInputs, SourceResponse
from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.wasabi import WasabiSourceConfig
from products.warehouse_sources.backend.temporal.data_imports.sources.wasabi.settings import (
+ AIR_JOBS,
+ AIR_JOBS_LOOKBACK_SECONDS,
BUCKET_UTILIZATIONS,
ENDPOINTS,
INCREMENTAL_FIELDS,
@@ -69,17 +71,19 @@ class WasabiSource(ResumableSource[WasabiSourceConfig, WasabiResumeConfig]):
force_refresh: bool = False,
api_version: str | None = None,
) -> list[SourceSchema]:
- # The date-window walk re-reads a boundary day each run, so append syncs would
- # duplicate rows — utilization endpoints are merge-only.
+ # The date-window walk and the AiR jobs date filter re-read a boundary day each run,
+ # so append syncs would duplicate rows — these endpoints are merge-only.
schemas = build_endpoint_schemas(
ENDPOINTS,
INCREMENTAL_FIELDS,
names,
- merge_only=(UTILIZATIONS, BUCKET_UTILIZATIONS),
+ merge_only=(UTILIZATIONS, BUCKET_UTILIZATIONS, AIR_JOBS),
)
for schema in schemas:
if schema.name in (UTILIZATIONS, BUCKET_UTILIZATIONS):
schema.default_incremental_lookback_seconds = UTILIZATION_LOOKBACK_SECONDS
+ elif schema.name == AIR_JOBS:
+ schema.default_incremental_lookback_seconds = AIR_JOBS_LOOKBACK_SECONDS
return schemas
def validate_credentials(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/tests/test_wasabi.py b/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/tests/test_wasabi.py
index 278081b368e4..abbaabdb385f 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/tests/test_wasabi.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/tests/test_wasabi.py
@@ -1,6 +1,7 @@
from datetime import UTC, date, datetime, timedelta
from typing import Any, cast
+import time_machine
from unittest import mock
from parameterized import parameterized
@@ -10,6 +11,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.tes
ScriptedResponse,
SourceDriver,
always,
+ route,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.wasabi import WasabiSourceConfig
from products.warehouse_sources.backend.temporal.data_imports.sources.wasabi.settings import (
@@ -224,3 +226,141 @@ class TestWasabiSourceTransport:
"field": "AcctNum",
}
assert child_resource["write_disposition"] == "replace"
+
+
+_SUB_INVOICE_PARENT = {
+ "AcctNum": 101,
+ "InvoiceNum": 900,
+ "CreateTime": "2026-02-01T00:00:00Z",
+ "PeriodStart": "2026-01-01T00:00:00Z",
+ "PeriodEnd": "2026-01-31T23:59:59Z",
+}
+
+
+def _sub_invoice_item(item_num: int, sub_invoice_num: int) -> dict[str, Any]:
+ return {
+ "SubInvoiceItemNum": item_num,
+ "SubInvoiceNum": sub_invoice_num,
+ "Type": "storage",
+ "DisplayName": "Timed active storage",
+ "Qty": 1.5,
+ "UnitCost": 6.99,
+ "Total": 10.49,
+ "Currency": "USD",
+ }
+
+
+def _air_job(job_id: str) -> dict[str, Any]:
+ return {
+ "JobId": job_id,
+ "AcctNum": 101,
+ "MediaRuntimeTotalSeconds": 120,
+ "CreatedAt": "2026-03-05 10:00:00",
+ "FinishedAt": "2026-03-05 10:05:00",
+ }
+
+
+class TestWasabiFanOutEndpoints:
+ @parameterized.expand(
+ [
+ ("details", "sub_account_invoice_items", ""),
+ ("regional", "sub_account_invoice_regional_items", "/regional"),
+ ]
+ )
+ def test_sub_invoice_items_walk_accounts_then_sub_invoices(self, _name: str, schema: str, suffix: str) -> None:
+ result = _driver().run(
+ schema,
+ route(
+ {
+ "/v1/accounts": [ScriptedResponse(json=[{"AcctNum": 101}, {"AcctNum": 102}])],
+ "/v1/accounts/101/invoices": [
+ ScriptedResponse(json=[{**_SUB_INVOICE_PARENT, "SubInvoiceNum": 5001}])
+ ],
+ "/v1/accounts/102/invoices": [
+ ScriptedResponse(json=[{**_SUB_INVOICE_PARENT, "AcctNum": 102, "SubInvoiceNum": 5002}])
+ ],
+ f"/v1/accounts/101/invoices/5001{suffix}": [
+ ScriptedResponse(
+ json={
+ "SubInvoice": {**_SUB_INVOICE_PARENT, "SubInvoiceNum": 5001},
+ "SubInvoiceItems": [_sub_invoice_item(1, 5001), _sub_invoice_item(2, 5001)],
+ }
+ )
+ ],
+ # A sub-invoice removed between its listing and the detail fetch.
+ f"/v1/accounts/102/invoices/5002{suffix}": [
+ ScriptedResponse(status=404, json={"Msg": "Not Found"})
+ ],
+ }
+ ),
+ )
+
+ assert result.raised is None
+ assert result.rows == [
+ {**_sub_invoice_item(1, 5001), **_SUB_INVOICE_PARENT},
+ {**_sub_invoice_item(2, 5001), **_SUB_INVOICE_PARENT},
+ ]
+
+ @parameterized.expand(
+ [
+ ("full_refresh", None, None, None),
+ ("incremental", "2026-03-05 10:05:00", "2026-03-05", (TODAY + timedelta(days=1)).isoformat()),
+ ]
+ )
+ def test_air_jobs_pages_each_account_until_an_empty_page(
+ self, _name: str, last_value: str | None, expected_from: str | None, expected_to: str | None
+ ) -> None:
+ with time_machine.travel(datetime.combine(TODAY, datetime.min.time(), UTC), tick=False):
+ result = _driver().run(
+ "air_jobs",
+ route(
+ {
+ "/v1/accounts": [ScriptedResponse(json=[{"AcctNum": 101}, {"AcctNum": 102}])],
+ "/v1/accounts/101/air-jobs": [
+ ScriptedResponse(
+ json=[
+ {
+ "Pagination": {
+ "PageSize": 100,
+ "PageNumber": 0,
+ "TotalRecords": 2,
+ "TotalPages": 2,
+ },
+ "Records": [_air_job("job-1")],
+ }
+ ]
+ ),
+ # Wasabi reports TotalPages as -1 on every page after the first.
+ ScriptedResponse(
+ json=[
+ {
+ "Pagination": {
+ "PageSize": 100,
+ "PageNumber": 1,
+ "TotalRecords": -1,
+ "TotalPages": -1,
+ },
+ "Records": [_air_job("job-2")],
+ }
+ ]
+ ),
+ ScriptedResponse(json=[{"Pagination": {}, "Records": []}]),
+ ],
+ # A bare object body is accepted too.
+ "/v1/accounts/102/air-jobs": [ScriptedResponse(json={"Pagination": {}, "Records": []})],
+ }
+ ),
+ should_use_incremental_field=last_value is not None,
+ incremental_field="FinishedAt",
+ db_incremental_field_last_value=last_value,
+ )
+
+ assert result.raised is None
+ assert result.rows == [_air_job("job-1"), _air_job("job-2")]
+ assert result.response is not None
+ # Rows restart at each account, so the run is not one ascending series.
+ assert result.response.sort_mode is None
+ job_requests = [r for r in result.requests if r.path.endswith("/air-jobs")]
+ assert [r.param("pageNumber") for r in job_requests] == ["0", "1", "2", "0"]
+ assert {r.param("from") for r in job_requests} == {expected_from}
+ assert {r.param("to") for r in job_requests} == {expected_to}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/wasabi.py b/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/wasabi.py
index 9234201763e9..f8620385eafe 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/wasabi.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/wasabi/wasabi.py
@@ -14,12 +14,17 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
rest_api_resource,
rest_api_resources,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
+ build_chained_resource,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (
BasePaginator,
+ PageNumberPaginator,
SinglePagePaginator,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (
ClientConfig,
+ Endpoint,
EndpointResource,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
@@ -117,19 +122,101 @@ def _client_config(api_key: str) -> ClientConfig:
}
-def _make_source_response(endpoint_config: WasabiEndpointConfig, resource: Any) -> SourceResponse:
+def _make_source_response(
+ endpoint_config: WasabiEndpointConfig, resource: Any, supports_resume: bool = True
+) -> SourceResponse:
return SourceResponse(
name=endpoint_config.name,
items=lambda: resource,
primary_keys=endpoint_config.primary_keys,
# Date windows walk oldest-first, so the per-batch max of the incremental field
- # advances monotonically across batches.
- sort_mode="asc",
+ # advances monotonically across batches. Fan-out rows restart per account.
+ sort_mode="asc" if endpoint_config.date_windowed else None,
partition_count=1,
partition_size=1,
partition_mode="datetime" if endpoint_config.partition_key else None,
partition_format="month" if endpoint_config.partition_key else None,
partition_keys=[endpoint_config.partition_key] if endpoint_config.partition_key else None,
+ supports_resume=supports_resume,
+ )
+
+
+def _paginator(endpoint_config: WasabiEndpointConfig) -> BasePaginator:
+ if endpoint_config.page_size is not None:
+ # TotalPages is only returned on page 0 (later pages report -1), so stop on an empty page instead.
+ return PageNumberPaginator(base_page=0, page_param="pageNumber")
+ return SinglePagePaginator()
+
+
+def _child_endpoint(
+ endpoint_config: WasabiEndpointConfig,
+ parent_name: str,
+ should_use_incremental_field: bool,
+ db_incremental_field_last_value: Optional[Any],
+) -> Endpoint:
+ params: dict[str, Any] = {
+ param: {"type": "resolve", "resource": parent_name, "field": field_name}
+ for param, field_name in endpoint_config.resolve_fields.items()
+ }
+ if endpoint_config.page_size is not None:
+ params["pageSize"] = endpoint_config.page_size
+ params["sortOrder"] = "asc"
+ if endpoint_config.incremental_fields and should_use_incremental_field:
+ params["from"] = _start_date(should_use_incremental_field, db_incremental_field_last_value).isoformat()
+ # `to` is exclusive, so tomorrow keeps jobs that finished today.
+ params["to"] = (datetime.now(UTC).date() + timedelta(days=1)).isoformat()
+ return {
+ "path": endpoint_config.path,
+ "params": params,
+ "data_selector": endpoint_config.data_selector,
+ "paginator": _paginator(endpoint_config),
+ }
+
+
+def _parent_resource(parent_config: WasabiEndpointConfig, endpoint: Endpoint) -> EndpointResource:
+ return {
+ "name": parent_config.name,
+ "table_name": parent_config.name,
+ "write_disposition": "replace",
+ "endpoint": endpoint,
+ "table_format": "delta",
+ }
+
+
+def _chained_resource(
+ endpoint_config: WasabiEndpointConfig,
+ parent_config: WasabiEndpointConfig,
+ api_key: str,
+ team_id: int,
+ job_id: str,
+) -> Any:
+ """Build the accounts -> sub-invoices -> sub-invoice detail chain and return the detail resource."""
+ assert parent_config.parent is not None
+ root_config = WASABI_ENDPOINTS[parent_config.parent]
+ root_resource = _parent_resource(
+ root_config,
+ {"path": root_config.path, "data_selector": "$", "paginator": SinglePagePaginator()},
+ )
+ middle_resource = _parent_resource(parent_config, _child_endpoint(parent_config, root_config.name, False, None))
+ child_endpoint = _child_endpoint(endpoint_config, parent_config.name, False, None)
+ # A sub-invoice or sub-account removed between its listing and this fetch must not sink the run.
+ child_endpoint["response_actions"] = [{"status_code": 404, "action": "ignore"}]
+ child_resource: EndpointResource = {
+ "name": endpoint_config.name,
+ "table_name": endpoint_config.name,
+ "write_disposition": "replace",
+ "include_from_parent": endpoint_config.include_from_parent,
+ "endpoint": child_endpoint,
+ "table_format": "delta",
+ }
+ return build_chained_resource(
+ resources=[root_resource, middle_resource, child_resource],
+ child_name=endpoint_config.name,
+ parent_name=parent_config.name,
+ parent_field_renames={field_name: field_name for field_name in endpoint_config.include_from_parent},
+ client_config=_client_config(api_key),
+ team_id=team_id,
+ job_id=job_id,
)
@@ -157,32 +244,30 @@ def wasabi_source(
if endpoint_config.parent is not None:
parent_config = WASABI_ENDPOINTS[endpoint_config.parent]
+ if parent_config.parent is not None:
+ # The chained walk saves no checkpoints, so a restart would repeat it from the start.
+ return _make_source_response(
+ endpoint_config,
+ _chained_resource(endpoint_config, parent_config, api_key, team_id, job_id),
+ supports_resume=False,
+ )
+
# Fan-out: iterate the sub-account list and fetch the child endpoint per account.
# Child rows natively carry AcctNum, so no parent-field injection is needed.
resources: list[str | EndpointResource] = [
- {
- "name": parent_config.name,
- "table_name": parent_config.name,
- "write_disposition": "replace",
- "endpoint": {
- "path": parent_config.path,
- "data_selector": "$",
- "paginator": SinglePagePaginator(),
- },
- "table_format": "delta",
- },
+ _parent_resource(
+ parent_config,
+ {"path": parent_config.path, "data_selector": "$", "paginator": SinglePagePaginator()},
+ ),
{
"name": endpoint_config.name,
"table_name": endpoint_config.name,
- "write_disposition": "replace",
- "endpoint": {
- "path": endpoint_config.path,
- "params": {
- "acct_num": {"type": "resolve", "resource": parent_config.name, "field": "AcctNum"},
- },
- "data_selector": "$",
- "paginator": SinglePagePaginator(),
- },
+ "write_disposition": {"disposition": "merge", "strategy": "upsert"}
+ if should_use_incremental_field
+ else "replace",
+ "endpoint": _child_endpoint(
+ endpoint_config, parent_config.name, should_use_incremental_field, db_incremental_field_last_value
+ ),
"table_format": "delta",
},
]