:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add zuora commerce entitlement tables (#120218)
Tom Owers6079c6d14111
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex b954dea9319d..eb942bf3064c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -13674,7 +13674,7 @@ Note: Tables are static (ZOOM_ENDPOINTS in products/warehouse_sources/backend/te## Zuora — gaps-Today (8): `accounts`, `credit_memos`, `invoices`, `orders`, `payments`, `products`, `refunds`, `subscriptions`+Today (11): `accounts`, `billable_metrics`, `credit_memos`, `entitlement_grants`, `features`, `invoices`, `orders`, `payments`, `products`, `refunds`, `subscriptions`Diffed against: <https://developer.zuora.com/yaml/apis/zuora-openapi-full-compact.yaml>@@ -13690,6 +13690,9 @@ Diffed against: <https://developer.zuora.com/yaml/apis/zuora-openapi-full-compac- [ ] `object-query/order-actions` — the individual actions inside each order we sync (create/amend/cancel/renew), giving orders analyzable structure (medium)- [ ] `object-query/order-line-items` — non-subscription order line items with quantity and amount, missing from the order headers we sync (medium)- [ ] `object-query/usages and object-query/processed-usages` — consumption records for usage-based charges - required to explain variable invoice amounts (medium)+- [x] `/commerce/billable-metrics` — billable metric definitions that measure entitlement usage (`billable_metrics`)+- [x] `/commerce/features` — catalog features granted through entitlements (`features`)+- [x] `/commerce/entitlements/grants` — runtime entitlement grants per account and subscription, paginated by `nextPageToken` (`entitlement_grants`)Note: The connector already uses Zuora's Object Query API (/object-query/<segment>, filtered on updateddate), so every gap listed is a same-shape, same-pagination addition - the spec exposes ~50 object-query collections and the source wires up 8. Also unsynced but lower value: payment-applications, refund-applications, payment-methods, taxation-items, billing-runs, payment-runs, prepaid-balances/prepaid-balance-transactions, invoice-schedules, ramps, summarystatements. Zuora also serves the spec as zuora-openapi-for-otc.yaml (order-to-cash only); I used the full compact build, version 2026-07-24.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/canonical_descriptions.pyindex 9e5de5e45b7c..067796779b5f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/canonical_descriptions.py@@ -151,4 +151,60 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {category="Category of the order (e.g. NewSale, Upsell).",),},+ "billable_metrics": {+ "description": "A billable metric that aggregates usage events to measure entitlement consumption or billing.",+ "docs_url": "https://developer.zuora.com/v1-api-reference/api/operation/getbillablemetrics",+ "columns": {+ "id": "Unique identifier of the billable metric.",+ "metricNumber": "System-generated number that identifies the billable metric. Values start with BM-.",+ "name": "Unique name of the billable metric.",+ "description": "Description of what the billable metric measures.",+ "kind": "Metric kinds: entitlement_kind, billable, or diagnostic.",+ "event_types": "Event types that the metric processes when it calculates usage.",+ "filters": "Filters applied to incoming events before aggregation.",+ "computed_properties": "Derived event properties used as aggregation inputs.",+ "aggregation": "How the metric aggregates events, for example a sum of an event property.",+ "group_keys": "Event properties used to group events before aggregation.",+ "status": "Lifecycle status of the billable metric.",+ },+ },+ "features": {+ "description": "A catalog feature that products grant to customers through entitlements.",+ "docs_url": "https://developer.zuora.com/v1-api-reference/api/operation/getfeatures",+ "columns": {+ "id": "Unique identifier of the feature.",+ "featureNumber": "System-generated number that identifies the feature.",+ "name": "Unique name of the feature.",+ "description": "Description of the feature.",+ "type": "Feature type: bool, enumType, numbered, or unmetered.",+ "status": "Lifecycle status of the feature: Active or Inactive.",+ "values": "Permitted values for an enumType feature.",+ "tags": "Tags used to search and check features as a group.",+ "unitOfMeasure": "Unit used to measure a numbered feature quantity, such as Each or GB.",+ "isMetered": "Whether a numbered feature tracks consumption.",+ "isCapped": "Whether a numbered metered feature stops consumption when the entitlement quantity is reached.",+ "metricId": "ID of the billable metric that measures usage for a numbered metered feature.",+ },+ },+ "entitlement_grants": {+ "description": "A runtime entitlement grant that gives an account access to a feature.",+ "docs_url": "https://developer.zuora.com/v1-api-reference/api/operation/getentitlementgrants",+ "columns": {+ "id": "Unique identifier of the entitlement grant.",+ "entitlementKey": "Unique key of the entitlement grant.",+ "entitlementNumber": "System-generated number that identifies the grant.",+ "accountKey": "Account number or account ID that owns the grant.",+ "subscriptionKey": "Subscription number or subscription ID associated with the grant.",+ "planKey": "Product rate plan ID or number associated with the grant.",+ "featureKey": "Unique key of the granted catalog feature.",+ "featureCode": "Feature code of the granted catalog feature.",+ "quantity": "Granted quantity for a numbered feature.",+ "startDate": "Date the grant became effective.",+ "endDate": "Date the grant ends.",+ "revokedDate": "Date the grant was revoked.",+ "state": "Current state of the grant: pending, active, paused, inactive, expired, or revoked.",+ "unit": "Unit of measure for a numbered feature grant.",+ "isDisabled": "Whether the grant is disabled.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/settings.pyindex 74d58f12341b..0efd14fe03ef 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/settings.py@@ -1,3 +1,5 @@+from posthog.dataclasses import frozen+from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType# Region/environment-specific REST hosts — credentials only work against@@ -38,7 +40,33 @@ ZUORA_ENDPOINTS: dict[str, str] = {"orders": "orders",}-ENDPOINTS = tuple(ZUORA_ENDPOINTS.keys())++@frozen+class ZuoraCommerceEndpoint:+ path: str+ data_selector: str+ # Only paginated lists return a nextPageToken; the others return every row in one response.+ paginated: bool = False+ # Set for a one-response list that takes a page size but returns no next-page token.+ page_size_param: str | None = None+++# Stream name -> Commerce API list endpoint. These lists expose no updated-date filter, so they+# sync as full refresh. The spec documents an alternate array key for each response, so the+# selectors match either key.+ZUORA_COMMERCE_ENDPOINTS: dict[str, ZuoraCommerceEndpoint] = {+ "billable_metrics": ZuoraCommerceEndpoint(+ path="commerce/billable-metrics",+ data_selector="(data[*]) | (response.data[*])",+ page_size_param="pagination.page_size",+ ),+ "features": ZuoraCommerceEndpoint(path="commerce/features", data_selector="(features[*]) | (items[*])"),+ "entitlement_grants": ZuoraCommerceEndpoint(+ path="commerce/entitlements/grants", data_selector="(grants[*]) | (items[*])", paginated=True+ ),+}++ENDPOINTS = (*ZUORA_ENDPOINTS, *ZUORA_COMMERCE_ENDPOINTS)INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {name: list(_UPDATED_DATE_INCREMENTAL_FIELDS) for name in ZUORA_ENDPOINTSdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/source.pyindex bc0b56d59a40..dc88ceff2437 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/source.py@@ -30,6 +30,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.zuora.settINCREMENTAL_FIELDS,)from products.warehouse_sources.backend.temporal.data_imports.sources.zuora.zuora import (+ ZUORA_INCOMPLETE_LIST_ERROR,ZuoraResumeConfig,validate_credentials as validate_zuora_credentials,zuora_source,@@ -74,6 +75,7 @@ class ZuoraSource(ResumableSource[ZuoraSourceConfig, ZuoraResumeConfig]):"400 Client Error: Bad Request for url: https://rest.": "Zuora rejected the request. If this occurs while connecting, check your client ID and client secret (and that they match the selected environment).","401 Client Error: Unauthorized for url: https://rest.": "Zuora authentication failed. Please check your client ID and client secret (and that they match the selected environment).","403 Client Error: Forbidden for url: https://rest.": "Zuora denied access. Please check that the OAuth client's user has permission for this object.",+ ZUORA_INCOMPLETE_LIST_ERROR: "Zuora returned more rows than one request can hold for this table, and its API has no way to fetch the rest. The sync stopped so the table is not saved incomplete. Contact support if this keeps happening.",}@propertydiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/tests/test_zuora.py b/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/tests/test_zuora.pyindex 6eb48c96c067..c4d6aefa4914 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/tests/test_zuora.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/tests/test_zuora.py@@ -15,9 +15,14 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.tesscripted_network,)from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.zuora import ZuoraSourceConfig-from products.warehouse_sources.backend.temporal.data_imports.sources.zuora.settings import ENDPOINTS, ZUORA_ENDPOINTS+from products.warehouse_sources.backend.temporal.data_imports.sources.zuora.settings import (+ ENDPOINTS,+ ZUORA_COMMERCE_ENDPOINTS,+ ZUORA_ENDPOINTS,+)from products.warehouse_sources.backend.temporal.data_imports.sources.zuora.source import ZuoraSourcefrom products.warehouse_sources.backend.temporal.data_imports.sources.zuora.zuora import (+ ZUORA_INCOMPLETE_LIST_ERROR,ZuoraResumeConfig,_base_url,_format_timestamp,@@ -44,7 +49,7 @@ def _driver() -> SourceDriver:def _data_requests(result: DriveResult) -> list[RecordedRequest]:- return [request for request in result.requests if request.path.startswith("/object-query/")]+ return [request for request in result.requests if request.path.startswith(("/object-query/", "/commerce/"))]class TestBaseUrl:@@ -97,19 +102,83 @@ class TestGetRows:assert request.param("filter[]") == "updateddate.GT:2024-01-02T03:04:05Z"assert "filter%5B%5D=updateddate.GT%3A2024-01-02T03%3A04%3A05Z" in request.url- def test_resumes_from_saved_cursor(self):+ @pytest.mark.parametrize(+ "endpoint, path, expected_query",+ [+ ("accounts", "/object-query/accounts", {"cursor": ("cur-9",)}),+ ("entitlement_grants", "/commerce/entitlements/grants", {"pageToken": ("cur-9",)}),+ ],+ )+ def test_resumes_from_saved_cursor(self, endpoint, path, expected_query):result = _driver().run(- "accounts",- route({"/oauth/token": [_token_response()], "/object-query/accounts": [_page([{"id": "a9"}])]}),+ endpoint,+ route({"/oauth/token": [_token_response()], path: [_page([{"id": "a9"}])]}),resume_state=ZuoraResumeConfig(cursor="cur-9"),)# The resumed run targets the saved page with the cursor alone.request = _data_requests(result)[0]- assert dict(request.query) == {"cursor": ("cur-9",)}- assert "cursor=cur-9" in request.url+ assert dict(request.query) == expected_queryassert "pageSize" not in request.url+ @pytest.mark.parametrize(+ "endpoint, path, body, expected_query",+ [+ ("features", "/commerce/features", {"features": [{"id": "f1"}, {"id": "f2"}]}, {}),+ ("features", "/commerce/features", {"items": [{"id": "f1"}, {"id": "f2"}]}, {}),+ (+ "billable_metrics",+ "/commerce/billable-metrics",+ {"data": [{"id": "f1"}, {"id": "f2"}]},+ {"pagination.page_size": ("99",)},+ ),+ (+ "billable_metrics",+ "/commerce/billable-metrics",+ {"response": {"data": [{"id": "f1"}, {"id": "f2"}]}},+ {"pagination.page_size": ("99",)},+ ),+ ],+ )+ def test_commerce_lists_read_either_documented_array_key(self, endpoint, path, body, expected_query):+ result = _driver().run(+ endpoint, route({"/oauth/token": [_token_response()], path: [ScriptedResponse(json=body)]})+ )++ assert [row["id"] for row in result.rows] == ["f1", "f2"]+ data_requests = _data_requests(result)+ assert len(data_requests) == 1+ assert dict(data_requests[0].query) == expected_query++ def test_billable_metrics_full_page_fails_before_yielding_rows(self):+ full_page = {"data": [{"id": f"bm-{index}"} for index in range(99)]}+ result = _driver().run(+ "billable_metrics",+ route(+ {"/oauth/token": [_token_response()], "/commerce/billable-metrics": [ScriptedResponse(json=full_page)]}+ ),+ )++ assert result.rows == []+ assert ZUORA_INCOMPLETE_LIST_ERROR in str(result.raised)++ def test_entitlement_grants_follow_next_page_token(self):+ result = _driver().run(+ "entitlement_grants",+ route(+ {+ "/oauth/token": [_token_response()],+ "/commerce/entitlements/grants": [+ ScriptedResponse(json={"grants": [{"id": "g1"}], "nextPageToken": "tok-2"}),+ ScriptedResponse(json={"grants": [{"id": "g2"}], "nextPageToken": None}),+ ],+ }+ ),+ )++ assert [row["id"] for row in result.rows] == ["g1", "g2"]+ assert [dict(request.query) for request in _data_requests(result)] == [{}, {"pageToken": ("tok-2",)}]+def test_mints_token_once_and_sends_bearer(self):result = _driver().run("accounts",@@ -130,4 +199,4 @@ class TestGetRows:class TestZuoraSourceResponse:def test_all_endpoints_have_paths(self):- assert set(ENDPOINTS) == set(ZUORA_ENDPOINTS.keys())+ assert set(ENDPOINTS) == set(ZUORA_ENDPOINTS) | set(ZUORA_COMMERCE_ENDPOINTS)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/zuora.py b/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/zuora.pyindex 190ae4d40c01..5e0f1776af6c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/zuora.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/zuora/zuora.py@@ -15,19 +15,27 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.jsonpath_utils import (find_values,)-from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import BasePaginator+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (+ BasePaginator,+ JSONResponseCursorPaginator,+ SinglePagePaginator,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (ClientConfig,Endpoint,+ PaginatorConfig,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.zuora.settings import (PAGE_SIZE,+ ZUORA_COMMERCE_ENDPOINTS,ZUORA_ENDPOINTS,ZUORA_ENVIRONMENT_HOSTS,)+ZUORA_INCOMPLETE_LIST_ERROR = "Zuora returned a full page for a list that has no next-page token"+@dataclasses.dataclassclass ZuoraResumeConfig:@@ -119,6 +127,61 @@ class ZuoraCursorPaginator(BasePaginator):return "ZuoraCursorPaginator()"+class ZuoraFullPageGuardPaginator(SinglePagePaginator):+ """Fetch a one-response list, and fail when the response fills the requested page.++ A full page can hide more rows that the API gives no way to fetch. The check runs before the+ page is yielded, so a truncated full refresh never replaces the table."""++ def __init__(self, page_size: int) -> None:+ super().__init__()+ self.page_size = page_size++ def update_state(self, response: Response, data: Optional[list[Any]] = None) -> None:+ if data is not None and len(data) >= self.page_size:+ raise ValueError(f"{ZUORA_INCOMPLETE_LIST_ERROR} ({len(data)} rows), so the table could be incomplete.")+ super().update_state(response, data)++ def __str__(self) -> str:+ return f"ZuoraFullPageGuardPaginator(page_size={self.page_size})"+++def _object_query_endpoint(+ endpoint: str, should_use_incremental_field: bool, db_incremental_field_last_value: Optional[Any]+) -> Endpoint:+ # Filters use lowercase field names (updateddate); rows come back camelCase (updatedDate).+ # Ascending sort lets the pipeline commit the watermark progressively as pages complete.+ params: dict[str, Any] = {"pageSize": PAGE_SIZE, "sort[]": "updateddate.ASC"}+ if should_use_incremental_field and db_incremental_field_last_value is not None:+ params["filter[]"] = f"updateddate.GT:{_format_timestamp(db_incremental_field_last_value)}"++ return {+ "path": f"object-query/{ZUORA_ENDPOINTS[endpoint]}",+ "params": params,+ "data_selector": "data",+ "paginator": ZuoraCursorPaginator(),+ }+++def _commerce_endpoint(endpoint: str) -> Endpoint:+ commerce = ZUORA_COMMERCE_ENDPOINTS[endpoint]+ params: dict[str, Any] = {}+ paginator: PaginatorConfig = "single_page"+ if commerce.paginated:+ paginator = JSONResponseCursorPaginator(+ cursor_path="nextPageToken", cursor_param="pageToken", raise_on_repeated_cursor=True+ )+ elif commerce.page_size_param is not None:+ params[commerce.page_size_param] = PAGE_SIZE+ paginator = ZuoraFullPageGuardPaginator(PAGE_SIZE)+ return {+ "path": commerce.path,+ "params": params,+ "data_selector": commerce.data_selector,+ "paginator": paginator,+ }++def validate_credentials(environment: str, client_id: str, client_secret: str) -> bool:"""Confirm the OAuth client credentials are valid by minting a token."""auth = _make_auth(environment, client_id, client_secret)@@ -145,17 +208,10 @@ def zuora_source(should_use_incremental_field: bool = False,db_incremental_field_last_value: Optional[Any] = None,) -> SourceResponse:- path_segment = ZUORA_ENDPOINTS[endpoint]auth = _make_auth(environment, client_id, client_secret)resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None- # Filters use lowercase field names (updateddate); rows come back camelCase (updatedDate).- # Ascending sort lets the pipeline commit the watermark progressively as pages complete.- params: dict[str, Any] = {"pageSize": PAGE_SIZE, "sort[]": "updateddate.ASC"}- if should_use_incremental_field and db_incremental_field_last_value is not None:- params["filter[]"] = f"updateddate.GT:{_format_timestamp(db_incremental_field_last_value)}"-initial_paginator_state: Optional[dict[str, Any]] = Noneif resume is not None and resume.cursor:initial_paginator_state = {"cursor": resume.cursor}@@ -166,12 +222,11 @@ def zuora_source(if state is not None and state.get("cursor") is not None:resumable_source_manager.save_state(ZuoraResumeConfig(cursor=state["cursor"]))- endpoint_config: Endpoint = {- "path": f"object-query/{path_segment}",- "params": params,- "data_selector": "data",- "paginator": ZuoraCursorPaginator(),- }+ endpoint_config = (+ _commerce_endpoint(endpoint)+ if endpoint in ZUORA_COMMERCE_ENDPOINTS+ else _object_query_endpoint(endpoint, should_use_incremental_field, db_incremental_field_last_value)+ )client_config: ClientConfig = {"base_url": _base_url(environment),"auth": auth,@@ -198,6 +253,6 @@ def zuora_source(primary_keys=["id"],partition_count=1,partition_size=1,- # Pages are requested sorted ascending by updateddate.+ # Object Query pages are requested sorted ascending by updateddate; Commerce lists are full refresh only.sort_mode="asc",)