:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add chargebee disputes, schedules, einvoices (#119212)
Tom Owersc80b50b89fb9
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 484d028181d6..e2d8fe0bc606 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -2301,14 +2301,17 @@ Note: Chameleon's published OpenAPI at developers.chameleon.io/api-reference/ope## Chargebee — gaps-Today (10): `CreditUnits`, `Customers`, `Events`, `Invoices`, `ItemPrices`, `Items`, `Meters`, `Orders`, `Subscriptions`, `Transactions`+Today (13): `CreditUnits`, `Customers`, `Disputes`, `Einvoices`, `Events`, `Invoices`, `ItemPrices`, `Items`, `Meters`, `Orders`, `PaymentSchedules`, `Subscriptions`, `Transactions`Diffed against: <https://raw.githubusercontent.com/chargebee/openapi/main/spec/chargebee_api_v2_pc_v2_spec.json>- [x] `GET /meters` — usage-based billing meters: how usage of each metered feature is measured (medium)- [x] `GET /credit_units` — credit unit definitions that credit-based billing grants and consumes (medium)+- [x] `GET /disputes` — chargebacks and inquiries raised against transactions (medium)+- [x] `GET /payment_schedules` — installment schedules that split an invoice's amount due (medium)+- [x] `GET /einvoices` — e-invoice records and provider status for each invoice or credit note (low)-Note: `metered_features` has only write endpoints (create, archive, reactivate, delete) and no list endpoint; the same configuration is readable through `GET /meters`, whose `id` equals the feature id. `ledger_entries` is a documented resource with no list endpoint. The credit ledger is listable through `GET /ledger_operations`.+Note: `metered_features` has only write endpoints (create, archive, reactivate, delete) and no list endpoint; the same configuration is readable through `GET /meters`, whose `id` equals the feature id. `ledger_entries` is a documented resource with no list endpoint. `GET /ledger_operations` lists the credit ledger, but every request needs a `subscription_id[is]` filter, so it is not a site-wide list endpoint and would need one request series per subscription.## Chargedesk — gapsdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/canonical_descriptions.pyindex 6da092651bc6..626667d3dced 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/canonical_descriptions.py@@ -54,6 +54,40 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {deleted="Whether the customer record has been deleted.",),},+ "Disputes": {+ "description": "A dispute a customer raised with their bank or card issuer against a transaction, such as a chargeback or an inquiry.",+ "docs_url": "https://apidocs.chargebee.com/docs/api/disputes",+ "columns": _columns(+ customer_id="Identifier of the customer who raised the dispute.",+ transaction_id="Identifier of the disputed transaction.",+ gateway_account_id="Identifier of the gateway account that processed the disputed transaction.",+ id_at_gateway="Identifier of the dispute at the payment gateway.",+ currency_code="Three-letter ISO currency code of the disputed amount.",+ amount="Disputed amount, in the smallest currency unit.",+ reason="Reason given for the dispute.",+ status="Status of the dispute: initiated, funds_withdrawn, in_review, cancelled, lost, or won.",+ type="Type of the dispute: chargeback or inquiry.",+ is_partial_dispute="Whether the dispute covers only part of the transaction amount.",+ resource_version="Version number that increments whenever the dispute is modified, as a Unix timestamp in milliseconds.",+ ),+ },+ "Einvoices": {+ "description": "An e-invoice record for an invoice or credit note, with its processing status at the connected e-invoicing provider.",+ "docs_url": "https://apidocs.chargebee.com/docs/api/einvoices",+ "columns": _columns(+ entity_type="Type of the parent document: invoice or credit_note.",+ entity_id="Identifier of the parent invoice or credit note.",+ reference_id="Identifier that the e-invoicing provider returned for this submission.",+ reference_number="Reference number that the Invoice Registration Portal (IRP) network assigned to the invoice.",+ status="Processing status of the e-invoice, such as scheduled, in_progress, success, failed, accepted, or rejected.",+ message="Details about the status. When the status is skipped or failed, this holds the reason or error.",+ resource_version="Version number that increments whenever the e-invoice is modified, as a Unix timestamp in milliseconds.",+ deleted="Whether the e-invoice has been deleted.",+ provider_references="Key-value pairs from the e-invoicing provider, such as a receipt message ID.",+ business_entity_id="Identifier of the business entity of the e-invoice. Same as the business entity of the parent document.",+ artifacts="E-invoice artifacts, such as application responses from the provider.",+ ),+ },"Events": {"description": "An event recording a change in Chargebee (subscription created, payment failed, etc.).","docs_url": "https://apidocs.chargebee.com/docs/api/events",@@ -173,6 +207,20 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {currency_code="Three-letter ISO currency code of the order.",),},+ "PaymentSchedules": {+ "description": "A payment schedule that splits the amount due on an invoice into installments, each collected on its own date.",+ "docs_url": "https://apidocs.chargebee.com/docs/api/payment_schedules",+ "columns": _columns(+ scheme_id="Identifier of the payment schedule scheme used to create the schedule.",+ entity_type="Type of the entity the schedule is based on. Always invoice.",+ entity_id="Identifier of the invoice the schedule is based on.",+ amount="Part of the invoice amount due that the schedule distributes, in the smallest currency unit.",+ resource_version="Version number that increments whenever the schedule is modified, as a Unix timestamp in milliseconds.",+ currency_code="Three-letter ISO currency code of the schedule amount.",+ schedule_entries="Installments of the schedule, each with its date, amount, and status.",+ reference_transactions="Transactions referenced against the schedule entries, most recent first.",+ ),+ },"Subscriptions": {"description": "A customer's recurring billing arrangement against one or more plans.","docs_url": "https://apidocs.chargebee.com/docs/api/subscriptions",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/chargebee.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/chargebee.pyindex 4f12ab9e44ce..a5ab73861af5 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/chargebee.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/chargebee.py@@ -75,6 +75,40 @@ def get_resource(},"table_format": "delta",},+ # Disputes only filter on `created_at`, which would miss status changes on existing disputes.+ "Disputes": {+ "name": "Disputes",+ "table_name": "disputes",+ "write_disposition": "replace",+ "endpoint": {+ "data_selector": "list[*].dispute",+ "path": "/v2/disputes",+ "params": {+ "limit": 100,+ },+ },+ "table_format": "delta",+ },+ "Einvoices": {+ "name": "Einvoices",+ "table_name": "einvoices",+ "write_disposition": {+ "disposition": "merge",+ "strategy": "upsert",+ }+ if should_use_incremental_field+ else "replace",+ "endpoint": {+ "data_selector": "list[*].einvoice",+ "path": "/v2/einvoices",+ "params": {+ "updated_at[after]": incremental_param("updated_at") if apply_incremental_filter else None,+ "sort_by[asc]": "updated_at",+ "limit": 100,+ },+ },+ "table_format": "delta",+ },# Note: it is possible to filter by event type, but for now we're# fetching all events"Events": {@@ -194,6 +228,25 @@ def get_resource(},"table_format": "delta",},+ "PaymentSchedules": {+ "name": "PaymentSchedules",+ "table_name": "payment_schedules",+ "write_disposition": {+ "disposition": "merge",+ "strategy": "upsert",+ }+ if should_use_incremental_field+ else "replace",+ "endpoint": {+ "data_selector": "list[*].payment_schedule",+ "path": "/v2/payment_schedules",+ "params": {+ "updated_at[after]": incremental_param("updated_at") if apply_incremental_filter else None,+ "limit": 100,+ },+ },+ "table_format": "delta",+ },"Subscriptions": {"name": "Subscriptions","table_name": "subscriptions",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/settings.pyindex fd3d9e1d4a13..2ac8b8d47840 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/settings.py@@ -3,23 +3,28 @@ from products.warehouse_sources.backend.types import IncrementalField, IncrementENDPOINTS = ("CreditUnits","Customers",+ "Disputes",+ "Einvoices","Events","Invoices","ItemPrices","Items","Meters","Orders",+ "PaymentSchedules","Subscriptions","Transactions",)INCREMENTAL_ENDPOINTS = ("Customers",+ "Einvoices","Events","Invoices","ItemPrices","Items","Orders",+ "PaymentSchedules","Subscriptions","Transactions",)@@ -33,6 +38,14 @@ INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {"field_type": IncrementalFieldType.Integer,},],+ "Einvoices": [+ {+ "label": "updated_at",+ "type": IncrementalFieldType.DateTime,+ "field": "updated_at",+ "field_type": IncrementalFieldType.Integer,+ },+ ],"Events": [{"label": "occurred_at",@@ -73,6 +86,14 @@ INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {"field_type": IncrementalFieldType.Integer,},],+ "PaymentSchedules": [+ {+ "label": "updated_at",+ "type": IncrementalFieldType.DateTime,+ "field": "updated_at",+ "field_type": IncrementalFieldType.Integer,+ },+ ],"Subscriptions": [{"label": "updated_at",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/tests/test_chargebee.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/tests/test_chargebee.pyindex ab51985d5367..6ed2ea5b11ef 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/tests/test_chargebee.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chargebee/tests/test_chargebee.py@@ -48,7 +48,11 @@ class TestChargebeeSourceResumeBehavior:@staticmethoddef _page(endpoint: str, row_id: str, next_offset: str | None = None) -> ScriptedResponse:- wrapper = {"CreditUnits": "credit_unit", "ItemPrices": "item_price"}.get(endpoint, endpoint[:-1].lower())+ wrapper = {+ "CreditUnits": "credit_unit",+ "ItemPrices": "item_price",+ "PaymentSchedules": "payment_schedule",+ }.get(endpoint, endpoint[:-1].lower())body: dict[str, Any] = {"list": [{wrapper: {"id": row_id}}]}if next_offset is not None:body["next_offset"] = next_offset@@ -149,7 +153,8 @@ class TestChargebeeSiteNameValidation:class TestChargebeeCatalogEndpoints:"""`Items` and `ItemPrices` carry the product catalog that `subscription_items` points at.- `CreditUnits` and `Meters` carry the usage-based billing catalog.+ `CreditUnits` and `Meters` carry the usage-based billing catalog. `Disputes`, `PaymentSchedules`+ and `Einvoices` follow the same nested shape.Their rows sit one level deeper than the response list, so a wrong `data_selector` orpath yields an empty table rather than an error.@@ -177,6 +182,24 @@ class TestChargebeeCatalogEndpoints:"meter",{"id": "api-calls", "type": "simple", "query": "SELECT SUM(api_calls) FROM events"},),+ (+ "Disputes",+ "/v2/disputes",+ "dispute",+ {"id": "dp_1", "transaction_id": "txn_1", "status": "in_review", "type": "chargeback"},+ ),+ (+ "PaymentSchedules",+ "/v2/payment_schedules",+ "payment_schedule",+ {"id": "ps_1", "entity_id": "inv_1", "schedule_entries": [{"id": "pse_1", "amount": 500}]},+ ),+ (+ "Einvoices",+ "/v2/einvoices",+ "einvoice",+ {"id": "einv_1", "entity_type": "credit_note", "entity_id": "cn_1", "status": "accepted"},+ ),],)def test_yields_the_nested_catalog_object(@@ -203,11 +226,13 @@ class TestChargebeeIncrementalFilter:CURSOR_PARAMS = {"Customers": "updated_at[after]",+ "Einvoices": "updated_at[after]","Events": "occurred_at[after]","Invoices": "updated_at[after]","ItemPrices": "updated_at[after]","Items": "updated_at[after]","Orders": "updated_at[after]",+ "PaymentSchedules": "updated_at[after]","Subscriptions": "updated_at[after]","Transactions": "updated_at[after]",}@@ -242,7 +267,7 @@ class TestChargebeeIncrementalFilter:assert result.requests[0].param(self.CURSOR_PARAMS[endpoint]) is Noneassert result.raised is None- @pytest.mark.parametrize("endpoint", ["CreditUnits", "Meters"])+ @pytest.mark.parametrize("endpoint", ["CreditUnits", "Disputes", "Meters"])def test_full_refresh_only_endpoints_ignore_incremental_watermark(self, endpoint: str) -> None:result = self._drive(endpoint, incremental=True, last_value=1750000000)