:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 5 missing tables to the Recharge source (#123657)
Tom Owers305d62456682
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 1765c645b23c..46d80a4e4121 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -10087,10 +10087,10 @@ Today (10): `addresses`, `charges`, `collections`, `customers`, `discounts`, `onDiffed against: <https://developer.rechargepayments.com/2021-11>-- [ ] `plans` — Lookup table resolving the plan IDs carried on subscriptions and onetimes (high)-- [ ] `events` — Recharge's account event log - subscription state transitions, cancellations and churn reasons over time (high)-- [ ] `credit_accounts and credit_adjustments` — Customer store-credit ledger; adjustments are the transaction-level rows behind each balance (medium)-- [ ] `bundle_selections` — Line-item contents of bundle subscriptions - without it a bundle subscription has no product breakdown (medium)+- [x] `plans` — Lookup table resolving the plan IDs carried on subscriptions and onetimes (high)+- [x] `events` — Recharge's account event log - subscription state transitions, cancellations and churn reasons over time (high)+- [x] `credit_accounts and credit_adjustments` — Customer store-credit ledger; adjustments are the transaction-level rows behind each balance (medium)+- [x] `bundle_selections` — Line-item contents of bundle subscriptions - without it a bundle subscription has no product breakdown (medium)- [ ] `checkouts` — Pre-conversion checkout records, required for abandoned-checkout and funnel analysis (medium)- [ ] `customer_entitlements` — Membership/perk entitlements per customer, a membership-program breakdown dimension (medium)- [ ] `metafields` — Custom key-value fields attached to customers, subscriptions and addresses that often hold merchant-specific segmentation (low)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/api_inventory.md b/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/api_inventory.mdindex 5206dbd0a531..80f954b31d2a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/api_inventory.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/api_inventory.md@@ -21,18 +21,23 @@ Source: Recharge Payments — subscription management for e-commerce.## Endpoints-| Schema | Path | Primary key | Partition key | Incremental | Notes |-| ----------------- | ------------------ | ----------- | ------------- | ----------- | ---------------------------------------------------------------------------------- |-| `customers` | `/customers` | `id` | `created_at` | ✅ | |-| `subscriptions` | `/subscriptions` | `id` | `created_at` | ✅ | |-| `orders` | `/orders` | `id` | `created_at` | ✅ | |-| `charges` | `/charges` | `id` | `created_at` | ✅ | 90-day retention for `processed_at` filtering (we use `updated_at`/`created_at`). |-| `addresses` | `/addresses` | `id` | `created_at` | ✅ | |-| `discounts` | `/discounts` | `id` | `created_at` | ✅ | |-| `onetimes` | `/onetimes` | `id` | `created_at` | ✅ | |-| `products` | `/products` | `id` | `created_at` | ➖ full | 2021-11 list endpoint takes only cursor + `limit`/`ids` — no `sort_by` or `*_min`. |-| `payment_methods` | `/payment_methods` | `id` | `created_at` | ✅ | Requires Recharge Pro / Custom plan. |-| `collections` | `/collections` | `id` | `created_at` | ➖ full | Server-side timestamp filter not reliably documented — full refresh. |+| Schema | Path | Primary key | Partition key | Incremental | Notes |+| -------------------- | --------------------- | ----------- | ------------- | ----------- | ---------------------------------------------------------------------------------------- |+| `customers` | `/customers` | `id` | `created_at` | ✅ | |+| `subscriptions` | `/subscriptions` | `id` | `created_at` | ✅ | |+| `orders` | `/orders` | `id` | `created_at` | ✅ | |+| `charges` | `/charges` | `id` | `created_at` | ✅ | 90-day retention for `processed_at` filtering (we use `updated_at`/`created_at`). |+| `addresses` | `/addresses` | `id` | `created_at` | ✅ | |+| `discounts` | `/discounts` | `id` | `created_at` | ✅ | |+| `onetimes` | `/onetimes` | `id` | `created_at` | ✅ | |+| `products` | `/products` | `id` | `created_at` | ➖ full | 2021-11 list endpoint takes only cursor + `limit`/`ids` — no `sort_by` or `*_min`. |+| `payment_methods` | `/payment_methods` | `id` | `created_at` | ✅ | Requires Recharge Pro / Custom plan. |+| `collections` | `/collections` | `id` | `created_at` | ➖ full | Server-side timestamp filter not reliably documented — full refresh. |+| `plans` | `/plans` | `id` | `created_at` | ✅ | Only `updated_at_min`. No `sort_by`, so rows come in API order. |+| `events` | `/events` | `id` | `created_at` | ✅ | Only `created_at_min`. No `sort_by`. Recharge keeps 7 days of events. Plus plan. |+| `credit_accounts` | `/credit_accounts` | `id` | `created_at` | ➖ full | Filters only on `expires_at`. No `sort_by`. Plus plan. |+| `credit_adjustments` | `/credit_adjustments` | `id` | `created_at` | ✅ | `updated_at_min` / `created_at_min`. Sort param has no documented name, so none is sent. |+| `bundle_selections` | `/bundle_selections` | `id` | `created_at` | ➖ full | Accepts `sort_by` (`id-asc`) but has no timestamp filter. Plus plan. |## Possible future work@@ -41,5 +46,4 @@ Source: Recharge Payments — subscription management for e-commerce.webhooks are **per-topic** (one webhook per `subscription/created`, `charge/created`, …),which doesn't map cleanly onto the one-webhook-per-source `create_webhook` / `delete_webhook`contract — so realtime webhook ingestion is intentionally deferred from this initial alpha.-- **Metafields / plan-specific resources** (bundle selections, credit adjustments) require a- fan-out or a Pro/Custom plan and are omitted from the initial endpoint set.+- **Metafields** are omitted from the endpoint set.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/canonical_descriptions.pyindex a095a7b858da..02c4844a89ce 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/canonical_descriptions.py@@ -172,4 +172,70 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {sort_order="Sort order applied to products in the collection.",),},+ "plans": {+ "description": "A plan that sets the subscription, prepaid or one-time purchase options of a product.",+ "docs_url": "https://developer.rechargepayments.com/2021-11/plans/plans_object",+ "columns": _columns(+ title="The title of the plan.",+ type="Type of plan (subscription, prepaid or onetime).",+ external_product_id="ID of the product in the connected ecommerce platform.",+ discount_amount="Discount applied to the product price when bought through this plan.",+ discount_type="How the discount amount applies (percentage or fixed_amount).",+ subscription_preferences="Subscription settings of the plan, such as the charge and order intervals.",+ channel_settings="Channels through which the plan is available.",+ sort_order="Position of the plan in a list of related plans.",+ deleted_at="Time at which the plan was deleted, if it was deleted.",+ ),+ },+ "events": {+ "description": "An action that took place in the store, such as an update to a subscription.",+ "docs_url": "https://developer.rechargepayments.com/2021-11/events/event_object",+ "columns": _columns(+ object_id="ID of the object the event is about.",+ object_type="Resource type of the object the event is about (e.g. customer, subscription).",+ customer_id="ID of the customer associated with the event, if any.",+ verb="The action of the event (e.g. created, updated).",+ description="Short summary of the event.",+ updated_attributes="Attributes changed by the event, with their previous and new values.",+ source="Details about the origin of the event.",+ custom_attributes="Custom key-value pairs passed to the Recharge API call that created the event.",+ ),+ },+ "credit_accounts": {+ "description": "A store-credit account that holds a credit balance for a customer.",+ "docs_url": "https://developer.rechargepayments.com/2021-11/credits/credit_account_object",+ "columns": _columns(+ customer_id="ID of the customer the credit account belongs to.",+ available_balance="Amount of credit available in the account.",+ initial_value="Value of the credit account when it was created.",+ currency_code="Currency code of the credit account.",+ expires_at="Time at which the credit account expires.",+ name="Name of the credit account.",+ type="Type of the credit account.",+ ),+ },+ "credit_adjustments": {+ "description": "A credit or debit entry that changes the balance of a credit account.",+ "docs_url": "https://developer.rechargepayments.com/2021-11/credits/credit_adjustment_list",+ "columns": _columns(+ credit_account_id="ID of the credit account the adjustment applies to.",+ amount="Amount of the adjustment.",+ type="Direction of the adjustment (credit or debit).",+ ending_balance="Balance of the credit account after the adjustment.",+ currency_code="Currency code of the adjustment.",+ note="Note that describes the adjustment.",+ created_by="Details about who or what created the adjustment.",+ ),+ },+ "bundle_selections": {+ "description": "The contents selected for a bundle that is linked to a subscription or one-time purchase.",+ "docs_url": "https://developer.rechargepayments.com/2021-11/bundle_selections/bundle_selection_object",+ "columns": _columns(+ bundle_variant_id="ID of the bundle variant used to validate the selection.",+ purchase_item_id="ID of the subscription or one-time purchase the selection belongs to.",+ external_product_id="ID of the bundle product in the connected ecommerce platform.",+ external_variant_id="ID of the bundle variant in the connected ecommerce platform.",+ items="Products selected in the bundle, with their quantities.",+ ),+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/recharge.py b/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/recharge.pyindex 3ce443e34625..872c772935b1 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/recharge.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/recharge.py@@ -258,7 +258,9 @@ def recharge_source(primary_keys=["id"],partition_count=1,partition_size=1,- sort_mode="asc",+ # Without `sort_by` the rows come in the API's default order, not by the+ # incremental field.+ sort_mode="asc" if config.supports_sort else None,partition_mode="datetime" if config.partition_key else None,partition_format="month" if config.partition_key else None,partition_keys=[config.partition_key] if config.partition_key else None,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/settings.pyindex cdf6ffeb7f15..edc2a0410154 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/settings.py@@ -124,6 +124,45 @@ RECHARGE_ENDPOINTS: dict[str, RechargeEndpointConfig] = {incremental_fields=[],supports_incremental=False,),+ # `/plans` filters on `updated_at_min` but documents no `sort_by` and no+ # `created_at_min`, so it syncs incrementally on `updated_at` in API order.+ "plans": RechargeEndpointConfig(+ name="plans",+ path="/plans",+ incremental_fields=[_timestamp_field("updated_at")],+ supports_sort=False,+ ),+ # Events are append-only and Recharge keeps only the last 7 days, so an+ # incremental sync on `created_at` is the only way to keep a longer history.+ "events": RechargeEndpointConfig(+ name="events",+ path="/events",+ incremental_fields=[_timestamp_field("created_at")],+ supports_sort=False,+ ),+ # `/credit_accounts` filters only on `expires_at` and documents no sort.+ "credit_accounts": RechargeEndpointConfig(+ name="credit_accounts",+ path="/credit_accounts",+ incremental_fields=[],+ supports_incremental=False,+ supports_sort=False,+ ),+ # The docs list the sort param of `/credit_adjustments` without a name, so+ # no `sort_by` is sent.+ "credit_adjustments": RechargeEndpointConfig(+ name="credit_adjustments",+ path="/credit_adjustments",+ incremental_fields=_UPDATED_AND_CREATED,+ supports_sort=False,+ ),+ # `/bundle_selections` accepts `sort_by` but has no timestamp filter.+ "bundle_selections": RechargeEndpointConfig(+ name="bundle_selections",+ path="/bundle_selections",+ incremental_fields=[],+ supports_incremental=False,+ ),}ENDPOINTS = tuple(RECHARGE_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/tests/test_recharge.py b/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/tests/test_recharge.pyindex 5a2dcd0324be..9f4d44b0148d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/tests/test_recharge.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/recharge/tests/test_recharge.py@@ -57,6 +57,23 @@ class TestBuildInitialParams:assert params["sort_by"] == "created_at-asc"assert "updated_at_min" not in params+ @parameterized.expand(+ [+ ("plans", "updated_at"),+ ("events", "created_at"),+ ("credit_adjustments", "updated_at"),+ ("credit_adjustments", "created_at"),+ ]+ )+ def test_incremental_on_unsortable_endpoint_omits_sort_by(self, endpoint: str, incremental_field: str) -> None:+ params = _build_initial_params(+ RECHARGE_ENDPOINTS[endpoint],+ should_use_incremental_field=True,+ db_incremental_field_last_value=datetime(2026, 1, 1, tzinfo=UTC),+ incremental_field=incremental_field,+ )+ assert params == {"limit": 250, f"{incremental_field}_min": "2026-01-01T00:00:00"}+@parameterized.expand([("default_endpoint_uses_max_page_size", "customers", 250),@@ -171,6 +188,25 @@ class TestRechargeSource:assert response is not Noneassert response.name == endpointassert response.primary_keys == ["id"]- assert response.sort_mode == "asc"assert response.partition_mode == "datetime"assert response.partition_keys == ["created_at"]++ @parameterized.expand(+ [+ ("sorted_endpoint", "customers", "asc"),+ ("unsorted_endpoint", "events", None),+ ]+ )+ def test_sort_mode_is_asc_only_when_the_request_sorts(+ self, _name: str, endpoint: str, expected_sort_mode: str | None+ ) -> None:+ response = _drive(+ endpoint,+ [_page(endpoint, [], None)],+ should_use_incremental_field=True,+ incremental_field="created_at",+ db_incremental_field_last_value=datetime(2026, 1, 1, tzinfo=UTC),+ ).response++ assert response is not None+ assert response.sort_mode == expected_sort_mode