:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse-sources): add e-conomic vat, payment type, voucher and totals tables (#119543)
Tom Owersa72257ea2d23
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex fb4bdf2623f8..558c83587faa 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -4168,7 +4168,7 @@ Note: The spec has only 35 operations and everything is promotion-scoped (the so## EConomic — gaps-Today (23): `accounting_year_entries`, `accounting_years`, `accounts`, `currencies`, `customer_groups`, `customers`, `departmental_distributions`, `departments`, `employees`, `invoices_booked`, `invoices_drafts`, `journal_entries`, `journals`, `orders_archived`, `orders_drafts`, `orders_sent`, `payment_terms`, `product_groups`, `products`, `supplier_groups`, `suppliers`, `units`, `vat_zones`+Today (29): `accounting_period_totals`, `accounting_year_entries`, `accounting_year_totals`, `accounting_years`, `accounts`, `currencies`, `customer_groups`, `customers`, `departmental_distributions`, `departments`, `employees`, `invoices_booked`, `invoices_drafts`, `journal_entries`, `journal_vouchers`, `journals`, `orders_archived`, `orders_drafts`, `orders_sent`, `payment_terms`, `payment_types`, `product_groups`, `products`, `supplier_groups`, `suppliers`, `units`, `vat_accounts`, `vat_types`, `vat_zones`Diffed against: <https://restdocs.e-conomic.com/>@@ -4176,13 +4176,13 @@ Diffed against: <https://restdocs.e-conomic.com/>- [x] `GET /accounting-years/{year}/entries` — full general-ledger entry stream for a financial year - the core fact table of the whole API (high)- [x] `GET /accounts/{accountNumber}/accounting-years/{year}/entries` — per-account ledger entries, letting you reconcile the accounts table already synced (high). Skipped: it returns a per-account subset of `/accounting-years/{year}/entries`, which is now synced as `accounting_year_entries` and carries `account` on every row, so a second table would only duplicate it.- [x] `GET /orders, /orders/drafts, /orders/sent, /orders/archived` — sales orders with their state slices - the pre-invoice half of the revenue pipeline is entirely missing (high)-- [ ] `GET /vat-types and GET /vat-accounts` — VAT code lookup resolving the vatZone/vatAccount references already carried on accounts, products and invoice lines (high)-- [ ] `GET /accounting-years/{year}/totals and /accounting-years/{year}/periods/{p}/totals` — account balances per year and per period - the trial-balance figures, without recomputing from entries (high)+- [x] `GET /vat-types and GET /vat-accounts` — VAT code lookup resolving the vatZone/vatAccount references already carried on accounts, products and invoice lines (high)+- [x] `GET /accounting-years/{year}/totals and /accounting-years/{year}/periods/{p}/totals` — account balances per year and per period - the trial-balance figures, without recomputing from entries (high)- [ ] `GET /accounting-years/{year}/periods` — accounting-period lookup that every entry and total is bucketed by (medium)- [ ] `GET /quotes, /quotes/drafts, /quotes/sent, /quotes/archived` — quotes and their lifecycle state, the top of the quote-to-cash funnel (medium)- [ ] `GET /customers/{customerNumber}/contacts` — customer contact records for joining people to the customers already synced (medium)-- [ ] `GET /payment-types` — payment type lookup referenced by invoices and payment terms (medium)-- [ ] `GET /journals/{journalNumber}/vouchers` — voucher records grouping entries, with attachment metadata (medium)+- [x] `GET /payment-types` — payment type lookup referenced by invoices and payment terms (medium)+- [x] `GET /journals/{journalNumber}/vouchers` — voucher records grouping entries, with attachment metadata (medium)- [ ] `GET /invoices/sent, /paid, /unpaid, /overdue, /not-due` — AR ageing state slices; derivable from booked invoices but these are the views finance actually asks for (medium)Note: Resource list derived from the JSON-schema filenames embedded in restdocs.e-conomic.com (they mirror the path tree exactly, e.g. journals.journalNumber.entries.get.schema.json). Also missing but below the cut: /customers/{n}/delivery-locations, /layouts, /products/{n}/pricing/currency-specific-sales-prices, /product-groups/{n}/sales-accounts, /invoices/totals/\*. app-roles and /self are config and were excluded.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/api_inventory.md b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/api_inventory.mdindex 33777d2e85b9..586685e3afe5 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/api_inventory.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/api_inventory.md@@ -33,31 +33,37 @@ live API (`https://restapi.e-conomic.com`) using e-conomic's public demo agreeme## Endpoint matrix-| Schema | Path | Primary key | Sort | Incremental | Notes |-| ---------------------------- | ---------------------------------- | ------------------------------------- | -------------------------------- | ---------------------------------- | -------------------------------------------------------------- |-| `customers` | `/customers` | `customerNumber` | `lastUpdated` | ✅ `lastUpdated` (DateTime) | |-| `customer_groups` | `/customer-groups` | `customerGroupNumber` | `customerGroupNumber` | full refresh | |-| `products` | `/products` | `productNumber` | `lastUpdated` | ✅ `lastUpdated` (DateTime) | |-| `product_groups` | `/product-groups` | `productGroupNumber` | `productGroupNumber` | full refresh | |-| `suppliers` | `/suppliers` | `supplierNumber` | `supplierNumber` | full refresh | no `lastUpdated` field; filter on it errors |-| `supplier_groups` | `/supplier-groups` | `supplierGroupNumber` | `supplierGroupNumber` | full refresh | |-| `accounts` | `/accounts` | `accountNumber` | `accountNumber` | full refresh | chart of accounts |-| `accounting_years` | `/accounting-years` | `year` | `year` | full refresh | |-| `journals` | `/journals` | `journalNumber` | `journalNumber` | full refresh | |-| `currencies` | `/currencies` | `code` | `code` | full refresh | |-| `payment_terms` | `/payment-terms` | `paymentTermsNumber` | _(none)_ | full refresh | `sort=paymentTermsNumber` → 400; tiny single-page table |-| `departments` | `/departments` | `departmentNumber` | `departmentNumber` | full refresh | |-| `departmental_distributions` | `/departmental-distributions` | `departmentalDistributionNumber` | `departmentalDistributionNumber` | full refresh | |-| `units` | `/units` | `unitNumber` | `unitNumber` | full refresh | |-| `vat_zones` | `/vat-zones` | `vatZoneNumber` | `vatZoneNumber` | full refresh | |-| `employees` | `/employees` | `employeeNumber` | `employeeNumber` | full refresh | |-| `invoices_booked` | `/invoices/booked` | `bookedInvoiceNumber` | `bookedInvoiceNumber` | ✅ `bookedInvoiceNumber` (Integer) | immutable; partition by stable `date` |-| `invoices_drafts` | `/invoices/drafts` | `draftInvoiceNumber` | `draftInvoiceNumber` | full refresh | has `lastUpdated` filter, but `sort=lastUpdated` → 400 |-| `orders_drafts` | `/orders/drafts` | `orderNumber` | `orderNumber` | full refresh | rows leave the table when the order is sent |-| `orders_sent` | `/orders/sent` | `orderNumber` | `orderNumber` | full refresh | rows leave the table when the order is archived |-| `orders_archived` | `/orders/archived` | `orderNumber` | `orderNumber` | full refresh | no `lastUpdated`; partition by `date` |-| `accounting_year_entries` | `/accounting-years/{year}/entries` | `accountingYear`, `entryNumber` | `entryNumber` | full refresh | fan-out over `accounting_years` via each year's `entries` link |-| `journal_entries` | `/journals/{n}/entries` | `journalNumber`, `journalEntryNumber` | `journalEntryNumber` | full refresh | fan-out over `journals`; unbooked entries only |+| Schema | Path | Primary key | Sort | Incremental | Notes |+| ---------------------------- | --------------------------------------------- | -------------------------------------------------- | -------------------------------- | ---------------------------------- | -------------------------------------------------------------- |+| `customers` | `/customers` | `customerNumber` | `lastUpdated` | ✅ `lastUpdated` (DateTime) | |+| `customer_groups` | `/customer-groups` | `customerGroupNumber` | `customerGroupNumber` | full refresh | |+| `products` | `/products` | `productNumber` | `lastUpdated` | ✅ `lastUpdated` (DateTime) | |+| `product_groups` | `/product-groups` | `productGroupNumber` | `productGroupNumber` | full refresh | |+| `suppliers` | `/suppliers` | `supplierNumber` | `supplierNumber` | full refresh | no `lastUpdated` field; filter on it errors |+| `supplier_groups` | `/supplier-groups` | `supplierGroupNumber` | `supplierGroupNumber` | full refresh | |+| `accounts` | `/accounts` | `accountNumber` | `accountNumber` | full refresh | chart of accounts |+| `accounting_years` | `/accounting-years` | `year` | `year` | full refresh | |+| `journals` | `/journals` | `journalNumber` | `journalNumber` | full refresh | |+| `currencies` | `/currencies` | `code` | `code` | full refresh | |+| `payment_terms` | `/payment-terms` | `paymentTermsNumber` | _(none)_ | full refresh | `sort=paymentTermsNumber` → 400; tiny single-page table |+| `departments` | `/departments` | `departmentNumber` | `departmentNumber` | full refresh | |+| `departmental_distributions` | `/departmental-distributions` | `departmentalDistributionNumber` | `departmentalDistributionNumber` | full refresh | |+| `units` | `/units` | `unitNumber` | `unitNumber` | full refresh | |+| `vat_zones` | `/vat-zones` | `vatZoneNumber` | `vatZoneNumber` | full refresh | |+| `vat_types` | `/vat-types` | `vatTypeNumber` | _(none)_ | full refresh | `sort` is accepted but ignored |+| `vat_accounts` | `/vat-accounts` | `vatCode` | _(none)_ | full refresh | `sort` is accepted but ignored |+| `payment_types` | `/payment-types` | `paymentTypeNumber` | _(none)_ | full refresh | `sort` is accepted but ignored |+| `employees` | `/employees` | `employeeNumber` | `employeeNumber` | full refresh | |+| `invoices_booked` | `/invoices/booked` | `bookedInvoiceNumber` | `bookedInvoiceNumber` | ✅ `bookedInvoiceNumber` (Integer) | immutable; partition by stable `date` |+| `invoices_drafts` | `/invoices/drafts` | `draftInvoiceNumber` | `draftInvoiceNumber` | full refresh | has `lastUpdated` filter, but `sort=lastUpdated` → 400 |+| `orders_drafts` | `/orders/drafts` | `orderNumber` | `orderNumber` | full refresh | rows leave the table when the order is sent |+| `orders_sent` | `/orders/sent` | `orderNumber` | `orderNumber` | full refresh | rows leave the table when the order is archived |+| `orders_archived` | `/orders/archived` | `orderNumber` | `orderNumber` | full refresh | no `lastUpdated`; partition by `date` |+| `accounting_year_entries` | `/accounting-years/{year}/entries` | `accountingYear`, `entryNumber` | `entryNumber` | full refresh | fan-out over `accounting_years` via each year's `entries` link |+| `journal_entries` | `/journals/{n}/entries` | `journalNumber`, `journalEntryNumber` | `journalEntryNumber` | full refresh | fan-out over `journals`; unbooked entries only |+| `journal_vouchers` | `/journals/{n}/vouchers` | `journalNumber`, `accountingYear`, `voucherNumber` | _(none)_ | full refresh | fan-out over `journals`; every `sort` field → 400 |+| `accounting_year_totals` | `/accounting-years/{year}/totals` | `accountingYear`, `accountNumber` | _(none)_ | full refresh | fan-out over `accounting_years` via each year's `totals` link |+| `accounting_period_totals` | `/accounting-years/{year}/periods/{p}/totals` | `accountingYear`, `periodNumber`, `accountNumber` | _(none)_ | full refresh | two-hop fan-out: years → periods → totals |## Throttling@@ -72,11 +78,18 @@ live API (`https://restapi.e-conomic.com`) using e-conomic's public demo agreemeyear is part of the primary key.- `journal_entries` reads `/journals/{journalNumber}/entries` once per journal. These are entries thatare not booked yet; booking moves them to the accounting-year entries.-- Both are full refresh. Resume state records which parents finished and the page of the current one.+- `journal_vouchers` reads `/journals/{journalNumber}/vouchers` once per journal. A voucher number is+ unique per journal and accounting year, so `accountingYear` is lifted out of its nested object into+ the primary key.+- `accounting_year_totals` reads each year's `totals` link. `accounting_period_totals` reads each year's+ `periods` link, then each period's `totals` link. Period numbers continue across accounting years+ (for example 25 to 36 for the third year). Both lift `account.accountNumber` into `accountNumber`.+- All are full refresh. Resume state records which parents finished and the page of the current one,+ except for `accounting_period_totals`: the framework keeps no resume state for a two-hop chain.- `/accounts/{accountNumber}/accounting-years/{year}/entries` is not synced: it returns a per-accountsubset of `accounting_year_entries`, which already carries `account`.## Not implemented (yet)-Other sub-resources (customer contacts, invoice lines, vouchers, totals, periods) and the separate+Other sub-resources (customer contacts, invoice lines, accounting-year vouchers, periods as a table) and the separatesub-APIs on `apis.e-conomic.com` (subscriptions, etc.) are out of scope for now.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/canonical_descriptions.pyindex 15f767de76c8..3a59abcfecae 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/canonical_descriptions.py@@ -347,4 +347,68 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"supplierInvoiceNumber": "Identifier of the supplier invoice, if any.",},},+ "vat_types": {+ "description": "A VAT type, which says whether a VAT account is used for sales, purchases or reverse charge.",+ "docs_url": "https://restdocs.e-conomic.com/#get-vat-types",+ "columns": {+ "vatTypeNumber": "Unique identifier of the VAT type.",+ "name": "Name of the VAT type.",+ },+ },+ "vat_accounts": {+ "description": "A VAT code with its rate and the accounts VAT is posted to. Resolves the vatAccount references on entries and other resources.",+ "docs_url": "https://restdocs.e-conomic.com/#get-vat-accounts",+ "columns": {+ "vatCode": "Unique VAT code.",+ "name": "Name of the VAT account.",+ "ratePercentage": "VAT rate in percent.",+ "vatType": "The VAT type of the VAT account.",+ "account": "The account VAT is posted to.",+ "contraAccount": "The contra account used for reverse-charge VAT, if any.",+ },+ },+ "payment_types": {+ "description": "A payment type, which says how a supplier or customer payment is made, for example bank transfer or IBAN.",+ "docs_url": "https://restdocs.e-conomic.com/#get-payment-types",+ "columns": {+ "paymentTypeNumber": "Unique identifier of the payment type.",+ "name": "Name of the payment type.",+ },+ },+ "journal_vouchers": {+ "description": "A voucher in a journal (daybook) that has not been booked yet. A voucher groups the entries posted together.",+ "docs_url": "https://restdocs.e-conomic.com/#get-journals-journalnumber-vouchers",+ "columns": {+ "journalNumber": "The journal the voucher is in.",+ "accountingYear": "The accounting year of the voucher, in the format YYYY or YYYY/YYYY.",+ "voucherNumber": "Identifier of the voucher within its journal and accounting year.",+ "entries": "The entries on the voucher, grouped by entry type.",+ "attachment": "Link to the attachment (for example a scanned receipt) on the voucher.",+ },+ },+ "accounting_year_totals": {+ "description": "The total posted to each account in an accounting year, in the base currency. Gives the trial balance for the year.",+ "docs_url": "https://restdocs.e-conomic.com/#get-accounting-years-accountingyear-totals",+ "columns": {+ "accountingYear": "The accounting year, in the format YYYY or YYYY/YYYY.",+ "accountNumber": "The account the total is for.",+ "account": "The account the total is for.",+ "totalInBaseCurrency": "Total of the entries on the account in the accounting year, in the base currency.",+ "fromDate": "First date of the accounting year.",+ "toDate": "Last date of the accounting year.",+ },+ },+ "accounting_period_totals": {+ "description": "The total posted to each account in an accounting period (usually a month), in the base currency.",+ "docs_url": "https://restdocs.e-conomic.com/#get-accounting-years-accountingyear-periods-accountingyearperiod-totals",+ "columns": {+ "accountingYear": "The accounting year of the period, in the format YYYY or YYYY/YYYY.",+ "periodNumber": "Identifier of the accounting period. Period numbers continue across accounting years.",+ "accountNumber": "The account the total is for.",+ "account": "The account the total is for.",+ "totalInBaseCurrency": "Total of the entries on the account in the period, in the base currency.",+ "fromDate": "First date of the period.",+ "toDate": "Last date of the period.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/e_conomic.py b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/e_conomic.pyindex 7917e2a53833..1be19a3c18a0 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/e_conomic.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/e_conomic.py@@ -1,6 +1,7 @@import dataclasses+from collections.abc import Callablefrom datetime import UTC, date, datetime-from typing import Any, Optional+from typing import Any, Optional, castfrom urllib.parse import urlparsefrom requests import PreparedRequest, Response@@ -12,17 +13,23 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.auth import AuthConfigBasefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (+ build_chained_resource,build_dependent_resource,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (JSONResponsePaginator,)-from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import ClientConfig+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.resource import Resource+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (+ ClientConfig,+ EndpointResource,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.e_conomic.settings import (E_CONOMIC_ENDPOINTS,+ E_CONOMIC_FANOUT_PARENTS,PAGE_SIZE,)@@ -110,12 +117,46 @@ class EConomicNextPagePaginator(JSONResponsePaginator):_assert_trusted_url(self._next_url)-def _add_entries_path(row: dict[str, Any]) -> dict[str, Any]:- row["entries_path"] = urlparse(row["entries"]).path+RowMap = Callable[[dict[str, Any]], dict[str, Any]]+++def _add_link_path(link: str) -> RowMap:+ # Accounting years spanning two calendar years use a different form in URLs than their `year`, so+ # child paths come from the links the API returns.+ def _mapper(row: dict[str, Any]) -> dict[str, Any]:+ row[f"{link}_path"] = urlparse(row[link]).path+ return row++ return _mapper+++def _add_period_fields(row: dict[str, Any]) -> dict[str, Any]:+ row["totals_path"] = urlparse(row["totals"]).path+ row["year"] = row["accountingYear"]["year"]return row-_PARENT_DATA_MAPS = {"accounting_year_entries": _add_entries_path}+def _add_account_number(row: dict[str, Any]) -> dict[str, Any]:+ row["accountNumber"] = row["account"]["accountNumber"]+ return row+++def _flatten_accounting_year(row: dict[str, Any]) -> dict[str, Any]:+ row["accountingYear"] = row["accountingYear"]["year"]+ return row+++_PARENT_DATA_MAPS: dict[str, RowMap] = {+ "accounting_year_entries": _add_link_path("entries"),+ "accounting_year_totals": _add_link_path("totals"),+}++# Lift the ids out of nested reference objects so they can be primary keys.+_CHILD_DATA_MAPS: dict[str, RowMap] = {+ "accounting_year_totals": _add_account_number,+ "accounting_period_totals": _add_account_number,+ "journal_vouchers": _flatten_accounting_year,+}def _client_config(app_secret_token: str, agreement_grant_token: str) -> ClientConfig:@@ -131,6 +172,85 @@ def _client_config(app_secret_token: str, agreement_grant_token: str) -> ClientC}+def _resolve(resource: str, field: str) -> dict[str, Any]:+ return {"type": "resolve", "resource": resource, "field": field}+++def _period_totals_resource(+ endpoint: str,+ client_config: ClientConfig,+ team_id: int,+ job_id: str,+) -> Resource:+ """Build accounting_years -> periods -> totals. A two-hop chain carries no resume state."""+ config = E_CONOMIC_ENDPOINTS[endpoint]+ fanout = config.fanout+ assert fanout is not None+ periods = E_CONOMIC_FANOUT_PARENTS[fanout.parent_name]+ periods_fanout = periods.fanout+ assert periods_fanout is not None+ years = E_CONOMIC_ENDPOINTS[periods_fanout.parent_name]++ resources: list[EndpointResource] = [+ {+ "name": years.name,+ "table_name": years.name,+ "write_disposition": "replace",+ "endpoint": {+ "path": years.path,+ "params": {"pagesize": years.page_size, **periods_fanout.parent_params},+ "data_selector": "collection",+ },+ "data_map": _add_link_path("periods"),+ "table_format": "delta",+ },+ {+ "name": periods.name,+ "table_name": periods.name,+ "write_disposition": "replace",+ "endpoint": {+ "path": periods.path,+ "params": {+ "pagesize": periods.page_size,+ periods_fanout.resolve_param: _resolve(years.name, periods_fanout.resolve_field),+ },+ "data_selector": "collection",+ "paginator": EConomicNextPagePaginator(),+ },+ "data_map": _add_period_fields,+ "table_format": "delta",+ },+ {+ "name": endpoint,+ "table_name": endpoint,+ "write_disposition": "replace",+ "include_from_parent": fanout.include_from_parent,+ "endpoint": {+ "path": config.path,+ "params": {+ "pagesize": config.page_size,+ fanout.resolve_param: _resolve(periods.name, fanout.resolve_field),+ },+ "data_selector": "collection",+ "paginator": EConomicNextPagePaginator(),+ },+ "table_format": "delta",+ },+ ]+ return cast(+ Resource,+ build_chained_resource(+ resources=resources,+ child_name=endpoint,+ parent_name=periods.name,+ parent_field_renames=fanout.parent_field_renames,+ client_config=client_config,+ team_id=team_id,+ job_id=job_id,+ ),+ )++def e_conomic_source(app_secret_token: str,agreement_grant_token: str,@@ -146,29 +266,40 @@ def e_conomic_source(resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else Noneif config.fanout is not None:-- def save_fanout_checkpoint(state: Optional[dict[str, Any]]) -> None:- if state:- resumable_source_manager.save_state(EConomicResumeConfig(fanout_state=state))-- child = build_dependent_resource(- endpoint_configs=E_CONOMIC_ENDPOINTS,- child_endpoint=endpoint,- fanout=config.fanout,- client_config=_client_config(app_secret_token, agreement_grant_token),- parent_data_map=_PARENT_DATA_MAPS.get(endpoint),- parent_endpoint_extra={"data_selector": "collection"},- # Explicit, because the framework treats a path ending in a placeholder ("{entries_path}") as a- # single-entity lookup and would otherwise fetch only its first page.- child_endpoint_extra={"data_selector": "collection", "paginator": EConomicNextPagePaginator()},- page_size_param="pagesize",- path_format_values={},- team_id=team_id,- job_id=job_id,- db_incremental_field_last_value=None,- resume_hook=save_fanout_checkpoint,- initial_paginator_state=resume.fanout_state if resume else None,- )+ client_config = _client_config(app_secret_token, agreement_grant_token)+ if config.fanout.parent_name in E_CONOMIC_FANOUT_PARENTS:+ child = _period_totals_resource(endpoint, client_config, team_id, job_id)+ else:++ def save_fanout_checkpoint(state: Optional[dict[str, Any]]) -> None:+ if state:+ resumable_source_manager.save_state(EConomicResumeConfig(fanout_state=state))++ child = cast(+ Resource,+ build_dependent_resource(+ endpoint_configs=E_CONOMIC_ENDPOINTS,+ child_endpoint=endpoint,+ fanout=config.fanout,+ client_config=client_config,+ parent_data_map=_PARENT_DATA_MAPS.get(endpoint),+ parent_endpoint_extra={"data_selector": "collection"},+ # Explicit, because the framework treats a path ending in a placeholder ("{entries_path}") as a+ # single-entity lookup and would otherwise fetch only its first page.+ child_endpoint_extra={"data_selector": "collection", "paginator": EConomicNextPagePaginator()},+ page_size_param="pagesize",+ path_format_values={},+ team_id=team_id,+ job_id=job_id,+ db_incremental_field_last_value=None,+ resume_hook=save_fanout_checkpoint,+ initial_paginator_state=resume.fanout_state if resume else None,+ ),+ )++ child_map = _CHILD_DATA_MAPS.get(endpoint)+ if child_map is not None:+ child.add_map(child_map)return SourceResponse(name=endpoint,items=lambda: child,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/settings.pyindex f2e9db6e2f18..c72ddb87969b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/settings.py@@ -139,6 +139,23 @@ E_CONOMIC_ENDPOINTS: dict[str, EConomicEndpointConfig] = {primary_keys=["vatZoneNumber"],sort="vatZoneNumber",),+ # The API accepts `sort` on these three lookups but ignores it, so no sort is sent. Each is a tiny+ # single-page table.+ "vat_types": EConomicEndpointConfig(+ name="vat_types",+ path="/vat-types",+ primary_keys=["vatTypeNumber"],+ ),+ "vat_accounts": EConomicEndpointConfig(+ name="vat_accounts",+ path="/vat-accounts",+ primary_keys=["vatCode"],+ ),+ "payment_types": EConomicEndpointConfig(+ name="payment_types",+ path="/payment-types",+ primary_keys=["paymentTypeNumber"],+ ),"employees": EConomicEndpointConfig(name="employees",path="/employees",@@ -220,6 +237,64 @@ E_CONOMIC_ENDPOINTS: dict[str, EConomicEndpointConfig] = {child_params={"sort": "journalEntryNumber"},),),+ # Unbooked vouchers, so full refresh like journal_entries. The API rejects every `sort` field here.+ "journal_vouchers": EConomicEndpointConfig(+ name="journal_vouchers",+ path="/journals/{journalNumber}/vouchers",+ # A voucher number is unique per journal and accounting year.+ primary_keys=["journalNumber", "accountingYear", "voucherNumber"],+ fanout=DependentEndpointConfig(+ parent_name="journals",+ resolve_param="journalNumber",+ resolve_field="journalNumber",+ include_from_parent=["journalNumber"],+ parent_field_renames={"journalNumber": "journalNumber"},+ parent_params={"sort": "journalNumber"},+ ),+ ),+ # Balances change while a year is open, so totals are full refresh. The API ignores `sort` here.+ "accounting_year_totals": EConomicEndpointConfig(+ name="accounting_year_totals",+ path="{totals_path}",+ primary_keys=["accountingYear", "accountNumber"],+ fanout=DependentEndpointConfig(+ parent_name="accounting_years",+ resolve_param="totals_path",+ resolve_field="totals_path",+ include_from_parent=["year"],+ parent_field_renames={"year": "accountingYear"},+ parent_params={"sort": "year"},+ ),+ ),+ "accounting_period_totals": EConomicEndpointConfig(+ name="accounting_period_totals",+ path="{totals_path}",+ primary_keys=["accountingYear", "periodNumber", "accountNumber"],+ fanout=DependentEndpointConfig(+ parent_name="accounting_year_periods",+ resolve_param="totals_path",+ resolve_field="totals_path",+ include_from_parent=["periodNumber", "year"],+ parent_field_renames={"periodNumber": "periodNumber", "year": "accountingYear"},+ ),+ ),+}++# Fan-out parents that are not tables themselves. `accounting_period_totals` reaches its totals through+# two hops: accounting_years -> accounting_year_periods -> totals.+E_CONOMIC_FANOUT_PARENTS: dict[str, EConomicEndpointConfig] = {+ "accounting_year_periods": EConomicEndpointConfig(+ name="accounting_year_periods",+ path="{periods_path}",+ primary_keys=["periodNumber"],+ fanout=DependentEndpointConfig(+ parent_name="accounting_years",+ resolve_param="periods_path",+ resolve_field="periods_path",+ include_from_parent=[],+ parent_params={"sort": "year"},+ ),+ ),}ENDPOINTS = tuple(E_CONOMIC_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/tests/test_e_conomic.py b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/tests/test_e_conomic.pyindex d9afd0c46d17..ce6fdd2d4bc1 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/tests/test_e_conomic.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/tests/test_e_conomic.py@@ -165,6 +165,85 @@ class TestAccountingYearEntriesFanout:assert "/accounting-years/2022_6_2023/entries" in result.saved_states[-1].fanout_state["completed"]+class TestTotalsAndVouchersFanout:+ def test_year_totals_walk_each_year_through_its_totals_link(self) -> None:+ result = _driver().run(+ "accounting_year_totals",+ [+ _page(+ [+ {"year": "2022", "totals": f"{E_CONOMIC_BASE_URL}/accounting-years/2022/totals"},+ {"year": "2022/2023", "totals": f"{E_CONOMIC_BASE_URL}/accounting-years/2022_6_2023/totals"},+ ]+ ),+ _page([{"account": {"accountNumber": 1000}, "totalInBaseCurrency": 5}]),+ _page([{"account": {"accountNumber": 1000}, "totalInBaseCurrency": 7}]),+ ],+ )++ assert result.raised is None+ assert [(r["accountingYear"], r["accountNumber"], r["totalInBaseCurrency"]) for r in result.rows] == [+ ("2022", 1000, 5),+ ("2022/2023", 1000, 7),+ ]+ assert result.paths[1:] == ["/accounting-years/2022/totals", "/accounting-years/2022_6_2023/totals"]++ def test_period_totals_walk_years_then_periods_then_totals(self) -> None:+ def period(number: int) -> dict[str, Any]:+ return {+ "periodNumber": number,+ "accountingYear": {"year": "2024"},+ # The API echoes the parent request's query string into these links.+ "totals": f"{E_CONOMIC_BASE_URL}/accounting-years/2024/periods/{number}/totals?pagesize=1000",+ }++ second_page = f"{E_CONOMIC_BASE_URL}/accounting-years/2024/periods/25/totals?skippages=1"+ result = _driver().run(+ "accounting_period_totals",+ [+ _page([{"year": "2024", "periods": f"{E_CONOMIC_BASE_URL}/accounting-years/2024/periods"}]),+ _page([period(25), period(26)]),+ _page([{"account": {"accountNumber": 1000}}], next_page=second_page),+ _page([{"account": {"accountNumber": 1010}}]),+ _page([{"account": {"accountNumber": 1000}}]),+ ],+ )++ assert result.raised is None+ assert [(r["accountingYear"], r["periodNumber"], r["accountNumber"]) for r in result.rows] == [+ ("2024", 25, 1000),+ ("2024", 25, 1010),+ ("2024", 26, 1000),+ ]+ assert result.paths == [+ "/accounting-years",+ "/accounting-years/2024/periods",+ "/accounting-years/2024/periods/25/totals",+ "/accounting-years/2024/periods/25/totals",+ "/accounting-years/2024/periods/26/totals",+ ]+ assert result.urls[3] == second_page++ def test_vouchers_carry_journal_and_flat_accounting_year(self) -> None:+ result = _driver().run(+ "journal_vouchers",+ [+ _page([{"journalNumber": 3}, {"journalNumber": 11}]),+ _page([{"voucherNumber": 20001, "accountingYear": {"year": "2022"}}]),+ _page([{"voucherNumber": 0, "accountingYear": {"year": "2024"}}]),+ ],+ )++ assert result.raised is None+ assert [(r["journalNumber"], r["accountingYear"], r["voucherNumber"]) for r in result.rows] == [+ (3, "2022", 20001),+ (11, "2024", 0),+ ]+ assert result.paths[1:] == ["/journals/3/vouchers", "/journals/11/vouchers"]+ # The vouchers endpoint rejects every sort field with a 400.+ assert result.params("sort")[1:] == [None, None]++class TestErrorHandling:@parameterized.expand([("server_error", 500), ("bad_gateway", 503), ("throttled", 429)])def test_retryable_status_is_retried(self, _name: str, status_code: int) -> None: