Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse-sources): add e-conomic vat, payment type, voucher and totals tables (#119543)

Tom Owersa72257ea2d23
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index fb4bdf2623f8..558c83587faa 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -4168,7 +4168,7 @@ Note: The spec has only 35 operations and everything is promotion-scoped (the so
## EConomic — gaps
-Today (23): `accounting_year_entries`, `accounting_years`, `accounts`, `currencies`, `customer_groups`, `customers`, `departmental_distributions`, `departments`, `employees`, `invoices_booked`, `invoices_drafts`, `journal_entries`, `journals`, `orders_archived`, `orders_drafts`, `orders_sent`, `payment_terms`, `product_groups`, `products`, `supplier_groups`, `suppliers`, `units`, `vat_zones`
+Today (29): `accounting_period_totals`, `accounting_year_entries`, `accounting_year_totals`, `accounting_years`, `accounts`, `currencies`, `customer_groups`, `customers`, `departmental_distributions`, `departments`, `employees`, `invoices_booked`, `invoices_drafts`, `journal_entries`, `journal_vouchers`, `journals`, `orders_archived`, `orders_drafts`, `orders_sent`, `payment_terms`, `payment_types`, `product_groups`, `products`, `supplier_groups`, `suppliers`, `units`, `vat_accounts`, `vat_types`, `vat_zones`
Diffed against: <https://restdocs.e-conomic.com/>
@@ -4176,13 +4176,13 @@ Diffed against: <https://restdocs.e-conomic.com/>
- [x] `GET /accounting-years/{year}/entries` — full general-ledger entry stream for a financial year - the core fact table of the whole API (high)
- [x] `GET /accounts/{accountNumber}/accounting-years/{year}/entries` — per-account ledger entries, letting you reconcile the accounts table already synced (high). Skipped: it returns a per-account subset of `/accounting-years/{year}/entries`, which is now synced as `accounting_year_entries` and carries `account` on every row, so a second table would only duplicate it.
- [x] `GET /orders, /orders/drafts, /orders/sent, /orders/archived` — sales orders with their state slices - the pre-invoice half of the revenue pipeline is entirely missing (high)
-- [ ] `GET /vat-types and GET /vat-accounts` — VAT code lookup resolving the vatZone/vatAccount references already carried on accounts, products and invoice lines (high)
-- [ ] `GET /accounting-years/{year}/totals and /accounting-years/{year}/periods/{p}/totals` — account balances per year and per period - the trial-balance figures, without recomputing from entries (high)
+- [x] `GET /vat-types and GET /vat-accounts` — VAT code lookup resolving the vatZone/vatAccount references already carried on accounts, products and invoice lines (high)
+- [x] `GET /accounting-years/{year}/totals and /accounting-years/{year}/periods/{p}/totals` — account balances per year and per period - the trial-balance figures, without recomputing from entries (high)
- [ ] `GET /accounting-years/{year}/periods` — accounting-period lookup that every entry and total is bucketed by (medium)
- [ ] `GET /quotes, /quotes/drafts, /quotes/sent, /quotes/archived` — quotes and their lifecycle state, the top of the quote-to-cash funnel (medium)
- [ ] `GET /customers/{customerNumber}/contacts` — customer contact records for joining people to the customers already synced (medium)
-- [ ] `GET /payment-types` — payment type lookup referenced by invoices and payment terms (medium)
-- [ ] `GET /journals/{journalNumber}/vouchers` — voucher records grouping entries, with attachment metadata (medium)
+- [x] `GET /payment-types` — payment type lookup referenced by invoices and payment terms (medium)
+- [x] `GET /journals/{journalNumber}/vouchers` — voucher records grouping entries, with attachment metadata (medium)
- [ ] `GET /invoices/sent, /paid, /unpaid, /overdue, /not-due` — AR ageing state slices; derivable from booked invoices but these are the views finance actually asks for (medium)
Note: Resource list derived from the JSON-schema filenames embedded in restdocs.e-conomic.com (they mirror the path tree exactly, e.g. journals.journalNumber.entries.get.schema.json). Also missing but below the cut: /customers/{n}/delivery-locations, /layouts, /products/{n}/pricing/currency-specific-sales-prices, /product-groups/{n}/sales-accounts, /invoices/totals/\*. app-roles and /self are config and were excluded.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/api_inventory.md b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/api_inventory.md
index 33777d2e85b9..586685e3afe5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/api_inventory.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/api_inventory.md
@@ -33,31 +33,37 @@ live API (`https://restapi.e-conomic.com`) using e-conomic's public demo agreeme
## Endpoint matrix
-| Schema | Path | Primary key | Sort | Incremental | Notes |
-| ---------------------------- | ---------------------------------- | ------------------------------------- | -------------------------------- | ---------------------------------- | -------------------------------------------------------------- |
-| `customers` | `/customers` | `customerNumber` | `lastUpdated` | ✅ `lastUpdated` (DateTime) | |
-| `customer_groups` | `/customer-groups` | `customerGroupNumber` | `customerGroupNumber` | full refresh | |
-| `products` | `/products` | `productNumber` | `lastUpdated` | ✅ `lastUpdated` (DateTime) | |
-| `product_groups` | `/product-groups` | `productGroupNumber` | `productGroupNumber` | full refresh | |
-| `suppliers` | `/suppliers` | `supplierNumber` | `supplierNumber` | full refresh | no `lastUpdated` field; filter on it errors |
-| `supplier_groups` | `/supplier-groups` | `supplierGroupNumber` | `supplierGroupNumber` | full refresh | |
-| `accounts` | `/accounts` | `accountNumber` | `accountNumber` | full refresh | chart of accounts |
-| `accounting_years` | `/accounting-years` | `year` | `year` | full refresh | |
-| `journals` | `/journals` | `journalNumber` | `journalNumber` | full refresh | |
-| `currencies` | `/currencies` | `code` | `code` | full refresh | |
-| `payment_terms` | `/payment-terms` | `paymentTermsNumber` | _(none)_ | full refresh | `sort=paymentTermsNumber` → 400; tiny single-page table |
-| `departments` | `/departments` | `departmentNumber` | `departmentNumber` | full refresh | |
-| `departmental_distributions` | `/departmental-distributions` | `departmentalDistributionNumber` | `departmentalDistributionNumber` | full refresh | |
-| `units` | `/units` | `unitNumber` | `unitNumber` | full refresh | |
-| `vat_zones` | `/vat-zones` | `vatZoneNumber` | `vatZoneNumber` | full refresh | |
-| `employees` | `/employees` | `employeeNumber` | `employeeNumber` | full refresh | |
-| `invoices_booked` | `/invoices/booked` | `bookedInvoiceNumber` | `bookedInvoiceNumber` | ✅ `bookedInvoiceNumber` (Integer) | immutable; partition by stable `date` |
-| `invoices_drafts` | `/invoices/drafts` | `draftInvoiceNumber` | `draftInvoiceNumber` | full refresh | has `lastUpdated` filter, but `sort=lastUpdated` → 400 |
-| `orders_drafts` | `/orders/drafts` | `orderNumber` | `orderNumber` | full refresh | rows leave the table when the order is sent |
-| `orders_sent` | `/orders/sent` | `orderNumber` | `orderNumber` | full refresh | rows leave the table when the order is archived |
-| `orders_archived` | `/orders/archived` | `orderNumber` | `orderNumber` | full refresh | no `lastUpdated`; partition by `date` |
-| `accounting_year_entries` | `/accounting-years/{year}/entries` | `accountingYear`, `entryNumber` | `entryNumber` | full refresh | fan-out over `accounting_years` via each year's `entries` link |
-| `journal_entries` | `/journals/{n}/entries` | `journalNumber`, `journalEntryNumber` | `journalEntryNumber` | full refresh | fan-out over `journals`; unbooked entries only |
+| Schema | Path | Primary key | Sort | Incremental | Notes |
+| ---------------------------- | --------------------------------------------- | -------------------------------------------------- | -------------------------------- | ---------------------------------- | -------------------------------------------------------------- |
+| `customers` | `/customers` | `customerNumber` | `lastUpdated` | ✅ `lastUpdated` (DateTime) | |
+| `customer_groups` | `/customer-groups` | `customerGroupNumber` | `customerGroupNumber` | full refresh | |
+| `products` | `/products` | `productNumber` | `lastUpdated` | ✅ `lastUpdated` (DateTime) | |
+| `product_groups` | `/product-groups` | `productGroupNumber` | `productGroupNumber` | full refresh | |
+| `suppliers` | `/suppliers` | `supplierNumber` | `supplierNumber` | full refresh | no `lastUpdated` field; filter on it errors |
+| `supplier_groups` | `/supplier-groups` | `supplierGroupNumber` | `supplierGroupNumber` | full refresh | |
+| `accounts` | `/accounts` | `accountNumber` | `accountNumber` | full refresh | chart of accounts |
+| `accounting_years` | `/accounting-years` | `year` | `year` | full refresh | |
+| `journals` | `/journals` | `journalNumber` | `journalNumber` | full refresh | |
+| `currencies` | `/currencies` | `code` | `code` | full refresh | |
+| `payment_terms` | `/payment-terms` | `paymentTermsNumber` | _(none)_ | full refresh | `sort=paymentTermsNumber` → 400; tiny single-page table |
+| `departments` | `/departments` | `departmentNumber` | `departmentNumber` | full refresh | |
+| `departmental_distributions` | `/departmental-distributions` | `departmentalDistributionNumber` | `departmentalDistributionNumber` | full refresh | |
+| `units` | `/units` | `unitNumber` | `unitNumber` | full refresh | |
+| `vat_zones` | `/vat-zones` | `vatZoneNumber` | `vatZoneNumber` | full refresh | |
+| `vat_types` | `/vat-types` | `vatTypeNumber` | _(none)_ | full refresh | `sort` is accepted but ignored |
+| `vat_accounts` | `/vat-accounts` | `vatCode` | _(none)_ | full refresh | `sort` is accepted but ignored |
+| `payment_types` | `/payment-types` | `paymentTypeNumber` | _(none)_ | full refresh | `sort` is accepted but ignored |
+| `employees` | `/employees` | `employeeNumber` | `employeeNumber` | full refresh | |
+| `invoices_booked` | `/invoices/booked` | `bookedInvoiceNumber` | `bookedInvoiceNumber` | ✅ `bookedInvoiceNumber` (Integer) | immutable; partition by stable `date` |
+| `invoices_drafts` | `/invoices/drafts` | `draftInvoiceNumber` | `draftInvoiceNumber` | full refresh | has `lastUpdated` filter, but `sort=lastUpdated` → 400 |
+| `orders_drafts` | `/orders/drafts` | `orderNumber` | `orderNumber` | full refresh | rows leave the table when the order is sent |
+| `orders_sent` | `/orders/sent` | `orderNumber` | `orderNumber` | full refresh | rows leave the table when the order is archived |
+| `orders_archived` | `/orders/archived` | `orderNumber` | `orderNumber` | full refresh | no `lastUpdated`; partition by `date` |
+| `accounting_year_entries` | `/accounting-years/{year}/entries` | `accountingYear`, `entryNumber` | `entryNumber` | full refresh | fan-out over `accounting_years` via each year's `entries` link |
+| `journal_entries` | `/journals/{n}/entries` | `journalNumber`, `journalEntryNumber` | `journalEntryNumber` | full refresh | fan-out over `journals`; unbooked entries only |
+| `journal_vouchers` | `/journals/{n}/vouchers` | `journalNumber`, `accountingYear`, `voucherNumber` | _(none)_ | full refresh | fan-out over `journals`; every `sort` field → 400 |
+| `accounting_year_totals` | `/accounting-years/{year}/totals` | `accountingYear`, `accountNumber` | _(none)_ | full refresh | fan-out over `accounting_years` via each year's `totals` link |
+| `accounting_period_totals` | `/accounting-years/{year}/periods/{p}/totals` | `accountingYear`, `periodNumber`, `accountNumber` | _(none)_ | full refresh | two-hop fan-out: years → periods → totals |
## Throttling
@@ -72,11 +78,18 @@ live API (`https://restapi.e-conomic.com`) using e-conomic's public demo agreeme
year is part of the primary key.
- `journal_entries` reads `/journals/{journalNumber}/entries` once per journal. These are entries that
are not booked yet; booking moves them to the accounting-year entries.
-- Both are full refresh. Resume state records which parents finished and the page of the current one.
+- `journal_vouchers` reads `/journals/{journalNumber}/vouchers` once per journal. A voucher number is
+ unique per journal and accounting year, so `accountingYear` is lifted out of its nested object into
+ the primary key.
+- `accounting_year_totals` reads each year's `totals` link. `accounting_period_totals` reads each year's
+ `periods` link, then each period's `totals` link. Period numbers continue across accounting years
+ (for example 25 to 36 for the third year). Both lift `account.accountNumber` into `accountNumber`.
+- All are full refresh. Resume state records which parents finished and the page of the current one,
+ except for `accounting_period_totals`: the framework keeps no resume state for a two-hop chain.
- `/accounts/{accountNumber}/accounting-years/{year}/entries` is not synced: it returns a per-account
subset of `accounting_year_entries`, which already carries `account`.
## Not implemented (yet)
-Other sub-resources (customer contacts, invoice lines, vouchers, totals, periods) and the separate
+Other sub-resources (customer contacts, invoice lines, accounting-year vouchers, periods as a table) and the separate
sub-APIs on `apis.e-conomic.com` (subscriptions, etc.) are out of scope for now.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/canonical_descriptions.py
index 15f767de76c8..3a59abcfecae 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/canonical_descriptions.py
@@ -347,4 +347,68 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"supplierInvoiceNumber": "Identifier of the supplier invoice, if any.",
},
},
+ "vat_types": {
+ "description": "A VAT type, which says whether a VAT account is used for sales, purchases or reverse charge.",
+ "docs_url": "https://restdocs.e-conomic.com/#get-vat-types",
+ "columns": {
+ "vatTypeNumber": "Unique identifier of the VAT type.",
+ "name": "Name of the VAT type.",
+ },
+ },
+ "vat_accounts": {
+ "description": "A VAT code with its rate and the accounts VAT is posted to. Resolves the vatAccount references on entries and other resources.",
+ "docs_url": "https://restdocs.e-conomic.com/#get-vat-accounts",
+ "columns": {
+ "vatCode": "Unique VAT code.",
+ "name": "Name of the VAT account.",
+ "ratePercentage": "VAT rate in percent.",
+ "vatType": "The VAT type of the VAT account.",
+ "account": "The account VAT is posted to.",
+ "contraAccount": "The contra account used for reverse-charge VAT, if any.",
+ },
+ },
+ "payment_types": {
+ "description": "A payment type, which says how a supplier or customer payment is made, for example bank transfer or IBAN.",
+ "docs_url": "https://restdocs.e-conomic.com/#get-payment-types",
+ "columns": {
+ "paymentTypeNumber": "Unique identifier of the payment type.",
+ "name": "Name of the payment type.",
+ },
+ },
+ "journal_vouchers": {
+ "description": "A voucher in a journal (daybook) that has not been booked yet. A voucher groups the entries posted together.",
+ "docs_url": "https://restdocs.e-conomic.com/#get-journals-journalnumber-vouchers",
+ "columns": {
+ "journalNumber": "The journal the voucher is in.",
+ "accountingYear": "The accounting year of the voucher, in the format YYYY or YYYY/YYYY.",
+ "voucherNumber": "Identifier of the voucher within its journal and accounting year.",
+ "entries": "The entries on the voucher, grouped by entry type.",
+ "attachment": "Link to the attachment (for example a scanned receipt) on the voucher.",
+ },
+ },
+ "accounting_year_totals": {
+ "description": "The total posted to each account in an accounting year, in the base currency. Gives the trial balance for the year.",
+ "docs_url": "https://restdocs.e-conomic.com/#get-accounting-years-accountingyear-totals",
+ "columns": {
+ "accountingYear": "The accounting year, in the format YYYY or YYYY/YYYY.",
+ "accountNumber": "The account the total is for.",
+ "account": "The account the total is for.",
+ "totalInBaseCurrency": "Total of the entries on the account in the accounting year, in the base currency.",
+ "fromDate": "First date of the accounting year.",
+ "toDate": "Last date of the accounting year.",
+ },
+ },
+ "accounting_period_totals": {
+ "description": "The total posted to each account in an accounting period (usually a month), in the base currency.",
+ "docs_url": "https://restdocs.e-conomic.com/#get-accounting-years-accountingyear-periods-accountingyearperiod-totals",
+ "columns": {
+ "accountingYear": "The accounting year of the period, in the format YYYY or YYYY/YYYY.",
+ "periodNumber": "Identifier of the accounting period. Period numbers continue across accounting years.",
+ "accountNumber": "The account the total is for.",
+ "account": "The account the total is for.",
+ "totalInBaseCurrency": "Total of the entries on the account in the period, in the base currency.",
+ "fromDate": "First date of the period.",
+ "toDate": "Last date of the period.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/e_conomic.py b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/e_conomic.py
index 7917e2a53833..1be19a3c18a0 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/e_conomic.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/e_conomic.py
@@ -1,6 +1,7 @@
import dataclasses
+from collections.abc import Callable
from datetime import UTC, date, datetime
-from typing import Any, Optional
+from typing import Any, Optional, cast
from urllib.parse import urlparse
from requests import PreparedRequest, Response
@@ -12,17 +13,23 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.auth import AuthConfigBase
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
+ build_chained_resource,
build_dependent_resource,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (
JSONResponsePaginator,
)
-from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import ClientConfig
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.resource import Resource
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (
+ ClientConfig,
+ EndpointResource,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probe
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
from products.warehouse_sources.backend.temporal.data_imports.sources.e_conomic.settings import (
E_CONOMIC_ENDPOINTS,
+ E_CONOMIC_FANOUT_PARENTS,
PAGE_SIZE,
)
@@ -110,12 +117,46 @@ class EConomicNextPagePaginator(JSONResponsePaginator):
_assert_trusted_url(self._next_url)
-def _add_entries_path(row: dict[str, Any]) -> dict[str, Any]:
- row["entries_path"] = urlparse(row["entries"]).path
+RowMap = Callable[[dict[str, Any]], dict[str, Any]]
+
+
+def _add_link_path(link: str) -> RowMap:
+ # Accounting years spanning two calendar years use a different form in URLs than their `year`, so
+ # child paths come from the links the API returns.
+ def _mapper(row: dict[str, Any]) -> dict[str, Any]:
+ row[f"{link}_path"] = urlparse(row[link]).path
+ return row
+
+ return _mapper
+
+
+def _add_period_fields(row: dict[str, Any]) -> dict[str, Any]:
+ row["totals_path"] = urlparse(row["totals"]).path
+ row["year"] = row["accountingYear"]["year"]
return row
-_PARENT_DATA_MAPS = {"accounting_year_entries": _add_entries_path}
+def _add_account_number(row: dict[str, Any]) -> dict[str, Any]:
+ row["accountNumber"] = row["account"]["accountNumber"]
+ return row
+
+
+def _flatten_accounting_year(row: dict[str, Any]) -> dict[str, Any]:
+ row["accountingYear"] = row["accountingYear"]["year"]
+ return row
+
+
+_PARENT_DATA_MAPS: dict[str, RowMap] = {
+ "accounting_year_entries": _add_link_path("entries"),
+ "accounting_year_totals": _add_link_path("totals"),
+}
+
+# Lift the ids out of nested reference objects so they can be primary keys.
+_CHILD_DATA_MAPS: dict[str, RowMap] = {
+ "accounting_year_totals": _add_account_number,
+ "accounting_period_totals": _add_account_number,
+ "journal_vouchers": _flatten_accounting_year,
+}
def _client_config(app_secret_token: str, agreement_grant_token: str) -> ClientConfig:
@@ -131,6 +172,85 @@ def _client_config(app_secret_token: str, agreement_grant_token: str) -> ClientC
}
+def _resolve(resource: str, field: str) -> dict[str, Any]:
+ return {"type": "resolve", "resource": resource, "field": field}
+
+
+def _period_totals_resource(
+ endpoint: str,
+ client_config: ClientConfig,
+ team_id: int,
+ job_id: str,
+) -> Resource:
+ """Build accounting_years -> periods -> totals. A two-hop chain carries no resume state."""
+ config = E_CONOMIC_ENDPOINTS[endpoint]
+ fanout = config.fanout
+ assert fanout is not None
+ periods = E_CONOMIC_FANOUT_PARENTS[fanout.parent_name]
+ periods_fanout = periods.fanout
+ assert periods_fanout is not None
+ years = E_CONOMIC_ENDPOINTS[periods_fanout.parent_name]
+
+ resources: list[EndpointResource] = [
+ {
+ "name": years.name,
+ "table_name": years.name,
+ "write_disposition": "replace",
+ "endpoint": {
+ "path": years.path,
+ "params": {"pagesize": years.page_size, **periods_fanout.parent_params},
+ "data_selector": "collection",
+ },
+ "data_map": _add_link_path("periods"),
+ "table_format": "delta",
+ },
+ {
+ "name": periods.name,
+ "table_name": periods.name,
+ "write_disposition": "replace",
+ "endpoint": {
+ "path": periods.path,
+ "params": {
+ "pagesize": periods.page_size,
+ periods_fanout.resolve_param: _resolve(years.name, periods_fanout.resolve_field),
+ },
+ "data_selector": "collection",
+ "paginator": EConomicNextPagePaginator(),
+ },
+ "data_map": _add_period_fields,
+ "table_format": "delta",
+ },
+ {
+ "name": endpoint,
+ "table_name": endpoint,
+ "write_disposition": "replace",
+ "include_from_parent": fanout.include_from_parent,
+ "endpoint": {
+ "path": config.path,
+ "params": {
+ "pagesize": config.page_size,
+ fanout.resolve_param: _resolve(periods.name, fanout.resolve_field),
+ },
+ "data_selector": "collection",
+ "paginator": EConomicNextPagePaginator(),
+ },
+ "table_format": "delta",
+ },
+ ]
+ return cast(
+ Resource,
+ build_chained_resource(
+ resources=resources,
+ child_name=endpoint,
+ parent_name=periods.name,
+ parent_field_renames=fanout.parent_field_renames,
+ client_config=client_config,
+ team_id=team_id,
+ job_id=job_id,
+ ),
+ )
+
+
def e_conomic_source(
app_secret_token: str,
agreement_grant_token: str,
@@ -146,29 +266,40 @@ def e_conomic_source(
resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
if config.fanout is not None:
-
- def save_fanout_checkpoint(state: Optional[dict[str, Any]]) -> None:
- if state:
- resumable_source_manager.save_state(EConomicResumeConfig(fanout_state=state))
-
- child = build_dependent_resource(
- endpoint_configs=E_CONOMIC_ENDPOINTS,
- child_endpoint=endpoint,
- fanout=config.fanout,
- client_config=_client_config(app_secret_token, agreement_grant_token),
- parent_data_map=_PARENT_DATA_MAPS.get(endpoint),
- parent_endpoint_extra={"data_selector": "collection"},
- # Explicit, because the framework treats a path ending in a placeholder ("{entries_path}") as a
- # single-entity lookup and would otherwise fetch only its first page.
- child_endpoint_extra={"data_selector": "collection", "paginator": EConomicNextPagePaginator()},
- page_size_param="pagesize",
- path_format_values={},
- team_id=team_id,
- job_id=job_id,
- db_incremental_field_last_value=None,
- resume_hook=save_fanout_checkpoint,
- initial_paginator_state=resume.fanout_state if resume else None,
- )
+ client_config = _client_config(app_secret_token, agreement_grant_token)
+ if config.fanout.parent_name in E_CONOMIC_FANOUT_PARENTS:
+ child = _period_totals_resource(endpoint, client_config, team_id, job_id)
+ else:
+
+ def save_fanout_checkpoint(state: Optional[dict[str, Any]]) -> None:
+ if state:
+ resumable_source_manager.save_state(EConomicResumeConfig(fanout_state=state))
+
+ child = cast(
+ Resource,
+ build_dependent_resource(
+ endpoint_configs=E_CONOMIC_ENDPOINTS,
+ child_endpoint=endpoint,
+ fanout=config.fanout,
+ client_config=client_config,
+ parent_data_map=_PARENT_DATA_MAPS.get(endpoint),
+ parent_endpoint_extra={"data_selector": "collection"},
+ # Explicit, because the framework treats a path ending in a placeholder ("{entries_path}") as a
+ # single-entity lookup and would otherwise fetch only its first page.
+ child_endpoint_extra={"data_selector": "collection", "paginator": EConomicNextPagePaginator()},
+ page_size_param="pagesize",
+ path_format_values={},
+ team_id=team_id,
+ job_id=job_id,
+ db_incremental_field_last_value=None,
+ resume_hook=save_fanout_checkpoint,
+ initial_paginator_state=resume.fanout_state if resume else None,
+ ),
+ )
+
+ child_map = _CHILD_DATA_MAPS.get(endpoint)
+ if child_map is not None:
+ child.add_map(child_map)
return SourceResponse(
name=endpoint,
items=lambda: child,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/settings.py
index f2e9db6e2f18..c72ddb87969b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/settings.py
@@ -139,6 +139,23 @@ E_CONOMIC_ENDPOINTS: dict[str, EConomicEndpointConfig] = {
primary_keys=["vatZoneNumber"],
sort="vatZoneNumber",
),
+ # The API accepts `sort` on these three lookups but ignores it, so no sort is sent. Each is a tiny
+ # single-page table.
+ "vat_types": EConomicEndpointConfig(
+ name="vat_types",
+ path="/vat-types",
+ primary_keys=["vatTypeNumber"],
+ ),
+ "vat_accounts": EConomicEndpointConfig(
+ name="vat_accounts",
+ path="/vat-accounts",
+ primary_keys=["vatCode"],
+ ),
+ "payment_types": EConomicEndpointConfig(
+ name="payment_types",
+ path="/payment-types",
+ primary_keys=["paymentTypeNumber"],
+ ),
"employees": EConomicEndpointConfig(
name="employees",
path="/employees",
@@ -220,6 +237,64 @@ E_CONOMIC_ENDPOINTS: dict[str, EConomicEndpointConfig] = {
child_params={"sort": "journalEntryNumber"},
),
),
+ # Unbooked vouchers, so full refresh like journal_entries. The API rejects every `sort` field here.
+ "journal_vouchers": EConomicEndpointConfig(
+ name="journal_vouchers",
+ path="/journals/{journalNumber}/vouchers",
+ # A voucher number is unique per journal and accounting year.
+ primary_keys=["journalNumber", "accountingYear", "voucherNumber"],
+ fanout=DependentEndpointConfig(
+ parent_name="journals",
+ resolve_param="journalNumber",
+ resolve_field="journalNumber",
+ include_from_parent=["journalNumber"],
+ parent_field_renames={"journalNumber": "journalNumber"},
+ parent_params={"sort": "journalNumber"},
+ ),
+ ),
+ # Balances change while a year is open, so totals are full refresh. The API ignores `sort` here.
+ "accounting_year_totals": EConomicEndpointConfig(
+ name="accounting_year_totals",
+ path="{totals_path}",
+ primary_keys=["accountingYear", "accountNumber"],
+ fanout=DependentEndpointConfig(
+ parent_name="accounting_years",
+ resolve_param="totals_path",
+ resolve_field="totals_path",
+ include_from_parent=["year"],
+ parent_field_renames={"year": "accountingYear"},
+ parent_params={"sort": "year"},
+ ),
+ ),
+ "accounting_period_totals": EConomicEndpointConfig(
+ name="accounting_period_totals",
+ path="{totals_path}",
+ primary_keys=["accountingYear", "periodNumber", "accountNumber"],
+ fanout=DependentEndpointConfig(
+ parent_name="accounting_year_periods",
+ resolve_param="totals_path",
+ resolve_field="totals_path",
+ include_from_parent=["periodNumber", "year"],
+ parent_field_renames={"periodNumber": "periodNumber", "year": "accountingYear"},
+ ),
+ ),
+}
+
+# Fan-out parents that are not tables themselves. `accounting_period_totals` reaches its totals through
+# two hops: accounting_years -> accounting_year_periods -> totals.
+E_CONOMIC_FANOUT_PARENTS: dict[str, EConomicEndpointConfig] = {
+ "accounting_year_periods": EConomicEndpointConfig(
+ name="accounting_year_periods",
+ path="{periods_path}",
+ primary_keys=["periodNumber"],
+ fanout=DependentEndpointConfig(
+ parent_name="accounting_years",
+ resolve_param="periods_path",
+ resolve_field="periods_path",
+ include_from_parent=[],
+ parent_params={"sort": "year"},
+ ),
+ ),
}
ENDPOINTS = tuple(E_CONOMIC_ENDPOINTS.keys())
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/tests/test_e_conomic.py b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/tests/test_e_conomic.py
index d9afd0c46d17..ce6fdd2d4bc1 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/tests/test_e_conomic.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/e_conomic/tests/test_e_conomic.py
@@ -165,6 +165,85 @@ class TestAccountingYearEntriesFanout:
assert "/accounting-years/2022_6_2023/entries" in result.saved_states[-1].fanout_state["completed"]
+class TestTotalsAndVouchersFanout:
+ def test_year_totals_walk_each_year_through_its_totals_link(self) -> None:
+ result = _driver().run(
+ "accounting_year_totals",
+ [
+ _page(
+ [
+ {"year": "2022", "totals": f"{E_CONOMIC_BASE_URL}/accounting-years/2022/totals"},
+ {"year": "2022/2023", "totals": f"{E_CONOMIC_BASE_URL}/accounting-years/2022_6_2023/totals"},
+ ]
+ ),
+ _page([{"account": {"accountNumber": 1000}, "totalInBaseCurrency": 5}]),
+ _page([{"account": {"accountNumber": 1000}, "totalInBaseCurrency": 7}]),
+ ],
+ )
+
+ assert result.raised is None
+ assert [(r["accountingYear"], r["accountNumber"], r["totalInBaseCurrency"]) for r in result.rows] == [
+ ("2022", 1000, 5),
+ ("2022/2023", 1000, 7),
+ ]
+ assert result.paths[1:] == ["/accounting-years/2022/totals", "/accounting-years/2022_6_2023/totals"]
+
+ def test_period_totals_walk_years_then_periods_then_totals(self) -> None:
+ def period(number: int) -> dict[str, Any]:
+ return {
+ "periodNumber": number,
+ "accountingYear": {"year": "2024"},
+ # The API echoes the parent request's query string into these links.
+ "totals": f"{E_CONOMIC_BASE_URL}/accounting-years/2024/periods/{number}/totals?pagesize=1000",
+ }
+
+ second_page = f"{E_CONOMIC_BASE_URL}/accounting-years/2024/periods/25/totals?skippages=1"
+ result = _driver().run(
+ "accounting_period_totals",
+ [
+ _page([{"year": "2024", "periods": f"{E_CONOMIC_BASE_URL}/accounting-years/2024/periods"}]),
+ _page([period(25), period(26)]),
+ _page([{"account": {"accountNumber": 1000}}], next_page=second_page),
+ _page([{"account": {"accountNumber": 1010}}]),
+ _page([{"account": {"accountNumber": 1000}}]),
+ ],
+ )
+
+ assert result.raised is None
+ assert [(r["accountingYear"], r["periodNumber"], r["accountNumber"]) for r in result.rows] == [
+ ("2024", 25, 1000),
+ ("2024", 25, 1010),
+ ("2024", 26, 1000),
+ ]
+ assert result.paths == [
+ "/accounting-years",
+ "/accounting-years/2024/periods",
+ "/accounting-years/2024/periods/25/totals",
+ "/accounting-years/2024/periods/25/totals",
+ "/accounting-years/2024/periods/26/totals",
+ ]
+ assert result.urls[3] == second_page
+
+ def test_vouchers_carry_journal_and_flat_accounting_year(self) -> None:
+ result = _driver().run(
+ "journal_vouchers",
+ [
+ _page([{"journalNumber": 3}, {"journalNumber": 11}]),
+ _page([{"voucherNumber": 20001, "accountingYear": {"year": "2022"}}]),
+ _page([{"voucherNumber": 0, "accountingYear": {"year": "2024"}}]),
+ ],
+ )
+
+ assert result.raised is None
+ assert [(r["journalNumber"], r["accountingYear"], r["voucherNumber"]) for r in result.rows] == [
+ (3, "2022", 20001),
+ (11, "2024", 0),
+ ]
+ assert result.paths[1:] == ["/journals/3/vouchers", "/journals/11/vouchers"]
+ # The vouchers endpoint rejects every sort field with a 400.
+ assert result.params("sort")[1:] == [None, None]
+
+
class TestErrorHandling:
@parameterized.expand([("server_error", 500), ("bad_gateway", 503), ("throttled", 429)])
def test_retryable_status_is_retried(self, _name: str, status_code: int) -> None: