:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add payout, disbursement and income tables to mono (#123052)
Tom Owersb909976b60bb
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 6a02b417203a..fee96f25e757 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -8189,10 +8189,10 @@ Diffed against: <https://docs.mono.co/llms.txt>- [x] `Fetch all Payments (GET /v2/payments/transactions)` — analyze payment collection revenue, failures, fees, and refunds separately from linked bank transactions (high)- [x] `Get all Mandates (GET /v3/payments/mandates)` — track direct debit authorizations, their states, and customer relationships for recurring collection analysis (high)- [x] `Retrieve All Debits (GET /v3/payments/mandates/debits)` — measure successful and failed mandate collections, collected amounts, and debit fees (high)-- [ ] `Payout (GET /v2/payments/payouts)` — reconcile payment collections with settlement amounts and payout status (high)-- [ ] `Payout Transactions (GET /v2/payments/payout/{payout}/transactions)` — match individual collections to settlements and explain payout totals (high)-- [ ] `Disbursements and distributions (GET /v3/payments/disburse/disbursements; GET /v3/payments/disburse/disbursements/{batch_id}/distributions)` — analyze outgoing payment batches and recipient distributions, including payment status and amounts (high)-- [ ] `Income Records (GET /v2/accounts/{id}/income-records)` — analyze completed income assessments and compare customer earning capacity over time (medium)+- [x] `Payout (GET /v2/payments/payouts)` — reconcile payment collections with settlement amounts and payout status (high). Added as `payouts`. Mono requires a `status` filter and documents only `settled`, so the table holds settled payouts+- [ ] `Payout Transactions (GET /v2/payments/payout/{payout}/transactions)` — match individual collections to settlements and explain payout totals (high). Skipped: the endpoint requires an `account` query parameter that payout records do not carry, so it cannot fan out from `payouts`+- [x] `Disbursements and distributions (GET /v3/payments/disburse/disbursements; GET /v3/payments/disburse/disbursements/{batch_id}/distributions)` — analyze outgoing payment batches and recipient distributions, including payment status and amounts (high). Added as `disbursements` and `disbursement_distributions`+- [x] `Income Records (GET /v2/accounts/{id}/income-records)` — analyze completed income assessments and compare customer earning capacity over time (medium). Added as `income_records`- [ ] `Assets (GET /v2/accounts/{id}/assets)` — analyze customer investment holdings that bank transaction tables do not expose (medium)- [ ] `Earnings (GET /v2/accounts/{id}/earnings)` — measure investment income and portfolio returns alongside customer holdings (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mono/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mono/canonical_descriptions.pyindex d2690bd67c0a..e68f7dd79f5b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mono/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mono/canonical_descriptions.py@@ -117,4 +117,60 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"date": "When the debit was made.",},},+ "payouts": {+ "description": "Settled payouts of DirectPay and Direct Debit collections to the business's settlement account.",+ "docs_url": "https://docs.mono.co/api/money-operations/payout",+ "columns": {+ "id": "The payout identifier.",+ "status": "The payout status. Mono returns only settled payouts for this table.",+ "amount": "The payout amount before fees, in the currency's smallest unit, such as kobo.",+ "fee": "The fee deducted from the payout, in the currency's smallest unit.",+ "settled_amount": "The amount paid to the settlement account after fees.",+ "settled_account": "The beneficiary bank, account name, and account number that received the payout.",+ "currency": "The payout currency.",+ "date": "The date of the payout.",+ "bank": "The identifier of the settlement bank account.",+ "processor": "The processor that sent the payout.",+ "message": "A message that describes the payout result.",+ },+ },+ "disbursements": {+ "description": "Disbursement batches that send funds from a source account to one or more recipients.",+ "docs_url": "https://docs.mono.co/api/disburse/fetch-all-disbursements",+ "columns": {+ "id": "The disbursement batch identifier.",+ "reference": "The reference the business gave the batch.",+ "source": "The source of the funds, such as mandate.",+ "status": "The batch status, such as processing or completed.",+ "total_amount": "The total amount of the batch in the currency's smallest unit, such as kobo.",+ "total_fee": "The total fee for the batch in the currency's smallest unit.",+ "created_at": "When the batch was created.",+ },+ },+ "disbursement_distributions": {+ "description": "Individual recipient payments inside a disbursement batch.",+ "docs_url": "https://docs.mono.co/api/disburse/fetch-all-distributions-in-batch",+ "columns": {+ "batch_id": "The identifier of the disbursement batch.",+ "id": "The distribution identifier.",+ "reference": "The reference of the distribution.",+ "account_number": "The recipient's bank account number.",+ "bank_code": "The code of the recipient's bank.",+ "amount": "The amount sent to the recipient in the currency's smallest unit, such as kobo.",+ "narration": "The narration sent with the payment.",+ "status": "The distribution status, such as completed.",+ "recipient_email": "The recipient's email address.",+ },+ },+ "income_records": {+ "description": "Completed income assessments of linked accounts.",+ "docs_url": "https://docs.mono.co/api/bank-data/accounts/income-records",+ "columns": {+ "account_id": "The identifier of the linked account.",+ "income": "The income assessment, with the income summary, the income streams, and the totals.",+ "app": "The identifier of the Mono app that requested the assessment.",+ "created_at": "When the assessment was created.",+ "updated_at": "When the assessment was last updated.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mono/mono.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mono/mono.pyindex f4f8accc0a49..5b51aaaefa7a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mono/mono.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mono/mono.py@@ -32,10 +32,13 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_from products.warehouse_sources.backend.temporal.data_imports.sources.mono.settings import (AUTH_ERROR,BASE_URL,+ DATA_SELECTORS,DATE_FILTERS,ENDPOINTS,+ FANOUT_PARENTS,INCREMENTAL_FIELDS,PAGE_SIZE,+ PAYOUT_STATUS,PERMISSION_ERROR,PRIMARY_KEYS,)@@ -135,8 +138,10 @@ def incremental_start(def endpoint_paginator(endpoint: str) -> PaginatorConfig | None:if endpoint in ("institutions", "mandate_debits"):return "single_page"- if endpoint == "payments":+ if endpoint in ("payments", "payouts"):return PageNumberPaginator(base_page=1, total_path="meta.paging.pages")+ if endpoint in ("disbursements", "disbursement_distributions"):+ return PageNumberPaginator(base_page=1, total_path="data.pagination.pages")if endpoint == "mandates":# The documented response has no paging metadata, so stop at the first empty page.return PageNumberPaginator(base_page=1)@@ -176,13 +181,18 @@ def mono_source("paginate": "true","limit": PAGE_SIZE,}+ elif endpoint in FANOUT_PARENTS:+ parent, placeholder = FANOUT_PARENTS[endpoint]+ params[placeholder] = {"type": "resolve", "resource": parent, "field": "id"}elif endpoint in ("payments", "mandates"):params["limit"] = PAGE_SIZE+ elif endpoint == "payouts":+ params["status"] = PAYOUT_STATUSendpoint_config: Endpoint = {"path": path,"params": params,- "data_selector": "data.payments" if endpoint == "payments" else "data",+ "data_selector": DATA_SELECTORS.get(endpoint, "data"),"data_selector_required": True,}paginator = endpoint_paginator(endpoint)@@ -199,14 +209,18 @@ def mono_source("endpoint": endpoint_config,}resources: list[str | EndpointResource] = []- if endpoint == "transactions":+ if endpoint in FANOUT_PARENTS:+ parent, _ = FANOUT_PARENTS[endpoint]+ parent_endpoint: Endpoint = {+ "path": schema_for_resource(ENDPOINTS, parent),+ "data_selector": DATA_SELECTORS.get(parent, "data"),+ "data_selector_required": True,+ }+ parent_paginator = endpoint_paginator(parent)+ if parent_paginator is not None:+ parent_endpoint["paginator"] = parent_paginatorresource["include_from_parent"] = ["id"]- resources.append(- {- "name": "accounts",- "endpoint": {"path": "accounts", "data_selector": "data", "data_selector_required": True},- }- )+ resources.append({"name": parent, "endpoint": parent_endpoint})resources.append(resource)def save_checkpoint(state: dict[str, Any] | None) -> None:@@ -223,8 +237,9 @@ def mono_source(initial_paginator_state=resume.paginator_state if resume else None,)result = next(item for item in built if item.name == endpoint)- if endpoint == "transactions":- result.add_map(rename_parent_fields("accounts", {"id": "account_id"}))+ if endpoint in FANOUT_PARENTS:+ parent, placeholder = FANOUT_PARENTS[endpoint]+ result.add_map(rename_parent_fields(parent, {"id": placeholder}))return SourceResponse(name=endpoint,items=lambda: result,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mono/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mono/settings.pyindex c69dcc8fac67..c7ad92dc394f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mono/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mono/settings.py@@ -1,7 +1,7 @@from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldTypeBASE_URL = "https://api.withmono.com"-# Relative paths follow the pinned API version. Mono serves bank coverage and Direct Debit only on v3.+# Relative paths follow the pinned API version. Mono serves bank coverage, Direct Debit and Disburse only on v3.ENDPOINTS = {"customers": "customers","accounts": "accounts",@@ -10,6 +10,10 @@ ENDPOINTS = {"payments": "payments/transactions","mandates": f"{BASE_URL}/v3/payments/mandates","mandate_debits": f"{BASE_URL}/v3/payments/mandates/debits",+ "payouts": "payments/payouts",+ "disbursements": f"{BASE_URL}/v3/payments/disburse/disbursements",+ "disbursement_distributions": f"{BASE_URL}/v3/payments/disburse/disbursements/{{batch_id}}/distributions",+ "income_records": "accounts/{account_id}/income-records",}PRIMARY_KEYS = {"customers": ["id"],@@ -19,7 +23,26 @@ PRIMARY_KEYS = {"payments": ["id"],"mandates": ["id"],"mandate_debits": ["mandate", "reference"],+ "payouts": ["id"],+ "disbursements": ["id"],+ "disbursement_distributions": ["batch_id", "id"],+ # Income records carry no id, so the account and the assessment time identify a record.+ "income_records": ["account_id", "created_at"],}+# Child endpoint -> (parent endpoint, path placeholder that takes the parent id).+FANOUT_PARENTS = {+ "transactions": ("accounts", "account_id"),+ "income_records": ("accounts", "account_id"),+ "disbursement_distributions": ("disbursements", "batch_id"),+}+DATA_SELECTORS = {+ "payments": "data.payments",+ "payouts": "data.payouts",+ "disbursements": "data.disbursements",+ "disbursement_distributions": "data.distributions",+}+# Mono requires a payout status and documents only "settled".+PAYOUT_STATUS = "settled"INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {"transactions": [{diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mono/tests/test_mono.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mono/tests/test_mono.pyindex fe8b25b16fd4..65f38e3043cb 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mono/tests/test_mono.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mono/tests/test_mono.py@@ -129,6 +129,16 @@ def payments_page(rows: list[dict[str, Any]], pages: int) -> ScriptedResponse:)+def payouts_page(rows: list[dict[str, Any]], pages: int) -> ScriptedResponse:+ return ScriptedResponse(+ json={"status": "successful", "data": {"payouts": rows}, "meta": {"paging": {"pages": pages, "next": None}}}+ )+++def disburse_page(key: str, rows: list[dict[str, Any]], pages: int) -> ScriptedResponse:+ return ScriptedResponse(json={"status": "successful", "data": {key: rows, "pagination": {"pages": pages}}})++@time_machine.travel("2026-06-15T12:00:00Z", tick=False)@pytest.mark.parametrize("endpoint,incremental,watermark,script,expected_path,expected_queries,expected_ids",@@ -178,6 +188,15 @@ def payments_page(rows: list[dict[str, Any]], pages: int) -> ScriptedResponse:[{"page": ("1",), "limit": ("100",), "start": ("09-06-2026",), "end": ("15-06-2026",)}],["payment-1"],),+ (+ "payouts",+ False,+ None,+ [payouts_page([{"id": "payout-1"}], 2), payouts_page([{"id": "payout-2"}], 2)],+ "/v2/payments/payouts",+ [{"page": (str(number),), "status": ("settled",)} for number in (1, 2)],+ ["payout-1", "payout-2"],+ ),("mandate_debits",False,@@ -218,6 +237,69 @@ def test_payment_and_coverage_tables_request_the_right_version_filters_and_pagesassert all(request.headers["mono-sec-key"] == CONFIG.api_key for request in result.requests)+ "endpoint,script,expected_requests,expected_keys",+ [+ (+ "income_records",+ [+ page([{"id": "account-a"}], "https://api.withmono.com/v2/accounts?page=2"),+ page(+ [{"created_at": "2025-05-23T09:48:43.276Z", "income": {"total_income": 16300}}],+ "https://api.withmono.com/v2/accounts/account-a/income-records?page=2",+ ),+ page([{"created_at": "2025-04-01T08:00:00.000Z", "income": {"total_income": 9000}}]),+ page([{"id": "account-b"}]),+ page([{"created_at": "2025-05-23T09:48:43.276Z", "income": {"total_income": 500}}]),+ ],+ [+ ("/v2/accounts", "1"),+ ("/v2/accounts/account-a/income-records", "1"),+ ("/v2/accounts/account-a/income-records", "2"),+ ("/v2/accounts", "2"),+ ("/v2/accounts/account-b/income-records", "1"),+ ],+ [+ ("account-a", "2025-05-23T09:48:43.276Z"),+ ("account-a", "2025-04-01T08:00:00.000Z"),+ ("account-b", "2025-05-23T09:48:43.276Z"),+ ],+ ),+ (+ "disbursement_distributions",+ [+ disburse_page("disbursements", [{"id": "batch-a"}], 2),+ disburse_page("distributions", [{"id": "distribution-1"}], 2),+ disburse_page("distributions", [{"id": "distribution-2"}], 2),+ disburse_page("disbursements", [{"id": "batch-b"}], 2),+ disburse_page("distributions", [{"id": "distribution-1"}], 1),+ ],+ [+ ("/v3/payments/disburse/disbursements", "1"),+ ("/v3/payments/disburse/disbursements/batch-a/distributions", "1"),+ ("/v3/payments/disburse/disbursements/batch-a/distributions", "2"),+ ("/v3/payments/disburse/disbursements", "2"),+ ("/v3/payments/disburse/disbursements/batch-b/distributions", "1"),+ ],+ [("batch-a", "distribution-1"), ("batch-a", "distribution-2"), ("batch-b", "distribution-1")],+ ),+ ],+)+def test_fanout_tables_walk_every_parent_page_and_key_rows_by_parent(+ endpoint: str,+ script: list[ScriptedResponse],+ expected_requests: list[tuple[str, str]],+ expected_keys: list[tuple[str, str]],+) -> None:+ result = driver().run(endpoint, script)++ assert result.raised is None+ assert [(request.path, request.param("page")) for request in result.requests] == expected_requests+ assert result.response is not None+ assert [tuple(row[key] for key in result.response.primary_keys or []) for row in result.rows] == expected_keys+ assert all(not any(key.startswith("_") for key in row) for row in result.rows)++@pytest.mark.parametrize("start,end,expected_query",[