:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 3 missing tables to the YNAB source (#120352)
Tom Owersdfbf02be46c7
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 93dfd0b25fa9..57426b9787b9 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -13457,9 +13457,9 @@ Today (9): `accounts`, `categories`, `category_groups`, `months`, `payee_locatioDiffed against: <https://api.ynab.com/papi/open_api_spec.yaml>-- [ ] `Plan month category details (GET /v1/plans/{plan_id}/months/{month})` — compare historical category budgets, spending, and available balances; month summaries lack category details (high)-- [ ] `Money movements (GET /v1/plans/{plan_id}/money_movements)` — track when money moved between budget categories, including amounts, notes, and the responsible user (high)-- [ ] `Money movement groups (GET /v1/plans/{plan_id}/money_movement_groups)` — group related budget adjustments by action, month, author, and note (medium)+- [x] `Plan month category details (GET /v1/plans/{plan_id}/months/{month})` — compare historical category budgets, spending, and available balances; month summaries lack category details (high)+- [x] `Money movements (GET /v1/plans/{plan_id}/money_movements)` — track when money moved between budget categories, including amounts, notes, and the responsible user (high)+- [x] `Money movement groups (GET /v1/plans/{plan_id}/money_movement_groups)` — group related budget adjustments by action, month, author, and note (medium)Note: Parsed every GET operation and the month and money movement schemas in the official OpenAPI specification. The existing months table contains summaries; month detail returns a category collection for each plan and month. Money movement collections need reads for each plan, while monthly variants repeat the same records. Excluded settings, user identity, transaction subsets, and write operations.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/canonical_descriptions.pyindex 3a0c4ce6c954..86a34382ab0a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/canonical_descriptions.py@@ -42,6 +42,41 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"docs_url": "https://api.ynab.com/v1#tag/months","columns": {"plan_id": "Plan containing this month.", "month": "First day of the plan month."},},+ "month_categories": {+ "description": "Categories with their assigned, activity, and available amounts for each plan month.",+ "docs_url": "https://api.ynab.com/v1#tag/months",+ "columns": {+ "plan_id": "Plan containing this category.",+ "month": "First day of the plan month that the amounts apply to.",+ "budgeted": "Amount assigned to the category in the month, in milliunits.",+ "activity": "Category activity in the month, in milliunits.",+ "balance": "Available category amount at the end of the month, in milliunits.",+ },+ },+ "money_movements": {+ "description": "Money moved between budget categories in each plan.",+ "docs_url": "https://api.ynab.com/v1#tag/money-movements",+ "columns": {+ "plan_id": "Plan containing this money movement.",+ "month": "Plan month of the money movement.",+ "moved_at": "Time the server processed the money movement.",+ "from_category_id": "Category the money was moved from.",+ "to_category_id": "Category the money was moved to.",+ "amount": "Amount moved, in milliunits.",+ "money_movement_group_id": "Group this money movement belongs to.",+ "performed_by_user_id": "User who moved the money.",+ },+ },+ "money_movement_groups": {+ "description": "Groups of related money movements, with their month, author, and note.",+ "docs_url": "https://api.ynab.com/v1#tag/money-movements",+ "columns": {+ "plan_id": "Plan containing this money movement group.",+ "group_created_at": "Time the money movement group was created.",+ "month": "Plan month of the money movement group.",+ "performed_by_user_id": "User who performed the money movement group.",+ },+ },"transactions": {"description": "Plan transactions with nested split transactions, excluding pending transactions.","docs_url": "https://api.ynab.com/v1#tag/transactions",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/settings.pyindex 09d7382454bf..b7c36740cde1 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/settings.py@@ -19,6 +19,8 @@ class YnabEndpoint:default_incremental_field: str | None = Nonepage_size: int = 0fanout: DependentEndpointConfig | None = None+ # Binds {month} to each month in the plan's first_month..last_month range.+ per_month: bool = FalsePLAN_FANOUT = DependentEndpointConfig(@@ -63,6 +65,36 @@ ENDPOINTS: dict[str, YnabEndpoint] = {partition_key="month",fanout=PLAN_FANOUT,),+ "month_categories": YnabEndpoint(+ name="month_categories",+ path="plans/{plan_id}/months/{month}",+ data_selector="data.month.categories",+ primary_keys=("plan_id", "month", "id"),+ partition_key="month",+ per_month=True,+ fanout=DependentEndpointConfig(+ parent_name="plans",+ resolve_param="plan_id",+ resolve_field="id",+ include_from_parent=["id", "month"],+ parent_field_renames={"id": "plan_id", "month": "month"},+ # A month inside the plan's range that YNAB has no record of returns 404.+ child_response_actions=[{"status_code": 404, "action": "ignore"}],+ ),+ ),+ "money_movements": YnabEndpoint(+ name="money_movements",+ path="plans/{plan_id}/money_movements",+ data_selector="data.money_movements",+ fanout=PLAN_FANOUT,+ ),+ "money_movement_groups": YnabEndpoint(+ name="money_movement_groups",+ path="plans/{plan_id}/money_movement_groups",+ data_selector="data.money_movement_groups",+ partition_key="group_created_at",+ fanout=PLAN_FANOUT,+ ),"transactions": YnabEndpoint(name="transactions",path="plans/{plan_id}/transactions",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/tests/test_ynab.py b/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/tests/test_ynab.pyindex 933ff406b825..e9813fd40078 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/tests/test_ynab.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/tests/test_ynab.py@@ -106,6 +106,18 @@ def _accounts_script() -> Any:{"scheduled_transactions": [{"id": "record-1", "frequency": "monthly"}]},{"id": "record-1", "frequency": "monthly"},),+ (+ "money_movements",+ "money_movements",+ {"money_movements": [{"id": "record-1", "amount": 5000, "to_category_id": "category-1"}]},+ {"id": "record-1", "amount": 5000, "to_category_id": "category-1"},+ ),+ (+ "money_movement_groups",+ "money_movement_groups",+ {"money_movement_groups": [{"id": "record-1", "group_created_at": "2024-01-02T03:04:05Z"}]},+ {"id": "record-1", "group_created_at": "2024-01-02T03:04:05Z"},+ ),],)def test_full_refresh_across_plans(@@ -150,6 +162,90 @@ def test_plans_list_and_empty_account(driver: SourceDriver) -> None:assert accounts.saved_states == []+def _month_detail(categories: list[dict[str, Any]]) -> ScriptedResponse:+ return ScriptedResponse(json={"data": {"month": {"categories": categories}}})+++ "plan,resume_state,expected_rows",+ [+ (+ {"id": "plan-1", "first_month": "2023-11-01", "last_month": "2024-01-01"},+ None,+ [("plan-1", "2023-11-01"), ("plan-1", "2023-12-01"), ("plan-1", "2024-01-01")],+ ),+ (+ {"id": "plan-1", "first_month": "2023-11-01", "last_month": "2024-01-01"},+ YnabResumeConfig(completed=["plans/plan-1/months/2023-11-01"]),+ [("plan-1", "2023-12-01"), ("plan-1", "2024-01-01")],+ ),+ ({"id": "plan-1", "first_month": "2024-01-01", "last_month": "2024-01-01"}, None, [("plan-1", "2024-01-01")]),+ ({"id": "plan-1", "first_month": None, "last_month": None}, None, []),+ ({"id": "plan-1"}, None, []),+ ],+)+def test_month_categories_cover_each_plan_month(+ plan: dict[str, Any],+ resume_state: YnabResumeConfig | None,+ expected_rows: list[tuple[str, str]],+ driver: SourceDriver,+) -> None:+ category = {"id": "category-1", "budgeted": 1000, "activity": -500, "balance": 500}+ result = driver.run(+ "month_categories",+ route(+ {+ "/v1/plans": [ScriptedResponse(json={"data": {"plans": [plan, {"id": "plan-2"}]}})],+ **{+ f"/v1/plans/plan-1/months/{month}": [_month_detail([category])]+ for month in ["2023-11-01", "2023-12-01", "2024-01-01"]+ },+ }+ ),+ resume_state=resume_state,+ )++ rows = [row for batch in result.items for row in batch]+ assert [(row["plan_id"], row["month"]) for row in rows] == expected_rows+ assert all(row["id"] == "category-1" and row["balance"] == 500 for row in rows)+ assert result.response is not None and result.response.primary_keys is not None+ assert len({tuple(row[key] for key in result.response.primary_keys) for row in rows}) == len(rows)+ assert result.paths == ["/v1/plans", *[f"/v1/plans/plan-1/months/{month}" for _, month in expected_rows]]+ assert all(dict(request.query) == {} for request in result.requests)+ if expected_rows:+ assert result.saved_states[-1].completed[-1] == f"plans/plan-1/months/{expected_rows[-1][1]}"+++def test_month_categories_skip_missing_month(driver: SourceDriver) -> None:+ result = driver.run(+ "month_categories",+ route(+ {+ "/v1/plans": [+ ScriptedResponse(+ json={+ "data": {+ "plans": [{"id": "plan-1", "first_month": "2024-01-01", "last_month": "2024-02-01"}]+ }+ }+ )+ ],+ "/v1/plans/plan-1/months/2024-01-01": [+ ScriptedResponse(status=404, json={"error": {"id": "404.2", "name": "resource_not_found"}})+ ],+ "/v1/plans/plan-1/months/2024-02-01": [_month_detail([{"id": "category-1"}])],+ }+ ),+ )++ assert result.raised is None+ assert result.items == [[{"id": "category-1", "plan_id": "plan-1", "month": "2024-02-01"}]]+ assert result.saved_states[-1].completed == [+ "plans/plan-1/months/2024-01-01",+ "plans/plan-1/months/2024-02-01",+ ]++@pytest.mark.parametrize("groups", [[], [{"id": "group-1", "categories": []}]])def test_empty_categories_complete_the_plan(groups: list[dict[str, Any]], driver: SourceDriver) -> None:result = driver.run(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/ynab.py b/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/ynab.pyindex 0a704f2e2e5c..eebeeff64f4b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/ynab.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/ynab/ynab.py@@ -1,3 +1,4 @@+from datetime import datefrom typing import Anyfrom requests.exceptions import HTTPError@@ -31,6 +32,20 @@ class YnabResumeConfig:completed: list[str]+def _plan_months(plan: dict[str, Any]) -> list[dict[str, Any]]:+ if not plan.get("first_month") or not plan.get("last_month"):+ return []+ month = date.fromisoformat(plan["first_month"]).replace(day=1)+ last_month = date.fromisoformat(plan["last_month"]).replace(day=1)+ months: list[dict[str, Any]] = []+ while month <= last_month:+ months.append({"id": plan["id"], "month": month.isoformat()})+ month = (+ month.replace(year=month.year + 1, month=1) if month.month == 12 else month.replace(month=month.month + 1)+ )+ return months++def validate_credentials(api_key: str, api_version: str, schema_name: str | None) -> tuple[bool, str | None]:client = RESTClient(base_url=f"https://api.ynab.com/{api_version}/",@@ -89,6 +104,12 @@ def ynab_source(# Wildcard selectors have no matches when every category group is empty."data_selector_required": endpoint != "categories",},+ child_params_extra=(+ {"month": {"type": "resolve", "resource": "plans", "field": "month"}}+ if endpoint_config.per_month+ else None+ ),+ parent_data_map=_plan_months if endpoint_config.per_month else None,resume_hook=save_checkpoint,initial_paginator_state={"completed": resume.completed} if resume else None,)