Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 4 missing tables to the Harvest source (#121119)

Tom Owers6bfcf60cb02c
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index c334952419ab..fcb64729c16e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -5712,18 +5712,18 @@ Note: Downloaded the official bundled OpenAPI specification and enumerated GET c
## Harvest — gaps
-Today (15): `clients`, `contacts`, `estimate_item_categories`, `estimates`, `expense_categories`, `expenses`, `invoice_item_categories`, `invoices`, `projects`, `roles`, `task_assignments`, `tasks`, `time_entries`, `user_assignments`, `users`
+Today (19): `clients`, `contacts`, `estimate_item_categories`, `estimates`, `expense_categories`, `expenses`, `invoice_item_categories`, `invoice_payments`, `invoices`, `projects`, `pto_requests`, `pto_types`, `roles`, `task_assignments`, `tasks`, `time_entries`, `user_assignments`, `user_cost_rates`, `users`
Diffed against: <https://help.getharvest.com/api-v2/>
-- [ ] `Invoice payments (GET /v2/invoices/{INVOICE_ID}/payments)` — reconcile invoice revenue with actual cash receipts, partial payments, and payment dates (high)
-- [ ] `User cost rates (GET /v2/users/{USER_ID}/cost_rates)` — track effective cost rate changes and analyze labor costs, including periods without logged time (medium)
+- [x] `Invoice payments (GET /v2/invoices/{INVOICE_ID}/payments)` — reconcile invoice revenue with actual cash receipts, partial payments, and payment dates (high) Shipped as `invoice_payments`, fanned out over `invoices`.
+- [x] `User cost rates (GET /v2/users/{USER_ID}/cost_rates)` — track effective cost rate changes and analyze labor costs, including periods without logged time (medium) Shipped as `user_cost_rates`, fanned out over `users` (full refresh only).
- [ ] `User billable rates (GET /v2/users/{USER_ID}/billable_rates)` — track billing rate changes and compare historical pricing across users and projects (medium)
- [ ] `Invoice messages (GET /v2/invoices/{INVOICE_ID}/messages)` — analyze invoice communication, payment reminders, and collection follow-up activity (medium)
- [ ] `Estimate messages (GET /v2/estimates/{estimate_ID}/messages)` — connect estimate follow-up activity with acceptance outcomes and sales cycle duration (medium)
-- [ ] `Time off requests (GET /v2/pto/requests)` — analyze approved leave, pending requests, absence patterns, and approval delays (medium)
+- [x] `Time off requests (GET /v2/pto/requests)` — analyze approved leave, pending requests, absence patterns, and approval delays (medium) Shipped as `pto_requests`.
- [ ] `Time off balances (GET /v2/pto/balances)` — measure accrued, used, pending, and remaining leave by employee and policy (medium)
-- [ ] `Time off policies (GET /v2/pto/types)` — identify leave categories and interpret policy allocation and accrual rules (medium)
+- [x] `Time off policies (GET /v2/pto/types)` — identify leave categories and interpret policy allocation and accrual rules (medium) Shipped as `pto_types` (active policies, full refresh).
- [ ] `Work schedules (GET /v2/pto/work_schedules)` — compare tracked hours and leave against expected working days and capacity (medium)
Note: Read the official API index and selected pages, extracting documented GET collection paths. Compared them with fifteen static tables; time entries include applied rates, while rate collections add effective history. Invoice, estimate, and rate collections need parent fan-out; leave collections require appropriate permissions. Excluded writes, settings, duplicate assignment views, and reports derivable from existing detail tables.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/canonical_descriptions.py
index fcc92024ab99..e3ed0faa5424 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/canonical_descriptions.py
@@ -122,6 +122,22 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
use_as_expense="Whether the category is used for expenses when generating an invoice.",
),
},
+ "invoice_payments": {
+ "description": "A payment recorded against an invoice. An invoice can have several partial payments.",
+ "docs_url": "https://help.getharvest.com/api-v2/invoices-api/invoices/invoice-payments/",
+ "columns": _columns(
+ invoice_id="ID of the invoice the payment was recorded against.",
+ amount="The amount of the payment.",
+ paid_at="Date and time the payment was made.",
+ paid_date="Date the payment was made.",
+ recorded_by="Name of the person who recorded the payment.",
+ recorded_by_email="Email of the person who recorded the payment.",
+ notes="Notes associated with the payment.",
+ transaction_id="Card authorization or PayPal transaction ID.",
+ payment_gateway="The ID and name of the payment gateway used to process the payment.",
+ created_at="Date and time the payment was recorded.",
+ ),
+ },
"invoices": {
"description": "An invoice issued to a client, including its line items.",
"docs_url": "https://help.getharvest.com/api-v2/invoices-api/invoices/invoices/",
@@ -158,6 +174,47 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
closed_at="Date and time the invoice was closed.",
),
},
+ "pto_requests": {
+ "description": "A time off request and its review status.",
+ "docs_url": "https://help.getharvest.com/api-v2/pto-api/pto/time-off-requests/",
+ "columns": _columns(
+ status="Status of the request: pending, approved, declined, changes_requested, or cancelled.",
+ start_date="Start date of the request.",
+ end_date="End date of the request.",
+ days="Total business days requested.",
+ business_days_at_submission="Working days at the time the request was submitted.",
+ start_time_minutes="Starting minute offset for a partial-day request.",
+ end_time_minutes="Ending minute offset for a partial-day request.",
+ notes="Notes from the person who submitted the request.",
+ review_reason="Reason given by the manager or administrator who reviewed the request.",
+ reviewed_at="Date and time the request was reviewed.",
+ user="The ID and name of the user the request is for.",
+ pto_type="The time off policy the request is booked against.",
+ reviewer="The ID and name of the reviewer, or null.",
+ request_days="Daily breakdown of the request, with the hours and day fraction for each date.",
+ coverage_warnings="Coworkers on shared projects who are also away on the requested dates.",
+ ),
+ },
+ "pto_types": {
+ "description": "A time off policy (leave category) and its accrual rules.",
+ "docs_url": "https://help.getharvest.com/api-v2/pto-api/pto/time-off-policies/",
+ "columns": _columns(
+ name="Name of the time off policy, for example Vacation or Sick Leave.",
+ color="Color assigned to the policy in the UI.",
+ icon="Icon assigned to the policy in the UI.",
+ accrual_type="How the allowance is granted: fixed (front-loaded), accrued (earned over time), or unlimited.",
+ accrual_schedule="How often leave accrues: monthly, biweekly, semimonthly, or hourly.",
+ accrual_first_payday="Anchor date for biweekly accruals.",
+ accrual_leave_hours="Leave hours earned per worked-hours base, for hourly accruals.",
+ accrual_worked_hours="Worked-hours base, for hourly accruals.",
+ accrual_max_hours_per_year="Maximum accrued hours per year, for hourly accruals.",
+ default_days_per_year="Default days per year credited to teammates.",
+ max_carryover_days="Maximum days that carry over between years.",
+ requires_approval="Whether requests under this policy need manager or administrator review.",
+ visible_to_all="Whether every teammate can see and book this policy.",
+ is_active="Whether the policy is active, or deactivated.",
+ ),
+ },
"projects": {
"description": "A project that time and expenses are tracked against.",
"docs_url": "https://help.getharvest.com/api-v2/projects-api/projects/projects/",
@@ -261,6 +318,16 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
budget="Budget in hours or currency for this user on the project.",
),
},
+ "user_cost_rates": {
+ "description": "A cost rate for a user, with the dates it was in effect.",
+ "docs_url": "https://help.getharvest.com/api-v2/users-api/users/cost-rates/",
+ "columns": _columns(
+ user_id="ID of the user the cost rate belongs to.",
+ amount="The amount of the cost rate.",
+ start_date="Date the cost rate takes effect.",
+ end_date="Date the cost rate stops being in effect, calculated by Harvest. Null for the current rate.",
+ ),
+ },
"users": {
"description": "A person in the Harvest account.",
"docs_url": "https://help.getharvest.com/api-v2/users-api/users/users/",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/harvest.py b/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/harvest.py
index 79f941dc370e..235c5cb2be98 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/harvest.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/harvest.py
@@ -1,15 +1,23 @@
import datetime
import dataclasses
-from typing import Any, Optional
+from collections.abc import Iterable
+from typing import Any, Optional, cast
from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source import (
RESTAPIConfig,
rest_api_resource,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
+ build_dependent_resource,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (
JSONResponsePaginator,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (
+ ClientConfig,
+ IncrementalConfig,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probe
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
@@ -27,12 +35,15 @@ HARVEST_USER_AGENT = "PostHog (https://posthog.com)"
REQUEST_TIMEOUT = (10.0, 60.0)
[email protected](frozen=True)
class HarvestResumeConfig:
# Absolute URL of the next page, taken from the response's `links.next`. Harvest's own
# docs tell integrators to follow that link rather than rebuild page/cursor params, since
# it picks whichever pagination style is cheapest for the endpoint.
- next_url: str
+ next_url: Optional[str] = None
+ # Per-parent sub-resources (invoice payments, user cost rates) checkpoint the shape
+ # `build_dependent_resource` emits: which parents finished and where the current one stopped.
+ fanout_state: Optional[dict[str, Any]] = None
def _to_iso8601(value: Any) -> Optional[str]:
@@ -77,6 +88,88 @@ def _build_params(endpoint: str, should_use_incremental_field: bool) -> dict[str
return params
+def _client_config(account_id: str, access_token: str, api_version: str) -> ClientConfig:
+ return {
+ "base_url": f"{HARVEST_BASE_URL}/{api_version}",
+ "headers": _build_headers(account_id),
+ "auth": {"type": "bearer", "token": access_token},
+ # Harvest returns a self-contained absolute next-page URL, which already carries
+ # `per_page` and `updated_since`, so the filter survives every page.
+ "paginator": JSONResponsePaginator(next_url_path="links.next"),
+ # Pagination and resume follow URLs supplied by the response body, so pin them to
+ # the API host and refuse redirects - a tampered link must not carry the bearer
+ # token somewhere else.
+ "allowed_hosts": [HARVEST_API_HOST],
+ "allow_redirects": False,
+ "request_timeout": REQUEST_TIMEOUT,
+ }
+
+
+def _updated_since(cursor_path: str) -> IncrementalConfig:
+ return {"cursor_path": cursor_path, "start_param": "updated_since", "convert": _to_iso8601}
+
+
+def _fanout_source(
+ endpoint: str,
+ client_config: ClientConfig,
+ team_id: int,
+ job_id: str,
+ resumable_source_manager: ResumableSourceManager[HarvestResumeConfig],
+ use_incremental: bool,
+ db_incremental_field_last_value: Optional[Any],
+) -> SourceResponse:
+ config = HARVEST_ENDPOINTS[endpoint]
+ assert config.fanout is not None
+ parent_config = HARVEST_ENDPOINTS[config.fanout.parent_name]
+
+ initial_state: Optional[dict[str, Any]] = None
+ if resumable_source_manager.can_resume():
+ resume = resumable_source_manager.load_state()
+ if resume is not None:
+ initial_state = resume.fanout_state
+
+ def save_checkpoint(state: Optional[dict[str, Any]]) -> None:
+ if state:
+ resumable_source_manager.save_state(HarvestResumeConfig(fanout_state=state))
+
+ # The parent listing always walks the full collection: a child can change without its
+ # parent's `updated_at` moving, so the child's own `updated_since` does the narrowing.
+ child = cast(
+ Iterable[Any],
+ build_dependent_resource(
+ endpoint_configs=HARVEST_ENDPOINTS,
+ child_endpoint=endpoint,
+ fanout=config.fanout,
+ client_config=client_config,
+ path_format_values={},
+ team_id=team_id,
+ job_id=job_id,
+ db_incremental_field_last_value=db_incremental_field_last_value if use_incremental else None,
+ should_use_incremental_field=use_incremental,
+ incremental_field="updated_at",
+ incremental_config_factory=_updated_since,
+ page_size_param="per_page",
+ parent_endpoint_extra={"data_selector": parent_config.data_key, "data_selector_required": True},
+ child_endpoint_extra={"data_selector": config.data_key, "data_selector_required": True},
+ resume_hook=save_checkpoint,
+ initial_paginator_state=initial_state,
+ ),
+ )
+
+ return SourceResponse(
+ name=endpoint,
+ items=lambda: child,
+ primary_keys=config.primary_keys,
+ partition_count=1,
+ partition_size=1,
+ partition_mode="datetime" if config.partition_key else None,
+ partition_format="month" if config.partition_key else None,
+ partition_keys=[config.partition_key] if config.partition_key else None,
+ # Rows arrive one parent at a time, so they are not one ordered series.
+ sort_mode=None,
+ )
+
+
def harvest_source(
account_id: str,
access_token: str,
@@ -90,22 +183,21 @@ def harvest_source(
) -> SourceResponse:
config = HARVEST_ENDPOINTS[endpoint]
use_incremental = should_use_incremental_field and config.supports_updated_since
+ client_config = _client_config(account_id, access_token, api_version)
+
+ if config.fanout is not None:
+ return _fanout_source(
+ endpoint,
+ client_config,
+ team_id,
+ job_id,
+ resumable_source_manager,
+ use_incremental,
+ db_incremental_field_last_value,
+ )
rest_config: RESTAPIConfig = {
- "client": {
- "base_url": f"{HARVEST_BASE_URL}/{api_version}",
- "headers": _build_headers(account_id),
- "auth": {"type": "bearer", "token": access_token},
- # Harvest returns a self-contained absolute next-page URL, which already carries
- # `per_page` and `updated_since`, so the filter survives every page.
- "paginator": JSONResponsePaginator(next_url_path="links.next"),
- # Pagination and resume follow URLs supplied by the response body, so pin them to
- # the API host and refuse redirects - a tampered link must not carry the bearer
- # token somewhere else.
- "allowed_hosts": [HARVEST_API_HOST],
- "allow_redirects": False,
- "request_timeout": REQUEST_TIMEOUT,
- },
+ "client": client_config,
"resources": [
{
"name": endpoint,
@@ -124,7 +216,7 @@ def harvest_source(
initial_paginator_state: Optional[dict[str, Any]] = None
if resumable_source_manager.can_resume():
resume = resumable_source_manager.load_state()
- if resume is not None:
+ if resume is not None and resume.next_url:
initial_paginator_state = {"next_url": resume.next_url}
def save_checkpoint(state: Optional[dict[str, Any]]) -> None:
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/settings.py
index 476604772b24..c7073b383583 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/settings.py
@@ -1,6 +1,10 @@
from dataclasses import dataclass, field
from typing import Optional
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
+ DependentEndpointConfig,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import ResponseAction
from products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import incremental_field
from products.warehouse_sources.backend.types import IncrementalField
@@ -28,6 +32,21 @@ class HarvestEndpointConfig:
# partitioning only adds overhead.
partition_key: Optional[str] = None
description: Optional[str] = None
+ # Set for sub-resources that Harvest only lists per parent object (e.g. an invoice's payments).
+ fanout: Optional[DependentEndpointConfig] = None
+
+ @property
+ def incremental_fields(self) -> list[IncrementalField]:
+ return [incremental_field("updated_at")] if self.supports_updated_since else []
+
+ @property
+ def default_incremental_field(self) -> Optional[str]:
+ return "updated_at" if self.supports_updated_since else None
+
+
+# A parent deleted between the parent listing and its child fetch answers 404. Treat it as an
+# empty child so one deleted invoice or user doesn't abort the whole fan-out.
+_MISSING_PARENT_IGNORE: list[ResponseAction] = [{"status_code": 404, "action": "ignore"}]
HARVEST_ENDPOINTS: dict[str, HarvestEndpointConfig] = {
@@ -76,6 +95,23 @@ HARVEST_ENDPOINTS: dict[str, HarvestEndpointConfig] = {
data_key="invoice_item_categories",
description="Categories available for invoice line items.",
),
+ "invoice_payments": HarvestEndpointConfig(
+ name="invoice_payments",
+ path="invoices/{invoice_id}/payments",
+ data_key="invoice_payments",
+ primary_keys=["invoice_id", "id"],
+ partition_key="created_at",
+ # Payment rows don't carry their invoice, so the parent id is injected as `invoice_id`.
+ fanout=DependentEndpointConfig(
+ parent_name="invoices",
+ resolve_param="invoice_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "invoice_id"},
+ child_response_actions=_MISSING_PARENT_IGNORE,
+ ),
+ description="Payments recorded against each invoice, including partial payments.",
+ ),
"invoices": HarvestEndpointConfig(
name="invoices",
path="invoices",
@@ -84,6 +120,22 @@ HARVEST_ENDPOINTS: dict[str, HarvestEndpointConfig] = {
partition_key="created_at",
description="Invoices issued to clients, including their line items.",
),
+ "pto_requests": HarvestEndpointConfig(
+ name="pto_requests",
+ path="pto/requests",
+ data_key="pto_requests",
+ partition_key="created_at",
+ description="Time off requests and their review status. Non-administrators only see their own requests.",
+ ),
+ "pto_types": HarvestEndpointConfig(
+ name="pto_types",
+ path="pto/types",
+ data_key="pto_types",
+ # The listing only returns active policies by default, so an incremental merge would never
+ # see a policy get deactivated. The table is tiny, so full refresh keeps it accurate.
+ supports_updated_since=False,
+ description="Time off policies (leave categories) and their accrual rules. Only active policies are listed.",
+ ),
"projects": HarvestEndpointConfig(
name="projects",
path="projects",
@@ -123,6 +175,23 @@ HARVEST_ENDPOINTS: dict[str, HarvestEndpointConfig] = {
data_key="user_assignments",
description="Users assigned to projects, with their rate and budget.",
),
+ "user_cost_rates": HarvestEndpointConfig(
+ name="user_cost_rates",
+ path="users/{user_id}/cost_rates",
+ data_key="cost_rates",
+ primary_keys=["user_id", "id"],
+ supports_updated_since=False,
+ fanout=DependentEndpointConfig(
+ parent_name="users",
+ resolve_param="user_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "user_id"},
+ child_response_actions=_MISSING_PARENT_IGNORE,
+ ),
+ description="Each user's cost rate history, with the dates each rate was in effect. Full refresh "
+ "only - the endpoint has no server-side modified-since filter. Needs an administrator.",
+ ),
"users": HarvestEndpointConfig(
name="users",
path="users",
@@ -137,8 +206,7 @@ ENDPOINTS = tuple(HARVEST_ENDPOINTS.keys())
# via `updated_since`. Endpoints without that filter advertise no incremental field, so
# `build_endpoint_schemas` marks them full refresh only.
INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {
- name: [incremental_field("updated_at")] if config.supports_updated_since else []
- for name, config in HARVEST_ENDPOINTS.items()
+ name: config.incremental_fields for name, config in HARVEST_ENDPOINTS.items()
}
DESCRIPTIONS: dict[str, str] = {
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/tests/test_harvest.py b/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/tests/test_harvest.py
index 8a14f402dd0f..00ce763a1ab0 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/tests/test_harvest.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/harvest/tests/test_harvest.py
@@ -6,6 +6,7 @@ from parameterized import parameterized
from products.warehouse_sources.backend.temporal.data_imports.sources.common.testing import (
ScriptedResponse,
SourceDriver,
+ route,
scripted_network,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.harvest import (
@@ -37,6 +38,9 @@ def _page(
return ScriptedResponse(json=body)
+_TOP_LEVEL_ENDPOINTS = [name for name in ENDPOINTS if HARVEST_ENDPOINTS[name].fanout is None]
+
+
def _driver() -> SourceDriver:
return SourceDriver(HarvestSource(), HarvestSourceConfig(account_id="123456", access_token="pat-secret"))
@@ -136,7 +140,7 @@ class TestHarvestTransport:
assert isinstance(result.raised, Exception)
assert "time_entries" in str(result.raised)
- @parameterized.expand(ENDPOINTS)
+ @parameterized.expand(_TOP_LEVEL_ENDPOINTS)
def test_every_endpoint_reads_its_own_envelope_and_path(self, endpoint: str) -> None:
config = HARVEST_ENDPOINTS[endpoint]
result = _driver().run(endpoint, [_page([{"id": 1}], data_key=config.data_key)])
@@ -150,6 +154,92 @@ class TestHarvestTransport:
assert response.name == endpoint
assert response.primary_keys == config.primary_keys
+ @parameterized.expand(
+ [
+ ("invoice_payments", "invoices", "invoice_id", True, None),
+ ("invoice_payments", "invoices", "invoice_id", False, None),
+ ("user_cost_rates", "users", "user_id", True, datetime.datetime(2026, 5, 4, tzinfo=datetime.UTC)),
+ (
+ "invoice_payments",
+ "invoices",
+ "invoice_id",
+ True,
+ datetime.datetime(2026, 5, 4, 9, 30, tzinfo=datetime.UTC),
+ ),
+ ]
+ )
+ def test_fanout_walks_every_parent_and_tags_rows_with_the_parent_id(
+ self, endpoint: str, parent: str, parent_key: str, incremental: bool, last_value: Any
+ ) -> None:
+ config = HARVEST_ENDPOINTS[endpoint]
+ parent_data_key = HARVEST_ENDPOINTS[parent].data_key
+ parent_next = f"https://{HARVEST_API_HOST}/v2/{parent}?cursor=p2"
+ result = _driver().run(
+ endpoint,
+ route(
+ {
+ f"/v2/{parent}": [
+ _page([{"id": 10}], data_key=parent_data_key, next_url=parent_next),
+ _page([{"id": 20}], data_key=parent_data_key),
+ ],
+ f"/v2/{config.path.format(**{parent_key: 10})}": [_page([{"id": 1}], data_key=config.data_key)],
+ f"/v2/{config.path.format(**{parent_key: 20})}": [_page([{"id": 2}], data_key=config.data_key)],
+ }
+ ),
+ should_use_incremental_field=incremental,
+ db_incremental_field_last_value=last_value,
+ )
+
+ assert result.raised is None
+ # Child ids are only unique per parent, so the parent id must land in the row for the key.
+ assert sorted((row[parent_key], row["id"]) for row in result.rows) == [(10, 1), (20, 2)]
+ response = result.response
+ assert response is not None
+ assert response.primary_keys == [parent_key, "id"]
+ assert response.sort_mode is None
+ # Filtering the parent listing would skip children that changed under an unchanged parent.
+ parent_requests = [r for r in result.requests if r.path == f"/v2/{parent}"]
+ assert [r.param("updated_since") for r in parent_requests] == [None, None]
+ child_requests = [r for r in result.requests if r.path != f"/v2/{parent}"]
+ expected = "2026-05-04T09:30:00Z" if last_value is not None and config.supports_updated_since else None
+ assert [r.param("updated_since") for r in child_requests] == [expected, expected]
+
+ @parameterized.expand([("invoice_payments", "invoices", "invoice_id"), ("user_cost_rates", "users", "user_id")])
+ def test_fanout_skips_a_parent_deleted_mid_sync(self, endpoint: str, parent: str, parent_key: str) -> None:
+ config = HARVEST_ENDPOINTS[endpoint]
+ result = _driver().run(
+ endpoint,
+ route(
+ {
+ f"/v2/{parent}": [_page([{"id": 10}, {"id": 20}], data_key=HARVEST_ENDPOINTS[parent].data_key)],
+ f"/v2/{config.path.format(**{parent_key: 10})}": [ScriptedResponse(status=404)],
+ f"/v2/{config.path.format(**{parent_key: 20})}": [_page([{"id": 2}], data_key=config.data_key)],
+ }
+ ),
+ )
+
+ # One 404 must not abort the fan-out and drop every later parent's rows.
+ assert result.raised is None
+ assert [(row[parent_key], row["id"]) for row in result.rows] == [(20, 2)]
+
+ def test_fanout_resume_skips_completed_parents(self) -> None:
+ result = _driver().run(
+ "invoice_payments",
+ route(
+ {
+ "/v2/invoices": [_page([{"id": 10}, {"id": 20}], data_key="invoices")],
+ "/v2/invoices/20/payments": [_page([{"id": 2}], data_key="invoice_payments")],
+ }
+ ),
+ resume_state=HarvestResumeConfig(
+ fanout_state={"completed": ["invoices/10/payments"], "current": None, "child_state": None}
+ ),
+ )
+
+ assert result.raised is None
+ assert "/v2/invoices/10/payments" not in result.paths
+ assert [row["id"] for row in result.rows] == [2]
+
@parameterized.expand([("time_entries", "created_at"), ("invoices", "created_at"), ("roles", None)])
def test_partitioning_matches_the_endpoint_config(self, endpoint: str, partition_key: str | None) -> None:
config = HARVEST_ENDPOINTS[endpoint]