:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse): add pipedrive deal products, changelog and lookup tables (#125448)
Tom Owers898063ae635f
diff --git a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt b/posthog/test/repo_invariants/dataclass_frozen_baseline.txtindex b51e60c064d1..5634a3cfb8c7 100644--- a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt+++ b/posthog/test/repo_invariants/dataclass_frozen_baseline.txt@@ -851,7 +851,6 @@1 products/warehouse_sources/backend/temporal/data_imports/sources/pinterest_ads/pinterest_ads.py1 products/warehouse_sources/backend/temporal/data_imports/sources/pinterest_ads/settings.py1 products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/pipedrive.py-1 products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/settings.py1 products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/pipeliner.py1 products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/settings.py1 products/warehouse_sources/backend/temporal/data_imports/sources/plaid/plaid.pydiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 657b05ed1d02..8f27341a160f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -9448,15 +9448,15 @@ Note: The synced alerts table maps to GET /actions. Every high-value gap is a pe## Pipedrive — gaps-Today (13): `activities`, `deal_fields`, `deals`, `leads`, `notes`, `organization_fields`, `organizations`, `person_fields`, `persons`, `pipelines`, `products`, `stages`, `users`+Today (17): `activities`, `activity_types`, `deal_changelog`, `deal_fields`, `deal_products`, `deals`, `lead_labels`, `leads`, `notes`, `organization_fields`, `organizations`, `person_fields`, `persons`, `pipelines`, `products`, `stages`, `users`Diffed against: <https://developers.pipedrive.com/docs/api/v2/openapi.yaml>-- [ ] `GET /deals/{id}/products (v2) and /deals/products` — deal line items - product, quantity, price, discount per deal; without them revenue can only be read as a single deal-level number (high)+- [x] `GET /deals/{id}/products (v2) and /deals/products` — deal line items - product, quantity, price, discount per deal; without them revenue can only be read as a single deal-level number (high). Added as `deal_products`, read through the batched `/deals/products` for every active and archived deal.- [ ] `GET /deals/archived (v2)` — /deals excludes archived deals, so the synced deals table silently loses closed/archived pipeline history (high)-- [ ] `GET /activityTypes (v1)` — lookup resolving the type key on every synced activity row (high)-- [ ] `GET /deals/{id}/flow and /deals/{id}/changelog (v1)` — per-field change history including stage moves - required for stage-transition, velocity and win-rate-over-time analysis (high)-- [ ] `GET /leadLabels (v1)` — lookup resolving the label IDs carried on the leads we already sync (high)+- [x] `GET /activityTypes (v1)` — lookup resolving the type key on every synced activity row (high). Added as `activity_types`.+- [x] `GET /deals/{id}/flow and /deals/{id}/changelog (v1)` — per-field change history including stage moves - required for stage-transition, velocity and win-rate-over-time analysis (high). Added as `deal_changelog`, read for every active and archived deal. `/deals/{id}/flow` skipped: it is a mixed timeline of activities, notes, files and field changes that the changelog and existing tables already cover.+- [x] `GET /leadLabels (v1)` — lookup resolving the label IDs carried on the leads we already sync (high). Added as `lead_labels`.- [ ] `GET /leadSources (v1)` — lookup resolving lead source values for attribution (medium)- [ ] `GET /currencies (v1)` — currency lookup for multi-currency deal and product values (medium)- [ ] `GET /pipelines/{id}/conversion_statistics and /movement_statistics (v1)` — Pipedrive's own funnel conversion and stage-movement metrics per pipeline (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/api_inventory.md b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/api_inventory.mdindex b452f392992c..d7d7b53108dc 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/api_inventory.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/api_inventory.md@@ -20,21 +20,34 @@ Reference: <https://developers.pipedrive.com/docs/api/v1>## Endpoints synced-| Schema | Path | Pagination | Primary key | Partition key |-| ------------------- | ---------------------------- | ---------- | ----------- | ------------- |-| deals | `/api/v2/deals` | cursor | id | add_time |-| persons | `/api/v2/persons` | cursor | id | add_time |-| organizations | `/api/v2/organizations` | cursor | id | add_time |-| products | `/api/v2/products` | cursor | id | add_time |-| pipelines | `/api/v2/pipelines` | cursor | id | add_time |-| stages | `/api/v2/stages` | cursor | id | add_time |-| activities | `/api/v2/activities` | cursor | id | add_time |-| notes | `/api/v1/notes` | offset | id | add_time |-| leads | `/api/v1/leads` | offset | id | add_time |-| users | `/api/v1/users` | offset | id | — |-| deal_fields | `/api/v1/dealFields` | offset | id | — |-| person_fields | `/api/v1/personFields` | offset | id | — |-| organization_fields | `/api/v1/organizationFields` | offset | id | — |+| Schema | Path | Pagination | Primary key | Partition key |+| ------------------- | ------------------------------ | ---------- | ------------------------ | ------------- |+| deals | `/api/v2/deals` | cursor | id | add_time |+| persons | `/api/v2/persons` | cursor | id | add_time |+| organizations | `/api/v2/organizations` | cursor | id | add_time |+| products | `/api/v2/products` | cursor | id | add_time |+| pipelines | `/api/v2/pipelines` | cursor | id | add_time |+| stages | `/api/v2/stages` | cursor | id | add_time |+| activities | `/api/v2/activities` | cursor | id | add_time |+| notes | `/api/v1/notes` | offset | id | add_time |+| leads | `/api/v1/leads` | offset | id | add_time |+| users | `/api/v1/users` | offset | id | — |+| deal_fields | `/api/v1/dealFields` | offset | id | — |+| person_fields | `/api/v1/personFields` | offset | id | — |+| organization_fields | `/api/v1/organizationFields` | offset | id | — |+| lead_labels | `/api/v1/leadLabels` | none | id | — |+| activity_types | `/api/v1/activityTypes` | none | id | — |+| deal_products | `/api/v2/deals/products` | cursor | deal_id, id | add_time |+| deal_changelog | `/api/v1/deals/{id}/changelog` | cursor | deal_id, field_key, time | time |++`lead_labels` and `activity_types` are documented as unpaginated and always return every row, so they are read in one request.++`deal_products` and `deal_changelog` fan out over every deal in `/api/v2/deals` and `/api/v2/deals/archived`, so a deal keeps its line items and history after it is archived.+`deal_products` sends up to 100 deal ids per request in the `deal_ids` param.+`deal_changelog` sends one request chain per deal, because the changelog has no batched form.+Changelog rows carry neither an id nor the deal id, so the source adds the `deal_id` it requested.+The resume state holds the deal list and page, the last finished deal id, and the cursor of the deal group in progress, so it does not grow with the account.+`/deals/{id}/flow` is not synced: it mixes activities, notes, files and field changes in one timeline, and the changelog and existing tables already cover those.## Source versionsdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/canonical_descriptions.pyindex df8223b9cfaa..0d9dafde2c5f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/canonical_descriptions.py@@ -224,4 +224,73 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"update_time": "Time at which the field was last updated.",},},+ "lead_labels": {+ "description": "A label that can be attached to leads in Pipedrive. Resolves the label IDs stored on each lead.",+ "docs_url": "https://developers.pipedrive.com/docs/api/v1/LeadLabels",+ "columns": {+ "id": "Unique identifier (UUID) for the lead label.",+ "name": "Name of the lead label.",+ "color": "Color of the lead label.",+ "add_time": "Time at which the lead label was created.",+ "update_time": "Time at which the lead label was last updated.",+ },+ },+ "activity_types": {+ "description": "An activity type (such as call, meeting, or email) configured in Pipedrive. Resolves the type key on each activity.",+ "docs_url": "https://developers.pipedrive.com/docs/api/v1/ActivityTypes",+ "columns": {+ "id": "Unique identifier for the activity type.",+ "name": "Name of the activity type.",+ "key_string": "Key of the activity type, matching the `type` value stored on activities.",+ "icon_key": "Icon used for the activity type.",+ "color": "Color of the activity type, as a hex value.",+ "order_nr": "Order number of the activity type.",+ "active_flag": "Whether the activity type is active.",+ "is_custom_flag": "Whether the activity type is a custom one.",+ "add_time": "Time at which the activity type was created.",+ "update_time": "Time at which the activity type was last updated.",+ },+ },+ "deal_products": {+ "description": "A product line item attached to a deal in Pipedrive, with its quantity, price, discount, and tax.",+ "docs_url": "https://developers.pipedrive.com/docs/api/v1/Deals#getDealProducts",+ "columns": {+ "id": "Unique identifier for the product attached to the deal.",+ "deal_id": "ID of the deal the product is attached to.",+ "product_id": "ID of the product.",+ "product_variation_id": "ID of the product variation, if any.",+ "name": "Name of the product as shown on the deal.",+ "item_price": "Price of one unit of the product.",+ "quantity": "Quantity of the product on the deal.",+ "discount": "Discount applied to the line item.",+ "discount_type": "Whether the discount is a percentage or an amount.",+ "tax": "Tax rate applied to the line item, as a percentage.",+ "tax_method": "How tax is applied (exclusive, inclusive, or none).",+ "sum": "Total value of the line item after discount and tax.",+ "currency": "Currency of the line item.",+ "comments": "Comments on the line item.",+ "is_enabled": "Whether the line item counts toward the deal value.",+ "billing_frequency": "How often the line item is billed (for example, one-time or monthly).",+ "billing_frequency_cycles": "Number of billing cycles for a recurring line item.",+ "billing_start_date": "Date on which billing of the line item starts.",+ "order_nr": "Position of the line item on the deal.",+ "add_time": "Time at which the product was attached to the deal.",+ "update_time": "Time at which the line item was last updated.",+ },+ },+ "deal_changelog": {+ "description": "A change to a field value on a deal in Pipedrive, such as a stage move. One row per field change.",+ "docs_url": "https://developers.pipedrive.com/docs/api/v1/Deals#getDealChangelog",+ "columns": {+ "deal_id": "ID of the deal that changed.",+ "field_key": "Key of the deal field that changed (for example, stage_id or status).",+ "old_value": "Value of the field before the change.",+ "new_value": "Value of the field after the change.",+ "actor_user_id": "ID of the user who made the change.",+ "time": "Time at which the change happened.",+ "change_source": "Where the change came from (for example, app, API, or automation).",+ "change_source_user_agent": "User agent of the client that made the change.",+ "is_bulk_update_flag": "Whether the change was part of a bulk update.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/pipedrive.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/pipedrive.pyindex af54b3ec6038..315288e54277 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/pipedrive.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/pipedrive.py@@ -2,6 +2,7 @@ import reimport secretsimport datetimeimport dataclasses+from collections.abc import Iteratorfrom typing import Any, Optionalfrom urllib.parse import urlencode@@ -24,25 +25,32 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resRESTAPIConfig,rest_api_resource,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.auth import APIKeyAuth+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.config_setup import (+ create_response_hooks,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (BasePaginator,JSONResponseCursorPaginator,OffsetPaginator,+ SinglePagePaginator,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.rest_client import RESTClient+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import ClientConfigfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.webhook_s3 import WebhookSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.pipedrive.settings import (+ DEAL_IDS_PER_REQUEST,+ DEAL_PARENT_PATHS,+ PAGE_SIZE,PipedriveEndpointConfig,endpoints_for_version,)LOGGER = structlog.get_logger(__name__)-# Pipedrive caps list pages at 500 items (default 100).-PAGE_SIZE = 500-# Webhook management lives on v1; there is no v2 equivalent.WEBHOOKS_PATH = "/api/v1/webhooks"# The payload format we register. v2 deliveries carry the entity under `data` in the same shape@@ -109,10 +117,17 @@ def _build_url(company_domain: str, path: str, params: dict[str, Any]) -> str:return f"{url}?{urlencode(clean_params)}"+def _cursor_paginator() -> JSONResponseCursorPaginator:+ return JSONResponseCursorPaginator(cursor_path="additional_data.next_cursor", cursor_param="cursor")++def _build_paginator(config: PipedriveEndpointConfig) -> BasePaginator:if config.pagination == "cursor":# v2 endpoints: opaque cursor in the response body; ``limit`` rides in the endpoint params.- return JSONResponseCursorPaginator(cursor_path="additional_data.next_cursor", cursor_param="cursor")+ return _cursor_paginator()+ if config.pagination == "none":+ # Documented as unpaginated: a full page must not send a second request at `start=500`.+ return SinglePagePaginator()# v1 endpoints: start/limit offset. No top-level total; termination is a short/empty page.return OffsetPaginator(limit=PAGE_SIZE,@@ -349,6 +364,115 @@ def delete_webhook(company_domain: str, api_token: str, webhook_url: str) -> Webreturn WebhookDeletionResult(success=True)+def _deal_fan_out_rows(+ client: RESTClient,+ config: PipedriveEndpointConfig,+ resumable_source_manager: ResumableSourceManager[PipedriveResumeConfig],+ initial_state: Optional[dict[str, Any]],+) -> Iterator[list[dict[str, Any]]]:+ """Read a deal child endpoint for every active and archived deal.++ Deals arrive in ascending id order, so the resume state stays small: the parent list and page+ being read, the last deal id whose rows are all yielded, and the cursor inside the deal group+ being read. The shared fan-out helper instead records every finished parent, which grows with+ the account.+ """+ state = initial_state or {}+ group_size = DEAL_IDS_PER_REQUEST if config.deal_fan_out == "batched" else 1+ # A deal deleted between the deals listing and this fetch has nothing left to read.+ hooks = create_response_hooks([{"status_code": 404, "action": "ignore"}], resource_name=config.name)+ start_parent = state.get("parent_index", 0)++ for parent_index, parent_path in enumerate(DEAL_PARENT_PATHS):+ if parent_index < start_parent:+ continue+ resuming = parent_index == start_parent+ progress: dict[str, Any] = {+ "parent_index": parent_index,+ "parent_page": state.get("parent_page") if resuming else None,+ "done_through": state.get("done_through") if resuming else None,+ "group": None,+ "child": None,+ }+ resume_group = state.get("group") if resuming else None+ resume_child = state.get("child") if resuming else None+ next_parent_page: dict[str, Any] = {}++ def stage(+ group: Optional[list[int]], child: Optional[dict[str, Any]], _progress: dict[str, Any] = progress+ ) -> None:+ _progress["group"] = group+ _progress["child"] = child+ resumable_source_manager.save_state(PipedriveResumeConfig(paginator_state=dict(_progress)))++ def on_parent_page(+ paginator_state: Optional[dict[str, Any]], has_next: bool, _next: dict[str, Any] = next_parent_page+ ) -> None:+ _next["state"] = paginator_state if has_next else None++ def read_group(+ group: list[int], child_initial: Optional[dict[str, Any]], _progress: dict[str, Any] = progress+ ) -> Iterator[list[dict[str, Any]]]:+ if config.deal_fan_out == "batched":+ path = config.path+ params: dict[str, Any] = {"deal_ids": ",".join(str(deal_id) for deal_id in group)}+ else:+ path = config.path.format(id=group[0])+ params = {}+ params["limit"] = PAGE_SIZE++ def on_child_page(+ paginator_state: Optional[dict[str, Any]],+ has_next: bool,+ _group: list[int] = group,+ ) -> None:+ # Staged before the page is yielded, so the cursor commits with the rows it covers.+ if has_next:+ stage(_group, paginator_state)+ else:+ _progress["done_through"] = _group[-1]+ stage(None, None)++ for rows in client.paginate(+ path=path,+ params=params,+ paginator=_cursor_paginator(),+ data_selector="data",+ hooks=hooks,+ initial_paginator_state=child_initial,+ page_state_hook=on_child_page,+ ):+ if config.deal_fan_out == "per_deal":+ for row in rows:+ row["deal_id"] = group[0]+ yield rows++ # A group with no rows stages nothing. Resuming from the last staged state reads such+ # groups again, which costs requests but yields no duplicate rows.+ _progress["done_through"] = group[-1]+ resumable_source_manager.safe_point()++ # Finish the saved group with its cursor before regrouping: deals added or deleted since the+ # crash would change the rebuilt groups, and restarting a half-read batch duplicates rows.+ if resume_group:+ yield from read_group(resume_group, resume_child)++ for deals in client.paginate(+ path=parent_path,+ params={"limit": PAGE_SIZE, "sort_by": "id", "sort_direction": "asc"},+ paginator=_cursor_paginator(),+ data_selector="data",+ initial_paginator_state=progress["parent_page"],+ page_state_hook=on_parent_page,+ ):+ done_through = progress["done_through"]+ deal_ids = [deal["id"] for deal in deals if done_through is None or deal["id"] > done_through]+ for start in range(0, len(deal_ids), group_size):+ yield from read_group(deal_ids[start : start + group_size], None)++ progress["parent_page"] = next_parent_page.get("state")++def pipedrive_source(company_domain: str,api_token: str,@@ -360,37 +484,19 @@ def pipedrive_source(db_incremental_field_last_value: Optional[Any] = None,api_version: str = "v2",) -> SourceResponse:- config = endpoints_for_version(api_version)[endpoint]-- params: dict[str, Any] = {}- if config.pagination == "cursor":- # The cursor paginator only injects ``cursor``; ``limit`` must be a static param.- params["limit"] = PAGE_SIZE-- rest_config: RESTAPIConfig = {- "client": {- "base_url": base_url(company_domain),- # Only the non-secret Accept header here; the token travels via framework auth so it's- # redacted from logged URLs, headers, sampled bodies, and raised error messages.- "headers": {"Accept": "application/json"},- "auth": {"type": "api_key", "api_key": api_token, "name": "x-api-token", "location": "header"},- "paginator": _build_paginator(config),- # base_url host (`{subdomain}.pipedrive.com`) is implicitly allowed; `[]` pins every- # request — including resume URLs — to it, matching the source's SSRF posture.- "allowed_hosts": [],- },- "resources": [- {- "name": endpoint,- "endpoint": {- "path": config.path,- "params": params,- # Mirrors the old `data.get("data") or []`: a 200 body without `data` yields an- # empty page rather than raising (not required).- "data_selector": "data",- },- }- ],+ endpoints = endpoints_for_version(api_version)+ config = endpoints[endpoint]++ client_config: ClientConfig = {+ "base_url": base_url(company_domain),+ # Only the non-secret Accept header here; the token travels via framework auth so it's+ # redacted from logged URLs, headers, sampled bodies, and raised error messages.+ "headers": {"Accept": "application/json"},+ "auth": {"type": "api_key", "api_key": api_token, "name": "x-api-token", "location": "header"},+ "paginator": _build_paginator(config),+ # base_url host (`{subdomain}.pipedrive.com`) is implicitly allowed; `[]` pins every+ # request — including resume URLs — to it, matching the source's SSRF posture.+ "allowed_hosts": [],}initial_paginator_state: Optional[dict[str, Any]] = None@@ -407,14 +513,46 @@ def pipedrive_source(if state:resumable_source_manager.save_state(PipedriveResumeConfig(paginator_state=state))- resource = rest_api_resource(- rest_config,- team_id,- job_id,- db_incremental_field_last_value,- resume_hook=save_checkpoint,- initial_paginator_state=initial_paginator_state,- )+ resource: Any+ column_hints = None+ if config.deal_fan_out is not None:+ client = RESTClient(+ base_url=base_url(company_domain),+ headers={"Accept": "application/json"},+ auth=APIKeyAuth(api_key=api_token, name="x-api-token", location="header"),+ allowed_hosts=[],+ )+ resource = _deal_fan_out_rows(client, config, resumable_source_manager, initial_paginator_state)+ else:+ params: dict[str, Any] = {}+ if config.pagination == "cursor":+ # The cursor paginator only injects ``cursor``; ``limit`` must be a static param.+ params["limit"] = PAGE_SIZE++ rest_config: RESTAPIConfig = {+ "client": client_config,+ "resources": [+ {+ "name": endpoint,+ "endpoint": {+ "path": config.path,+ "params": params,+ # Mirrors the old `data.get("data") or []`: a 200 body without `data` yields an+ # empty page rather than raising (not required).+ "data_selector": "data",+ },+ }+ ],+ }+ resource = rest_api_resource(+ rest_config,+ team_id,+ job_id,+ db_incremental_field_last_value,+ resume_hook=save_checkpoint,+ initial_paginator_state=initial_paginator_state,+ )+ column_hints = resource.column_hintswebhook_enabled = async_to_sync(webhook_source_manager.webhook_enabled)()@@ -428,11 +566,11 @@ def pipedrive_source(return SourceResponse(name=endpoint,items=items,- primary_keys=[config.primary_key],+ primary_keys=config.primary_keys,partition_count=1,partition_size=1,partition_mode="datetime" if config.partition_key else None,partition_format="week" if config.partition_key else None,partition_keys=[config.partition_key] if config.partition_key else None,- column_hints=resource.column_hints,+ column_hints=column_hints,)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/settings.pyindex 97404e6f45fc..f9999bbefff5 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/settings.py@@ -1,26 +1,40 @@-from dataclasses import dataclass+from dataclasses import dataclass, fieldfrom typing import Literal, Optional-PaginationStyle = Literal["cursor", "offset"]+PaginationStyle = Literal["cursor", "offset", "none"]+# "batched" reads one request per group of deals through a `deal_ids` query param;+# "per_deal" binds a single deal id into the `{id}` path placeholder.+DealFanOut = Literal["batched", "per_deal"]+# Pipedrive caps list pages at 500 items (default 100).+PAGE_SIZE = 500+# `/api/v2/deals/products` accepts at most 100 deal ids per request.+DEAL_IDS_PER_REQUEST = 100+# `/api/v2/deals` leaves out archived deals, so deal fan-outs read both lists to keep the line+# items and history of a deal after it is archived.+DEAL_PARENT_PATHS = ("/api/v2/deals", "/api/v2/deals/archived")-@dataclass++@dataclass(frozen=True)class PipedriveEndpointConfig:name: strpath: str- # v2 endpoints use cursor pagination; v1 endpoints use start/limit offset pagination.+ # v2 endpoints and the v1 changelog use cursor pagination; other v1 collections use+ # start/limit offset pagination. "none" is for lookups documented as always returning every row.pagination: PaginationStyle- primary_key: str = "id"+ primary_keys: list[str] = field(default_factory=lambda: ["id"])# Stable creation-time field used for datetime partitioning. Never an update_time-style# field, which would rewrite partitions on every sync. `None` disables partitioning for# endpoints (users, *_fields) whose rows have no stable creation timestamp.partition_key: Optional[str] = "add_time"+ # Set for endpoints read for every deal in `DEAL_PARENT_PATHS`.+ deal_fan_out: Optional[DealFanOut] = None# Endpoints whose path and pagination are identical across every supported source version.-# Their v1 collection endpoints have no v2 replacement (notes, leads, users, *_fields) or were-# already on v2 (deals, persons, organizations, products, pipelines, stages), so they don't-# change between our `v1` and `v2` labels.+# Their v1 collection endpoints have no v2 replacement (notes, leads, users, *_fields, lookups,+# the deal changelog) or were already on v2 (deals, persons, organizations, products, pipelines,+# stages, deal products), so they don't change between our `v1` and `v2` labels._SHARED_ENDPOINTS: dict[str, PipedriveEndpointConfig] = {"deals": PipedriveEndpointConfig(name="deals", path="/api/v2/deals", pagination="cursor"),"persons": PipedriveEndpointConfig(name="persons", path="/api/v2/persons", pagination="cursor"),@@ -40,6 +54,28 @@ _SHARED_ENDPOINTS: dict[str, PipedriveEndpointConfig] = {"organization_fields": PipedriveEndpointConfig(name="organization_fields", path="/api/v1/organizationFields", pagination="offset", partition_key=None),+ "lead_labels": PipedriveEndpointConfig(+ name="lead_labels", path="/api/v1/leadLabels", pagination="none", partition_key=None+ ),+ "activity_types": PipedriveEndpointConfig(+ name="activity_types", path="/api/v1/activityTypes", pagination="none", partition_key=None+ ),+ "deal_products": PipedriveEndpointConfig(+ name="deal_products",+ path="/api/v2/deals/products",+ pagination="cursor",+ primary_keys=["deal_id", "id"],+ deal_fan_out="batched",+ ),+ # Changelog rows carry no id or deal reference, so the source adds the deal id it requested.+ "deal_changelog": PipedriveEndpointConfig(+ name="deal_changelog",+ path="/api/v1/deals/{id}/changelog",+ pagination="cursor",+ primary_keys=["deal_id", "field_key", "time"],+ partition_key="time",+ deal_fan_out="per_deal",+ ),}# `activities` is the only endpoint that differs by version: Pipedrive deprecated the v1diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/tests/test_pipedrive.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/tests/test_pipedrive.pyindex fb704c82eeab..e21eeaedb090 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/tests/test_pipedrive.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/tests/test_pipedrive.py@@ -59,7 +59,11 @@ def _make_manager(resume_state: PipedriveResumeConfig | None = None) -> mock.Magdef _wire(- session: mock.MagicMock, responses: list[Response], *, prepared_url: Optional[str] = None+ session: mock.MagicMock,+ responses: list[Response],+ *,+ prepared_url: Optional[str] = None,+ urls: Optional[list[str]] = None,) -> list[dict[str, Any]]:"""Wire a mock session; return a list capturing each request's params AT SEND TIME.@@ -72,6 +76,8 @@ def _wire(def _prepare(request: Any) -> mock.MagicMock:param_snapshots.append(dict(request.params or {}))+ if urls is not None:+ urls.append(request.url)prepared = mock.MagicMock()prepared.url = prepared_url or "https://acme.pipedrive.com/api/v2/deals"return prepared@@ -315,14 +321,179 @@ class TestPagination:_rows(_source("deals", _make_manager()))+class TestLookupEndpoints:+ @pytest.mark.parametrize("endpoint", ["lead_labels", "activity_types"])+ @mock.patch(CLIENT_SESSION_PATCH)+ def test_reads_every_row_in_one_request(self, MockSession: mock.MagicMock, endpoint: str) -> None:+ session = MockSession.return_value+ full_page = [{"id": i} for i in range(PAGE_SIZE + 100)]+ params = _wire(session, [_response(full_page)])++ rows = _rows(_source(endpoint, _make_manager()))++ assert len(rows) == PAGE_SIZE + 100+ assert session.send.call_count == 1+ assert params == [{}]+++BASE = "https://acme.pipedrive.com"+DEAL_LIST_PARAMS = {"limit": PAGE_SIZE, "sort_by": "id", "sort_direction": "asc"}+++class TestDealFanOut:+ @mock.patch(CLIENT_SESSION_PATCH)+ def test_deal_products_batches_active_and_archived_deals(self, MockSession: mock.MagicMock) -> None:+ session = MockSession.return_value+ urls: list[str] = []+ params = _wire(+ session,+ [+ _response([{"id": 1}, {"id": 2}], {"next_cursor": None}),+ _response([{"id": 10, "deal_id": 1}], {"next_cursor": "c2"}),+ _response([{"id": 11, "deal_id": 2}], {"next_cursor": None}),+ _response([{"id": 3}], {"next_cursor": None}),+ _response([{"id": 12, "deal_id": 3}], {"next_cursor": None}),+ ],+ urls=urls,+ )+ manager = _make_manager()++ rows = _rows(_source("deal_products", manager))++ assert [row["id"] for row in rows] == [10, 11, 12]+ assert urls == [+ f"{BASE}/api/v2/deals",+ f"{BASE}/api/v2/deals/products",+ f"{BASE}/api/v2/deals/products",+ f"{BASE}/api/v2/deals/archived",+ f"{BASE}/api/v2/deals/products",+ ]+ assert params == [+ DEAL_LIST_PARAMS,+ {"deal_ids": "1,2", "limit": PAGE_SIZE},+ {"deal_ids": "1,2", "limit": PAGE_SIZE, "cursor": "c2"},+ DEAL_LIST_PARAMS,+ {"deal_ids": "3", "limit": PAGE_SIZE},+ ]+ assert manager.save_state.call_args.args[0] == PipedriveResumeConfig(+ paginator_state={"parent_index": 1, "parent_page": None, "done_through": 3, "group": None, "child": None}+ )++ @mock.patch(CLIENT_SESSION_PATCH)+ def test_deal_changelog_reads_each_deal_and_skips_a_deleted_one(self, MockSession: mock.MagicMock) -> None:+ session = MockSession.return_value+ urls: list[str] = []+ change = {"field_key": "stage_id", "old_value": "1", "new_value": "2", "time": "2024-05-01 10:00:00"}+ _wire(+ session,+ [+ _response([{"id": 7}, {"id": 8}], {"next_cursor": None}),+ _response([dict(change)], {"next_cursor": None}),+ _response(None, drop_data=True, status=404),+ _response([], {"next_cursor": None}),+ ],+ urls=urls,+ )++ rows = _rows(_source("deal_changelog", _make_manager()))++ assert rows == [{**change, "deal_id": 7}]+ assert urls == [+ f"{BASE}/api/v2/deals",+ f"{BASE}/api/v1/deals/7/changelog",+ f"{BASE}/api/v1/deals/8/changelog",+ f"{BASE}/api/v2/deals/archived",+ ]++ @mock.patch(CLIENT_SESSION_PATCH)+ def test_resume_continues_inside_the_saved_deal(self, MockSession: mock.MagicMock) -> None:+ session = MockSession.return_value+ urls: list[str] = []+ params = _wire(+ session,+ [+ _response([{"field_key": "status", "time": "2024-05-02 09:00:00"}], {"next_cursor": None}),+ _response([{"id": 5}, {"id": 6}, {"id": 7}], {"next_cursor": None}),+ _response([{"field_key": "value", "time": "2024-05-03 09:00:00"}], {"next_cursor": None}),+ ],+ urls=urls,+ )+ manager = _make_manager(+ PipedriveResumeConfig(+ paginator_state={+ "parent_index": 1,+ "parent_page": {"cursor": "p2"},+ "done_through": 5,+ "group": [6],+ "child": {"cursor": "c9"},+ }+ )+ )++ rows = _rows(_source("deal_changelog", manager))++ assert [row["deal_id"] for row in rows] == [6, 7]+ assert urls == [+ f"{BASE}/api/v1/deals/6/changelog",+ f"{BASE}/api/v2/deals/archived",+ f"{BASE}/api/v1/deals/7/changelog",+ ]+ assert params == [+ {"limit": PAGE_SIZE, "cursor": "c9"},+ {**DEAL_LIST_PARAMS, "cursor": "p2"},+ {"limit": PAGE_SIZE},+ ]++ @mock.patch(CLIENT_SESSION_PATCH)+ def test_resume_finishes_the_saved_batch_when_its_deals_changed(self, MockSession: mock.MagicMock) -> None:+ session = MockSession.return_value+ urls: list[str] = []+ params = _wire(+ session,+ [+ _response([{"id": 31, "deal_id": 2}], {"next_cursor": None}),+ # Deal 2 was deleted and deal 4 added since the crash, so a rebuilt batch would differ.+ _response([{"id": 1}, {"id": 3}, {"id": 4}], {"next_cursor": None}),+ _response([{"id": 41, "deal_id": 4}], {"next_cursor": None}),+ _response([], {"next_cursor": None}),+ ],+ urls=urls,+ )+ manager = _make_manager(+ PipedriveResumeConfig(+ paginator_state={+ "parent_index": 0,+ "parent_page": None,+ "done_through": None,+ "group": [1, 2, 3],+ "child": {"cursor": "c2"},+ }+ )+ )++ rows = _rows(_source("deal_products", manager))++ assert [row["id"] for row in rows] == [31, 41]+ assert urls[:3] == [+ f"{BASE}/api/v2/deals/products",+ f"{BASE}/api/v2/deals",+ f"{BASE}/api/v2/deals/products",+ ]+ assert params[0] == {"deal_ids": "1,2,3", "limit": PAGE_SIZE, "cursor": "c2"}+ assert params[2] == {"deal_ids": "4", "limit": PAGE_SIZE}++class TestPipedriveSourcePartitioning:@pytest.mark.parametrize(- "endpoint, expected_partition_keys, expected_mode",+ "endpoint, expected_primary_keys, expected_partition_keys, expected_mode",[- ("deals", ["add_time"], "datetime"),- ("activities", ["add_time"], "datetime"),- ("users", None, None),- ("deal_fields", None, None),+ ("deals", ["id"], ["add_time"], "datetime"),+ ("activities", ["id"], ["add_time"], "datetime"),+ ("users", ["id"], None, None),+ ("deal_fields", ["id"], None, None),+ ("lead_labels", ["id"], None, None),+ ("deal_products", ["deal_id", "id"], ["add_time"], "datetime"),+ ("deal_changelog", ["deal_id", "field_key", "time"], ["time"], "datetime"),],)@mock.patch(CLIENT_SESSION_PATCH)@@ -330,12 +501,13 @@ class TestPipedriveSourcePartitioning:self,MockSession: mock.MagicMock,endpoint: str,+ expected_primary_keys: list[str],expected_partition_keys: list[str] | None,expected_mode: str | None,) -> None:response = _source(endpoint, _make_manager())assert response.name == endpoint- assert response.primary_keys == ["id"]+ assert response.primary_keys == expected_primary_keysassert response.partition_keys == expected_partition_keysassert response.partition_mode == expected_modeassert response.partition_format == ("week" if expected_mode else None)