:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse): add pipedrive archived deals, mailbox and projects tables (#126282)
Tom Owers67fa5851bb66
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex c7928131b61e..7221d647014c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -9448,22 +9448,22 @@ Note: The synced alerts table maps to GET /actions. Every high-value gap is a pe## Pipedrive — gaps-Today (17): `activities`, `activity_types`, `deal_changelog`, `deal_fields`, `deal_products`, `deals`, `lead_labels`, `leads`, `notes`, `organization_fields`, `organizations`, `person_fields`, `persons`, `pipelines`, `products`, `stages`, `users`+Today (25): `activities`, `activity_types`, `archived_deals`, `archived_projects`, `deal_changelog`, `deal_fields`, `deal_products`, `deals`, `lead_labels`, `leads`, `mail_messages`, `mail_threads`, `notes`, `organization_fields`, `organizations`, `person_fields`, `persons`, `pipelines`, `products`, `project_boards`, `project_phases`, `project_tasks`, `projects`, `stages`, `users`Diffed against: <https://developers.pipedrive.com/docs/api/v2/openapi.yaml>- [x] `GET /deals/{id}/products (v2) and /deals/products` — deal line items - product, quantity, price, discount per deal; without them revenue can only be read as a single deal-level number (high). Added as `deal_products`, read through the batched `/deals/products` for every active and archived deal.-- [ ] `GET /deals/archived (v2)` — /deals excludes archived deals, so the synced deals table silently loses closed/archived pipeline history (high)+- [x] `GET /deals/archived (v2)` — /deals excludes archived deals, so the synced deals table silently loses closed/archived pipeline history (high). Added as `archived_deals`.- [x] `GET /activityTypes (v1)` — lookup resolving the type key on every synced activity row (high). Added as `activity_types`.- [x] `GET /deals/{id}/flow and /deals/{id}/changelog (v1)` — per-field change history including stage moves - required for stage-transition, velocity and win-rate-over-time analysis (high). Added as `deal_changelog`, read for every active and archived deal. `/deals/{id}/flow` skipped: it is a mixed timeline of activities, notes, files and field changes that the changelog and existing tables already cover.- [x] `GET /leadLabels (v1)` — lookup resolving the label IDs carried on the leads we already sync (high). Added as `lead_labels`.- [ ] `GET /leadSources (v1)` — lookup resolving lead source values for attribution (medium)- [ ] `GET /currencies (v1)` — currency lookup for multi-currency deal and product values (medium)-- [ ] `GET /pipelines/{id}/conversion_statistics and /movement_statistics (v1)` — Pipedrive's own funnel conversion and stage-movement metrics per pipeline (medium)+- [x] `GET /pipelines/{id}/conversion_statistics and /movement_statistics (v1)` — Pipedrive's own funnel conversion and stage-movement metrics per pipeline (medium). Skipped: each call returns one aggregate for a required date range and a single user, not rows, and `deals` with `deal_changelog` hold the data it summarizes.- [ ] `GET /callLogs (v1)` — call activity with duration and outcome, not covered by the activities table (medium)- [ ] `GET /goals and /goals/{id}/results (v1)` — sales targets and attainment, the natural denominator for deal reporting (medium)-- [ ] `GET /mailbox/mailThreads and /mailbox/mailThreads/{id}/mailMessages (v1)` — email engagement tied to deals and persons (medium)-- [ ] `GET /projects, /projects/{id}/tasks, /projects/boards, /projects/phases (v1 and v2)` — the Projects product is entirely absent, including its board/phase lookups (medium)+- [x] `GET /mailbox/mailThreads and /mailbox/mailThreads/{id}/mailMessages (v1)` — email engagement tied to deals and persons (medium). Added as `mail_threads` (every folder, one row per thread) and `mail_messages` (read for every thread).+- [x] `GET /projects, /projects/{id}/tasks, /projects/boards, /projects/phases (v1 and v2)` — the Projects product is entirely absent, including its board/phase lookups (medium). Added as `projects` and `archived_projects` (v2), and `project_tasks` (`/api/v1/tasks`, every task in one list), `project_boards` and `project_phases` on v1, because their v2 endpoints are still in beta.Note: Diffed against both specs - v2 (https://developers.pipedrive.com/docs/api/v2/openapi.yaml) and v1 (https://developers.pipedrive.com/docs/api/v1/openapi.yaml) - because collection reads are split across them: v2 owns deals/persons/organizations/products/stages/pipelines/activities, v1 still owns activityTypes, currencies, leadLabels, leadSources, goals, callLogs, mailbox and the deal changelog/flow. Deliberately excluded as config: filters, roles, permissionSets, legacyTeams, userSettings, webhooks, files.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/api_inventory.md b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/api_inventory.mdindex d7d7b53108dc..32d61057b3f6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/api_inventory.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/api_inventory.md@@ -20,25 +20,33 @@ Reference: <https://developers.pipedrive.com/docs/api/v1>## Endpoints synced-| Schema | Path | Pagination | Primary key | Partition key |-| ------------------- | ------------------------------ | ---------- | ------------------------ | ------------- |-| deals | `/api/v2/deals` | cursor | id | add_time |-| persons | `/api/v2/persons` | cursor | id | add_time |-| organizations | `/api/v2/organizations` | cursor | id | add_time |-| products | `/api/v2/products` | cursor | id | add_time |-| pipelines | `/api/v2/pipelines` | cursor | id | add_time |-| stages | `/api/v2/stages` | cursor | id | add_time |-| activities | `/api/v2/activities` | cursor | id | add_time |-| notes | `/api/v1/notes` | offset | id | add_time |-| leads | `/api/v1/leads` | offset | id | add_time |-| users | `/api/v1/users` | offset | id | — |-| deal_fields | `/api/v1/dealFields` | offset | id | — |-| person_fields | `/api/v1/personFields` | offset | id | — |-| organization_fields | `/api/v1/organizationFields` | offset | id | — |-| lead_labels | `/api/v1/leadLabels` | none | id | — |-| activity_types | `/api/v1/activityTypes` | none | id | — |-| deal_products | `/api/v2/deals/products` | cursor | deal_id, id | add_time |-| deal_changelog | `/api/v1/deals/{id}/changelog` | cursor | deal_id, field_key, time | time |+| Schema | Path | Pagination | Primary key | Partition key |+| ------------------- | ----------------------------------------------- | ---------- | ------------------------ | ------------- |+| deals | `/api/v2/deals` | cursor | id | add_time |+| archived_deals | `/api/v2/deals/archived` | cursor | id | add_time |+| persons | `/api/v2/persons` | cursor | id | add_time |+| organizations | `/api/v2/organizations` | cursor | id | add_time |+| products | `/api/v2/products` | cursor | id | add_time |+| pipelines | `/api/v2/pipelines` | cursor | id | add_time |+| stages | `/api/v2/stages` | cursor | id | add_time |+| activities | `/api/v2/activities` | cursor | id | add_time |+| notes | `/api/v1/notes` | offset | id | add_time |+| leads | `/api/v1/leads` | offset | id | add_time |+| users | `/api/v1/users` | offset | id | — |+| deal_fields | `/api/v1/dealFields` | offset | id | — |+| person_fields | `/api/v1/personFields` | offset | id | — |+| organization_fields | `/api/v1/organizationFields` | offset | id | — |+| lead_labels | `/api/v1/leadLabels` | none | id | — |+| activity_types | `/api/v1/activityTypes` | none | id | — |+| deal_products | `/api/v2/deals/products` | cursor | deal_id, id | add_time |+| deal_changelog | `/api/v1/deals/{id}/changelog` | cursor | deal_id, field_key, time | time |+| mail_threads | `/api/v1/mailbox/mailThreads` | offset | id | add_time |+| mail_messages | `/api/v1/mailbox/mailThreads/{id}/mailMessages` | none | id | add_time |+| projects | `/api/v2/projects` | cursor | id | add_time |+| archived_projects | `/api/v2/projects/archived` | cursor | id | add_time |+| project_tasks | `/api/v1/tasks` | cursor | id | add_time |+| project_boards | `/api/v1/projects/boards` | none | id | — |+| project_phases | `/api/v1/projects/phases` | none | id | — |`lead_labels` and `activity_types` are documented as unpaginated and always return every row, so they are read in one request.@@ -47,6 +55,21 @@ Reference: <https://developers.pipedrive.com/docs/api/v1>`deal_changelog` sends one request chain per deal, because the changelog has no batched form.Changelog rows carry neither an id nor the deal id, so the source adds the `deal_id` it requested.The resume state holds the deal list and page, the last finished deal id, and the cursor of the deal group in progress, so it does not grow with the account.+`archived_deals` and `archived_projects` read the archived lists, which `/api/v2/deals` and `/api/v2/projects` leave out.++`mail_threads` reads `/mailbox/mailThreads` once per folder (`inbox`, `sent`, `archive`, `drafts`).+A thread is listed in every folder that holds one of its messages, so a thread already read from an earlier folder is skipped.+`mail_messages` reads `/mailbox/mailThreads/{id}/mailMessages` for every thread in the same walk, without message bodies.+Threads come newest message first, so the resume state is the folder, the page, and the ids of the threads of that page that are done.+A resume lists the earlier folders and pages again to rebuild the threads already read, so the state does not grow with the mailbox.+The mailbox only holds mail synced by the user who owns the API token.++`project_tasks`, `project_boards` and `project_phases` stay on v1, because their v2 endpoints (`/api/v2/tasks`, `/api/v2/boards`, `/api/v2/phases`) are still in beta.+`project_phases` reads `/projects/phases` once per board, because the endpoint requires a `board_id`.+Its resume state is the ids of the boards done, so a board deleted between attempts does not shift the rest.++`/pipelines/{id}/conversion_statistics` and `/pipelines/{id}/movement_statistics` are not synced: they return one aggregate for a required date range and a single user, not a collection of rows, and `deals` with `deal_changelog` hold the data they summarize.+`/deals/{id}/flow` is not synced: it mixes activities, notes, files and field changes in one timeline, and the changelog and existing tables already cover those.## Source versionsdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/canonical_descriptions.pyindex 0d9dafde2c5f..98cd1781249f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/canonical_descriptions.py@@ -293,4 +293,137 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"is_bulk_update_flag": "Whether the change was part of a bulk update.",},},+ "mail_threads": {+ "description": "An email conversation synced to Pipedrive's mailbox, which can be linked to a deal or lead. One row per thread across the inbox, sent, archive, and drafts folders.",+ "docs_url": "https://developers.pipedrive.com/docs/api/v1/Mailbox#getMailThreads",+ "columns": {+ "id": "Unique identifier for the mail thread.",+ "account_id": "ID of the connected mail account.",+ "user_id": "ID of the user the thread is assigned to.",+ "subject": "Subject of the thread.",+ "snippet": "Short excerpt of the latest message.",+ "parties": "Senders and recipients taking part in the thread.",+ "folders": "Mail folders that hold messages from the thread.",+ "message_count": "Number of messages in the thread.",+ "read_flag": "Whether the thread has been read.",+ "has_sent_flag": "Whether the thread contains a sent message.",+ "has_draft_flag": "Whether the thread contains a draft.",+ "archived_flag": "Whether the thread is archived.",+ "deleted_flag": "Whether the thread is deleted.",+ "mail_tracking_status": "Open tracking status of the thread.",+ "first_message_timestamp": "Time of the first message in the thread.",+ "last_message_timestamp": "Time of the latest message in the thread.",+ "last_message_sent_timestamp": "Time the latest message was sent from the thread.",+ "last_message_received_timestamp": "Time the latest message was received in the thread.",+ "deal_id": "ID of the deal the thread is linked to.",+ "deal_status": "Status of the linked deal.",+ "lead_id": "ID of the lead the thread is linked to.",+ "add_time": "Time at which the thread was added to Pipedrive.",+ "update_time": "Time at which the thread was last updated in Pipedrive.",+ },+ },+ "mail_messages": {+ "description": "A single email message in a Pipedrive mail thread. Message bodies are not included.",+ "docs_url": "https://developers.pipedrive.com/docs/api/v1/Mailbox#getMailThreadMessages",+ "columns": {+ "id": "Unique identifier for the mail message.",+ "mail_thread_id": "ID of the mail thread the message belongs to.",+ "account_id": "ID of the connected mail account.",+ "user_id": "ID of the user the message is assigned to.",+ "subject": "Subject of the message.",+ "snippet": "Short excerpt of the message.",+ "from": "Senders of the message.",+ "to": "Recipients of the message.",+ "cc": "Recipients copied on the message.",+ "bcc": "Recipients blind-copied on the message.",+ "sent_flag": "Whether the message was sent.",+ "sent_from_pipedrive_flag": "Whether the message was sent from Pipedrive.",+ "read_flag": "Whether the message has been read.",+ "mail_tracking_status": "Open tracking status of the message.",+ "message_time": "Time at which the message was received or created.",+ "add_time": "Time at which the message was added to Pipedrive.",+ "update_time": "Time at which the message was last updated in Pipedrive.",+ },+ },+ "projects": {+ "description": "A project in Pipedrive Projects, usually delivery work that follows a won deal. Archived projects are in the archived_projects table.",+ "docs_url": "https://developers.pipedrive.com/docs/api/v1/Projects#getProjects",+ "columns": {+ "id": "Unique identifier for the project.",+ "title": "Title of the project.",+ "description": "Description of the project.",+ "status": "Status of the project (open, completed, canceled, or deleted).",+ "board_id": "ID of the project board the project is on.",+ "phase_id": "ID of the board phase the project is in.",+ "owner_id": "ID of the user who owns the project.",+ "start_date": "Start date of the project.",+ "end_date": "End date of the project.",+ "deal_ids": "IDs of the deals linked to the project.",+ "person_ids": "IDs of the persons linked to the project.",+ "org_ids": "IDs of the organizations linked to the project.",+ "label_ids": "IDs of the labels on the project.",+ "health_status": "Health status of the project.",+ "add_time": "Time at which the project was created.",+ "update_time": "Time at which the project was last updated.",+ "status_change_time": "Time of the latest status change of the project.",+ "archive_time": "Time at which the project was archived, or null if it is not archived.",+ "custom_fields": "Values of the custom fields on the project, keyed by field key.",+ },+ },+ "project_tasks": {+ "description": "A task inside a Pipedrive project.",+ "docs_url": "https://developers.pipedrive.com/docs/api/v1/Tasks#getTasks",+ "columns": {+ "id": "Unique identifier for the task.",+ "title": "Title of the task.",+ "description": "Description of the task.",+ "project_id": "ID of the project the task belongs to.",+ "parent_task_id": "ID of the parent task, or null for a top-level task.",+ "creator_id": "ID of the user who created the task.",+ "assignee_ids": "IDs of the users assigned to the task.",+ "is_done": "Whether the task is done.",+ "is_milestone": "Whether the task is a milestone.",+ "priority": "Priority of the task.",+ "start_date": "Start date of the task.",+ "due_date": "Due date of the task.",+ "add_time": "Time at which the task was created.",+ "update_time": "Time at which the task was last updated.",+ "marked_as_done_time": "Time at which the task was marked as done.",+ },+ },+ "project_boards": {+ "description": "A board in Pipedrive Projects. Resolves the board_id on projects and project phases.",+ "docs_url": "https://developers.pipedrive.com/docs/api/v1/ProjectBoards#getProjectsBoards",+ "columns": {+ "id": "Unique identifier for the board.",+ "name": "Name of the board.",+ "order_nr": "Display order of the board.",+ "add_time": "Time at which the board was created.",+ "update_time": "Time at which the board was last updated.",+ },+ },+ "project_phases": {+ "description": "A phase (column) on a Pipedrive project board. Resolves the phase_id on projects.",+ "docs_url": "https://developers.pipedrive.com/docs/api/v1/ProjectPhases#getProjectsPhases",+ "columns": {+ "id": "Unique identifier for the phase.",+ "name": "Name of the phase.",+ "board_id": "ID of the board the phase belongs to.",+ "order_nr": "Position of the phase on its board.",+ "add_time": "Time at which the phase was created.",+ "update_time": "Time at which the phase was last updated.",+ },+ },+}++# The archived lists return the same objects as the active lists.+CANONICAL_DESCRIPTIONS["archived_deals"] = {+ **CANONICAL_DESCRIPTIONS["deals"],+ "description": "An archived sales deal in Pipedrive. The deals table leaves these out.",+ "docs_url": "https://developers.pipedrive.com/docs/api/v1/Deals#getArchivedDeals",+}+CANONICAL_DESCRIPTIONS["archived_projects"] = {+ **CANONICAL_DESCRIPTIONS["projects"],+ "description": "An archived project in Pipedrive Projects. The projects table leaves these out.",+ "docs_url": "https://developers.pipedrive.com/docs/api/v1/Projects#getArchivedProjects",}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/pipedrive.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/pipedrive.pyindex 315288e54277..331dc740da1c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/pipedrive.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/pipedrive.py@@ -44,7 +44,10 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.webfrom products.warehouse_sources.backend.temporal.data_imports.sources.pipedrive.settings import (DEAL_IDS_PER_REQUEST,DEAL_PARENT_PATHS,+ MAIL_FOLDERS,+ MAIL_THREADS_PATH,PAGE_SIZE,+ PROJECT_BOARDS_PATH,PipedriveEndpointConfig,endpoints_for_version,)@@ -121,13 +124,7 @@ def _cursor_paginator() -> JSONResponseCursorPaginator:return JSONResponseCursorPaginator(cursor_path="additional_data.next_cursor", cursor_param="cursor")-def _build_paginator(config: PipedriveEndpointConfig) -> BasePaginator:- if config.pagination == "cursor":- # v2 endpoints: opaque cursor in the response body; ``limit`` rides in the endpoint params.- return _cursor_paginator()- if config.pagination == "none":- # Documented as unpaginated: a full page must not send a second request at `start=500`.- return SinglePagePaginator()+def _offset_paginator(maximum_offset: Optional[int] = None) -> OffsetPaginator:# v1 endpoints: start/limit offset. No top-level total; termination is a short/empty page.return OffsetPaginator(limit=PAGE_SIZE,@@ -135,10 +132,21 @@ def _build_paginator(config: PipedriveEndpointConfig) -> BasePaginator:offset_param="start",limit_param="limit",total_path=None,+ maximum_offset=maximum_offset,stop_after_empty_page=True,)+def _build_paginator(config: PipedriveEndpointConfig) -> BasePaginator:+ if config.pagination == "cursor":+ # v2 endpoints: opaque cursor in the response body; ``limit`` rides in the endpoint params.+ return _cursor_paginator()+ if config.pagination == "none":+ # Documented as unpaginated: a full page must not send a second request at `start=500`.+ return SinglePagePaginator()+ return _offset_paginator()++def validate_credentials(company_domain: str, api_token: str) -> Optional[int]:"""Return the status code of a cheap authenticated probe, or ``None`` on transport error.@@ -473,6 +481,145 @@ def _deal_fan_out_rows(progress["parent_page"] = next_parent_page.get("state")+def _list_thread_ids(client: RESTClient, folder: str, stop_offset: Optional[int] = None) -> set[Any]:+ """List the thread ids of a folder, or of its pages before `stop_offset` when it is set."""+ if stop_offset is not None and stop_offset <= 0:+ return set()+ ids: set[Any] = set()+ for threads in client.paginate(+ path=MAIL_THREADS_PATH,+ params={"folder": folder},+ paginator=_offset_paginator(maximum_offset=stop_offset),+ data_selector="data",+ ):+ ids.update(thread.get("id") for thread in threads)+ return ids+++def _mail_rows(+ client: RESTClient,+ config: PipedriveEndpointConfig,+ resumable_source_manager: ResumableSourceManager[PipedriveResumeConfig],+ initial_state: Optional[dict[str, Any]],+) -> Iterator[list[dict[str, Any]]]:+ """Read the threads of every mail folder, or for `mail_messages` the messages of each thread.++ A thread is listed in every folder that holds one of its messages, so a thread already read+ from an earlier folder is skipped. Threads come newest message first rather than by id, so the+ resume state is the folder, the page being read, and the ids of the threads of that page that+ are done. A resume lists the folders and pages before it again to rebuild the skipped threads,+ which keeps the state from growing with the mailbox.+ """+ state = initial_state or {}+ start_folder = state.get("folder_index", 0)+ seen: set[Any] = set(state.get("done_ids", []))+ if initial_state:+ for folder in MAIL_FOLDERS[:start_folder]:+ seen |= _list_thread_ids(client, folder)+ resume_page = state.get("page")+ if start_folder < len(MAIL_FOLDERS) and resume_page:+ seen |= _list_thread_ids(client, MAIL_FOLDERS[start_folder], stop_offset=resume_page.get("offset"))+ # A thread deleted between the folder listing and this fetch has nothing left to read.+ hooks = create_response_hooks([{"status_code": 404, "action": "ignore"}], resource_name=config.name)++ def stage(progress: dict[str, Any]) -> None:+ resumable_source_manager.save_state(PipedriveResumeConfig(paginator_state=progress))++ def stage_page_done(folder_index: int, next_page_state: Optional[dict[str, Any]]) -> None:+ if next_page_state:+ stage({"folder_index": folder_index, "page": next_page_state})+ else:+ stage({"folder_index": folder_index + 1})++ for folder_index, folder in enumerate(MAIL_FOLDERS):+ if folder_index < start_folder:+ continue+ resuming = folder_index == start_folder+ page: Optional[dict[str, Any]] = state.get("page") if resuming else None+ done_in_page: list[Any] = list(state.get("done_ids", [])) if resuming else []+ next_page: dict[str, Any] = {}++ def on_page(+ paginator_state: Optional[dict[str, Any]], has_next: bool, _next: dict[str, Any] = next_page+ ) -> None:+ _next["state"] = paginator_state if has_next else None++ for threads in client.paginate(+ path=MAIL_THREADS_PATH,+ params={"folder": folder},+ paginator=_offset_paginator(),+ data_selector="data",+ initial_paginator_state=page,+ page_state_hook=on_page,+ ):+ if config.custom_read == "mail_threads":+ new_threads = [thread for thread in threads if thread.get("id") not in seen]+ seen.update(thread.get("id") for thread in new_threads)+ stage_page_done(folder_index, next_page.get("state"))+ if new_threads:+ yield new_threads+ else:+ resumable_source_manager.safe_point()+ else:+ for thread in threads:+ thread_id = thread.get("id")+ if thread_id in seen:+ continue+ seen.add(thread_id)+ messages: list[dict[str, Any]] = []+ for rows in client.paginate(+ path=config.path.format(id=thread_id),+ paginator=SinglePagePaginator(),+ data_selector="data",+ hooks=hooks,+ ):+ messages.extend(rows)+ if messages:+ done_in_page.append(thread_id)+ stage({"folder_index": folder_index, "page": page, "done_ids": list(done_in_page)})+ yield messages+ else:+ resumable_source_manager.safe_point()+ # Every thread of the page is done, so the next attempt can start at the next page.+ stage_page_done(folder_index, next_page.get("state"))+ resumable_source_manager.safe_point()++ page = next_page.get("state")+ done_in_page = []+++def _project_phase_rows(+ client: RESTClient,+ config: PipedriveEndpointConfig,+ resumable_source_manager: ResumableSourceManager[PipedriveResumeConfig],+ initial_state: Optional[dict[str, Any]],+) -> Iterator[list[dict[str, Any]]]:+ """Read the phases of every project board. The resume state is the ids of the boards done."""+ done_boards: list[Any] = list((initial_state or {}).get("done_board_ids", []))+ # A board deleted between the board listing and this fetch has no phases left to read.+ hooks = create_response_hooks([{"status_code": 404, "action": "ignore"}], resource_name=config.name)++ for boards in client.paginate(path=PROJECT_BOARDS_PATH, paginator=SinglePagePaginator(), data_selector="data"):+ for board in boards:+ if board["id"] in done_boards:+ continue+ for phases in client.paginate(+ path=config.path,+ params={"board_id": board["id"]},+ paginator=SinglePagePaginator(),+ data_selector="data",+ hooks=hooks,+ ):+ if phases:+ done_boards.append(board["id"])+ resumable_source_manager.save_state(+ PipedriveResumeConfig(paginator_state={"done_board_ids": list(done_boards)})+ )+ yield phases+ else:+ resumable_source_manager.safe_point()++def pipedrive_source(company_domain: str,api_token: str,@@ -515,14 +662,19 @@ def pipedrive_source(resource: Anycolumn_hints = None- if config.deal_fan_out is not None:+ if config.deal_fan_out is not None or config.custom_read is not None:client = RESTClient(base_url=base_url(company_domain),headers={"Accept": "application/json"},auth=APIKeyAuth(api_key=api_token, name="x-api-token", location="header"),allowed_hosts=[],)- resource = _deal_fan_out_rows(client, config, resumable_source_manager, initial_paginator_state)+ if config.custom_read == "project_phases":+ resource = _project_phase_rows(client, config, resumable_source_manager, initial_paginator_state)+ elif config.custom_read is not None:+ resource = _mail_rows(client, config, resumable_source_manager, initial_paginator_state)+ else:+ resource = _deal_fan_out_rows(client, config, resumable_source_manager, initial_paginator_state)else:params: dict[str, Any] = {}if config.pagination == "cursor":diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/settings.pyindex f9999bbefff5..2b64613f00df 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/settings.py@@ -5,6 +5,9 @@ PaginationStyle = Literal["cursor", "offset", "none"]# "batched" reads one request per group of deals through a `deal_ids` query param;# "per_deal" binds a single deal id into the `{id}` path placeholder.DealFanOut = Literal["batched", "per_deal"]+# Endpoints that need more than one collection walk: mail threads are listed per folder, mail+# messages per thread, and project phases per board.+CustomRead = Literal["mail_threads", "mail_messages", "project_phases"]# Pipedrive caps list pages at 500 items (default 100).PAGE_SIZE = 500@@ -13,6 +16,13 @@ DEAL_IDS_PER_REQUEST = 100# `/api/v2/deals` leaves out archived deals, so deal fan-outs read both lists to keep the line# items and history of a deal after it is archived.DEAL_PARENT_PATHS = ("/api/v2/deals", "/api/v2/deals/archived")+# `/mailbox/mailThreads` lists one folder per request, and a thread is listed in every folder that+# holds one of its messages.+MAIL_FOLDERS = ("inbox", "sent", "archive", "drafts")+MAIL_THREADS_PATH = "/api/v1/mailbox/mailThreads"+# Project boards, phases and tasks stay on v1 because their v2 endpoints are still in beta.+# `/projects/phases` requires a `board_id`, so phases are read once per board.+PROJECT_BOARDS_PATH = "/api/v1/projects/boards"@dataclass(frozen=True)@@ -29,14 +39,18 @@ class PipedriveEndpointConfig:partition_key: Optional[str] = "add_time"# Set for endpoints read for every deal in `DEAL_PARENT_PATHS`.deal_fan_out: Optional[DealFanOut] = None+ custom_read: Optional[CustomRead] = None# Endpoints whose path and pagination are identical across every supported source version.# Their v1 collection endpoints have no v2 replacement (notes, leads, users, *_fields, lookups,-# the deal changelog) or were already on v2 (deals, persons, organizations, products, pipelines,-# stages, deal products), so they don't change between our `v1` and `v2` labels.+# the deal changelog, the mailbox, project boards, phases and tasks) or were already on v2 (deals, persons, organizations, products,+# pipelines, stages, deal products, projects), so they don't change between our `v1` and `v2` labels._SHARED_ENDPOINTS: dict[str, PipedriveEndpointConfig] = {"deals": PipedriveEndpointConfig(name="deals", path="/api/v2/deals", pagination="cursor"),+ "archived_deals": PipedriveEndpointConfig(+ name="archived_deals", path="/api/v2/deals/archived", pagination="cursor"+ ),"persons": PipedriveEndpointConfig(name="persons", path="/api/v2/persons", pagination="cursor"),"organizations": PipedriveEndpointConfig(name="organizations", path="/api/v2/organizations", pagination="cursor"),"products": PipedriveEndpointConfig(name="products", path="/api/v2/products", pagination="cursor"),@@ -76,6 +90,30 @@ _SHARED_ENDPOINTS: dict[str, PipedriveEndpointConfig] = {partition_key="time",deal_fan_out="per_deal",),+ "mail_threads": PipedriveEndpointConfig(+ name="mail_threads", path=MAIL_THREADS_PATH, pagination="offset", custom_read="mail_threads"+ ),+ "mail_messages": PipedriveEndpointConfig(+ name="mail_messages",+ path="/api/v1/mailbox/mailThreads/{id}/mailMessages",+ pagination="none",+ custom_read="mail_messages",+ ),+ "projects": PipedriveEndpointConfig(name="projects", path="/api/v2/projects", pagination="cursor"),+ "archived_projects": PipedriveEndpointConfig(+ name="archived_projects", path="/api/v2/projects/archived", pagination="cursor"+ ),+ "project_tasks": PipedriveEndpointConfig(name="project_tasks", path="/api/v1/tasks", pagination="cursor"),+ "project_boards": PipedriveEndpointConfig(+ name="project_boards", path=PROJECT_BOARDS_PATH, pagination="none", partition_key=None+ ),+ "project_phases": PipedriveEndpointConfig(+ name="project_phases",+ path="/api/v1/projects/phases",+ pagination="none",+ partition_key=None,+ custom_read="project_phases",+ ),}# `activities` is the only endpoint that differs by version: Pipedrive deprecated the v1diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/tests/test_pipedrive.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/tests/test_pipedrive.pyindex e21eeaedb090..30d841991e99 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/tests/test_pipedrive.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipedrive/tests/test_pipedrive.py@@ -322,7 +322,7 @@ class TestPagination:class TestLookupEndpoints:- @pytest.mark.parametrize("endpoint", ["lead_labels", "activity_types"])+ @pytest.mark.parametrize("endpoint", ["lead_labels", "activity_types", "project_boards"])@mock.patch(CLIENT_SESSION_PATCH)def test_reads_every_row_in_one_request(self, MockSession: mock.MagicMock, endpoint: str) -> None:session = MockSession.return_value@@ -483,6 +483,169 @@ class TestDealFanOut:assert params[2] == {"deal_ids": "4", "limit": PAGE_SIZE}+MAIL_THREADS_URL = f"{BASE}/api/v1/mailbox/mailThreads"+++def _folder_params(folder: str, start: int = 0) -> dict[str, Any]:+ return {"folder": folder, "start": start, "limit": PAGE_SIZE}+++class TestMailbox:+ @mock.patch(CLIENT_SESSION_PATCH)+ def test_mail_threads_reads_every_folder_once_per_thread(self, MockSession: mock.MagicMock) -> None:+ session = MockSession.return_value+ params = _wire(+ session,+ [+ _response([{"id": 1}, {"id": 2}]),+ # Thread 2 holds an inbox and a sent message, so it lists in both folders.+ _response([{"id": 2}, {"id": 3}]),+ _response([]),+ _response(None),+ ],+ )+ manager = _make_manager()++ rows = _rows(_source("mail_threads", manager))++ assert [row["id"] for row in rows] == [1, 2, 3]+ assert params == [_folder_params(folder) for folder in ("inbox", "sent", "archive", "drafts")]+ assert manager.save_state.call_args.args[0] == PipedriveResumeConfig(paginator_state={"folder_index": 4})++ @mock.patch(CLIENT_SESSION_PATCH)+ def test_mail_messages_reads_each_thread_and_skips_a_deleted_one(self, MockSession: mock.MagicMock) -> None:+ session = MockSession.return_value+ urls: list[str] = []+ _wire(+ session,+ [+ _response([{"id": 1}, {"id": 2}]),+ _response([{"id": 10, "mail_thread_id": 1}, {"id": 11, "mail_thread_id": 1}]),+ _response(None, drop_data=True, status=404),+ _response([{"id": 1}]),+ _response([]),+ _response([]),+ ],+ urls=urls,+ )+ manager = _make_manager()++ rows = _rows(_source("mail_messages", manager))++ assert [row["id"] for row in rows] == [10, 11]+ assert urls == [+ MAIL_THREADS_URL,+ f"{MAIL_THREADS_URL}/1/mailMessages",+ f"{MAIL_THREADS_URL}/2/mailMessages",+ MAIL_THREADS_URL,+ MAIL_THREADS_URL,+ MAIL_THREADS_URL,+ ]+ assert [call.args[0].paginator_state for call in manager.save_state.call_args_list] == [+ {"folder_index": 0, "page": None, "done_ids": [1]},+ # Inbox is done, and the inbox thread listed again in sent is skipped.+ {"folder_index": 1},+ {"folder_index": 2},+ {"folder_index": 3},+ {"folder_index": 4},+ ]++ @mock.patch(CLIENT_SESSION_PATCH)+ def test_mail_messages_resume_continues_inside_the_saved_page(self, MockSession: mock.MagicMock) -> None:+ session = MockSession.return_value+ urls: list[str] = []+ params = _wire(+ session,+ [+ # The resume lists the inbox and the sent pages before the saved one to rebuild+ # the threads already read.+ _response([{"id": 7}]),+ _response([{"id": 4}]),+ # A reply moved thread 6 ahead of thread 5, which was done before the crash, and+ # thread 7 was already read from the inbox.+ _response([{"id": 6}, {"id": 5}, {"id": 7}]),+ _response([{"id": 60, "mail_thread_id": 6}]),+ _response([]),+ _response([]),+ ],+ urls=urls,+ )+ manager = _make_manager(+ PipedriveResumeConfig(paginator_state={"folder_index": 1, "page": {"offset": 500}, "done_ids": [5]})+ )++ rows = _rows(_source("mail_messages", manager))++ assert [row["id"] for row in rows] == [60]+ assert urls == [MAIL_THREADS_URL] * 3 + [f"{MAIL_THREADS_URL}/6/mailMessages"] + [MAIL_THREADS_URL] * 2+ assert params[:3] == [_folder_params("inbox"), _folder_params("sent"), _folder_params("sent", start=500)]+ assert params[4:] == [_folder_params("archive"), _folder_params("drafts")]+ assert [call.args[0].paginator_state for call in manager.save_state.call_args_list] == [+ {"folder_index": 1, "page": {"offset": 500}, "done_ids": [5, 6]},+ {"folder_index": 2},+ {"folder_index": 3},+ {"folder_index": 4},+ ]++ @mock.patch(CLIENT_SESSION_PATCH)+ def test_mail_threads_resume_skips_threads_read_before_the_crash(self, MockSession: mock.MagicMock) -> None:+ session = MockSession.return_value+ _wire(+ session,+ [+ _response([{"id": 1}, {"id": 2}]),+ _response([{"id": 2}, {"id": 3}]),+ _response([]),+ _response(None),+ ],+ )+ manager = _make_manager(PipedriveResumeConfig(paginator_state={"folder_index": 1}))++ rows = _rows(_source("mail_threads", manager))++ assert [row["id"] for row in rows] == [3]+++class TestProjectPhases:+ @pytest.mark.parametrize(+ "resume_state, boards, phase_pages, expected_phase_boards, expected_ids, expected_done",+ [+ (None, [1, 2, 3], [[{"id": 11}], [], [{"id": 31}, {"id": 32}]], [1, 2, 3], [11, 31, 32], [1, 3]),+ # Board 1 was deleted after it was read, so board 2 is now first in the listing.+ ({"done_board_ids": [1]}, [2, 3], [[{"id": 21}], [{"id": 31}]], [2, 3], [21, 31], [1, 2, 3]),+ ({"done_board_ids": [1, 2]}, [1, 2, 3], [[{"id": 31}]], [3], [31], [1, 2, 3]),+ ],+ )+ @mock.patch(CLIENT_SESSION_PATCH)+ def test_reads_the_phases_of_each_board(+ self,+ MockSession: mock.MagicMock,+ resume_state: Optional[dict[str, Any]],+ boards: list[int],+ phase_pages: list[list[dict[str, Any]]],+ expected_phase_boards: list[int],+ expected_ids: list[int],+ expected_done: list[int],+ ) -> None:+ session = MockSession.return_value+ urls: list[str] = []+ params = _wire(+ session,+ [_response([{"id": board_id} for board_id in boards]), *(_response(page) for page in phase_pages)],+ urls=urls,+ )+ manager = _make_manager(PipedriveResumeConfig(paginator_state=resume_state) if resume_state else None)++ rows = _rows(_source("project_phases", manager))++ assert [row["id"] for row in rows] == expected_ids+ assert urls == [f"{BASE}/api/v1/projects/boards"] + [f"{BASE}/api/v1/projects/phases"] * len(phase_pages)+ assert params[1:] == [{"board_id": board_id} for board_id in expected_phase_boards]+ assert manager.save_state.call_args.args[0] == PipedriveResumeConfig(+ paginator_state={"done_board_ids": expected_done}+ )++class TestPipedriveSourcePartitioning:@pytest.mark.parametrize("endpoint, expected_primary_keys, expected_partition_keys, expected_mode",