:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse_sources): support intercom api version 2.17 (#119203)
Tom Owerse664588b5f44
diff --git a/.agents/skills/warehouse-source-new-version/SKILL.md b/.agents/skills/warehouse-source-new-version/SKILL.mdindex 854b0664b2a9..304730c89716 100644--- a/.agents/skills/warehouse-source-new-version/SKILL.md+++ b/.agents/skills/warehouse-source-new-version/SKILL.md@@ -116,6 +116,7 @@ The opposite gap also occurs: a reference rendered from one version-blind OpenAP- A version that changes the pagination _mechanism_ (e.g. offset→cursor, or one opaque cursor param for another, since a token one generation issued means nothing to the other) also changes the resumable checkpoint's persisted shape. Add a new defaulted field to the resume-config dataclass rather than repurposing the existing one — so state persisted under the old version still decodes — and make the seed and save-checkpoint logic dispatch on the resolved version. Reusing one field across incompatible mechanisms corrupts in-flight resumes of the other version. When a table stays on its old wire under the new pin, key the check on the wire that issued the cursor, not on the pin. When both mechanisms are positional with a fixed page size (offset↔page number), keep the existing field instead: let the new version's paginator derive its page from the stored offset, so one checkpoint means the same row under either pin.- Other products can read a synced table's columns by name, e.g. a signal emitter registered on `(source_type, table)` in `products/signals/backend/emission/registry.py`. Grep for the source type outside its directory before a new version renames or nests a column such a reader uses, and keep that column under its old name on the new version's rows.- When the vendor renames a collection between versions, keep the schema/table name set identical across versions and put the rename in a per-version path on the endpoint config — otherwise discovery diffs orphan the table on repin.+- A table gated to an explicit list of versions (an endpoint added by an earlier bump) drops off a new default unless the new label joins that list. Before flipping the default, extend every such list for each endpoint the new version still serves.- When the new version has no equivalent for an old endpoint (a dropped collection, not a rename), keep that table only on the versions that serve it and let the table set differ by version — never map it to a guessed path just to keep the sets equal, since docs are the sole source of truth and an unverified path makes a new source surface a table that 404s at sync time.- The registry catalog invariants (`sources/tests/test_source_catalog_invariants.py`) read only the default version's `get_schemas`, so a new default that drops a table makes that table's canonical description look stray. Keep the description, since pins on the old version still sync and render that table, and add the source to `DESCRIPTIONS_NOT_IN_SCHEMAS`.- A source with no dispatch may currently read its version from a constant on a shared model (e.g. the OAuth `Integration` model) that also drives version-independent flows like OAuth token minting. Repoint only the sync request path onto the resolved pin; leave that constant, since bumping it changes those other flows' version with a blast radius beyond this source.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 2d502c6d081d..a443bcfd8e67 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -6444,7 +6444,7 @@ Note: Parsed the official Swagger resource index and individual JSON definitions## Intercom — gaps-Today (25): `activity_logs`, `admins`, `articles`, `audiences`, `collections`, `companies`, `company_attributes`, `company_segments`, `contact_attributes`, `contacts`, `content_snippets`, `conversation_attributes`, `conversation_parts`, `conversations`, `help_centers`, `macros`, `news_items`, `newsfeeds`, `segments`, `subscription_types`, `tags`, `teams`, `ticket_states`, `ticket_types`, `tickets`+Today (30): `activity_logs`, `admins`, `articles`, `audiences`, `collections`, `companies`, `company_attributes`, `company_segments`, `contact_activities`, `contact_attributes`, `contacts`, `content_snippets`, `conversation_attributes`, `conversation_hitl_events`, `conversation_parts`, `conversations`, `deleted_contacts`, `dynamic_contents`, `folders`, `help_centers`, `macros`, `news_items`, `newsfeeds`, `segments`, `subscription_types`, `tags`, `teams`, `ticket_states`, `ticket_types`, `tickets`Diffed against: <https://github.com/intercom/Intercom-OpenAPI/blob/main/descriptions/2.16/api.intercom.io.yaml>@@ -6455,6 +6455,14 @@ The main spec-verified diff is in [COVERAGE_GAPS.md](COVERAGE_GAPS.md#intercom--- [x] `GET /audiences` — saved audience definitions that content and messages target (low). Added as `audiences`.- [x] `GET /content_snippets` — snippets that feed Fin and Copilot answers (medium). Added as `content_snippets`.+These endpoints were announced in the 2.17 changelog and are only in the 2.17 description, so the tables only show for sources pinned to 2.17.++- [x] `GET /contacts/activities` — workspace feed of contact lifecycle events: created, role updated, merged, deleted (medium). Added as `contact_activities`.+- [x] `GET /contacts/deleted` — archived, blocked, merged, and deleted contacts with the reason (medium). Added as `deleted_contacts`.+- [x] `GET /folders` — folders that organize articles and other content, decoding `articles.folder_id` (low). Added as `folders`.+- [x] `GET /dynamic_contents` — reusable Help Center content referenced from article bodies (low). Added as `dynamic_contents`.+- [x] `GET /conversations/{id}/hitl_events` — Fin human-in-the-loop consultation events per conversation (medium). Added as `conversation_hitl_events`, fanned out over `conversations` and incremental on the parent's `updated_at`.+## Intruder — adequateToday (8): `fixed_occurrences`, `issue_targets`, `issues`, `occurrences`, `scan_schedules`, `scans`, `tags`, `targets`diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/canonical_descriptions.pyindex db2af17cd350..8d0323669870 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/canonical_descriptions.py@@ -416,4 +416,78 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"audience_ids": "Identifiers of the audiences the snippet targets.",},},+ "contact_activities": {+ "description": "A contact lifecycle event in the workspace, such as a contact being created, changing role, merged, or deleted.",+ "docs_url": "https://developers.intercom.com/docs/references/rest-api/api.intercom.io/contacts/listcontactactivities",+ "columns": {+ "id": "Unique identifier for the activity event.",+ "type": "The object type, always 'contact_activity'.",+ "event_type": "The type of activity event, such as contact.created or contact.merged.",+ "contact_id": "Identifier of the contact the event relates to.",+ "external_id": "The external identifier of the contact, if one is set.",+ "role": "The role of the contact at the time of the event: user, lead, or visitor.",+ "merge_source_id": "For merge events, the identifier of the contact that was merged away.",+ "merge_target_id": "For merge events, the identifier of the contact that was merged into.",+ "failure_reason": "For failed events, the reason the activity did not succeed.",+ "created_at": "Time at which the event occurred, as a Unix timestamp.",+ },+ },+ "deleted_contacts": {+ "description": "A contact that is no longer active in the workspace, with the reason it is inactive.",+ "docs_url": "https://developers.intercom.com/docs/references/rest-api/api.intercom.io/contacts/listdeletedcontacts",+ "columns": {+ "id": "Unique identifier for the contact.",+ "type": "The object type, always 'contact'.",+ "external_id": "The identifier for the contact provided by the client. Null for permanently deleted contacts.",+ "email": "The contact's email address. Null for permanently deleted contacts.",+ "role": "The role of the contact.",+ "status": "Why the contact is inactive: archived, blocked, merged, pending_deletion, or permanently_deleted.",+ "merged_into_id": "Identifier of the contact this one was merged into. Present only when status is merged.",+ "updated_at": "Time at which the contact record was last updated, as a Unix timestamp. Not the deletion time.",+ },+ },+ "folders": {+ "description": "A folder that organizes articles and other content into a hierarchy.",+ "docs_url": "https://developers.intercom.com/docs/references/rest-api/api.intercom.io/folders/listfolders",+ "columns": {+ "id": "Unique identifier for the folder.",+ "name": "The name of the folder.",+ "description": "The description of the folder.",+ "emoji": "An emoji associated with the folder.",+ "parent_folder_id": "Identifier of the parent folder, or null for a root folder.",+ "content_count": "The number of content items in the folder.",+ "created_at": "Time at which the folder was created, as a Unix timestamp.",+ "updated_at": "Time at which the folder was last updated, as a Unix timestamp.",+ },+ },+ "dynamic_contents": {+ "description": "Reusable Help Center content with a variation per locale, and optionally per brand, that articles reference.",+ "docs_url": "https://developers.intercom.com/docs/references/rest-api/api.intercom.io/dynamic-content/listdynamiccontents",+ "columns": {+ "id": "Unique identifier for the dynamic content.",+ "type": "The object type, always 'dynamic_content'.",+ "name": "The name of the dynamic content.",+ "key": "The identifier articles use to reference the dynamic content.",+ "content_type": "Whether the content is a short inline value (inline) or one or more blocks of content (block).",+ "default_locale": "The locale whose variation is used when there is no variation for the requested locale.",+ "variations": "The variations of the dynamic content, one per locale and brand.",+ "created_at": "Time at which the dynamic content was created, as a Unix timestamp.",+ "updated_at": "Time at which the dynamic content was last updated, as a Unix timestamp.",+ },+ },+ "conversation_hitl_events": {+ "description": "An event from a Fin human-in-the-loop consultation, where Fin asks a teammate before it replies in a conversation.",+ "docs_url": "https://developers.intercom.com/docs/references/rest-api/api.intercom.io/conversations/listhitlevents",+ "columns": {+ "conversation_id": "Identifier of the conversation the event belongs to.",+ "hitl_cycle_id": "Identifier of the consultation. All events of one consultation share it.",+ "event_type": "What happened, such as fin_consultation_started, fin_draft_proposed, or fin_consultation_finished.",+ "round_index": "The feedback round the event belongs to, starting at 1. 0 for events outside a round, such as the start and end of the consultation.",+ "conversation_part_id": "The conversation part the event relates to, where one exists.",+ "rule_id": "The escalation rule that triggered the consultation, as recorded when it began. Null if no rule triggered it.",+ "rule_name": "The rule's name as recorded when the consultation began.",+ "created_at": "Time at which the event occurred, as a Unix timestamp. Approximate for fin_draft_proposed.",+ "data": "Fields specific to the event type, such as the draft, the teammate input, or the consultation outcome.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/intercom.py b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/intercom.pyindex ce4b9436be7d..a3e3ee1f9c10 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/intercom.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/intercom.py@@ -2,13 +2,14 @@ import timeimport dataclassesfrom collections.abc import AsyncIterable, Callable, Iterable, Iteratorfrom datetime import UTC, datetime-from typing import Any, Optional, cast+from typing import Any, ClassVar, Optional, castimport structlogfrom requests import Request, Response, Sessionfrom requests.exceptions import HTTPErrorfrom urllib3.util.retry import Retry+from products.warehouse_sources.backend.temporal.data_imports.sources.common.cursor import SourceCursorManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (DEFAULT_RETRY,make_tracked_session,@@ -62,6 +63,13 @@ class IntercomResumeConfig:page: int | None = None[email protected](frozen=True)+class IntercomConversationCursor:+ cursor_kind: ClassVar[str] = "intercom_conversation_updated_at"+ # `updated_at` of the newest conversation whose HITL events this run read.+ updated_at: int++def _load_resume_state(resumable_source_manager: ResumableSourceManager[IntercomResumeConfig] | None,) -> IntercomResumeConfig | None:@@ -344,7 +352,7 @@ def _build_paginator(cfg: IntercomEndpointConfig) -> BasePaginator:if cfg.paginator_kind == "pages":return IntercomPagesPaginator()if cfg.paginator_kind == "page_number":- return PageNumberPaginator(base_page=1, total_path="total_pages")+ return PageNumberPaginator(base_page=1, total_path=cfg.total_pages_path)return SinglePagePaginator()@@ -479,20 +487,27 @@ def _intercom_post(session: Session, path_or_url: str, body: dict[str, Any]) ->return _request_with_rate_limit_retry(do)+def _fetch_conversation_parts(session: Session, conversation: dict[str, Any]) -> list[dict[str, Any]]:+ full = _intercom_get(session, f"/conversations/{conversation['id']}")+ return (full.get("conversation_parts") or {}).get("conversation_parts") or []++def _conversation_parts_generator(session: Session,incremental_field: str,db_incremental_field_last_value: Optional[Any],resumable_source_manager: ResumableSourceManager[IntercomResumeConfig] | None = None,+ fetch_children: Callable[[Session, dict[str, Any]], list[dict[str, Any]]] = _fetch_conversation_parts,) -> Iterator[dict[str, Any]]:- """Yield each conversation part. Parents come from `POST /conversations/search`- with the same `updated_at >` server-side filter the conversations endpoint uses,- so only parents whose timestamp advanced are refetched. The part rows carry their- own `updated_at`, so the pipeline's cursor watermark advances per-part.- `conversation_id` is injected onto each row for joinability.+ """Yield each conversation part, or the rows `fetch_children` returns for each+ conversation. Parents come from `POST /conversations/search` with the same+ `updated_at >` server-side filter the conversations endpoint uses, so only parents+ whose timestamp advanced are refetched. Part rows carry their own `updated_at`, so+ the pipeline's cursor watermark advances per-part. `conversation_id` is injected onto+ each row for joinability.The resume cursor is the parent page plus the conversations in it that are done,- staged after each conversation's parts are yielded. A resume refetches that page+ staged after each conversation's children are yielded. A resume refetches that pageand skips the done conversations."""body = _build_search_body(INTERCOM_ENDPOINTS["conversations"], incremental_field, db_incremental_field_last_value)completed: list[str] = []@@ -510,8 +525,8 @@ def _conversation_parts_generator(resumable_source_manager.save_state(IntercomResumeConfig(cursor=page_cursor, query_value=query_value, completed_conversation_ids=list(done)))- # Every part the staged cursor covers is yielded and nothing is buffered here. Many- # conversations have no parts, so the walk can make many requests between rows.+ # Every row the staged cursor covers is yielded and nothing is buffered here. Many+ # conversations have no children, so the walk can make many requests between rows.resumable_source_manager.safe_point()while True:@@ -523,16 +538,15 @@ def _conversation_parts_generator(if conversation_id in done_ids:continuetry:- full = _intercom_get(session, f"/conversations/{conv['id']}")+ children = fetch_children(session, conv)except HTTPError as exc:if not _is_not_found(exc):raiselogger.warning("intercom_conversation_not_found", conversation_id=conv["id"])else:- parts = (full.get("conversation_parts") or {}).get("conversation_parts") or []- for part in parts:- part["conversation_id"] = conv["id"]- yield part+ for child in children:+ child["conversation_id"] = conv["id"]+ yield childcompleted.append(conversation_id)done_ids.add(conversation_id)stage(page_cursor, completed)@@ -714,12 +728,44 @@ def _company_segments_generator(resumable_source_manager.safe_point()+def _conversation_hitl_events_generator(+ session: Session,+ should_use_incremental_field: bool,+ cursor_manager: SourceCursorManager[IntercomConversationCursor] | None,+ resumable_source_manager: ResumableSourceManager[IntercomResumeConfig] | None = None,+) -> Iterator[dict[str, Any]]:+ """Yield the HITL events of each conversation, with `conversation_id` and `round_index` injected.++ The events carry only `created_at`, and many workspaces have none, so a watermark taken from+ the rows would never move. The source cursor holds the newest parent `updated_at` the walk+ reached instead, and the next run searches only conversations updated after it."""+ stored = cursor_manager.load() if should_use_incremental_field and cursor_manager is not None else None++ def fetch_events(session: Session, conversation: dict[str, Any]) -> list[dict[str, Any]]:+ events = _intercom_get(session, f"/conversations/{conversation['id']}/hitl_events").get("data") or []+ for event in events:+ # Rounds of one consultation share every other key column, sometimes `created_at` too.+ event["round_index"] = (event.get("data") or {}).get("round_index") or 0+ if cursor_manager is not None and conversation.get("updated_at") is not None:+ cursor_manager.stage(IntercomConversationCursor(updated_at=int(conversation["updated_at"])))+ return events++ return _conversation_parts_generator(+ session,+ "updated_at",+ stored.updated_at if stored is not None else None,+ resumable_source_manager,+ fetch_children=fetch_events,+ )++def _substream_items(session: Session,endpoint: str,incremental_field: str | None,db_incremental_field_last_value: Optional[Any],resumable_source_manager: ResumableSourceManager[IntercomResumeConfig] | None = None,+ cursor_manager: SourceCursorManager[IntercomConversationCursor] | None = None,) -> Iterator[dict[str, Any]]:if endpoint == "conversation_parts":if not incremental_field:@@ -730,6 +776,10 @@ def _substream_items(return _conversation_parts_generator(session, incremental_field, db_incremental_field_last_value, resumable_source_manager)+ if endpoint == "conversation_hitl_events":+ return _conversation_hitl_events_generator(+ session, incremental_field is not None, cursor_manager, resumable_source_manager+ )if endpoint == "company_segments":return _company_segments_generator(session, resumable_source_manager)raise ValueError(f"Unknown Intercom substream endpoint: {endpoint}")@@ -797,6 +847,7 @@ def intercom_source(should_use_incremental_field: bool = False,incremental_field: str | None = None,db_incremental_field_last_value: Optional[Any] = None,+ cursor_manager: SourceCursorManager[IntercomConversationCursor] | None = None,) -> SourceResponse:cfg = INTERCOM_ENDPOINTS[endpoint]if cfg.api_versions is not None and api_version not in cfg.api_versions:@@ -817,7 +868,12 @@ def intercom_source(had_resume_state = resumable_source_manager.can_resume()try:yield from _substream_items(- session, endpoint, incremental_field, db_incremental_field_last_value, resumable_source_manager+ session,+ endpoint,+ incremental_field,+ db_incremental_field_last_value,+ resumable_source_manager,+ cursor_manager,)except HTTPError as exc:if had_resume_state and _is_invalid_resume_cursor(exc):diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/settings.pyindex 94d568076af8..72cac1f5d429 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/settings.py@@ -26,6 +26,8 @@ class IntercomEndpointConfig:partition_count: int = 1partition_size: int = 1page_size: int = 150+ # Where page-number endpoints report their page count: top level, or nested under `pages`.+ total_pages_path: str = "total_pages"sort_mode: SortMode = "asc"extra_params: dict[str, str] = field(default_factory=dict)# Query-string param Intercom expects for the cursor filter (e.g. "created_at_after"@@ -40,8 +42,8 @@ class IntercomEndpointConfig:# Endpoint contracts validated against the live Intercom REST API (version 2.13).# Selectors, pagination shape, and partition keys reflect what the API actually# returns — not the docs.-# The endpoints with `api_versions=("2.16",)` exist only on 2.16 and follow-# Intercom's published 2.16 OpenAPI description.+# The endpoints with `api_versions` set exist only from 2.16 or 2.17 on and follow+# Intercom's published OpenAPI description for that version.## Real-incremental endpoints (server-side filter honored): contacts, conversations,# tickets, activity_logs, macros, conversation_parts (via parent). Everything else is@@ -285,7 +287,7 @@ INTERCOM_ENDPOINTS: dict[str, IntercomEndpointConfig] = {partition_mode="md5",partition_format=None,extra_params={"include_archived": "true"},- api_versions=("2.16",),+ api_versions=("2.16", "2.17"),),"macros": IntercomEndpointConfig(# `GET /macros` — saved replies. Paged by a `pages.next.starting_after` cursor,@@ -298,7 +300,7 @@ INTERCOM_ENDPOINTS: dict[str, IntercomEndpointConfig] = {paginator_kind="pages",sort_mode="desc",incremental_query_param="updated_since",- api_versions=("2.16",),+ api_versions=("2.16", "2.17"),),"audiences": IntercomEndpointConfig(# `GET /audiences` — saved audience definitions (targeting predicates). Paged by@@ -308,7 +310,7 @@ INTERCOM_ENDPOINTS: dict[str, IntercomEndpointConfig] = {data_selector="data",paginator_kind="page_number",page_size=50,- api_versions=("2.16",),+ api_versions=("2.16", "2.17"),),"content_snippets": IntercomEndpointConfig(# `GET /content_snippets` — snippets that feed Fin and Copilot answers. Same@@ -318,7 +320,68 @@ INTERCOM_ENDPOINTS: dict[str, IntercomEndpointConfig] = {data_selector="data",paginator_kind="page_number",page_size=50,- api_versions=("2.16",),+ api_versions=("2.16", "2.17"),+ ),+ "contact_activities": IntercomEndpointConfig(+ # `GET /contacts/activities` — workspace feed of contact lifecycle events (created,+ # role updated, merged, deleted). Oldest first, paged by a `pages.next.starting_after`+ # cursor, with no timestamp filter, so full refresh.+ name="contact_activities",+ path="/contacts/activities",+ data_selector="data",+ paginator_kind="pages",+ api_versions=("2.17",),+ ),+ "deleted_contacts": IntercomEndpointConfig(+ # `GET /contacts/deleted` — contacts that are archived, blocked, merged, or deleted.+ # Rows carry only `updated_at`, which changes, so partition on the id.+ name="deleted_contacts",+ path="/contacts/deleted",+ data_selector="data",+ paginator_kind="page_number",+ partition_key="id",+ partition_mode="md5",+ partition_format=None,+ total_pages_path="pages.total_pages",+ api_versions=("2.17",),+ ),+ "folders": IntercomEndpointConfig(+ # `GET /folders` — the folders articles and other content are organized into+ # (decodes `articles.folder_id`). Intercom documents no `per_page` maximum, so this+ # uses the 50 its examples show.+ name="folders",+ path="/folders",+ data_selector="data",+ paginator_kind="page_number",+ page_size=50,+ total_pages_path="pages.total_pages",+ api_versions=("2.17",),+ ),+ "dynamic_contents": IntercomEndpointConfig(+ # `GET /dynamic_contents` — reusable Help Center content with a variation per locale+ # and brand. Most recently updated first, `per_page` max 50, no timestamp filter.+ # Needs the `read_dynamic_content` scope and the feature enabled on the workspace.+ name="dynamic_contents",+ path="/dynamic_contents",+ data_selector="data",+ paginator_kind="page_number",+ page_size=50,+ sort_mode="desc",+ api_versions=("2.17",),+ ),+ "conversation_hitl_events": IntercomEndpointConfig(+ # Substream of `conversations`: `GET /conversations/{id}/hitl_events` for each parent+ # the conversations search returns. Events have no id of their own. Every event of one+ # consultation shares `hitl_cycle_id`, and each type occurs once per consultation or+ # once per feedback round, hence the composite key. Each yielded event carries+ # `conversation_id` and `round_index` (0 for event types outside a round).+ name="conversation_hitl_events",+ path="",+ data_selector="",+ paginator_kind="substream",+ parent_endpoint="conversations",+ primary_keys=["conversation_id", "hitl_cycle_id", "event_type", "round_index"],+ api_versions=("2.17",),),}@@ -334,6 +397,11 @@ ENDPOINTS = tuple(INTERCOM_ENDPOINTS.keys())# - conversation_parts: child of conversations; the parent's `updated_at >`# filter is server-honored, so parents whose timestamp didn't advance# aren't refetched.+# - conversation_hitl_events: child of conversations, filtered on the parent's+# `updated_at` like conversation_parts, but from a source cursor rather than the+# rows' `created_at`. Intercom lists the events only once a consultation ends,+# and the end (a reply, an escalation, a takeover) updates the conversation, so+# a parent with new events is refetched.## Companies, segments, and company_segments are intentionally excluded: their# list endpoints don't accept a timestamp filter, and a "client-side cursor"@@ -382,6 +450,14 @@ INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {"field_type": IncrementalFieldType.Integer,}],+ "conversation_hitl_events": [+ {+ "label": "created_at",+ "type": IncrementalFieldType.DateTime,+ "field": "created_at",+ "field_type": IncrementalFieldType.Integer,+ }+ ],"macros": [{"label": "updated_at",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/source.pyindex 98d096ca0614..a4d8d548e58b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/source.py@@ -13,6 +13,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.basfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.canonical_descriptions import (CanonicalDescriptions,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.cursor import CursorSourcefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.mixins import OAuthMixinfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistryfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager@@ -22,6 +23,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_IntercomSourceConfig,)from products.warehouse_sources.backend.temporal.data_imports.sources.intercom.intercom import (+ IntercomConversationCursor,IntercomResumeConfig,intercom_source,validate_credentials as validate_intercom_credentials,@@ -34,7 +36,11 @@ from products.warehouse_sources.backend.types import ExternalDataSourceType@SourceRegistry.register-class IntercomSource(ResumableSource[IntercomSourceConfig, IntercomResumeConfig], OAuthMixin):+class IntercomSource(+ ResumableSource[IntercomSourceConfig, IntercomResumeConfig],+ CursorSource[IntercomConversationCursor],+ OAuthMixin,+):codebase_detection = detect(env=("^INTERCOM_",@@ -83,8 +89,8 @@ class IntercomSource(ResumableSource[IntercomSourceConfig, IntercomResumeConfig])lists_tables_without_credentials = True # static endpoint catalog — safe for public docs- supported_versions = ("2.13", "2.15", "2.16")- default_version = "2.16"+ supported_versions = ("2.13", "2.15", "2.16", "2.17")+ default_version = "2.17"api_docs_url = "https://developers.intercom.com/docs/references/rest-api"@property@@ -218,6 +224,14 @@ class IntercomSource(ResumableSource[IntercomSourceConfig, IntercomResumeConfig]integration.access_token, schema_name=schema_name, api_version=self.resolve_api_version(api_version))+ def cursor_class(self) -> type[IntercomConversationCursor]:+ return IntercomConversationCursor++ def merge_cursors(+ self, current: IntercomConversationCursor, candidate: IntercomConversationCursor+ ) -> IntercomConversationCursor:+ return max(current, candidate, key=lambda cursor: cursor.updated_at)+def get_resumable_source_manager(self, inputs: SourceInputs) -> ResumableSourceManager[IntercomResumeConfig]:return ResumableSourceManager[IntercomResumeConfig](inputs, IntercomResumeConfig)@@ -244,4 +258,8 @@ class IntercomSource(ResumableSource[IntercomSourceConfig, IntercomResumeConfig]db_incremental_field_last_value=inputs.db_incremental_field_last_valueif inputs.should_use_incremental_fieldelse None,+ # Only the HITL events table keeps a source cursor; see `_conversation_hitl_events_generator`.+ cursor_manager=self.get_cursor_manager(inputs)+ if inputs.schema_name == "conversation_hitl_events"+ else None,)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom.py b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom.pyindex f5eec365173b..e6016fd5a641 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom.py@@ -8,6 +8,7 @@ from unittest import mockfrom requests import Responsefrom requests.exceptions import HTTPError+from products.warehouse_sources.backend.temporal.data_imports.sources.common.cursor import SourceCursorManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (JSONResponseCursorPaginator,JSONResponsePaginator,@@ -29,6 +30,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_from products.warehouse_sources.backend.temporal.data_imports.sources.intercom import intercom as intercom_modulefrom products.warehouse_sources.backend.temporal.data_imports.sources.intercom.intercom import (INTERCOM_API_BASE,+ IntercomConversationCursor,IntercomPagesPaginator,IntercomResumeConfig,IntercomSearchPaginator,@@ -73,6 +75,8 @@ def _drive() -> DriveResult:source = IntercomSource()config = IntercomSourceConfig(intercom_integration_id=456)+ # The pipeline gives every schema of a cursor source a manager, with no cursor on a first run.+ inputs.setdefault("source_cursor", SourceCursorManager(IntercomConversationCursor, None, source))with mock.patch.object(source, "get_oauth_integration", return_value=mock.MagicMock(access_token="token")):return SourceDriver(source, config).run(endpoint, script, resume_state=resume_state, **inputs)@@ -437,7 +441,7 @@ class TestIntercomSource:@pytest.mark.parametrize("endpoint", list(INTERCOM_ENDPOINTS.keys()))def test_source_response_metadata(self, endpoint: str):cfg = INTERCOM_ENDPOINTS[endpoint]- result = _drive(endpoint, [], api_version="2.16")+ result = _drive(endpoint, [], api_version="2.17")response = result.responseassert response is not None@@ -448,6 +452,20 @@ class TestIntercomSource:# A companies scroll expires within a minute and cannot restart mid-walk.assert response.supports_resume is (cfg.paginator_kind != "scroll")+ @pytest.mark.parametrize("pin", ["2.13", "2.15", "2.16", "2.17"])+ @pytest.mark.parametrize(+ "endpoint,response",+ [+ ("admins", ScriptedResponse(json={"admins": []})),+ ("conversation_parts", ScriptedResponse(json={"conversations": [], "pages": {}})),+ ],+ )+ def test_sync_requests_send_the_pinned_version(self, pin: str, endpoint: str, response: ScriptedResponse):+ result = _drive(endpoint, [response], api_version=pin)++ assert result.raised is None+ assert [request.headers["intercom-version"] for request in result.requests] == [pin]+@pytest.mark.parametrize("endpoint", [name for name, cfg in INTERCOM_ENDPOINTS.items() if cfg.api_versions is not None])@@ -598,29 +616,33 @@ class TestResumableRestEndpoints:assert result.urls[0] == expected_url-def _numbered_page(ids: list[int], page: int, total_pages: int) -> ScriptedResponse:- return ScriptedResponse(- json={- "type": "list",- "data": [{"id": i} for i in ids],- "page": page,- "per_page": 50,- "total_pages": total_pages,- }- )+def _numbered_page(ids: list[int], page: int, total_pages: int, nested: bool = False) -> ScriptedResponse:+ counts = {"page": page, "per_page": 50, "total_pages": total_pages}+ body: dict[str, Any] = {"type": "list", "data": [{"id": i} for i in ids]}+ body.update({"pages": {"type": "pages", **counts}} if nested else counts)+ return ScriptedResponse(json=body)class TestPageNumberEndpoints:- @pytest.mark.parametrize("endpoint", ["audiences", "content_snippets"])- def test_fresh_run_walks_every_page_and_stops_at_total_pages(self, endpoint: str):- responses = [_numbered_page([1], 1, 2), _numbered_page([2], 2, 2)]+ @pytest.mark.parametrize(+ "endpoint,per_page,nested",+ [+ ("audiences", "50", False),+ ("content_snippets", "50", False),+ ("dynamic_contents", "50", False),+ ("folders", "50", True),+ ("deleted_contacts", "150", True),+ ],+ )+ def test_fresh_run_walks_every_page_and_stops_at_total_pages(self, endpoint: str, per_page: str, nested: bool):+ responses = [_numbered_page([1], 1, 2, nested), _numbered_page([2], 2, 2, nested)]result = _drive(endpoint, responses)assert result.raised is Noneassert [row["id"] for row in result.rows] == [1, 2]# The last page is known from `total_pages`, so no trailing empty-page request goes out.- assert result.params("per_page") == ["50", "50"]+ assert result.params("per_page") == [per_page, per_page]assert result.params("page") == ["1", "2"]assert result.saved_states == [IntercomResumeConfig(page=2)]@@ -760,3 +782,55 @@ class TestResumableConversationParts:assert body["pagination"]["starting_after"] == "p2"assert body["query"]["value"] == 1600000000assert result.paths == ["/conversations/search", "/conversations/c4"]++ def test_hitl_events_walk_parents_from_the_source_cursor_and_advance_it(self):+ cursor_manager = SourceCursorManager(+ IntercomConversationCursor, IntercomConversationCursor(updated_at=1700000000), IntercomSource()+ )+ rounds = [+ {"hitl_cycle_id": "h1", "event_type": "fin_consultation_started", "created_at": 1700000050, "data": {}},+ *(+ {+ "hitl_cycle_id": "h1",+ "event_type": "fin_teammate_input_received",+ "created_at": 1700000060,+ "data": {"round_index": i},+ }+ for i in (1, 2)+ ),+ ]+ result = _drive(+ "conversation_hitl_events",+ [+ ScriptedResponse(+ json={+ "conversations": [+ {"id": "c1", "updated_at": 1700000100},+ {"id": "c2", "updated_at": 1700000200},+ ],+ "pages": {"next": None},+ }+ ),+ ScriptedResponse(json={"type": "list", "data": rounds}),+ ScriptedResponse(json={"type": "list", "data": []}),+ ],+ should_use_incremental_field=True,+ incremental_field="created_at",+ db_incremental_field_last_value=1700000060,+ source_cursor=cursor_manager,+ )++ assert result.raised is None+ assert [(e["conversation_id"], e["event_type"], e["round_index"]) for e in result.rows] == [+ ("c1", "fin_consultation_started", 0),+ ("c1", "fin_teammate_input_received", 1),+ ("c1", "fin_teammate_input_received", 2),+ ]+ assert result.requests[0].json()["query"] == {"field": "updated_at", "operator": ">", "value": 1700000000}+ assert result.paths == [+ "/conversations/search",+ "/conversations/c1/hitl_events",+ "/conversations/c2/hitl_events",+ ]+ # A conversation with no events still moves the cursor, so the next run skips it.+ assert cursor_manager.staged == IntercomConversationCursor(updated_at=1700000200)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom_source.pyindex 5c8b0ee1768e..a8ba786516a0 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom_source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom_source.py@@ -26,7 +26,7 @@ class TestIntercomSource:def test_default_version_is_latest(self):# New sources are stamped with the default; keep it on the newest supported version.- assert self.source.default_version == "2.16"+ assert self.source.default_version == "2.17"assert self.source.default_version in self.source.supported_versions@pytest.mark.parametrize(@@ -44,19 +44,33 @@ class TestIntercomSource:retryable_errors = self.source.get_retryable_errors()assert any(key in error_msg for key in retryable_errors)- @pytest.mark.parametrize("pin", ["2.13", "2.15"])- def test_get_schemas_hides_tables_the_pinned_version_does_not_serve(self, pin: str):+ @pytest.mark.parametrize(+ "pin,has_macros,has_contact_activities",+ [+ ("2.13", False, False),+ ("2.15", False, False),+ ("2.16", True, False),+ ("2.17", True, True),+ (None, True, True),+ ],+ )+ def test_get_schemas_hides_tables_the_pinned_version_does_not_serve(+ self, pin: str | None, has_macros: bool, has_contact_activities: bool+ ):names = {s.name for s in self.source.get_schemas(self.config, self.team_id, api_version=pin)}- assert names == {name for name, cfg in INTERCOM_ENDPOINTS.items() if cfg.api_versions is None}- assert "macros" not in names+ assert {name for name, cfg in INTERCOM_ENDPOINTS.items() if cfg.api_versions is None} <= names+ assert ("macros" in names) is has_macros+ assert ("contact_activities" in names) is has_contact_activitiesdef test_get_schemas_names_filter(self):schemas = self.source.get_schemas(self.config, self.team_id, names=["contacts", "companies"])assert {s.name for s in schemas} == {"contacts", "companies"}- @pytest.mark.parametrize("pin,expected", [("2.13", "2.13"), ("2.15", "2.15"), ("2.16", "2.16"), (None, "2.16")])+ @pytest.mark.parametrize(+ "pin,expected", [("2.13", "2.13"), ("2.15", "2.15"), ("2.16", "2.16"), ("2.17", "2.17"), (None, "2.17")]+ )@mock.patch("products.warehouse_sources.backend.temporal.data_imports.sources.intercom.source.IntercomSource.get_oauth_integration")@@ -125,6 +139,7 @@ class TestIntercomSource:should_use_incremental_field=True,incremental_field="updated_at",db_incremental_field_last_value="1700000000",+ cursor_manager=None,)@mock.patch("products.warehouse_sources.backend.temporal.data_imports.sources.intercom.source.intercom_source")