Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse_sources): support intercom api version 2.17 (#119203)

Tom Owerse664588b5f44
diff --git a/.agents/skills/warehouse-source-new-version/SKILL.md b/.agents/skills/warehouse-source-new-version/SKILL.md
index 854b0664b2a9..304730c89716 100644
--- a/.agents/skills/warehouse-source-new-version/SKILL.md
+++ b/.agents/skills/warehouse-source-new-version/SKILL.md
@@ -116,6 +116,7 @@ The opposite gap also occurs: a reference rendered from one version-blind OpenAP
- A version that changes the pagination _mechanism_ (e.g. offset→cursor, or one opaque cursor param for another, since a token one generation issued means nothing to the other) also changes the resumable checkpoint's persisted shape. Add a new defaulted field to the resume-config dataclass rather than repurposing the existing one — so state persisted under the old version still decodes — and make the seed and save-checkpoint logic dispatch on the resolved version. Reusing one field across incompatible mechanisms corrupts in-flight resumes of the other version. When a table stays on its old wire under the new pin, key the check on the wire that issued the cursor, not on the pin. When both mechanisms are positional with a fixed page size (offset↔page number), keep the existing field instead: let the new version's paginator derive its page from the stored offset, so one checkpoint means the same row under either pin.
- Other products can read a synced table's columns by name, e.g. a signal emitter registered on `(source_type, table)` in `products/signals/backend/emission/registry.py`. Grep for the source type outside its directory before a new version renames or nests a column such a reader uses, and keep that column under its old name on the new version's rows.
- When the vendor renames a collection between versions, keep the schema/table name set identical across versions and put the rename in a per-version path on the endpoint config — otherwise discovery diffs orphan the table on repin.
+- A table gated to an explicit list of versions (an endpoint added by an earlier bump) drops off a new default unless the new label joins that list. Before flipping the default, extend every such list for each endpoint the new version still serves.
- When the new version has no equivalent for an old endpoint (a dropped collection, not a rename), keep that table only on the versions that serve it and let the table set differ by version — never map it to a guessed path just to keep the sets equal, since docs are the sole source of truth and an unverified path makes a new source surface a table that 404s at sync time.
- The registry catalog invariants (`sources/tests/test_source_catalog_invariants.py`) read only the default version's `get_schemas`, so a new default that drops a table makes that table's canonical description look stray. Keep the description, since pins on the old version still sync and render that table, and add the source to `DESCRIPTIONS_NOT_IN_SCHEMAS`.
- A source with no dispatch may currently read its version from a constant on a shared model (e.g. the OAuth `Integration` model) that also drives version-independent flows like OAuth token minting. Repoint only the sync request path onto the resolved pin; leave that constant, since bumping it changes those other flows' version with a blast radius beyond this source.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 2d502c6d081d..a443bcfd8e67 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -6444,7 +6444,7 @@ Note: Parsed the official Swagger resource index and individual JSON definitions
## Intercom — gaps
-Today (25): `activity_logs`, `admins`, `articles`, `audiences`, `collections`, `companies`, `company_attributes`, `company_segments`, `contact_attributes`, `contacts`, `content_snippets`, `conversation_attributes`, `conversation_parts`, `conversations`, `help_centers`, `macros`, `news_items`, `newsfeeds`, `segments`, `subscription_types`, `tags`, `teams`, `ticket_states`, `ticket_types`, `tickets`
+Today (30): `activity_logs`, `admins`, `articles`, `audiences`, `collections`, `companies`, `company_attributes`, `company_segments`, `contact_activities`, `contact_attributes`, `contacts`, `content_snippets`, `conversation_attributes`, `conversation_hitl_events`, `conversation_parts`, `conversations`, `deleted_contacts`, `dynamic_contents`, `folders`, `help_centers`, `macros`, `news_items`, `newsfeeds`, `segments`, `subscription_types`, `tags`, `teams`, `ticket_states`, `ticket_types`, `tickets`
Diffed against: <https://github.com/intercom/Intercom-OpenAPI/blob/main/descriptions/2.16/api.intercom.io.yaml>
@@ -6455,6 +6455,14 @@ The main spec-verified diff is in [COVERAGE_GAPS.md](COVERAGE_GAPS.md#intercom--
- [x] `GET /audiences` — saved audience definitions that content and messages target (low). Added as `audiences`.
- [x] `GET /content_snippets` — snippets that feed Fin and Copilot answers (medium). Added as `content_snippets`.
+These endpoints were announced in the 2.17 changelog and are only in the 2.17 description, so the tables only show for sources pinned to 2.17.
+
+- [x] `GET /contacts/activities` — workspace feed of contact lifecycle events: created, role updated, merged, deleted (medium). Added as `contact_activities`.
+- [x] `GET /contacts/deleted` — archived, blocked, merged, and deleted contacts with the reason (medium). Added as `deleted_contacts`.
+- [x] `GET /folders` — folders that organize articles and other content, decoding `articles.folder_id` (low). Added as `folders`.
+- [x] `GET /dynamic_contents` — reusable Help Center content referenced from article bodies (low). Added as `dynamic_contents`.
+- [x] `GET /conversations/{id}/hitl_events` — Fin human-in-the-loop consultation events per conversation (medium). Added as `conversation_hitl_events`, fanned out over `conversations` and incremental on the parent's `updated_at`.
+
## Intruder — adequate
Today (8): `fixed_occurrences`, `issue_targets`, `issues`, `occurrences`, `scan_schedules`, `scans`, `tags`, `targets`
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/canonical_descriptions.py
index db2af17cd350..8d0323669870 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/canonical_descriptions.py
@@ -416,4 +416,78 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"audience_ids": "Identifiers of the audiences the snippet targets.",
},
},
+ "contact_activities": {
+ "description": "A contact lifecycle event in the workspace, such as a contact being created, changing role, merged, or deleted.",
+ "docs_url": "https://developers.intercom.com/docs/references/rest-api/api.intercom.io/contacts/listcontactactivities",
+ "columns": {
+ "id": "Unique identifier for the activity event.",
+ "type": "The object type, always 'contact_activity'.",
+ "event_type": "The type of activity event, such as contact.created or contact.merged.",
+ "contact_id": "Identifier of the contact the event relates to.",
+ "external_id": "The external identifier of the contact, if one is set.",
+ "role": "The role of the contact at the time of the event: user, lead, or visitor.",
+ "merge_source_id": "For merge events, the identifier of the contact that was merged away.",
+ "merge_target_id": "For merge events, the identifier of the contact that was merged into.",
+ "failure_reason": "For failed events, the reason the activity did not succeed.",
+ "created_at": "Time at which the event occurred, as a Unix timestamp.",
+ },
+ },
+ "deleted_contacts": {
+ "description": "A contact that is no longer active in the workspace, with the reason it is inactive.",
+ "docs_url": "https://developers.intercom.com/docs/references/rest-api/api.intercom.io/contacts/listdeletedcontacts",
+ "columns": {
+ "id": "Unique identifier for the contact.",
+ "type": "The object type, always 'contact'.",
+ "external_id": "The identifier for the contact provided by the client. Null for permanently deleted contacts.",
+ "email": "The contact's email address. Null for permanently deleted contacts.",
+ "role": "The role of the contact.",
+ "status": "Why the contact is inactive: archived, blocked, merged, pending_deletion, or permanently_deleted.",
+ "merged_into_id": "Identifier of the contact this one was merged into. Present only when status is merged.",
+ "updated_at": "Time at which the contact record was last updated, as a Unix timestamp. Not the deletion time.",
+ },
+ },
+ "folders": {
+ "description": "A folder that organizes articles and other content into a hierarchy.",
+ "docs_url": "https://developers.intercom.com/docs/references/rest-api/api.intercom.io/folders/listfolders",
+ "columns": {
+ "id": "Unique identifier for the folder.",
+ "name": "The name of the folder.",
+ "description": "The description of the folder.",
+ "emoji": "An emoji associated with the folder.",
+ "parent_folder_id": "Identifier of the parent folder, or null for a root folder.",
+ "content_count": "The number of content items in the folder.",
+ "created_at": "Time at which the folder was created, as a Unix timestamp.",
+ "updated_at": "Time at which the folder was last updated, as a Unix timestamp.",
+ },
+ },
+ "dynamic_contents": {
+ "description": "Reusable Help Center content with a variation per locale, and optionally per brand, that articles reference.",
+ "docs_url": "https://developers.intercom.com/docs/references/rest-api/api.intercom.io/dynamic-content/listdynamiccontents",
+ "columns": {
+ "id": "Unique identifier for the dynamic content.",
+ "type": "The object type, always 'dynamic_content'.",
+ "name": "The name of the dynamic content.",
+ "key": "The identifier articles use to reference the dynamic content.",
+ "content_type": "Whether the content is a short inline value (inline) or one or more blocks of content (block).",
+ "default_locale": "The locale whose variation is used when there is no variation for the requested locale.",
+ "variations": "The variations of the dynamic content, one per locale and brand.",
+ "created_at": "Time at which the dynamic content was created, as a Unix timestamp.",
+ "updated_at": "Time at which the dynamic content was last updated, as a Unix timestamp.",
+ },
+ },
+ "conversation_hitl_events": {
+ "description": "An event from a Fin human-in-the-loop consultation, where Fin asks a teammate before it replies in a conversation.",
+ "docs_url": "https://developers.intercom.com/docs/references/rest-api/api.intercom.io/conversations/listhitlevents",
+ "columns": {
+ "conversation_id": "Identifier of the conversation the event belongs to.",
+ "hitl_cycle_id": "Identifier of the consultation. All events of one consultation share it.",
+ "event_type": "What happened, such as fin_consultation_started, fin_draft_proposed, or fin_consultation_finished.",
+ "round_index": "The feedback round the event belongs to, starting at 1. 0 for events outside a round, such as the start and end of the consultation.",
+ "conversation_part_id": "The conversation part the event relates to, where one exists.",
+ "rule_id": "The escalation rule that triggered the consultation, as recorded when it began. Null if no rule triggered it.",
+ "rule_name": "The rule's name as recorded when the consultation began.",
+ "created_at": "Time at which the event occurred, as a Unix timestamp. Approximate for fin_draft_proposed.",
+ "data": "Fields specific to the event type, such as the draft, the teammate input, or the consultation outcome.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/intercom.py b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/intercom.py
index ce4b9436be7d..a3e3ee1f9c10 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/intercom.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/intercom.py
@@ -2,13 +2,14 @@ import time
import dataclasses
from collections.abc import AsyncIterable, Callable, Iterable, Iterator
from datetime import UTC, datetime
-from typing import Any, Optional, cast
+from typing import Any, ClassVar, Optional, cast
import structlog
from requests import Request, Response, Session
from requests.exceptions import HTTPError
from urllib3.util.retry import Retry
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.cursor import SourceCursorManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import (
DEFAULT_RETRY,
make_tracked_session,
@@ -62,6 +63,13 @@ class IntercomResumeConfig:
page: int | None = None
[email protected](frozen=True)
+class IntercomConversationCursor:
+ cursor_kind: ClassVar[str] = "intercom_conversation_updated_at"
+ # `updated_at` of the newest conversation whose HITL events this run read.
+ updated_at: int
+
+
def _load_resume_state(
resumable_source_manager: ResumableSourceManager[IntercomResumeConfig] | None,
) -> IntercomResumeConfig | None:
@@ -344,7 +352,7 @@ def _build_paginator(cfg: IntercomEndpointConfig) -> BasePaginator:
if cfg.paginator_kind == "pages":
return IntercomPagesPaginator()
if cfg.paginator_kind == "page_number":
- return PageNumberPaginator(base_page=1, total_path="total_pages")
+ return PageNumberPaginator(base_page=1, total_path=cfg.total_pages_path)
return SinglePagePaginator()
@@ -479,20 +487,27 @@ def _intercom_post(session: Session, path_or_url: str, body: dict[str, Any]) ->
return _request_with_rate_limit_retry(do)
+def _fetch_conversation_parts(session: Session, conversation: dict[str, Any]) -> list[dict[str, Any]]:
+ full = _intercom_get(session, f"/conversations/{conversation['id']}")
+ return (full.get("conversation_parts") or {}).get("conversation_parts") or []
+
+
def _conversation_parts_generator(
session: Session,
incremental_field: str,
db_incremental_field_last_value: Optional[Any],
resumable_source_manager: ResumableSourceManager[IntercomResumeConfig] | None = None,
+ fetch_children: Callable[[Session, dict[str, Any]], list[dict[str, Any]]] = _fetch_conversation_parts,
) -> Iterator[dict[str, Any]]:
- """Yield each conversation part. Parents come from `POST /conversations/search`
- with the same `updated_at >` server-side filter the conversations endpoint uses,
- so only parents whose timestamp advanced are refetched. The part rows carry their
- own `updated_at`, so the pipeline's cursor watermark advances per-part.
- `conversation_id` is injected onto each row for joinability.
+ """Yield each conversation part, or the rows `fetch_children` returns for each
+ conversation. Parents come from `POST /conversations/search` with the same
+ `updated_at >` server-side filter the conversations endpoint uses, so only parents
+ whose timestamp advanced are refetched. Part rows carry their own `updated_at`, so
+ the pipeline's cursor watermark advances per-part. `conversation_id` is injected onto
+ each row for joinability.
The resume cursor is the parent page plus the conversations in it that are done,
- staged after each conversation's parts are yielded. A resume refetches that page
+ staged after each conversation's children are yielded. A resume refetches that page
and skips the done conversations."""
body = _build_search_body(INTERCOM_ENDPOINTS["conversations"], incremental_field, db_incremental_field_last_value)
completed: list[str] = []
@@ -510,8 +525,8 @@ def _conversation_parts_generator(
resumable_source_manager.save_state(
IntercomResumeConfig(cursor=page_cursor, query_value=query_value, completed_conversation_ids=list(done))
)
- # Every part the staged cursor covers is yielded and nothing is buffered here. Many
- # conversations have no parts, so the walk can make many requests between rows.
+ # Every row the staged cursor covers is yielded and nothing is buffered here. Many
+ # conversations have no children, so the walk can make many requests between rows.
resumable_source_manager.safe_point()
while True:
@@ -523,16 +538,15 @@ def _conversation_parts_generator(
if conversation_id in done_ids:
continue
try:
- full = _intercom_get(session, f"/conversations/{conv['id']}")
+ children = fetch_children(session, conv)
except HTTPError as exc:
if not _is_not_found(exc):
raise
logger.warning("intercom_conversation_not_found", conversation_id=conv["id"])
else:
- parts = (full.get("conversation_parts") or {}).get("conversation_parts") or []
- for part in parts:
- part["conversation_id"] = conv["id"]
- yield part
+ for child in children:
+ child["conversation_id"] = conv["id"]
+ yield child
completed.append(conversation_id)
done_ids.add(conversation_id)
stage(page_cursor, completed)
@@ -714,12 +728,44 @@ def _company_segments_generator(
resumable_source_manager.safe_point()
+def _conversation_hitl_events_generator(
+ session: Session,
+ should_use_incremental_field: bool,
+ cursor_manager: SourceCursorManager[IntercomConversationCursor] | None,
+ resumable_source_manager: ResumableSourceManager[IntercomResumeConfig] | None = None,
+) -> Iterator[dict[str, Any]]:
+ """Yield the HITL events of each conversation, with `conversation_id` and `round_index` injected.
+
+ The events carry only `created_at`, and many workspaces have none, so a watermark taken from
+ the rows would never move. The source cursor holds the newest parent `updated_at` the walk
+ reached instead, and the next run searches only conversations updated after it."""
+ stored = cursor_manager.load() if should_use_incremental_field and cursor_manager is not None else None
+
+ def fetch_events(session: Session, conversation: dict[str, Any]) -> list[dict[str, Any]]:
+ events = _intercom_get(session, f"/conversations/{conversation['id']}/hitl_events").get("data") or []
+ for event in events:
+ # Rounds of one consultation share every other key column, sometimes `created_at` too.
+ event["round_index"] = (event.get("data") or {}).get("round_index") or 0
+ if cursor_manager is not None and conversation.get("updated_at") is not None:
+ cursor_manager.stage(IntercomConversationCursor(updated_at=int(conversation["updated_at"])))
+ return events
+
+ return _conversation_parts_generator(
+ session,
+ "updated_at",
+ stored.updated_at if stored is not None else None,
+ resumable_source_manager,
+ fetch_children=fetch_events,
+ )
+
+
def _substream_items(
session: Session,
endpoint: str,
incremental_field: str | None,
db_incremental_field_last_value: Optional[Any],
resumable_source_manager: ResumableSourceManager[IntercomResumeConfig] | None = None,
+ cursor_manager: SourceCursorManager[IntercomConversationCursor] | None = None,
) -> Iterator[dict[str, Any]]:
if endpoint == "conversation_parts":
if not incremental_field:
@@ -730,6 +776,10 @@ def _substream_items(
return _conversation_parts_generator(
session, incremental_field, db_incremental_field_last_value, resumable_source_manager
)
+ if endpoint == "conversation_hitl_events":
+ return _conversation_hitl_events_generator(
+ session, incremental_field is not None, cursor_manager, resumable_source_manager
+ )
if endpoint == "company_segments":
return _company_segments_generator(session, resumable_source_manager)
raise ValueError(f"Unknown Intercom substream endpoint: {endpoint}")
@@ -797,6 +847,7 @@ def intercom_source(
should_use_incremental_field: bool = False,
incremental_field: str | None = None,
db_incremental_field_last_value: Optional[Any] = None,
+ cursor_manager: SourceCursorManager[IntercomConversationCursor] | None = None,
) -> SourceResponse:
cfg = INTERCOM_ENDPOINTS[endpoint]
if cfg.api_versions is not None and api_version not in cfg.api_versions:
@@ -817,7 +868,12 @@ def intercom_source(
had_resume_state = resumable_source_manager.can_resume()
try:
yield from _substream_items(
- session, endpoint, incremental_field, db_incremental_field_last_value, resumable_source_manager
+ session,
+ endpoint,
+ incremental_field,
+ db_incremental_field_last_value,
+ resumable_source_manager,
+ cursor_manager,
)
except HTTPError as exc:
if had_resume_state and _is_invalid_resume_cursor(exc):
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/settings.py
index 94d568076af8..72cac1f5d429 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/settings.py
@@ -26,6 +26,8 @@ class IntercomEndpointConfig:
partition_count: int = 1
partition_size: int = 1
page_size: int = 150
+ # Where page-number endpoints report their page count: top level, or nested under `pages`.
+ total_pages_path: str = "total_pages"
sort_mode: SortMode = "asc"
extra_params: dict[str, str] = field(default_factory=dict)
# Query-string param Intercom expects for the cursor filter (e.g. "created_at_after"
@@ -40,8 +42,8 @@ class IntercomEndpointConfig:
# Endpoint contracts validated against the live Intercom REST API (version 2.13).
# Selectors, pagination shape, and partition keys reflect what the API actually
# returns — not the docs.
-# The endpoints with `api_versions=("2.16",)` exist only on 2.16 and follow
-# Intercom's published 2.16 OpenAPI description.
+# The endpoints with `api_versions` set exist only from 2.16 or 2.17 on and follow
+# Intercom's published OpenAPI description for that version.
#
# Real-incremental endpoints (server-side filter honored): contacts, conversations,
# tickets, activity_logs, macros, conversation_parts (via parent). Everything else is
@@ -285,7 +287,7 @@ INTERCOM_ENDPOINTS: dict[str, IntercomEndpointConfig] = {
partition_mode="md5",
partition_format=None,
extra_params={"include_archived": "true"},
- api_versions=("2.16",),
+ api_versions=("2.16", "2.17"),
),
"macros": IntercomEndpointConfig(
# `GET /macros` — saved replies. Paged by a `pages.next.starting_after` cursor,
@@ -298,7 +300,7 @@ INTERCOM_ENDPOINTS: dict[str, IntercomEndpointConfig] = {
paginator_kind="pages",
sort_mode="desc",
incremental_query_param="updated_since",
- api_versions=("2.16",),
+ api_versions=("2.16", "2.17"),
),
"audiences": IntercomEndpointConfig(
# `GET /audiences` — saved audience definitions (targeting predicates). Paged by
@@ -308,7 +310,7 @@ INTERCOM_ENDPOINTS: dict[str, IntercomEndpointConfig] = {
data_selector="data",
paginator_kind="page_number",
page_size=50,
- api_versions=("2.16",),
+ api_versions=("2.16", "2.17"),
),
"content_snippets": IntercomEndpointConfig(
# `GET /content_snippets` — snippets that feed Fin and Copilot answers. Same
@@ -318,7 +320,68 @@ INTERCOM_ENDPOINTS: dict[str, IntercomEndpointConfig] = {
data_selector="data",
paginator_kind="page_number",
page_size=50,
- api_versions=("2.16",),
+ api_versions=("2.16", "2.17"),
+ ),
+ "contact_activities": IntercomEndpointConfig(
+ # `GET /contacts/activities` — workspace feed of contact lifecycle events (created,
+ # role updated, merged, deleted). Oldest first, paged by a `pages.next.starting_after`
+ # cursor, with no timestamp filter, so full refresh.
+ name="contact_activities",
+ path="/contacts/activities",
+ data_selector="data",
+ paginator_kind="pages",
+ api_versions=("2.17",),
+ ),
+ "deleted_contacts": IntercomEndpointConfig(
+ # `GET /contacts/deleted` — contacts that are archived, blocked, merged, or deleted.
+ # Rows carry only `updated_at`, which changes, so partition on the id.
+ name="deleted_contacts",
+ path="/contacts/deleted",
+ data_selector="data",
+ paginator_kind="page_number",
+ partition_key="id",
+ partition_mode="md5",
+ partition_format=None,
+ total_pages_path="pages.total_pages",
+ api_versions=("2.17",),
+ ),
+ "folders": IntercomEndpointConfig(
+ # `GET /folders` — the folders articles and other content are organized into
+ # (decodes `articles.folder_id`). Intercom documents no `per_page` maximum, so this
+ # uses the 50 its examples show.
+ name="folders",
+ path="/folders",
+ data_selector="data",
+ paginator_kind="page_number",
+ page_size=50,
+ total_pages_path="pages.total_pages",
+ api_versions=("2.17",),
+ ),
+ "dynamic_contents": IntercomEndpointConfig(
+ # `GET /dynamic_contents` — reusable Help Center content with a variation per locale
+ # and brand. Most recently updated first, `per_page` max 50, no timestamp filter.
+ # Needs the `read_dynamic_content` scope and the feature enabled on the workspace.
+ name="dynamic_contents",
+ path="/dynamic_contents",
+ data_selector="data",
+ paginator_kind="page_number",
+ page_size=50,
+ sort_mode="desc",
+ api_versions=("2.17",),
+ ),
+ "conversation_hitl_events": IntercomEndpointConfig(
+ # Substream of `conversations`: `GET /conversations/{id}/hitl_events` for each parent
+ # the conversations search returns. Events have no id of their own. Every event of one
+ # consultation shares `hitl_cycle_id`, and each type occurs once per consultation or
+ # once per feedback round, hence the composite key. Each yielded event carries
+ # `conversation_id` and `round_index` (0 for event types outside a round).
+ name="conversation_hitl_events",
+ path="",
+ data_selector="",
+ paginator_kind="substream",
+ parent_endpoint="conversations",
+ primary_keys=["conversation_id", "hitl_cycle_id", "event_type", "round_index"],
+ api_versions=("2.17",),
),
}
@@ -334,6 +397,11 @@ ENDPOINTS = tuple(INTERCOM_ENDPOINTS.keys())
# - conversation_parts: child of conversations; the parent's `updated_at >`
# filter is server-honored, so parents whose timestamp didn't advance
# aren't refetched.
+# - conversation_hitl_events: child of conversations, filtered on the parent's
+# `updated_at` like conversation_parts, but from a source cursor rather than the
+# rows' `created_at`. Intercom lists the events only once a consultation ends,
+# and the end (a reply, an escalation, a takeover) updates the conversation, so
+# a parent with new events is refetched.
#
# Companies, segments, and company_segments are intentionally excluded: their
# list endpoints don't accept a timestamp filter, and a "client-side cursor"
@@ -382,6 +450,14 @@ INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {
"field_type": IncrementalFieldType.Integer,
}
],
+ "conversation_hitl_events": [
+ {
+ "label": "created_at",
+ "type": IncrementalFieldType.DateTime,
+ "field": "created_at",
+ "field_type": IncrementalFieldType.Integer,
+ }
+ ],
"macros": [
{
"label": "updated_at",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/source.py
index 98d096ca0614..a4d8d548e58b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/source.py
@@ -13,6 +13,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.bas
from products.warehouse_sources.backend.temporal.data_imports.sources.common.canonical_descriptions import (
CanonicalDescriptions,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.cursor import CursorSource
from products.warehouse_sources.backend.temporal.data_imports.sources.common.mixins import OAuthMixin
from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistry
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
@@ -22,6 +23,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_
IntercomSourceConfig,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.intercom.intercom import (
+ IntercomConversationCursor,
IntercomResumeConfig,
intercom_source,
validate_credentials as validate_intercom_credentials,
@@ -34,7 +36,11 @@ from products.warehouse_sources.backend.types import ExternalDataSourceType
@SourceRegistry.register
-class IntercomSource(ResumableSource[IntercomSourceConfig, IntercomResumeConfig], OAuthMixin):
+class IntercomSource(
+ ResumableSource[IntercomSourceConfig, IntercomResumeConfig],
+ CursorSource[IntercomConversationCursor],
+ OAuthMixin,
+):
codebase_detection = detect(
env=(
"^INTERCOM_",
@@ -83,8 +89,8 @@ class IntercomSource(ResumableSource[IntercomSourceConfig, IntercomResumeConfig]
)
lists_tables_without_credentials = True # static endpoint catalog — safe for public docs
- supported_versions = ("2.13", "2.15", "2.16")
- default_version = "2.16"
+ supported_versions = ("2.13", "2.15", "2.16", "2.17")
+ default_version = "2.17"
api_docs_url = "https://developers.intercom.com/docs/references/rest-api"
@property
@@ -218,6 +224,14 @@ class IntercomSource(ResumableSource[IntercomSourceConfig, IntercomResumeConfig]
integration.access_token, schema_name=schema_name, api_version=self.resolve_api_version(api_version)
)
+ def cursor_class(self) -> type[IntercomConversationCursor]:
+ return IntercomConversationCursor
+
+ def merge_cursors(
+ self, current: IntercomConversationCursor, candidate: IntercomConversationCursor
+ ) -> IntercomConversationCursor:
+ return max(current, candidate, key=lambda cursor: cursor.updated_at)
+
def get_resumable_source_manager(self, inputs: SourceInputs) -> ResumableSourceManager[IntercomResumeConfig]:
return ResumableSourceManager[IntercomResumeConfig](inputs, IntercomResumeConfig)
@@ -244,4 +258,8 @@ class IntercomSource(ResumableSource[IntercomSourceConfig, IntercomResumeConfig]
db_incremental_field_last_value=inputs.db_incremental_field_last_value
if inputs.should_use_incremental_field
else None,
+ # Only the HITL events table keeps a source cursor; see `_conversation_hitl_events_generator`.
+ cursor_manager=self.get_cursor_manager(inputs)
+ if inputs.schema_name == "conversation_hitl_events"
+ else None,
)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom.py b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom.py
index f5eec365173b..e6016fd5a641 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom.py
@@ -8,6 +8,7 @@ from unittest import mock
from requests import Response
from requests.exceptions import HTTPError
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.cursor import SourceCursorManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (
JSONResponseCursorPaginator,
JSONResponsePaginator,
@@ -29,6 +30,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_
from products.warehouse_sources.backend.temporal.data_imports.sources.intercom import intercom as intercom_module
from products.warehouse_sources.backend.temporal.data_imports.sources.intercom.intercom import (
INTERCOM_API_BASE,
+ IntercomConversationCursor,
IntercomPagesPaginator,
IntercomResumeConfig,
IntercomSearchPaginator,
@@ -73,6 +75,8 @@ def _drive(
) -> DriveResult:
source = IntercomSource()
config = IntercomSourceConfig(intercom_integration_id=456)
+ # The pipeline gives every schema of a cursor source a manager, with no cursor on a first run.
+ inputs.setdefault("source_cursor", SourceCursorManager(IntercomConversationCursor, None, source))
with mock.patch.object(source, "get_oauth_integration", return_value=mock.MagicMock(access_token="token")):
return SourceDriver(source, config).run(endpoint, script, resume_state=resume_state, **inputs)
@@ -437,7 +441,7 @@ class TestIntercomSource:
@pytest.mark.parametrize("endpoint", list(INTERCOM_ENDPOINTS.keys()))
def test_source_response_metadata(self, endpoint: str):
cfg = INTERCOM_ENDPOINTS[endpoint]
- result = _drive(endpoint, [], api_version="2.16")
+ result = _drive(endpoint, [], api_version="2.17")
response = result.response
assert response is not None
@@ -448,6 +452,20 @@ class TestIntercomSource:
# A companies scroll expires within a minute and cannot restart mid-walk.
assert response.supports_resume is (cfg.paginator_kind != "scroll")
+ @pytest.mark.parametrize("pin", ["2.13", "2.15", "2.16", "2.17"])
+ @pytest.mark.parametrize(
+ "endpoint,response",
+ [
+ ("admins", ScriptedResponse(json={"admins": []})),
+ ("conversation_parts", ScriptedResponse(json={"conversations": [], "pages": {}})),
+ ],
+ )
+ def test_sync_requests_send_the_pinned_version(self, pin: str, endpoint: str, response: ScriptedResponse):
+ result = _drive(endpoint, [response], api_version=pin)
+
+ assert result.raised is None
+ assert [request.headers["intercom-version"] for request in result.requests] == [pin]
+
@pytest.mark.parametrize(
"endpoint", [name for name, cfg in INTERCOM_ENDPOINTS.items() if cfg.api_versions is not None]
)
@@ -598,29 +616,33 @@ class TestResumableRestEndpoints:
assert result.urls[0] == expected_url
-def _numbered_page(ids: list[int], page: int, total_pages: int) -> ScriptedResponse:
- return ScriptedResponse(
- json={
- "type": "list",
- "data": [{"id": i} for i in ids],
- "page": page,
- "per_page": 50,
- "total_pages": total_pages,
- }
- )
+def _numbered_page(ids: list[int], page: int, total_pages: int, nested: bool = False) -> ScriptedResponse:
+ counts = {"page": page, "per_page": 50, "total_pages": total_pages}
+ body: dict[str, Any] = {"type": "list", "data": [{"id": i} for i in ids]}
+ body.update({"pages": {"type": "pages", **counts}} if nested else counts)
+ return ScriptedResponse(json=body)
class TestPageNumberEndpoints:
- @pytest.mark.parametrize("endpoint", ["audiences", "content_snippets"])
- def test_fresh_run_walks_every_page_and_stops_at_total_pages(self, endpoint: str):
- responses = [_numbered_page([1], 1, 2), _numbered_page([2], 2, 2)]
+ @pytest.mark.parametrize(
+ "endpoint,per_page,nested",
+ [
+ ("audiences", "50", False),
+ ("content_snippets", "50", False),
+ ("dynamic_contents", "50", False),
+ ("folders", "50", True),
+ ("deleted_contacts", "150", True),
+ ],
+ )
+ def test_fresh_run_walks_every_page_and_stops_at_total_pages(self, endpoint: str, per_page: str, nested: bool):
+ responses = [_numbered_page([1], 1, 2, nested), _numbered_page([2], 2, 2, nested)]
result = _drive(endpoint, responses)
assert result.raised is None
assert [row["id"] for row in result.rows] == [1, 2]
# The last page is known from `total_pages`, so no trailing empty-page request goes out.
- assert result.params("per_page") == ["50", "50"]
+ assert result.params("per_page") == [per_page, per_page]
assert result.params("page") == ["1", "2"]
assert result.saved_states == [IntercomResumeConfig(page=2)]
@@ -760,3 +782,55 @@ class TestResumableConversationParts:
assert body["pagination"]["starting_after"] == "p2"
assert body["query"]["value"] == 1600000000
assert result.paths == ["/conversations/search", "/conversations/c4"]
+
+ def test_hitl_events_walk_parents_from_the_source_cursor_and_advance_it(self):
+ cursor_manager = SourceCursorManager(
+ IntercomConversationCursor, IntercomConversationCursor(updated_at=1700000000), IntercomSource()
+ )
+ rounds = [
+ {"hitl_cycle_id": "h1", "event_type": "fin_consultation_started", "created_at": 1700000050, "data": {}},
+ *(
+ {
+ "hitl_cycle_id": "h1",
+ "event_type": "fin_teammate_input_received",
+ "created_at": 1700000060,
+ "data": {"round_index": i},
+ }
+ for i in (1, 2)
+ ),
+ ]
+ result = _drive(
+ "conversation_hitl_events",
+ [
+ ScriptedResponse(
+ json={
+ "conversations": [
+ {"id": "c1", "updated_at": 1700000100},
+ {"id": "c2", "updated_at": 1700000200},
+ ],
+ "pages": {"next": None},
+ }
+ ),
+ ScriptedResponse(json={"type": "list", "data": rounds}),
+ ScriptedResponse(json={"type": "list", "data": []}),
+ ],
+ should_use_incremental_field=True,
+ incremental_field="created_at",
+ db_incremental_field_last_value=1700000060,
+ source_cursor=cursor_manager,
+ )
+
+ assert result.raised is None
+ assert [(e["conversation_id"], e["event_type"], e["round_index"]) for e in result.rows] == [
+ ("c1", "fin_consultation_started", 0),
+ ("c1", "fin_teammate_input_received", 1),
+ ("c1", "fin_teammate_input_received", 2),
+ ]
+ assert result.requests[0].json()["query"] == {"field": "updated_at", "operator": ">", "value": 1700000000}
+ assert result.paths == [
+ "/conversations/search",
+ "/conversations/c1/hitl_events",
+ "/conversations/c2/hitl_events",
+ ]
+ # A conversation with no events still moves the cursor, so the next run skips it.
+ assert cursor_manager.staged == IntercomConversationCursor(updated_at=1700000200)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom_source.py
index 5c8b0ee1768e..a8ba786516a0 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom_source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/intercom/tests/test_intercom_source.py
@@ -26,7 +26,7 @@ class TestIntercomSource:
def test_default_version_is_latest(self):
# New sources are stamped with the default; keep it on the newest supported version.
- assert self.source.default_version == "2.16"
+ assert self.source.default_version == "2.17"
assert self.source.default_version in self.source.supported_versions
@pytest.mark.parametrize(
@@ -44,19 +44,33 @@ class TestIntercomSource:
retryable_errors = self.source.get_retryable_errors()
assert any(key in error_msg for key in retryable_errors)
- @pytest.mark.parametrize("pin", ["2.13", "2.15"])
- def test_get_schemas_hides_tables_the_pinned_version_does_not_serve(self, pin: str):
+ @pytest.mark.parametrize(
+ "pin,has_macros,has_contact_activities",
+ [
+ ("2.13", False, False),
+ ("2.15", False, False),
+ ("2.16", True, False),
+ ("2.17", True, True),
+ (None, True, True),
+ ],
+ )
+ def test_get_schemas_hides_tables_the_pinned_version_does_not_serve(
+ self, pin: str | None, has_macros: bool, has_contact_activities: bool
+ ):
names = {s.name for s in self.source.get_schemas(self.config, self.team_id, api_version=pin)}
- assert names == {name for name, cfg in INTERCOM_ENDPOINTS.items() if cfg.api_versions is None}
- assert "macros" not in names
+ assert {name for name, cfg in INTERCOM_ENDPOINTS.items() if cfg.api_versions is None} <= names
+ assert ("macros" in names) is has_macros
+ assert ("contact_activities" in names) is has_contact_activities
def test_get_schemas_names_filter(self):
schemas = self.source.get_schemas(self.config, self.team_id, names=["contacts", "companies"])
assert {s.name for s in schemas} == {"contacts", "companies"}
- @pytest.mark.parametrize("pin,expected", [("2.13", "2.13"), ("2.15", "2.15"), ("2.16", "2.16"), (None, "2.16")])
+ @pytest.mark.parametrize(
+ "pin,expected", [("2.13", "2.13"), ("2.15", "2.15"), ("2.16", "2.16"), ("2.17", "2.17"), (None, "2.17")]
+ )
@mock.patch(
"products.warehouse_sources.backend.temporal.data_imports.sources.intercom.source.IntercomSource.get_oauth_integration"
)
@@ -125,6 +139,7 @@ class TestIntercomSource:
should_use_incremental_field=True,
incremental_field="updated_at",
db_incremental_field_last_value="1700000000",
+ cursor_manager=None,
)
@mock.patch("products.warehouse_sources.backend.temporal.data_imports.sources.intercom.source.intercom_source")