:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 2 missing tables to the Chatwoot source (#123743)
Tom Owers5ceb16349629
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 476707a530f5..6a9c8076022a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -2409,17 +2409,17 @@ Note: No machine-readable spec is published; I enumerated the full /reference/\*## Chatwoot — gaps-Today (11): `agents`, `contacts`, `conversations`, `custom_attribute_definitions`, `inbox_members`, `inboxes`, `labels`, `messages`, `reporting_events`, `team_members`, `teams`+Today (13): `agents`, `audit_logs`, `contact_labels`, `contacts`, `conversations`, `custom_attribute_definitions`, `inbox_members`, `inboxes`, `labels`, `messages`, `reporting_events`, `team_members`, `teams`Diffed against: <https://raw.githubusercontent.com/chatwoot/chatwoot/develop/swagger/swagger.json>- [x] `/api/v1/accounts/{account_id}/reporting_events (and .../conversations/{conversation_id}/reporting_events)` — the per-conversation event stream (first_response, conversation_resolved, reply_time) that every support SLA metric is computed from (high)- [x] `/api/v1/accounts/{account_id}/teams/{team_id}/team_members` — lookup join table resolving which agents belong to which of the already-synced teams (high)- [x] `/api/v1/accounts/{account_id}/inbox_members/{inbox_id}` — lookup join table resolving agent-to-inbox assignment for routing and load analysis (high)-- [ ] `/api/v2/accounts/{account_id}/summary_reports/agent | inbox | team | channel` — vendor-computed agent/inbox/team performance rollups (resolution counts, response times) (medium)-- [ ] `/api/v2/accounts/{account_id}/reports/conversations and /reports/summary` — conversation volume and first-response-time timeseries, the product's headline dashboard numbers (medium)-- [ ] `/api/v1/accounts/{account_id}/conversations/{conversation_id}/labels and /contacts/{id}/labels` — the label-to-conversation and label-to-contact join tables; the labels table alone cannot be joined to anything (medium)-- [ ] `/api/v1/accounts/{account_id}/audit_logs` — who changed what in the account, useful for change-vs-metric correlation (low)+- ~~`/api/v2/accounts/{account_id}/summary_reports/agent | inbox | team | channel` — vendor-computed agent/inbox/team performance rollups (resolution counts, response times) (medium)~~ — skipped: each call returns averages over a caller-chosen since/until window with no stable row identity, and `reporting_events` already syncs the per-conversation events these averages come from+- ~~`/api/v2/accounts/{account_id}/reports/conversations and /reports/summary` — conversation volume and first-response-time timeseries, the product's headline dashboard numbers (medium)~~ — skipped: neither is a timeseries. `/reports/conversations` returns live open/unattended/unassigned counts, and `/reports/summary` returns one aggregate for a caller-chosen window, both derivable from `conversations` and `reporting_events`+- [x] `/api/v1/accounts/{account_id}/conversations/{conversation_id}/labels and /contacts/{id}/labels` — the label-to-conversation and label-to-contact join tables; the labels table alone cannot be joined to anything (medium). Added as `contact_labels`, read through the contacts list's `labels[]` filter, one walk per label. Conversation labels need no table: every `conversations` row already carries a `labels` array+- [x] `/api/v1/accounts/{account_id}/audit_logs` — who changed what in the account, useful for change-vs-metric correlation (low). Added as `audit_logs`, incremental on `created_at`Note: Vendor's own swagger.json in the chatwoot/chatwoot repo, 90 paths. Coverage of the core objects is good; what's missing is the reporting/event layer and the membership join tables.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/canonical_descriptions.pyindex 7667b373c082..417656d38171 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/canonical_descriptions.py@@ -169,4 +169,36 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"updated_at": "When the definition was last updated (ISO 8601 string).",},},+ "contact_labels": {+ "description": "One label attached to one contact. Join to labels on label_id and to contacts on contact_id.",+ "docs_url": "https://developers.chatwoot.com/api-reference/contact-labels/list-labels",+ "columns": {+ "label_id": "Id of the label.",+ "label": "The label text, as it appears in the labels column of conversations.",+ "contact_id": "Id of the contact the label is attached to.",+ },+ },+ "audit_logs": {+ "description": "One create, update, or delete of a record in the account, with the user who made it and the changed values.",+ "docs_url": "https://developers.chatwoot.com/api-reference/audit-logs/list-audit-logs-in-account",+ "columns": {+ "id": "Unique identifier for the audit log entry.",+ "auditable_id": "Id of the record that changed.",+ "auditable_type": "Type of the record that changed, such as Inbox, Team, or Macro.",+ "auditable": "The record's current data. Null for message entries.",+ "associated_id": "Id of the associated object, usually the account.",+ "associated_type": "Type of the associated object.",+ "user_id": "Id of the user who made the change.",+ "user_type": "Type of the user who made the change.",+ "username": "Email or username of the user who made the change.",+ "action": "What happened to the record: create, update, or destroy.",+ "audited_changes": "The changed attributes and their values.",+ "version": "Version number of the record after this change.",+ "comment": "Optional comment stored with the entry.",+ "request_uuid": "Id of the request that made the change.",+ "remote_address": "IP address the change came from, masked unless IP logging is turned on.",+ "location": "Location derived from the IP address, when IP logging is off.",+ "created_at": "Unix timestamp when the change was made.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/chatwoot.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/chatwoot.pyindex 7852966ac09a..fa54875f0287 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/chatwoot.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/chatwoot.py@@ -48,7 +48,7 @@ MAX_MESSAGE_PAGES_PER_CONVERSATION = 10_000# hostile server multiply requests without bound (conversations × pages). This aggregate ceiling# bounds the total message pages one sync fetches regardless of how many conversations it enumerates.MAX_MESSAGE_PAGES_PER_SYNC = 1_000_000-# Member fan-out issues one request per team / inbox, so the parent list bounds the request count.+# Fan-outs issue at least one request per team / inbox / label, so the parent list bounds the request count.MAX_MEMBER_FANOUT_PARENTS = 100_000# The host is customer-controlled (self-hosted Chatwoot), so a malicious or misconfigured server@@ -67,6 +67,7 @@ RESPONSE_TOO_LARGE_ERROR = "Chatwoot response body was too large"RESPONSE_TOO_SLOW_ERROR = "Chatwoot response download was too slow"INVALID_ACCOUNT_ID_ERROR = "Chatwoot account ID must be a number"REPORTING_EVENTS_UNAVAILABLE_ERROR = "Chatwoot reporting events are not available on this instance"+AUDIT_LOGS_UNAVAILABLE_ERROR = "Chatwoot audit logs are not available on this instance"# Authority chars that let `urlparse` and the HTTP client disagree on where the host ends. A userinfo# separator or a (raw or percent-encoded) backslash in `https://169.254.169.254\@attacker.example`@@ -100,17 +101,18 @@ class ChatwootResumeConfig:# last message id already yielded within it (the next request sends `after=<message_id>`).conversation_id: int | None = Noneafter: int | None = None- # Member fan-out bookmark: the team / inbox currently being walked.+ # Fan-out bookmark: the team / inbox / label currently being walked. contact_labels pairs it+ # with `page` within that label.parent_id: int | None = None- # reporting_events only: the window end the interrupted attempt paginated against. Page+ # reporting_events and audit_logs: the window end the interrupted attempt paginated against. Page# numbers only line up against the same window, so a resumed attempt must reuse it rather# than recompute "now".until: int | None = None@frozen-class ReportingEventsWindow:- """The since/until bounds, in unix seconds, of one reporting_events page walk."""+class SinceUntilWindow:+ """The since/until bounds, in unix seconds, of one reporting_events or audit_logs page walk."""since: intuntil: int@@ -477,19 +479,22 @@ def _get_member_rows(resumable_source_manager.save_state(ChatwootResumeConfig(parent_id=remaining[index + 1]))-def _reporting_events_window(db_incremental_field_last_value: Any) -> ReportingEventsWindow:- """The unix-second bounds for one reporting_events walk.+def _since_until_window(db_incremental_field_last_value: Any) -> SinceUntilWindow:+ """The unix-second bounds for one reporting_events or audit_logs walk.- `until` is pinned at the start of the walk because Chatwoot orders this endpoint newest-first- across page numbers: without a fixed upper bound, an event created mid-walk shifts every later- page down and the rows at each boundary are never fetched. Chatwoot ignores the filter unless- both bounds are present, so a full refresh sends epoch 0 rather than omitting `since`.+ `until` is pinned at the start of the walk because page numbers only stay stable against a+ fixed row set: reporting_events is ordered newest-first, so without a fixed upper bound an+ event created mid-walk shifts every later page down and the rows at each boundary are never+ fetched. reporting_events ignores the filter unless both bounds are present, so a full+ refresh sends epoch 0 rather than omitting `since`. The watermark is an ISO string for+ reporting_events and a unix int for audit_logs."""- watermark = parse_datetime_value(db_incremental_field_last_value)- return ReportingEventsWindow(- since=max(0, int(watermark.timestamp())) if watermark is not None else 0,- until=int(datetime.now(UTC).timestamp()),- )+ if isinstance(db_incremental_field_last_value, int) and not isinstance(db_incremental_field_last_value, bool):+ since = max(0, db_incremental_field_last_value)+ else:+ watermark = parse_datetime_value(db_incremental_field_last_value)+ since = max(0, int(watermark.timestamp())) if watermark is not None else 0+ return SinceUntilWindow(since=since, until=int(datetime.now(UTC).timestamp()))def _get_reporting_event_rows(@@ -500,7 +505,7 @@ def _get_reporting_event_rows(resumable_source_manager: ResumableSourceManager[ChatwootResumeConfig],db_incremental_field_last_value: Any,) -> Iterator[list[dict[str, Any]]]:- window = _reporting_events_window(db_incremental_field_last_value)+ window = _since_until_window(db_incremental_field_last_value)resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else Nonepage = 1@@ -536,6 +541,126 @@ def _get_reporting_event_rows(logger.warning(f"Chatwoot: reporting_events hit the {MAX_LIST_PAGES}-page cap; rows beyond it were skipped")+def _get_audit_log_rows(+ session: requests.Session,+ base_url: str,+ config: ChatwootEndpointConfig,+ logger: FilteringBoundLogger,+ resumable_source_manager: ResumableSourceManager[ChatwootResumeConfig],+ db_incremental_field_last_value: Any,+) -> Iterator[list[dict[str, Any]]]:+ window = _since_until_window(db_incremental_field_last_value)++ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None+ page = 1+ if resume is not None and resume.page is not None:+ page = resume.page+ if resume.until is not None:+ window = dataclasses.replace(window, until=resume.until)+ logger.debug(f"Chatwoot: resuming audit_logs from page {page} of the window ending {window.until}")++ while page <= MAX_LIST_PAGES:+ params = {**config.params, "since": window.since, "until": window.until, "page": page}+ url = _build_url(base_url, config.path, params)+ try:+ data = _fetch_json(session, url, logger)+ except requests.HTTPError as exc:+ # The route only exists on enterprise builds, so a 404 means the instance has no audit+ # logs, not that the account id is wrong.+ if exc.response is not None and exc.response.status_code == 404:+ raise Exception(f"{AUDIT_LOGS_UNAVAILABLE_ERROR}: GET {config.path} returned 404") from exc+ raise++ items = _extract_items(data, config.data_path, url)+ if not items:+ return++ page += 1+ resumable_source_manager.save_state(ChatwootResumeConfig(page=page, until=window.until))+ yield items++ total_entries = data.get("total_entries") if isinstance(data, dict) else None+ per_page = data.get("per_page") if isinstance(data, dict) else None+ if isinstance(total_entries, int) and isinstance(per_page, int) and per_page > 0:+ if (page - 1) * per_page >= total_entries:+ return+ logger.warning(f"Chatwoot: audit_logs hit the {MAX_LIST_PAGES}-page cap; rows beyond it were skipped")+++def _list_labels(session: requests.Session, base_url: str, logger: FilteringBoundLogger) -> list[tuple[int, str]]:+ """(id, title) of every label in the account, the parents of the contact_labels walk."""+ labels = CHATWOOT_ENDPOINTS["labels"]+ url = _build_url(base_url, labels.path, dict(labels.params))+ items = _extract_items(_fetch_json(session, url, logger), labels.data_path, url)+ result = [+ (item["id"], item["title"])+ for item in items+ if item.get("id") is not None and isinstance(item.get("title"), str) and item["title"]+ ]+ if len(result) > MAX_MEMBER_FANOUT_PARENTS:+ logger.warning(+ f"Chatwoot: labels returned more than the {MAX_MEMBER_FANOUT_PARENTS}-parent fan-out cap; "+ "rows beyond it were skipped"+ )+ return result[:MAX_MEMBER_FANOUT_PARENTS]+ return result+++def _get_contact_label_rows(+ session: requests.Session,+ base_url: str,+ config: ChatwootEndpointConfig,+ logger: FilteringBoundLogger,+ resumable_source_manager: ResumableSourceManager[ChatwootResumeConfig],+) -> Iterator[list[dict[str, Any]]]:+ """Page the contacts list once per label, filtered to that label, emitting one row per pair.++ The `labels[]` filter on the contacts list costs one request per 15 tagged contacts, where the+ per-contact /contacts/{id}/labels endpoint would cost one request per contact.+ """+ labels = _list_labels(session, base_url, logger)+ label_ids = [label_id for label_id, _ in labels]++ # If the bookmarked label is gone, start over from the first one — merge dedupes re-pulled+ # rows on the primary key.+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None+ remaining = labels+ resume_page: int | None = None+ if resume is not None and resume.parent_id is not None and resume.parent_id in label_ids:+ remaining = labels[label_ids.index(resume.parent_id) :]+ resume_page = resume.page+ logger.debug(f"Chatwoot: resuming contact_labels from label_id={resume.parent_id}, page={resume_page}")++ for index, (label_id, title) in enumerate(remaining):+ page = resume_page or 1+ resume_page = None # only the resumed-into label uses the saved page++ if index > 0:+ # Every row of the previous label is yielded, so this label is a safe place to resume+ # from. A run of labels with no contacts would otherwise make many requests with no yield.+ resumable_source_manager.save_state(ChatwootResumeConfig(parent_id=label_id, page=1))+ resumable_source_manager.safe_point()++ while page <= MAX_LIST_PAGES:+ url = _build_url(base_url, config.path, {**config.params, "labels[]": title, "page": page})+ items = _extract_items(_fetch_json(session, url, logger), config.data_path, url)+ if not items:+ break+ rows = [+ {"label_id": label_id, "label": title, "contact_id": item["id"]}+ for item in items+ if item.get("id") is not None+ ]+ page += 1+ resumable_source_manager.save_state(ChatwootResumeConfig(parent_id=label_id, page=page))+ if rows:+ yield rows+ else:+ logger.warning(+ f"Chatwoot: label {label_id} hit the {MAX_LIST_PAGES}-page contact cap; rows beyond it were skipped"+ )++def get_rows(host: str | None,account_id: str | int | None,@@ -574,6 +699,16 @@ def get_rows()return+ if config.kind == "audit_logs":+ yield from _get_audit_log_rows(+ session, base_url, config, logger, resumable_source_manager, db_incremental_field_last_value+ )+ return++ if config.kind == "contact_labels":+ yield from _get_contact_label_rows(session, base_url, config, logger, resumable_source_manager)+ return+resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else Nonepage = resume.page if resume is not None and resume.page is not None else 1if resume is not None:diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/settings.pyindex 1adbf42addbf..0c4aa95e79e2 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/settings.py@@ -10,6 +10,8 @@ DEFAULT_HOST = "https://app.chatwoot.com"CONTACTS_PAGE_SIZE = 15# `after`-cursor pages on the messages endpoint return up to 100 rows (MessageFinder#messages_after).MESSAGES_PAGE_SIZE = 100+# The contact_labels walk reads the contacts list once per label, filtered to that label.+CONTACT_LABELS_PARAMS = {"sort": "created_at", "include_contact_inboxes": "false"}# The REST API returns `message_type` as an integer enum; webhook payloads carry the string form.MESSAGE_TYPE_TO_INT = {"incoming": 0, "outgoing": 1, "activity": 2, "template": 3}@@ -22,10 +24,11 @@ class ChatwootEndpointConfig:# placeholder filled from the parent endpoint's rows.path: str# "paged": page-number list. "single": one unpaginated response. "messages": the- # per-conversation fan-out with an id-based `after` cursor. "reporting_events": a- # page-number list bounded by a since/until window. "member_fanout": one unpaginated- # request per parent row, stamping the parent id onto each child row.- kind: Literal["paged", "single", "messages", "reporting_events", "member_fanout"]+ # per-conversation fan-out with an id-based `after` cursor. "reporting_events" and+ # "audit_logs": page-number lists bounded by a since/until window. "member_fanout": one+ # unpaginated request per parent row, stamping the parent id onto each child row.+ # "contact_labels": a paged contacts walk per label, one row per (label, contact) pair.+ kind: Literal["paged", "single", "messages", "reporting_events", "audit_logs", "member_fanout", "contact_labels"]# Where the row list lives in the response body; () means the body is a bare JSON array.data_path: tuple[str, ...] = ()# Query params sent on every request.@@ -38,16 +41,16 @@ class ChatwootEndpointConfig:description: Optional[str] = Noneincremental_fields: list[IncrementalField] = field(default_factory=list)# Order rows arrive in, which the pipeline's watermark handling depends on.- sort_mode: SortMode = "asc"+ sort_mode: Optional[SortMode] = "asc"# member_fanout only: the endpoint whose rows supply {parent_id}, and the column each child# row records it in. Agent ids repeat across parents, so the column is part of the primary key.parent: Optional[str] = Noneparent_id_field: Optional[str] = None-# Only reporting_events declares incremental sync: it is the one endpoint with a server-side-# timestamp filter (since/until on created_at). Every other list endpoint is a full walk, and-# webhooks are the delta path for conversations and messages.+# Only reporting_events and audit_logs declare incremental sync: they are the endpoints with a+# server-side timestamp filter (since/until on created_at). Every other list endpoint is a full+# walk, and webhooks are the delta path for conversations and messages.CHATWOOT_ENDPOINTS: dict[str, ChatwootEndpointConfig] = {"conversations": ChatwootEndpointConfig(name="conversations",@@ -163,6 +166,45 @@ CHATWOOT_ENDPOINTS: dict[str, ChatwootEndpointConfig] = {kind="single",description="Custom attribute definitions for conversations, contacts, and companies.",),+ "contact_labels": ChatwootEndpointConfig(+ name="contact_labels",+ path="/contacts",+ kind="contact_labels",+ data_path=("payload",),+ params=CONTACT_LABELS_PARAMS,+ parent="labels",+ parent_id_field="label_id",+ primary_keys=["label_id", "contact_id"],+ description=(+ "Which labels are attached to which contacts. One row per label on a contact, joinable "+ "to labels on label_id and to contacts on contact_id. Conversation labels are already "+ "in the labels column of the conversations table."+ ),+ ),+ "audit_logs": ChatwootEndpointConfig(+ name="audit_logs",+ path="/audit_logs",+ kind="audit_logs",+ data_path=("audit_logs",),+ params={"sort": "asc"},+ partition_key="created_at",+ incremental_fields=[+ {+ "label": "created_at",+ "type": IncrementalFieldType.DateTime,+ "field": "created_at",+ "field_type": IncrementalFieldType.Integer,+ }+ ],+ # Older self-hosted instances ignore sort/since/until and return newest-first, so the+ # order is not guaranteed.+ sort_mode=None,+ description=(+ "Who created, changed, or deleted what in the account, with the changed values. "+ "Requires an administrator token, and the endpoint is absent on Chatwoot "+ "community-edition builds."+ ),+ ),}ENDPOINTS = tuple(CHATWOOT_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/source.pyindex db622baceba6..acd0325b990d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/source.py@@ -15,6 +15,7 @@ from products.warehouse_sources.backend.facade.source_config import (SourceFieldInputConfigType,)from products.warehouse_sources.backend.temporal.data_imports.sources.chatwoot.chatwoot import (+ AUDIT_LOGS_UNAVAILABLE_ERROR,HOST_NOT_ALLOWED_ERROR,HTTP_NOT_ALLOWED_ERROR,INVALID_ACCOUNT_ID_ERROR,@@ -188,6 +189,7 @@ If automatic creation failed, note that only Chatwoot administrators can manage# so retrying never changes the outcome."422 Client Error": "Chatwoot rejected a request while syncing this data. Please contact support if this persists.",REPORTING_EVENTS_UNAVAILABLE_ERROR: "This Chatwoot instance does not serve reporting events. The endpoint only exists on Chatwoot Cloud and enterprise builds, so deselect the reporting_events table to sync the rest.",+ AUDIT_LOGS_UNAVAILABLE_ERROR: "This Chatwoot instance does not serve audit logs. The endpoint only exists on Chatwoot Cloud and enterprise builds, so deselect the audit_logs table to sync the rest.",INVALID_ACCOUNT_ID_ERROR: "The Chatwoot account ID must be a number (the number in your Chatwoot URL, e.g. app.chatwoot.com/app/accounts/1). Update the source configuration.",HOST_NOT_ALLOWED_ERROR: "The Chatwoot host is not allowed. Please use a publicly reachable instance URL.",HTTP_NOT_ALLOWED_ERROR: "The Chatwoot host must use HTTPS. Please update the instance URL to use https://.",@@ -204,8 +206,8 @@ If automatic creation failed, note that only Chatwoot administrators can manageforce_refresh: bool = False,api_version: str | None = None,) -> list[SourceSchema]:- # Only reporting_events has a server-side timestamp filter to sync incrementally against;- # the rest are full refresh, and conversations and messages also support webhook-fed deltas.+ # Only reporting_events and audit_logs have a server-side timestamp filter to sync+ # incrementally against; the rest are full refresh, and conversations and messages also support webhook-fed deltas.schemas = [SourceSchema(name=endpoint,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/tests/test_chatwoot.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/tests/test_chatwoot.pyindex 48ea36d64c71..9cbdf0200488 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/tests/test_chatwoot.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/tests/test_chatwoot.py@@ -9,6 +9,7 @@ import requestsfrom products.warehouse_sources.backend.temporal.data_imports.pipelines.core.arrow_utils import table_from_py_listfrom products.warehouse_sources.backend.temporal.data_imports.sources.chatwoot.chatwoot import (+ AUDIT_LOGS_UNAVAILABLE_ERROR,REPORTING_EVENTS_UNAVAILABLE_ERROR,ChatwootResumeConfig,_normalize_account_id,@@ -413,6 +414,123 @@ class TestGetRowsReportingEvents:]+class TestGetRowsAuditLogs:+ @pytest.mark.parametrize(+ "watermark, expected_since",+ [+ (None, 0),+ # The created_at watermark is the unix int the API returns.+ (1767323045, 1767323045),+ ("2026-01-02T03:04:05+00:00", 1767323045),+ ("not-a-date", 0),+ ],+ )+ def test_walks_pages_within_a_pinned_ascending_window(self, watermark, expected_since):+ result = _driver().run(+ "audit_logs",+ [+ ScriptedResponse(+ json={"per_page": 2, "total_entries": 3, "current_page": 1, "audit_logs": [{"id": 1}, {"id": 2}]}+ ),+ ScriptedResponse(+ json={"per_page": 2, "total_entries": 3, "current_page": 2, "audit_logs": [{"id": 3}]}+ ),+ ],+ incremental_field="created_at",+ db_incremental_field_last_value=watermark,+ )++ assert [row["id"] for row in result.rows] == [1, 2, 3]+ # total_entries ends the walk, so no third request is made.+ assert result.params("page") == ["1", "2"]+ assert result.params("sort") == ["asc", "asc"]+ assert result.params("since") == [str(expected_since)] * 2+ untils = set(result.params("until"))+ assert len(untils) == 1+ until = int(untils.pop() or "0")+ assert result.committed_states == [+ ChatwootResumeConfig(page=2, until=until),+ ChatwootResumeConfig(page=3, until=until),+ ]++ def test_missing_totals_fall_back_to_the_empty_page(self):+ result = _driver().run(+ "audit_logs",+ [ScriptedResponse(json={"audit_logs": [{"id": 1}]}), ScriptedResponse(json={"audit_logs": []})],+ )++ assert [row["id"] for row in result.rows] == [1]+ assert result.params("page") == ["1", "2"]++ def test_resume_reuses_the_saved_page_and_window(self):+ result = _driver().run(+ "audit_logs",+ [ScriptedResponse(json={"audit_logs": []})],+ resume_state=ChatwootResumeConfig(page=4, until=1767000000),+ incremental_field="created_at",+ db_incremental_field_last_value=1766000000,+ )++ assert result.params("page") == ["4"]+ assert result.params("until") == ["1767000000"]+ assert result.params("since") == ["1766000000"]++ def test_404_reports_the_endpoint_as_unavailable_not_a_missing_account(self):+ result = _driver().run("audit_logs", [ScriptedResponse(status=404, json={"error": "Not Found"})])++ assert isinstance(result.raised, Exception)+ assert AUDIT_LOGS_UNAVAILABLE_ERROR in str(result.raised)+++class TestGetRowsContactLabels:+ LABELS = {"payload": [{"id": 1, "title": "vip"}, {"id": 2, "title": "churn_risk"}, {"id": 3, "title": "beta"}]}++ def test_walks_each_label_and_commits_past_labels_with_no_contacts(self):+ result = _driver().run(+ "contact_labels",+ [+ ScriptedResponse(json=self.LABELS),+ ScriptedResponse(json={"meta": {}, "payload": [{"id": 10, "email": "[email protected]"}, {"id": 11}]}),+ ScriptedResponse(json={"meta": {}, "payload": []}),+ ScriptedResponse(json={"meta": {}, "payload": []}),+ ScriptedResponse(json={"meta": {}, "payload": [{"id": 10}]}),+ ScriptedResponse(json={"meta": {}, "payload": []}),+ ],+ )++ # A slim join row: the contact's own columns stay in the contacts table.+ assert result.rows == [+ {"label_id": 1, "label": "vip", "contact_id": 10},+ {"label_id": 1, "label": "vip", "contact_id": 11},+ {"label_id": 3, "label": "beta", "contact_id": 10},+ ]+ assert result.paths == ["/api/v1/accounts/1/labels", *["/api/v1/accounts/1/contacts"] * 5]+ assert result.params("labels[]") == [None, "vip", "vip", "churn_risk", "beta", "beta"]+ assert result.params("page") == [None, "1", "2", "1", "1", "2"]+ # Label 2 yields no rows, so only the safe point commits the move on to label 3.+ assert ChatwootResumeConfig(parent_id=3, page=1) in result.committed_states++ @pytest.mark.parametrize(+ "resume_state, expected_requests",+ [+ (ChatwootResumeConfig(parent_id=2, page=3), [("churn_risk", "3"), ("beta", "1")]),+ # A bookmark for a label that no longer exists must restart the walk, not skip everything.+ (ChatwootResumeConfig(parent_id=999, page=3), [("vip", "1"), ("churn_risk", "1"), ("beta", "1")]),+ ],+ )+ def test_resumes_from_bookmarked_label_and_page(self, resume_state, expected_requests):+ result = _driver().run(+ "contact_labels",+ [+ ScriptedResponse(json=self.LABELS),+ *[ScriptedResponse(json={"meta": {}, "payload": []}) for _ in expected_requests],+ ],+ resume_state=resume_state,+ )++ assert list(zip(result.params("labels[]"), result.params("page")))[1:] == expected_requests++class TestWebhookTableTransformer:def test_message_events_are_normalized_to_the_rest_shape(self):transform = make_webhook_table_transformer("messages")