Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 2 missing tables to the Chatwoot source (#123743)

Tom Owers5ceb16349629
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 476707a530f5..6a9c8076022a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -2409,17 +2409,17 @@ Note: No machine-readable spec is published; I enumerated the full /reference/\*
## Chatwoot — gaps
-Today (11): `agents`, `contacts`, `conversations`, `custom_attribute_definitions`, `inbox_members`, `inboxes`, `labels`, `messages`, `reporting_events`, `team_members`, `teams`
+Today (13): `agents`, `audit_logs`, `contact_labels`, `contacts`, `conversations`, `custom_attribute_definitions`, `inbox_members`, `inboxes`, `labels`, `messages`, `reporting_events`, `team_members`, `teams`
Diffed against: <https://raw.githubusercontent.com/chatwoot/chatwoot/develop/swagger/swagger.json>
- [x] `/api/v1/accounts/{account_id}/reporting_events (and .../conversations/{conversation_id}/reporting_events)` — the per-conversation event stream (first_response, conversation_resolved, reply_time) that every support SLA metric is computed from (high)
- [x] `/api/v1/accounts/{account_id}/teams/{team_id}/team_members` — lookup join table resolving which agents belong to which of the already-synced teams (high)
- [x] `/api/v1/accounts/{account_id}/inbox_members/{inbox_id}` — lookup join table resolving agent-to-inbox assignment for routing and load analysis (high)
-- [ ] `/api/v2/accounts/{account_id}/summary_reports/agent | inbox | team | channel` — vendor-computed agent/inbox/team performance rollups (resolution counts, response times) (medium)
-- [ ] `/api/v2/accounts/{account_id}/reports/conversations and /reports/summary` — conversation volume and first-response-time timeseries, the product's headline dashboard numbers (medium)
-- [ ] `/api/v1/accounts/{account_id}/conversations/{conversation_id}/labels and /contacts/{id}/labels` — the label-to-conversation and label-to-contact join tables; the labels table alone cannot be joined to anything (medium)
-- [ ] `/api/v1/accounts/{account_id}/audit_logs` — who changed what in the account, useful for change-vs-metric correlation (low)
+- ~~`/api/v2/accounts/{account_id}/summary_reports/agent | inbox | team | channel` — vendor-computed agent/inbox/team performance rollups (resolution counts, response times) (medium)~~ — skipped: each call returns averages over a caller-chosen since/until window with no stable row identity, and `reporting_events` already syncs the per-conversation events these averages come from
+- ~~`/api/v2/accounts/{account_id}/reports/conversations and /reports/summary` — conversation volume and first-response-time timeseries, the product's headline dashboard numbers (medium)~~ — skipped: neither is a timeseries. `/reports/conversations` returns live open/unattended/unassigned counts, and `/reports/summary` returns one aggregate for a caller-chosen window, both derivable from `conversations` and `reporting_events`
+- [x] `/api/v1/accounts/{account_id}/conversations/{conversation_id}/labels and /contacts/{id}/labels` — the label-to-conversation and label-to-contact join tables; the labels table alone cannot be joined to anything (medium). Added as `contact_labels`, read through the contacts list's `labels[]` filter, one walk per label. Conversation labels need no table: every `conversations` row already carries a `labels` array
+- [x] `/api/v1/accounts/{account_id}/audit_logs` — who changed what in the account, useful for change-vs-metric correlation (low). Added as `audit_logs`, incremental on `created_at`
Note: Vendor's own swagger.json in the chatwoot/chatwoot repo, 90 paths. Coverage of the core objects is good; what's missing is the reporting/event layer and the membership join tables.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/canonical_descriptions.py
index 7667b373c082..417656d38171 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/canonical_descriptions.py
@@ -169,4 +169,36 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"updated_at": "When the definition was last updated (ISO 8601 string).",
},
},
+ "contact_labels": {
+ "description": "One label attached to one contact. Join to labels on label_id and to contacts on contact_id.",
+ "docs_url": "https://developers.chatwoot.com/api-reference/contact-labels/list-labels",
+ "columns": {
+ "label_id": "Id of the label.",
+ "label": "The label text, as it appears in the labels column of conversations.",
+ "contact_id": "Id of the contact the label is attached to.",
+ },
+ },
+ "audit_logs": {
+ "description": "One create, update, or delete of a record in the account, with the user who made it and the changed values.",
+ "docs_url": "https://developers.chatwoot.com/api-reference/audit-logs/list-audit-logs-in-account",
+ "columns": {
+ "id": "Unique identifier for the audit log entry.",
+ "auditable_id": "Id of the record that changed.",
+ "auditable_type": "Type of the record that changed, such as Inbox, Team, or Macro.",
+ "auditable": "The record's current data. Null for message entries.",
+ "associated_id": "Id of the associated object, usually the account.",
+ "associated_type": "Type of the associated object.",
+ "user_id": "Id of the user who made the change.",
+ "user_type": "Type of the user who made the change.",
+ "username": "Email or username of the user who made the change.",
+ "action": "What happened to the record: create, update, or destroy.",
+ "audited_changes": "The changed attributes and their values.",
+ "version": "Version number of the record after this change.",
+ "comment": "Optional comment stored with the entry.",
+ "request_uuid": "Id of the request that made the change.",
+ "remote_address": "IP address the change came from, masked unless IP logging is turned on.",
+ "location": "Location derived from the IP address, when IP logging is off.",
+ "created_at": "Unix timestamp when the change was made.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/chatwoot.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/chatwoot.py
index 7852966ac09a..fa54875f0287 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/chatwoot.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/chatwoot.py
@@ -48,7 +48,7 @@ MAX_MESSAGE_PAGES_PER_CONVERSATION = 10_000
# hostile server multiply requests without bound (conversations × pages). This aggregate ceiling
# bounds the total message pages one sync fetches regardless of how many conversations it enumerates.
MAX_MESSAGE_PAGES_PER_SYNC = 1_000_000
-# Member fan-out issues one request per team / inbox, so the parent list bounds the request count.
+# Fan-outs issue at least one request per team / inbox / label, so the parent list bounds the request count.
MAX_MEMBER_FANOUT_PARENTS = 100_000
# The host is customer-controlled (self-hosted Chatwoot), so a malicious or misconfigured server
@@ -67,6 +67,7 @@ RESPONSE_TOO_LARGE_ERROR = "Chatwoot response body was too large"
RESPONSE_TOO_SLOW_ERROR = "Chatwoot response download was too slow"
INVALID_ACCOUNT_ID_ERROR = "Chatwoot account ID must be a number"
REPORTING_EVENTS_UNAVAILABLE_ERROR = "Chatwoot reporting events are not available on this instance"
+AUDIT_LOGS_UNAVAILABLE_ERROR = "Chatwoot audit logs are not available on this instance"
# Authority chars that let `urlparse` and the HTTP client disagree on where the host ends. A userinfo
# separator or a (raw or percent-encoded) backslash in `https://169.254.169.254\@attacker.example`
@@ -100,17 +101,18 @@ class ChatwootResumeConfig:
# last message id already yielded within it (the next request sends `after=<message_id>`).
conversation_id: int | None = None
after: int | None = None
- # Member fan-out bookmark: the team / inbox currently being walked.
+ # Fan-out bookmark: the team / inbox / label currently being walked. contact_labels pairs it
+ # with `page` within that label.
parent_id: int | None = None
- # reporting_events only: the window end the interrupted attempt paginated against. Page
+ # reporting_events and audit_logs: the window end the interrupted attempt paginated against. Page
# numbers only line up against the same window, so a resumed attempt must reuse it rather
# than recompute "now".
until: int | None = None
@frozen
-class ReportingEventsWindow:
- """The since/until bounds, in unix seconds, of one reporting_events page walk."""
+class SinceUntilWindow:
+ """The since/until bounds, in unix seconds, of one reporting_events or audit_logs page walk."""
since: int
until: int
@@ -477,19 +479,22 @@ def _get_member_rows(
resumable_source_manager.save_state(ChatwootResumeConfig(parent_id=remaining[index + 1]))
-def _reporting_events_window(db_incremental_field_last_value: Any) -> ReportingEventsWindow:
- """The unix-second bounds for one reporting_events walk.
+def _since_until_window(db_incremental_field_last_value: Any) -> SinceUntilWindow:
+ """The unix-second bounds for one reporting_events or audit_logs walk.
- `until` is pinned at the start of the walk because Chatwoot orders this endpoint newest-first
- across page numbers: without a fixed upper bound, an event created mid-walk shifts every later
- page down and the rows at each boundary are never fetched. Chatwoot ignores the filter unless
- both bounds are present, so a full refresh sends epoch 0 rather than omitting `since`.
+ `until` is pinned at the start of the walk because page numbers only stay stable against a
+ fixed row set: reporting_events is ordered newest-first, so without a fixed upper bound an
+ event created mid-walk shifts every later page down and the rows at each boundary are never
+ fetched. reporting_events ignores the filter unless both bounds are present, so a full
+ refresh sends epoch 0 rather than omitting `since`. The watermark is an ISO string for
+ reporting_events and a unix int for audit_logs.
"""
- watermark = parse_datetime_value(db_incremental_field_last_value)
- return ReportingEventsWindow(
- since=max(0, int(watermark.timestamp())) if watermark is not None else 0,
- until=int(datetime.now(UTC).timestamp()),
- )
+ if isinstance(db_incremental_field_last_value, int) and not isinstance(db_incremental_field_last_value, bool):
+ since = max(0, db_incremental_field_last_value)
+ else:
+ watermark = parse_datetime_value(db_incremental_field_last_value)
+ since = max(0, int(watermark.timestamp())) if watermark is not None else 0
+ return SinceUntilWindow(since=since, until=int(datetime.now(UTC).timestamp()))
def _get_reporting_event_rows(
@@ -500,7 +505,7 @@ def _get_reporting_event_rows(
resumable_source_manager: ResumableSourceManager[ChatwootResumeConfig],
db_incremental_field_last_value: Any,
) -> Iterator[list[dict[str, Any]]]:
- window = _reporting_events_window(db_incremental_field_last_value)
+ window = _since_until_window(db_incremental_field_last_value)
resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
page = 1
@@ -536,6 +541,126 @@ def _get_reporting_event_rows(
logger.warning(f"Chatwoot: reporting_events hit the {MAX_LIST_PAGES}-page cap; rows beyond it were skipped")
+def _get_audit_log_rows(
+ session: requests.Session,
+ base_url: str,
+ config: ChatwootEndpointConfig,
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[ChatwootResumeConfig],
+ db_incremental_field_last_value: Any,
+) -> Iterator[list[dict[str, Any]]]:
+ window = _since_until_window(db_incremental_field_last_value)
+
+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
+ page = 1
+ if resume is not None and resume.page is not None:
+ page = resume.page
+ if resume.until is not None:
+ window = dataclasses.replace(window, until=resume.until)
+ logger.debug(f"Chatwoot: resuming audit_logs from page {page} of the window ending {window.until}")
+
+ while page <= MAX_LIST_PAGES:
+ params = {**config.params, "since": window.since, "until": window.until, "page": page}
+ url = _build_url(base_url, config.path, params)
+ try:
+ data = _fetch_json(session, url, logger)
+ except requests.HTTPError as exc:
+ # The route only exists on enterprise builds, so a 404 means the instance has no audit
+ # logs, not that the account id is wrong.
+ if exc.response is not None and exc.response.status_code == 404:
+ raise Exception(f"{AUDIT_LOGS_UNAVAILABLE_ERROR}: GET {config.path} returned 404") from exc
+ raise
+
+ items = _extract_items(data, config.data_path, url)
+ if not items:
+ return
+
+ page += 1
+ resumable_source_manager.save_state(ChatwootResumeConfig(page=page, until=window.until))
+ yield items
+
+ total_entries = data.get("total_entries") if isinstance(data, dict) else None
+ per_page = data.get("per_page") if isinstance(data, dict) else None
+ if isinstance(total_entries, int) and isinstance(per_page, int) and per_page > 0:
+ if (page - 1) * per_page >= total_entries:
+ return
+ logger.warning(f"Chatwoot: audit_logs hit the {MAX_LIST_PAGES}-page cap; rows beyond it were skipped")
+
+
+def _list_labels(session: requests.Session, base_url: str, logger: FilteringBoundLogger) -> list[tuple[int, str]]:
+ """(id, title) of every label in the account, the parents of the contact_labels walk."""
+ labels = CHATWOOT_ENDPOINTS["labels"]
+ url = _build_url(base_url, labels.path, dict(labels.params))
+ items = _extract_items(_fetch_json(session, url, logger), labels.data_path, url)
+ result = [
+ (item["id"], item["title"])
+ for item in items
+ if item.get("id") is not None and isinstance(item.get("title"), str) and item["title"]
+ ]
+ if len(result) > MAX_MEMBER_FANOUT_PARENTS:
+ logger.warning(
+ f"Chatwoot: labels returned more than the {MAX_MEMBER_FANOUT_PARENTS}-parent fan-out cap; "
+ "rows beyond it were skipped"
+ )
+ return result[:MAX_MEMBER_FANOUT_PARENTS]
+ return result
+
+
+def _get_contact_label_rows(
+ session: requests.Session,
+ base_url: str,
+ config: ChatwootEndpointConfig,
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[ChatwootResumeConfig],
+) -> Iterator[list[dict[str, Any]]]:
+ """Page the contacts list once per label, filtered to that label, emitting one row per pair.
+
+ The `labels[]` filter on the contacts list costs one request per 15 tagged contacts, where the
+ per-contact /contacts/{id}/labels endpoint would cost one request per contact.
+ """
+ labels = _list_labels(session, base_url, logger)
+ label_ids = [label_id for label_id, _ in labels]
+
+ # If the bookmarked label is gone, start over from the first one — merge dedupes re-pulled
+ # rows on the primary key.
+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
+ remaining = labels
+ resume_page: int | None = None
+ if resume is not None and resume.parent_id is not None and resume.parent_id in label_ids:
+ remaining = labels[label_ids.index(resume.parent_id) :]
+ resume_page = resume.page
+ logger.debug(f"Chatwoot: resuming contact_labels from label_id={resume.parent_id}, page={resume_page}")
+
+ for index, (label_id, title) in enumerate(remaining):
+ page = resume_page or 1
+ resume_page = None # only the resumed-into label uses the saved page
+
+ if index > 0:
+ # Every row of the previous label is yielded, so this label is a safe place to resume
+ # from. A run of labels with no contacts would otherwise make many requests with no yield.
+ resumable_source_manager.save_state(ChatwootResumeConfig(parent_id=label_id, page=1))
+ resumable_source_manager.safe_point()
+
+ while page <= MAX_LIST_PAGES:
+ url = _build_url(base_url, config.path, {**config.params, "labels[]": title, "page": page})
+ items = _extract_items(_fetch_json(session, url, logger), config.data_path, url)
+ if not items:
+ break
+ rows = [
+ {"label_id": label_id, "label": title, "contact_id": item["id"]}
+ for item in items
+ if item.get("id") is not None
+ ]
+ page += 1
+ resumable_source_manager.save_state(ChatwootResumeConfig(parent_id=label_id, page=page))
+ if rows:
+ yield rows
+ else:
+ logger.warning(
+ f"Chatwoot: label {label_id} hit the {MAX_LIST_PAGES}-page contact cap; rows beyond it were skipped"
+ )
+
+
def get_rows(
host: str | None,
account_id: str | int | None,
@@ -574,6 +699,16 @@ def get_rows(
)
return
+ if config.kind == "audit_logs":
+ yield from _get_audit_log_rows(
+ session, base_url, config, logger, resumable_source_manager, db_incremental_field_last_value
+ )
+ return
+
+ if config.kind == "contact_labels":
+ yield from _get_contact_label_rows(session, base_url, config, logger, resumable_source_manager)
+ return
+
resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
page = resume.page if resume is not None and resume.page is not None else 1
if resume is not None:
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/settings.py
index 1adbf42addbf..0c4aa95e79e2 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/settings.py
@@ -10,6 +10,8 @@ DEFAULT_HOST = "https://app.chatwoot.com"
CONTACTS_PAGE_SIZE = 15
# `after`-cursor pages on the messages endpoint return up to 100 rows (MessageFinder#messages_after).
MESSAGES_PAGE_SIZE = 100
+# The contact_labels walk reads the contacts list once per label, filtered to that label.
+CONTACT_LABELS_PARAMS = {"sort": "created_at", "include_contact_inboxes": "false"}
# The REST API returns `message_type` as an integer enum; webhook payloads carry the string form.
MESSAGE_TYPE_TO_INT = {"incoming": 0, "outgoing": 1, "activity": 2, "template": 3}
@@ -22,10 +24,11 @@ class ChatwootEndpointConfig:
# placeholder filled from the parent endpoint's rows.
path: str
# "paged": page-number list. "single": one unpaginated response. "messages": the
- # per-conversation fan-out with an id-based `after` cursor. "reporting_events": a
- # page-number list bounded by a since/until window. "member_fanout": one unpaginated
- # request per parent row, stamping the parent id onto each child row.
- kind: Literal["paged", "single", "messages", "reporting_events", "member_fanout"]
+ # per-conversation fan-out with an id-based `after` cursor. "reporting_events" and
+ # "audit_logs": page-number lists bounded by a since/until window. "member_fanout": one
+ # unpaginated request per parent row, stamping the parent id onto each child row.
+ # "contact_labels": a paged contacts walk per label, one row per (label, contact) pair.
+ kind: Literal["paged", "single", "messages", "reporting_events", "audit_logs", "member_fanout", "contact_labels"]
# Where the row list lives in the response body; () means the body is a bare JSON array.
data_path: tuple[str, ...] = ()
# Query params sent on every request.
@@ -38,16 +41,16 @@ class ChatwootEndpointConfig:
description: Optional[str] = None
incremental_fields: list[IncrementalField] = field(default_factory=list)
# Order rows arrive in, which the pipeline's watermark handling depends on.
- sort_mode: SortMode = "asc"
+ sort_mode: Optional[SortMode] = "asc"
# member_fanout only: the endpoint whose rows supply {parent_id}, and the column each child
# row records it in. Agent ids repeat across parents, so the column is part of the primary key.
parent: Optional[str] = None
parent_id_field: Optional[str] = None
-# Only reporting_events declares incremental sync: it is the one endpoint with a server-side
-# timestamp filter (since/until on created_at). Every other list endpoint is a full walk, and
-# webhooks are the delta path for conversations and messages.
+# Only reporting_events and audit_logs declare incremental sync: they are the endpoints with a
+# server-side timestamp filter (since/until on created_at). Every other list endpoint is a full
+# walk, and webhooks are the delta path for conversations and messages.
CHATWOOT_ENDPOINTS: dict[str, ChatwootEndpointConfig] = {
"conversations": ChatwootEndpointConfig(
name="conversations",
@@ -163,6 +166,45 @@ CHATWOOT_ENDPOINTS: dict[str, ChatwootEndpointConfig] = {
kind="single",
description="Custom attribute definitions for conversations, contacts, and companies.",
),
+ "contact_labels": ChatwootEndpointConfig(
+ name="contact_labels",
+ path="/contacts",
+ kind="contact_labels",
+ data_path=("payload",),
+ params=CONTACT_LABELS_PARAMS,
+ parent="labels",
+ parent_id_field="label_id",
+ primary_keys=["label_id", "contact_id"],
+ description=(
+ "Which labels are attached to which contacts. One row per label on a contact, joinable "
+ "to labels on label_id and to contacts on contact_id. Conversation labels are already "
+ "in the labels column of the conversations table."
+ ),
+ ),
+ "audit_logs": ChatwootEndpointConfig(
+ name="audit_logs",
+ path="/audit_logs",
+ kind="audit_logs",
+ data_path=("audit_logs",),
+ params={"sort": "asc"},
+ partition_key="created_at",
+ incremental_fields=[
+ {
+ "label": "created_at",
+ "type": IncrementalFieldType.DateTime,
+ "field": "created_at",
+ "field_type": IncrementalFieldType.Integer,
+ }
+ ],
+ # Older self-hosted instances ignore sort/since/until and return newest-first, so the
+ # order is not guaranteed.
+ sort_mode=None,
+ description=(
+ "Who created, changed, or deleted what in the account, with the changed values. "
+ "Requires an administrator token, and the endpoint is absent on Chatwoot "
+ "community-edition builds."
+ ),
+ ),
}
ENDPOINTS = tuple(CHATWOOT_ENDPOINTS.keys())
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/source.py
index db622baceba6..acd0325b990d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/source.py
@@ -15,6 +15,7 @@ from products.warehouse_sources.backend.facade.source_config import (
SourceFieldInputConfigType,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.chatwoot.chatwoot import (
+ AUDIT_LOGS_UNAVAILABLE_ERROR,
HOST_NOT_ALLOWED_ERROR,
HTTP_NOT_ALLOWED_ERROR,
INVALID_ACCOUNT_ID_ERROR,
@@ -188,6 +189,7 @@ If automatic creation failed, note that only Chatwoot administrators can manage
# so retrying never changes the outcome.
"422 Client Error": "Chatwoot rejected a request while syncing this data. Please contact support if this persists.",
REPORTING_EVENTS_UNAVAILABLE_ERROR: "This Chatwoot instance does not serve reporting events. The endpoint only exists on Chatwoot Cloud and enterprise builds, so deselect the reporting_events table to sync the rest.",
+ AUDIT_LOGS_UNAVAILABLE_ERROR: "This Chatwoot instance does not serve audit logs. The endpoint only exists on Chatwoot Cloud and enterprise builds, so deselect the audit_logs table to sync the rest.",
INVALID_ACCOUNT_ID_ERROR: "The Chatwoot account ID must be a number (the number in your Chatwoot URL, e.g. app.chatwoot.com/app/accounts/1). Update the source configuration.",
HOST_NOT_ALLOWED_ERROR: "The Chatwoot host is not allowed. Please use a publicly reachable instance URL.",
HTTP_NOT_ALLOWED_ERROR: "The Chatwoot host must use HTTPS. Please update the instance URL to use https://.",
@@ -204,8 +206,8 @@ If automatic creation failed, note that only Chatwoot administrators can manage
force_refresh: bool = False,
api_version: str | None = None,
) -> list[SourceSchema]:
- # Only reporting_events has a server-side timestamp filter to sync incrementally against;
- # the rest are full refresh, and conversations and messages also support webhook-fed deltas.
+ # Only reporting_events and audit_logs have a server-side timestamp filter to sync
+ # incrementally against; the rest are full refresh, and conversations and messages also support webhook-fed deltas.
schemas = [
SourceSchema(
name=endpoint,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/tests/test_chatwoot.py b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/tests/test_chatwoot.py
index 48ea36d64c71..9cbdf0200488 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/tests/test_chatwoot.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/chatwoot/tests/test_chatwoot.py
@@ -9,6 +9,7 @@ import requests
from products.warehouse_sources.backend.temporal.data_imports.pipelines.core.arrow_utils import table_from_py_list
from products.warehouse_sources.backend.temporal.data_imports.sources.chatwoot.chatwoot import (
+ AUDIT_LOGS_UNAVAILABLE_ERROR,
REPORTING_EVENTS_UNAVAILABLE_ERROR,
ChatwootResumeConfig,
_normalize_account_id,
@@ -413,6 +414,123 @@ class TestGetRowsReportingEvents:
]
+class TestGetRowsAuditLogs:
+ @pytest.mark.parametrize(
+ "watermark, expected_since",
+ [
+ (None, 0),
+ # The created_at watermark is the unix int the API returns.
+ (1767323045, 1767323045),
+ ("2026-01-02T03:04:05+00:00", 1767323045),
+ ("not-a-date", 0),
+ ],
+ )
+ def test_walks_pages_within_a_pinned_ascending_window(self, watermark, expected_since):
+ result = _driver().run(
+ "audit_logs",
+ [
+ ScriptedResponse(
+ json={"per_page": 2, "total_entries": 3, "current_page": 1, "audit_logs": [{"id": 1}, {"id": 2}]}
+ ),
+ ScriptedResponse(
+ json={"per_page": 2, "total_entries": 3, "current_page": 2, "audit_logs": [{"id": 3}]}
+ ),
+ ],
+ incremental_field="created_at",
+ db_incremental_field_last_value=watermark,
+ )
+
+ assert [row["id"] for row in result.rows] == [1, 2, 3]
+ # total_entries ends the walk, so no third request is made.
+ assert result.params("page") == ["1", "2"]
+ assert result.params("sort") == ["asc", "asc"]
+ assert result.params("since") == [str(expected_since)] * 2
+ untils = set(result.params("until"))
+ assert len(untils) == 1
+ until = int(untils.pop() or "0")
+ assert result.committed_states == [
+ ChatwootResumeConfig(page=2, until=until),
+ ChatwootResumeConfig(page=3, until=until),
+ ]
+
+ def test_missing_totals_fall_back_to_the_empty_page(self):
+ result = _driver().run(
+ "audit_logs",
+ [ScriptedResponse(json={"audit_logs": [{"id": 1}]}), ScriptedResponse(json={"audit_logs": []})],
+ )
+
+ assert [row["id"] for row in result.rows] == [1]
+ assert result.params("page") == ["1", "2"]
+
+ def test_resume_reuses_the_saved_page_and_window(self):
+ result = _driver().run(
+ "audit_logs",
+ [ScriptedResponse(json={"audit_logs": []})],
+ resume_state=ChatwootResumeConfig(page=4, until=1767000000),
+ incremental_field="created_at",
+ db_incremental_field_last_value=1766000000,
+ )
+
+ assert result.params("page") == ["4"]
+ assert result.params("until") == ["1767000000"]
+ assert result.params("since") == ["1766000000"]
+
+ def test_404_reports_the_endpoint_as_unavailable_not_a_missing_account(self):
+ result = _driver().run("audit_logs", [ScriptedResponse(status=404, json={"error": "Not Found"})])
+
+ assert isinstance(result.raised, Exception)
+ assert AUDIT_LOGS_UNAVAILABLE_ERROR in str(result.raised)
+
+
+class TestGetRowsContactLabels:
+ LABELS = {"payload": [{"id": 1, "title": "vip"}, {"id": 2, "title": "churn_risk"}, {"id": 3, "title": "beta"}]}
+
+ def test_walks_each_label_and_commits_past_labels_with_no_contacts(self):
+ result = _driver().run(
+ "contact_labels",
+ [
+ ScriptedResponse(json=self.LABELS),
+ ScriptedResponse(json={"meta": {}, "payload": [{"id": 10, "email": "[email protected]"}, {"id": 11}]}),
+ ScriptedResponse(json={"meta": {}, "payload": []}),
+ ScriptedResponse(json={"meta": {}, "payload": []}),
+ ScriptedResponse(json={"meta": {}, "payload": [{"id": 10}]}),
+ ScriptedResponse(json={"meta": {}, "payload": []}),
+ ],
+ )
+
+ # A slim join row: the contact's own columns stay in the contacts table.
+ assert result.rows == [
+ {"label_id": 1, "label": "vip", "contact_id": 10},
+ {"label_id": 1, "label": "vip", "contact_id": 11},
+ {"label_id": 3, "label": "beta", "contact_id": 10},
+ ]
+ assert result.paths == ["/api/v1/accounts/1/labels", *["/api/v1/accounts/1/contacts"] * 5]
+ assert result.params("labels[]") == [None, "vip", "vip", "churn_risk", "beta", "beta"]
+ assert result.params("page") == [None, "1", "2", "1", "1", "2"]
+ # Label 2 yields no rows, so only the safe point commits the move on to label 3.
+ assert ChatwootResumeConfig(parent_id=3, page=1) in result.committed_states
+
+ @pytest.mark.parametrize(
+ "resume_state, expected_requests",
+ [
+ (ChatwootResumeConfig(parent_id=2, page=3), [("churn_risk", "3"), ("beta", "1")]),
+ # A bookmark for a label that no longer exists must restart the walk, not skip everything.
+ (ChatwootResumeConfig(parent_id=999, page=3), [("vip", "1"), ("churn_risk", "1"), ("beta", "1")]),
+ ],
+ )
+ def test_resumes_from_bookmarked_label_and_page(self, resume_state, expected_requests):
+ result = _driver().run(
+ "contact_labels",
+ [
+ ScriptedResponse(json=self.LABELS),
+ *[ScriptedResponse(json={"meta": {}, "payload": []}) for _ in expected_requests],
+ ],
+ resume_state=resume_state,
+ )
+
+ assert list(zip(result.params("labels[]"), result.params("page")))[1:] == expected_requests
+
+
class TestWebhookTableTransformer:
def test_message_events_are_normalized_to_the_rest_shape(self):
transform = make_webhook_table_transformer("messages")