:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse_sources): add omnisend segments, automations and analytics tables (#123168)
Tom Owerse0a219b41094
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 24de3231cdad..182c1e63a205 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -8629,10 +8629,10 @@ Today (4): `campaigns`, `categories`, `contacts`, `products`Diffed against: <https://api-docs.omnisend.com/llms.txt>-- [ ] `POST /analytics/statistics (Statistics)` — Aggregated delivery, engagement, revenue-attribution and audience-growth metrics grouped by event date — the vendor's headline marketing numbers, absent entirely today (high)-- [ ] `POST /analytics/reports (Reports)` — Same metric family grouped by send date with campaign/automation dimensions, so campaigns can be ranked by performance rather than just listed (high)-- [ ] `GET /segments and GET /segments/{segmentID}/statistics` — Segment definitions plus member counts — the lookup that resolves the segment ids automations and campaigns target (high)-- [ ] `GET /automations (list automation workflows)` — Automated flows are the other half of Omnisend sending volume; only one-off campaigns are synced today (high)+- [x] `POST /analytics/statistics (Statistics)` — Aggregated delivery, engagement, revenue-attribution and audience-growth metrics grouped by event date — the vendor's headline marketing numbers, absent entirely today (high)+- [x] `POST /analytics/reports (Reports)` — Same metric family grouped by send date with campaign/automation dimensions, so campaigns can be ranked by performance rather than just listed (high)+- [x] `GET /segments and GET /segments/{segmentID}/statistics` — Segment definitions plus member counts — the lookup that resolves the segment ids automations and campaigns target (high)+- [x] `GET /automations (list automation workflows)` — Automated flows are the other half of Omnisend sending volume; only one-off campaigns are synced today (high)- [ ] `GET /brands/current` — Single-row account/brand context (currency, store connection) that makes revenue figures interpretable (low)Note: The connector targets v3 (https://api.omnisend.com/v3). The gaps above are on the current 2026-03-15 version (header-based versioning, base https://api.omnisend.com/api/), which added Segments, Automations, Analytics, Email Templates and Images — so closing them means reaching into a newer API version, not just adding v3 paths. Events are write-only (POST /events), so there is no queryable event stream. Email templates, universal layouts, images and batches were excluded as assets/plumbing.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/api_inventory.md b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/api_inventory.mdindex 2e1aeaa0759b..87a8d7127fdd 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/api_inventory.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/api_inventory.md@@ -21,14 +21,16 @@ The same API key works for both versions; only the header changes. `limit` defau## List endpoints-| Schema | v3 path / key / PK | 2026-03-15 path / key / PK | Partition (stable) |-| ---------- | ------------------------------------------- | --------------------------------------------------- | ------------------ |-| contacts | `/contacts` / `contacts` / `contactID` | `/contacts` / `contacts` / `id` | `createdAt` |-| campaigns | `/campaigns` / `campaign` / `campaignID` | `/campaigns` / `campaigns` / `id` | `createdAt` |-| carts | `/carts` / `carts` / `cartID` | — (write-only events) | `createdAt` |-| orders | `/orders` / `orders` / `orderID` | — (write-only events) | `createdAt` |-| products | `/products` / `products` / `productID` | `/products` / `products` / `id` | `createdAt` |-| categories | `/categories` / `categories` / `categoryID` | `/product-categories` / `categories` / `categoryID` | — |+| Schema | v3 path / key / PK | 2026-03-15 path / key / PK | Partition (stable) |+| ----------- | ------------------------------------------- | --------------------------------------------------- | ------------------ |+| contacts | `/contacts` / `contacts` / `contactID` | `/contacts` / `contacts` / `id` | `createdAt` |+| campaigns | `/campaigns` / `campaign` / `campaignID` | `/campaigns` / `campaigns` / `id` | `createdAt` |+| carts | `/carts` / `carts` / `cartID` | — (write-only events) | `createdAt` |+| orders | `/orders` / `orders` / `orderID` | — (write-only events) | `createdAt` |+| products | `/products` / `products` / `productID` | `/products` / `products` / `id` | `createdAt` |+| categories | `/categories` / `categories` / `categoryID` | `/product-categories` / `categories` / `categoryID` | — |+| automations | — | `/automations` / `automations` / `id` | `createdAt` |+| segments | — | `/segments` / `segments` / `segmentID` | `createdAt` |v3 endpoint existence was confirmed against the live API (all return non-404 without a key).v3 response array keys follow the plural `<resource>` convention **except `/campaigns`, which@@ -38,9 +40,40 @@ Other 2026-03-15 changes visible in synced rows: product and variant prices arestore currency (v3: integer cents), product `images` are URL strings, and campaign statisticsmoved off the campaign object into the Analytics API.+`automations` and `segments` are cursor paginated and sorted `createdAt` ascending.+`automations` uses `view=summary`, which drops the `blocks` tree: webhook action blocks return+their header values (often third-party credentials) verbatim. `/segments`+caps `limit` at 50. Each `segments` row also carries `contactsCount` from+`GET /segments/{segmentID}/statistics`, one request per segment. A segment deleted between the+two calls keeps a null count. Segment reads share a 100 req/min limit.++## Analytics tables++| Schema | Request | Row grain | Primary key |+| -------------------- | ---------------------------- | ----------------------------------------------------------- | ----------------------------------------------------------- |+| analytics_reports | `POST /analytics/reports` | send date × `marketingActivityType` × `marketingActivityID` | `timestamp`, `marketingActivityType`, `marketingActivityID` |+| analytics_statistics | `POST /analytics/statistics` | event date (account totals) | `timestamp` |++- Both use `day` granularity in UTC. A query may span at most 60 days, so we send 56-day+ windows, 4 queries per request (a request must span under 12 months). `to` is exclusive.+- `/analytics/reports` requires `dateRange.interval: "custom"` with explicit dates;+ `/analytics/statistics` has no `interval` field.+- History reaches back 5 years, the API's lookback limit. A full read is about 9 requests.+- Rate limit: 10 req/min and **55 req/day per brand**. Incremental sync on `timestamp` keeps a+ run to one request, so it is the advised mode; frequent full refreshes can use up the daily budget.+- Incremental runs re-read a 30-day lookback window (`default_incremental_lookback_seconds`),+ because opens, clicks and attributed orders keep landing on a send date after the send.+ Merge only: append would duplicate the re-read days.+- A read that returns no rows sets no watermark, so each completed read also stores its end date+ as a source cursor (`omnisend_analytics_read_through`). The next incremental run without a+ watermark starts from that date minus the lookback, not 5 years back.+- Only count metrics are requested. Rates and averages are derivable and can't be summed.+ `analytics_reports` leaves out `totalOrders`/`totalRevenue`, which allow no breakdown.+## Sync mode-All endpoints ship **full refresh (replace)**.+All list endpoints ship **full refresh (replace)**. The analytics tables support incremental+merge on `timestamp`, because their date range is a server-side filter.Omnisend documents `updatedAtFrom` as a server-side filter on `/contacts`, but with hardrestrictions (it cannot be combined with `email`, `phone`, `status`, `segmentID`, ordiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/canonical_descriptions.pyindex 64b2fa5296ef..3fce9ab5686d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/canonical_descriptions.py@@ -103,4 +103,87 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"updatedAt": "Time at which the category was last updated.",},},+ "automations": {+ "description": "An automation workflow in Omnisend — a trigger plus the blocks (messages, delays, splits) it runs.",+ "docs_url": "https://api-docs.omnisend.com/reference/get_automations",+ "columns": {+ "id": "Unique identifier for the automation workflow.",+ "name": "The automation workflow's name.",+ "isEnabled": "Whether the automation workflow is enabled.",+ "trigger": "Trigger configuration: the event and filters that start the workflow.",+ "exitConditions": "Events that remove contacts from the workflow.",+ "settings": "Workflow settings such as frequency and overlap limiters.",+ "brandID": "Identifier of the brand that owns the workflow.",+ "url": "Link to the workflow editor in the Omnisend app.",+ "enabledAt": "Time at which the workflow was last enabled.",+ "disabledAt": "Time at which the workflow was last disabled.",+ "createdAt": "Time at which the workflow was created.",+ "updatedAt": "Time at which the workflow was last updated.",+ },+ },+ "segments": {+ "description": "A contact segment in Omnisend, with its member count.",+ "docs_url": "https://api-docs.omnisend.com/reference/get_segments",+ "columns": {+ "segmentID": "Unique identifier for the segment.",+ "name": "The segment's name.",+ "status": "Processing status of the segment (ready, building, archived).",+ "conditionGroups": "Condition groups that define which contacts match the segment.",+ "isStarred": "Whether the segment is marked as a favourite.",+ "contactsCount": "Number of contacts that match the segment, from the segment statistics endpoint.",+ "archivedAt": "Time at which the segment was archived, if archived.",+ "createdAt": "Time at which the segment was created.",+ "updatedAt": "Time at which the segment was last updated.",+ },+ },+ "analytics_reports": {+ "description": "Daily message performance per campaign or automation workflow, grouped by the date the messages were sent.",+ "docs_url": "https://api-docs.omnisend.com/reference/reports",+ "columns": {+ "timestamp": "Date (UTC) the messages were sent.",+ "marketingActivityType": "Kind of marketing activity (Campaign or Automation).",+ "marketingActivityID": "Identifier of the campaign or automation workflow.",+ "sent": "Messages sent.",+ "failed": "Messages that failed to deliver (bounces, invalid addresses).",+ "sentCost": "Cost of the messages sent. SMS only.",+ "markedAsSpamUnique": "Unique messages marked as spam by recipients.",+ "unsubscribedUnique": "Unique messages that resulted in an unsubscribe.",+ "openedUnique": "Unique messages opened at least once.",+ "opened": "Total opens, including repeat opens of the same message.",+ "clickedUnique": "Unique messages clicked at least once.",+ "clicked": "Total clicks, including repeat clicks on the same message.",+ "attributedOrdersUnique": "Unique messages with at least one attributed order.",+ "attributedOrders": "Placed orders attributed to messages.",+ "attributedRevenue": "Revenue from orders attributed to messages.",+ },+ },+ "analytics_statistics": {+ "description": "Daily account-level marketing statistics, grouped by the date each event happened.",+ "docs_url": "https://api-docs.omnisend.com/reference/statistics",+ "columns": {+ "timestamp": "Date (UTC) the events happened.",+ "sent": "Messages sent.",+ "failed": "Messages that failed to deliver (bounces, invalid addresses).",+ "sentCost": "Cost of the messages sent. SMS only.",+ "markedAsSpamUnique": "Unique messages marked as spam by recipients.",+ "unsubscribedUnique": "Unique messages that resulted in an unsubscribe.",+ "openedUnique": "Unique messages opened at least once.",+ "opened": "Total opens, including repeat opens of the same message.",+ "clickedUnique": "Unique messages clicked at least once.",+ "clicked": "Total clicks, including repeat clicks on the same message.",+ "attributedOrdersUnique": "Unique messages with at least one attributed order.",+ "attributedOrders": "Placed orders attributed to messages.",+ "attributedRevenue": "Revenue from orders attributed to messages.",+ "totalOrders": "All placed orders, attributed or not.",+ "totalRevenue": "Revenue from all placed orders, attributed or not.",+ "totalOrderedProductUnits": "Product units ordered across all orders.",+ "attributedOrderedProductUnits": "Product units from orders attributed to messages.",+ "subscribedEmail": "Contacts who joined the email list.",+ "subscribedSms": "Contacts who joined the SMS list.",+ "subscribedPush": "Contacts who joined the push list.",+ "unsubscribedEmail": "Contacts who left the email list.",+ "unsubscribedSms": "Contacts who left the SMS list.",+ "unsubscribedPush": "Contacts who left the push list.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/omnisend.py b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/omnisend.pyindex 69790c7a8ca0..4fb07a52a16f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/omnisend.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/omnisend.py@@ -1,17 +1,26 @@import dataclasses-from typing import Any, Optional+from collections.abc import Iterator+from datetime import UTC, date, datetime, timedelta+from typing import Any, ClassVar, Optionalfrom requests import Response+from requests.exceptions import HTTPError+from posthog.dataclasses import frozen++from products.warehouse_sources.backend.temporal.data_imports.sources.common.cursor import SourceCursorManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_sessionfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source import (RESTAPIConfig,+ RESTClient,rest_api_resource,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.auth import APIKeyAuthfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (BasePaginator,JSONResponseCursorPaginator,JSONResponsePaginator,+ SinglePagePaginator,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import ClientConfigfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager@@ -21,6 +30,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.omnisend.sOMNISEND_2026_03_15,OMNISEND_ENDPOINTS,OMNISEND_V3,+ OmnisendAnalyticsQuery,OmnisendWire,)@@ -40,6 +50,12 @@ UNSUPPORTED_ENDPOINT_ERROR = "Omnisend API version has no list endpoint for this# 400 req/min general rate limit.PAGE_SIZE = 250+# Omnisend rejects an analytics `dateRange.from` more than 5 years back.+ANALYTICS_HISTORY_DAYS = 5 * 365 - 5+# Daily granularity caps one query at 60 days, and a request at 4 queries spanning under 12 months.+ANALYTICS_WINDOW_DAYS = 56+ANALYTICS_WINDOWS_PER_REQUEST = 4+@dataclasses.dataclass(frozen=True)class OmnisendResumeConfig:@@ -47,6 +63,25 @@ class OmnisendResumeConfig:next_url: Optional[str] = None# Opaque `paging.cursors.after` token for the cursor-paginated 2026-03-15 endpoints.cursor: Optional[str] = None+ # Last segment on the `cursor` page whose row was handed on.+ last_segment_id: Optional[str] = None+ # First day of the next analytics request, as an ISO date.+ window_start: Optional[str] = None+++@frozen+class OmnisendAnalyticsCursor:+ """The exclusive end date of the last analytics read that completed."""++ cursor_kind: ClassVar[str] = "omnisend_analytics_read_through"++ read_through: str+++@frozen+class _AnalyticsWindow:+ start: date+ end: dateclass _OmnisendCursorPaginator(JSONResponseCursorPaginator):@@ -120,6 +155,189 @@ def _client_config(api_key: str, api_version: str, wire: OmnisendWire) -> Client}+def _rest_client(api_key: str, api_version: str) -> RESTClient:+ auth = _auth_header(api_key, api_version)+ return RESTClient(+ base_url=_base_url(api_version),+ headers=_headers(api_version),+ auth=APIKeyAuth(api_key=auth.value, name=auth.name, location="header"),+ paginator=SinglePagePaginator(),+ allowed_hosts=[],+ allow_redirects=False,+ request_timeout=(10, 60),+ )+++def _fetch_object(+ client: RESTClient,+ path: str,+ method: str = "get",+ params: Optional[dict[str, Any]] = None,+ json: Optional[dict[str, Any]] = None,+) -> dict[str, Any]:+ pages = list(client.paginate(path=path, method=method, params=params, json=json, data_selector="$"))+ body = pages[0][0] if pages and pages[0] else None+ if not isinstance(body, dict):+ raise ValueError(f"Omnisend returned an unexpected response from {path}. Expected a JSON object.")+ return body+++def _segment_contacts_count(client: RESTClient, segment_id: Any) -> Optional[int]:+ if not segment_id:+ return None+ try:+ statistics = _fetch_object(client, f"/segments/{segment_id}/statistics")+ except HTTPError as e:+ # A segment deleted between the list call and this one.+ if e.response is not None and e.response.status_code == 404:+ return None+ raise+ return statistics.get("contactsCount")+++def _iter_segments(+ api_key: str,+ api_version: str,+ wire: OmnisendWire,+ resumable_source_manager: ResumableSourceManager[OmnisendResumeConfig],+) -> Iterator[list[dict[str, Any]]]:+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None+ cursor = resume.cursor if resume is not None else None+ done_through = resume.last_segment_id if resume is not None else None++ client = _rest_client(api_key, api_version)+ try:+ while True:+ params: dict[str, Any] = {"limit": wire.page_size or PAGE_SIZE, **wire.params}+ if cursor:+ params["after"] = cursor+ page = _fetch_object(client, wire.path, params=params)+ segments = page.get(wire.data_key)+ if not isinstance(segments, list):+ raise ValueError(f"Omnisend response from {wire.path} has no '{wire.data_key}' list")++ start = 0+ # Skip by id, not position: if the checkpointed segment is gone, replay the whole page.+ if done_through is not None:+ ids = [segment.get("segmentID") for segment in segments]+ if done_through in ids:+ start = ids.index(done_through) + 1+ done_through = None++ for segment in segments[start:]:+ segment_id = segment.get("segmentID")+ row = {**segment, "contactsCount": _segment_contacts_count(client, segment_id)}+ resumable_source_manager.save_state(OmnisendResumeConfig(cursor=cursor, last_segment_id=segment_id))+ yield [row]++ paging = page.get("paging") or {}+ next_cursor = (paging.get("cursors") or {}).get("after")+ if paging.get("hasMore") is False or not next_cursor:+ break+ if next_cursor == cursor:+ raise ValueError("Omnisend segments pagination is not advancing (repeated cursor)")+ cursor = next_cursor+ resumable_source_manager.safe_point()+ finally:+ client.session.close()+++def _as_date(value: Any) -> date:+ if isinstance(value, datetime):+ return value.date()+ if isinstance(value, date):+ return value+ return date.fromisoformat(str(value)[:10])+++def _analytics_start(+ today: date,+ should_use_incremental_field: bool,+ db_incremental_field_last_value: Any,+ stored_cursor: Optional[OmnisendAnalyticsCursor],+ lookback: timedelta,+) -> date:+ earliest = today - timedelta(days=ANALYTICS_HISTORY_DAYS)+ if not should_use_incremental_field:+ return earliest+ if db_incremental_field_last_value is not None:+ return max(earliest, _as_date(db_incremental_field_last_value))+ # Reads that returned no rows set no watermark, so the stored read-through date bounds the next read.+ if stored_cursor is not None:+ return max(earliest, date.fromisoformat(stored_cursor.read_through) - lookback)+ return earliest+++def _analytics_windows(start: date, end: date) -> list[_AnalyticsWindow]:+ windows = []+ window_start = start+ while window_start < end:+ window_end = min(window_start + timedelta(days=ANALYTICS_WINDOW_DAYS), end)+ windows.append(_AnalyticsWindow(start=window_start, end=window_end))+ window_start = window_end+ return windows+++def _analytics_body(query: OmnisendAnalyticsQuery, windows: list[_AnalyticsWindow]) -> dict[str, Any]:+ queries = []+ for index, window in enumerate(windows):+ date_range: dict[str, str] = {"from": f"{window.start}T00:00:00Z", "to": f"{window.end}T00:00:00Z"}+ if query.date_range_interval:+ date_range["interval"] = query.date_range_interval+ queries.append(+ {+ "alias": f"window-{index}",+ "metrics": [{"name": metric} for metric in query.metrics],+ "dimensions": [+ {"name": "timestamp", "granularity": "day"},+ *({"name": dimension} for dimension in query.dimensions),+ ],+ "dateRange": date_range,+ }+ )+ return {"queries": queries}+++def _iter_analytics(+ api_key: str,+ api_version: str,+ wire: OmnisendWire,+ query: OmnisendAnalyticsQuery,+ start: date,+ end: date,+ resumable_source_manager: ResumableSourceManager[OmnisendResumeConfig],+ cursor_manager: Optional[SourceCursorManager[OmnisendAnalyticsCursor]],+) -> Iterator[list[dict[str, Any]]]:+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None+ if resume is not None and resume.window_start:+ start = max(start, date.fromisoformat(resume.window_start))++ windows = _analytics_windows(start, end)+ client = _rest_client(api_key, api_version)+ try:+ for offset in range(0, len(windows), ANALYTICS_WINDOWS_PER_REQUEST):+ batch = windows[offset : offset + ANALYTICS_WINDOWS_PER_REQUEST]+ response = _fetch_object(client, wire.path, method="post", json=_analytics_body(query, batch))+ results = response.get(wire.data_key)+ if not isinstance(results, list):+ raise ValueError(f"Omnisend response from {wire.path} has no '{wire.data_key}' list")+ rows = [row for result in results for row in (result.get("rows") or [])]++ next_offset = offset + ANALYTICS_WINDOWS_PER_REQUEST+ if next_offset < len(windows):+ resumable_source_manager.save_state(+ OmnisendResumeConfig(window_start=windows[next_offset].start.isoformat())+ )+ if rows:+ yield rows+ else:+ resumable_source_manager.safe_point()+ if cursor_manager is not None:+ cursor_manager.stage(OmnisendAnalyticsCursor(read_through=end.isoformat()))+ finally:+ client.session.close()++def validate_credentials(api_key: str, api_version: str) -> tuple[bool, int | None]:"""Cheap probe to confirm the API key is genuine. Returns (is_valid, status_code)."""auth = _auth_header(api_key, api_version)@@ -138,9 +356,56 @@ def omnisend_source(job_id: str,api_version: str,resumable_source_manager: ResumableSourceManager[OmnisendResumeConfig],+ should_use_incremental_field: bool = False,+ db_incremental_field_last_value: Any = None,+ analytics_cursor_manager: Optional[SourceCursorManager[OmnisendAnalyticsCursor]] = None,) -> SourceResponse:config = OMNISEND_ENDPOINTS[endpoint]wire = _wire(endpoint, api_version)+ partition_keys = [config.partition_key] if config.partition_key else None++ if config.analytics is not None:+ query = config.analytics+ today = datetime.now(UTC).date()+ start = _analytics_start(+ today,+ should_use_incremental_field,+ db_incremental_field_last_value,+ analytics_cursor_manager.load() if analytics_cursor_manager is not None else None,+ timedelta(seconds=config.default_incremental_lookback_seconds or 0),+ )+ return SourceResponse(+ name=endpoint,+ # `to` is exclusive, so tomorrow's midnight includes today's completed hours.+ items=lambda: _iter_analytics(+ api_key,+ api_version,+ wire,+ query,+ start,+ today + timedelta(days=1),+ resumable_source_manager,+ analytics_cursor_manager,+ ),+ primary_keys=[wire.primary_key, *query.dimensions],+ partition_count=1,+ partition_size=1,+ partition_mode="datetime",+ partition_format="month",+ partition_keys=partition_keys,+ )++ if endpoint == "segments":+ return SourceResponse(+ name=endpoint,+ items=lambda: _iter_segments(api_key, api_version, wire, resumable_source_manager),+ primary_keys=[wire.primary_key],+ partition_count=1,+ partition_size=1,+ partition_mode="datetime",+ partition_format="month",+ partition_keys=partition_keys,+ )rest_config: RESTAPIConfig = {"client": _client_config(api_key, api_version, wire),@@ -149,7 +414,7 @@ def omnisend_source("name": endpoint,"endpoint": {"path": wire.path,- "params": {"limit": PAGE_SIZE},+ "params": {"limit": wire.page_size or PAGE_SIZE, **wire.params},"data_selector": wire.data_key,# A 200 body missing the envelope key means the response shape changed — fail# loud instead of silently syncing 0 rows.@@ -183,7 +448,7 @@ def omnisend_source(rest_config,team_id,job_id,- None, # every Omnisend endpoint is full refresh+ None, # every Omnisend list endpoint is full refreshresume_hook=save_checkpoint,initial_paginator_state=initial_paginator_state,)@@ -196,7 +461,7 @@ def omnisend_source(partition_size=1,partition_mode="datetime" if config.partition_key else None,partition_format="month" if config.partition_key else None,- partition_keys=[config.partition_key] if config.partition_key else None,+ partition_keys=partition_keys,sort_mode="asc",column_hints=resource.column_hints,)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/settings.pyindex 2b33fad62d8e..e7241741807a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/settings.py@@ -1,7 +1,8 @@+from collections.abc import Mappingfrom dataclasses import dataclass, field-from typing import Optional+from typing import Any, Optional-from products.warehouse_sources.backend.types import IncrementalField+from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldTypeOMNISEND_V3 = "v3"# Header-versioned (`Omnisend-Version`) on the `/api` base path, with `Authorization: Omnisend-API-Key`.@@ -15,6 +16,17 @@ class OmnisendWire:primary_key: str# Opaque `paging.cursors.after` tokens instead of a fully-formed `paging.next` URL.cursor_paginated: bool = False+ params: Mapping[str, Any] = field(default_factory=dict)+ page_size: Optional[int] = None+++@dataclass(frozen=True)+class OmnisendAnalyticsQuery:+ # Breakdown dimensions next to the daily `timestamp`; together they key a row.+ dimensions: tuple[str, ...]+ metrics: tuple[str, ...]+ # `/analytics/reports` needs `dateRange.interval`; `/analytics/statistics` has no such field.+ date_range_interval: Optional[str] = None@dataclass(frozen=True)@@ -26,9 +38,56 @@ class OmnisendEndpointConfig:# Stable creation-time field used for delta partitioning. Never a mutable field# like updatedAt — partitions would rewrite on every sync.partition_key: Optional[str] = None- # Advertised incremental options. Empty = full refresh only (see api_inventory.md:- # Omnisend's only server-side timestamp filter is unverified, so we ship full refresh).+ # Advertised incremental options. Empty = full refresh only (see api_inventory.md: the list+ # endpoints' timestamp filters are unverified, so they ship full refresh).incremental_fields: list[IncrementalField] = field(default_factory=list)+ analytics: Optional[OmnisendAnalyticsQuery] = None+ default_incremental_lookback_seconds: Optional[int] = None+++# Count metrics only: rates and per-order averages are derivable and can't be summed across rows.+_ACTIVITY_METRICS = (+ "sent",+ "failed",+ "sentCost",+ "markedAsSpamUnique",+ "unsubscribedUnique",+ "openedUnique",+ "opened",+ "clickedUnique",+ "clicked",+ "attributedOrdersUnique",+ "attributedOrders",+ "attributedRevenue",+)++# Every statistics metric supports the timestamp dimension, so a daily account-level row can carry+# store sales and audience growth alongside message activity.+_ACCOUNT_STATISTICS_METRICS = (+ *_ACTIVITY_METRICS,+ "totalOrders",+ "totalRevenue",+ "totalOrderedProductUnits",+ "attributedOrderedProductUnits",+ "subscribedEmail",+ "subscribedSms",+ "subscribedPush",+ "unsubscribedEmail",+ "unsubscribedSms",+ "unsubscribedPush",+)++_ANALYTICS_INCREMENTAL_FIELDS: list[IncrementalField] = [+ {+ "label": "timestamp",+ "type": IncrementalFieldType.Date,+ "field": "timestamp",+ "field_type": IncrementalFieldType.Date,+ },+]++# Opens, clicks and attributed orders keep landing on a send date for weeks after the send.+_ANALYTICS_LOOKBACK_SECONDS = 30 * 24 * 60 * 60OMNISEND_ENDPOINTS: dict[str, OmnisendEndpointConfig] = {@@ -81,6 +140,61 @@ OMNISEND_ENDPOINTS: dict[str, OmnisendEndpointConfig] = {),},),+ "automations": OmnisendEndpointConfig(+ name="automations",+ wires={+ OMNISEND_2026_03_15: OmnisendWire(+ path="/automations",+ data_key="automations",+ primary_key="id",+ cursor_paginated=True,+ # The summary view drops the `blocks` tree, whose webhook actions carry header values+ # (often third-party credentials) verbatim.+ params={"sort": "createdAt", "direction": "asc", "view": "summary"},+ ),+ },+ partition_key="createdAt",+ ),+ "segments": OmnisendEndpointConfig(+ name="segments",+ wires={+ OMNISEND_2026_03_15: OmnisendWire(+ path="/segments",+ data_key="segments",+ primary_key="segmentID",+ cursor_paginated=True,+ params={"sort": "createdAt", "direction": "asc"},+ page_size=50,+ ),+ },+ partition_key="createdAt",+ ),+ "analytics_reports": OmnisendEndpointConfig(+ name="analytics_reports",+ wires={+ OMNISEND_2026_03_15: OmnisendWire(path="/analytics/reports", data_key="reports", primary_key="timestamp"),+ },+ partition_key="timestamp",+ incremental_fields=_ANALYTICS_INCREMENTAL_FIELDS,+ analytics=OmnisendAnalyticsQuery(+ dimensions=("marketingActivityType", "marketingActivityID"),+ metrics=_ACTIVITY_METRICS,+ date_range_interval="custom",+ ),+ default_incremental_lookback_seconds=_ANALYTICS_LOOKBACK_SECONDS,+ ),+ "analytics_statistics": OmnisendEndpointConfig(+ name="analytics_statistics",+ wires={+ OMNISEND_2026_03_15: OmnisendWire(+ path="/analytics/statistics", data_key="statistics", primary_key="timestamp"+ ),+ },+ partition_key="timestamp",+ incremental_fields=_ANALYTICS_INCREMENTAL_FIELDS,+ analytics=OmnisendAnalyticsQuery(dimensions=(), metrics=_ACCOUNT_STATISTICS_METRICS),+ default_incremental_lookback_seconds=_ANALYTICS_LOOKBACK_SECONDS,+ ),}ENDPOINTS = tuple(OMNISEND_ENDPOINTS.keys())@@ -93,3 +207,12 @@ def endpoints_for_version(api_version: str) -> tuple[str, ...]:INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {name: config.incremental_fields for name, config in OMNISEND_ENDPOINTS.items()}++# Analytics rows for recent days are restated on each read, so only a merge keeps one row per key.+MERGE_ONLY_ENDPOINTS = tuple(name for name, config in OMNISEND_ENDPOINTS.items() if config.analytics is not None)++DEFAULT_INCREMENTAL_LOOKBACK_SECONDS: dict[str, int] = {+ name: config.default_incremental_lookback_seconds+ for name, config in OMNISEND_ENDPOINTS.items()+ if config.default_incremental_lookback_seconds is not None+}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/source.pyindex 510a1646b4e8..1cb3616ad495 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/source.py@@ -16,6 +16,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.basfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.canonical_descriptions import (CanonicalDescriptions,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.cursor import CursorSourcefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistryfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import (@@ -28,12 +29,15 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_)from products.warehouse_sources.backend.temporal.data_imports.sources.omnisend.omnisend import (UNSUPPORTED_ENDPOINT_ERROR,+ OmnisendAnalyticsCursor,OmnisendResumeConfig,omnisend_source,validate_credentials as validate_omnisend_credentials,)from products.warehouse_sources.backend.temporal.data_imports.sources.omnisend.settings import (+ DEFAULT_INCREMENTAL_LOOKBACK_SECONDS,INCREMENTAL_FIELDS,+ MERGE_ONLY_ENDPOINTS,OMNISEND_2026_03_15,OMNISEND_ENDPOINTS,OMNISEND_V3,@@ -43,7 +47,10 @@ from products.warehouse_sources.backend.types import ExternalDataSourceType@SourceRegistry.register-class OmnisendSource(ResumableSource[OmnisendSourceConfig, OmnisendResumeConfig]):+class OmnisendSource(+ ResumableSource[OmnisendSourceConfig, OmnisendResumeConfig],+ CursorSource[OmnisendAnalyticsCursor],+):codebase_detection = detect(env=("^OMNISEND_",), script_hosts=("omnisnippet1.com",))supported_versions = (OMNISEND_V3, OMNISEND_2026_03_15)@@ -103,7 +110,11 @@ You can create an API key in your [Omnisend account settings](https://app.omniseapi_version: str | None = None,) -> list[SourceSchema]:return build_endpoint_schemas(- endpoints_for_version(self.resolve_api_version(api_version)), INCREMENTAL_FIELDS, names+ endpoints_for_version(self.resolve_api_version(api_version)),+ INCREMENTAL_FIELDS,+ names,+ merge_only=MERGE_ONLY_ENDPOINTS,+ default_incremental_lookback_seconds=DEFAULT_INCREMENTAL_LOOKBACK_SECONDS,)def validate_credentials(@@ -134,6 +145,14 @@ You can create an API key in your [Omnisend account settings](https://app.omniseUNSUPPORTED_ENDPOINT_ERROR: None,}+ def cursor_class(self) -> type[OmnisendAnalyticsCursor]:+ return OmnisendAnalyticsCursor++ def merge_cursors(+ self, current: OmnisendAnalyticsCursor, candidate: OmnisendAnalyticsCursor+ ) -> OmnisendAnalyticsCursor:+ return max(current, candidate, key=lambda cursor: cursor.read_through)+def get_resumable_source_manager(self, inputs: SourceInputs) -> ResumableSourceManager[OmnisendResumeConfig]:return ResumableSourceManager[OmnisendResumeConfig](inputs, OmnisendResumeConfig)@@ -150,4 +169,7 @@ You can create an API key in your [Omnisend account settings](https://app.omnisejob_id=inputs.job_id,api_version=self.resolve_api_version(inputs.api_version),resumable_source_manager=resumable_source_manager,+ should_use_incremental_field=inputs.should_use_incremental_field,+ db_incremental_field_last_value=inputs.db_incremental_field_last_value,+ analytics_cursor_manager=self.get_cursor_manager(inputs) if inputs.source_cursor is not None else None,)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/tests/test_omnisend.py b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/tests/test_omnisend.pyindex 8f98e1b47ef7..6dfffcd08cb0 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/tests/test_omnisend.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/tests/test_omnisend.py@@ -1,9 +1,11 @@+from datetime import UTC, date, datetime, timedeltafrom typing import Anyimport pytestfrom requests.exceptions import HTTPError+from products.warehouse_sources.backend.temporal.data_imports.sources.common.cursor import SourceCursorManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.testing import (ScriptedResponse,SourceDriver,@@ -13,9 +15,11 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_OmnisendSourceConfig,)from products.warehouse_sources.backend.temporal.data_imports.sources.omnisend.omnisend import (+ ANALYTICS_HISTORY_DAYS,OMNISEND_API_BASE_URL,OMNISEND_BASE_URL,UNSUPPORTED_ENDPOINT_ERROR,+ OmnisendAnalyticsCursor,OmnisendResumeConfig,validate_credentials,)@@ -72,7 +76,7 @@ class TestPagination:assert result.saved_states == []assert result.raised is None- @pytest.mark.parametrize("endpoint", ["contacts", "campaigns"])+ @pytest.mark.parametrize("endpoint", ["contacts", "campaigns", "automations"])def test_2026_follows_after_cursor_and_saves_it(self, endpoint: str) -> None:result = _driver().run(endpoint,@@ -90,6 +94,17 @@ class TestPagination:assert result.saved_states == [OmnisendResumeConfig(cursor="c1")]assert result.raised is None+ def test_automations_request_the_summary_view_without_webhook_blocks(self) -> None:+ result = _driver().run(+ "automations",+ [_cursor_page([{"id": "a1"}], after=None, has_more=False, key="automations")],+ api_version=OMNISEND_2026_03_15,+ )++ assert result.requests[0].param("view") == "summary"+ assert result.requests[0].param("include") is None+ assert result.raised is None+def test_2026_stops_when_has_more_is_false_even_with_a_cursor(self) -> None:result = _driver().run("contacts", [_cursor_page([{"id": "1"}], after="c1", has_more=False)], api_version=OMNISEND_2026_03_15@@ -112,6 +127,174 @@ class TestPagination:assert result.raised is None+class TestSegments:+ def test_walks_pages_and_adds_each_segment_contacts_count(self) -> None:+ def respond(request: Any) -> ScriptedResponse:+ if request.path == "/api/segments":+ if request.param("after") is None:+ return _cursor_page([{"segmentID": "s1"}, {"segmentID": "s2"}], "c1", True, key="segments")+ return _cursor_page([{"segmentID": "s3"}], None, False, key="segments")+ if request.path == "/api/segments/s2/statistics":+ return ScriptedResponse(status=404, json={"error": "Segment not found"})+ return ScriptedResponse(json={"contactsCount": int(request.path.split("/")[3][1:]) * 10})++ result = _driver().run("segments", respond, api_version=OMNISEND_2026_03_15)++ assert result.raised is None+ assert result.requests[0].param("limit") == "50"+ assert result.requests[0].param("sort") == "createdAt"+ assert result.requests[0].param("direction") == "asc"+ assert result.rows == [+ {"segmentID": "s1", "contactsCount": 10},+ {"segmentID": "s2", "contactsCount": None},+ {"segmentID": "s3", "contactsCount": 30},+ ]+ assert result.saved_states[-1] == OmnisendResumeConfig(cursor="c1", last_segment_id="s3")+ assert result.response is not None+ assert result.response.primary_keys == ["segmentID"]++ @pytest.mark.parametrize(+ ("last_segment_id", "expected_rows"),+ [+ ("s1", [{"segmentID": "s2", "contactsCount": 5}]),+ # The checkpointed segment was deleted, so the page is replayed.+ ("gone", [{"segmentID": "s1", "contactsCount": 5}, {"segmentID": "s2", "contactsCount": 5}]),+ ],+ )+ def test_resume_skips_segments_already_handed_on(+ self, last_segment_id: str, expected_rows: list[dict[str, Any]]+ ) -> None:+ def respond(request: Any) -> ScriptedResponse:+ if request.path == "/api/segments":+ return _cursor_page([{"segmentID": "s1"}, {"segmentID": "s2"}], None, False, key="segments")+ return ScriptedResponse(json={"contactsCount": 5})++ result = _driver().run(+ "segments",+ respond,+ resume_state=OmnisendResumeConfig(cursor="c4", last_segment_id=last_segment_id),+ api_version=OMNISEND_2026_03_15,+ )++ assert result.raised is None+ assert result.requests[0].param("after") == "c4"+ assert result.rows == expected_rows+++def _analytics_response(key: str, request: Any) -> ScriptedResponse:+ results = [+ {"alias": query["alias"], "rows": [{"timestamp": query["dateRange"]["from"][:10], "sent": 1}]}+ for query in request.json()["queries"]+ ]+ return ScriptedResponse(json={key: results})+++class TestAnalytics:+ @pytest.mark.parametrize(+ ("endpoint", "data_key", "interval", "dimensions"),+ [+ ("analytics_reports", "reports", "custom", ["timestamp", "marketingActivityType", "marketingActivityID"]),+ ("analytics_statistics", "statistics", None, ["timestamp"]),+ ],+ )+ def test_full_refresh_walks_the_whole_history_in_batched_windows(+ self, endpoint: str, data_key: str, interval: str | None, dimensions: list[str]+ ) -> None:+ today = datetime.now(UTC).date()+ result = _driver().run(+ endpoint, lambda request: _analytics_response(data_key, request), api_version=OMNISEND_2026_03_15+ )++ assert result.raised is None+ assert all(request.method == "POST" for request in result.requests)+ assert {request.path for request in result.requests} == {f"/api/analytics/{data_key}"}+ queries = [query for request in result.requests for query in request.json()["queries"]]+ assert all(len(request.json()["queries"]) <= 4 for request in result.requests)+ assert queries[0]["dateRange"]["from"] == f"{today - timedelta(days=ANALYTICS_HISTORY_DAYS)}T00:00:00Z"+ assert queries[-1]["dateRange"]["to"] == f"{today + timedelta(days=1)}T00:00:00Z"+ for previous, current in zip(queries, queries[1:]):+ assert previous["dateRange"]["to"] == current["dateRange"]["from"]+ for query in queries:+ span = date.fromisoformat(query["dateRange"]["to"][:10]) - date.fromisoformat(+ query["dateRange"]["from"][:10]+ )+ assert span <= timedelta(days=60)+ assert query["dateRange"].get("interval") == interval+ assert [dimension["name"] for dimension in query["dimensions"]] == dimensions+ assert len(result.rows) == len(queries)+ assert result.response is not None+ assert result.response.primary_keys == dimensions++ @pytest.mark.parametrize(+ "last_value",+ [date(2026, 3, 1), datetime(2026, 3, 1, 12, tzinfo=UTC), "2026-03-01"],+ )+ def test_incremental_reads_from_the_watermark(self, last_value: Any) -> None:+ result = _driver().run(+ "analytics_statistics",+ lambda request: _analytics_response("statistics", request),+ api_version=OMNISEND_2026_03_15,+ incremental_field="timestamp",+ db_incremental_field_last_value=last_value,+ )++ assert result.raised is None+ assert result.requests[0].json()["queries"][0]["dateRange"]["from"] == "2026-03-01T00:00:00Z"++ @pytest.mark.parametrize(+ ("last_value", "read_through", "expected_from"),+ [+ # No rows ever came back, so only the stored read-through date bounds the read.+ (None, "2026-05-01", "2026-04-01"),+ (date(2026, 4, 20), "2026-05-01", "2026-04-20"),+ ],+ )+ def test_incremental_read_stores_its_end_for_runs_without_a_watermark(+ self, last_value: date | None, read_through: str, expected_from: str+ ) -> None:+ cursor_manager = SourceCursorManager(+ OmnisendAnalyticsCursor, OmnisendAnalyticsCursor(read_through=read_through), OmnisendSource()+ )+ result = _driver().run(+ "analytics_reports",+ lambda request: ScriptedResponse(json={"reports": [{"alias": "window-0", "rows": []}]}),+ api_version=OMNISEND_2026_03_15,+ incremental_field="timestamp",+ db_incremental_field_last_value=last_value,+ source_cursor=cursor_manager,+ )++ assert result.raised is None+ assert result.requests[0].json()["queries"][0]["dateRange"]["from"] == f"{expected_from}T00:00:00Z"+ tomorrow = datetime.now(UTC).date() + timedelta(days=1)+ assert cursor_manager.staged == OmnisendAnalyticsCursor(read_through=tomorrow.isoformat())++ def test_resume_starts_at_the_saved_window_and_saves_the_next(self) -> None:+ today = datetime.now(UTC).date()+ resume_start = today - timedelta(days=300)+ result = _driver().run(+ "analytics_reports",+ lambda request: ScriptedResponse(json={"reports": [{"alias": "window-0", "rows": []}]}),+ resume_state=OmnisendResumeConfig(window_start=resume_start.isoformat()),+ api_version=OMNISEND_2026_03_15,+ )++ assert result.raised is None+ assert len(result.requests) == 2+ assert result.requests[0].json()["queries"][0]["dateRange"]["from"] == f"{resume_start}T00:00:00Z"+ second_from = result.requests[1].json()["queries"][0]["dateRange"]["from"][:10]+ assert result.saved_states == [OmnisendResumeConfig(window_start=second_from)]++ def test_missing_envelope_key_raises(self) -> None:+ result = _driver().run(+ "analytics_reports",+ lambda request: ScriptedResponse(json={"unexpected": []}),+ api_version=OMNISEND_2026_03_15,+ )++ assert isinstance(result.raised, ValueError)++class TestResume:def test_resume_seeds_starting_url(self) -> None:result = _driver().run(@@ -244,7 +427,7 @@ class TestSourceResponse:assert UNSUPPORTED_ENDPOINT_ERROR in str(result.raised)def test_every_endpoint_partition_key_is_stable(self) -> None:- # Partition keys must be creation-time fields, never mutable ones.+ # Partition keys must be creation-time fields or analytics date buckets, never mutable ones.for config in OMNISEND_ENDPOINTS.values():if config.partition_key is not None:- assert config.partition_key == "createdAt"+ assert config.partition_key == ("timestamp" if config.analytics else "createdAt")diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/tests/test_omnisend_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/tests/test_omnisend_source.pyindex 6593ad241dc1..57d1168d3f08 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/tests/test_omnisend_source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/omnisend/tests/test_omnisend_source.py@@ -20,8 +20,32 @@ class TestOmnisendSource:("api_version", "expected_tables"),[(OMNISEND_V3, {"contacts", "campaigns", "carts", "orders", "products", "categories"}),- (OMNISEND_2026_03_15, {"contacts", "campaigns", "products", "categories"}),- (None, {"contacts", "campaigns", "products", "categories"}),+ (+ OMNISEND_2026_03_15,+ {+ "contacts",+ "campaigns",+ "products",+ "categories",+ "automations",+ "segments",+ "analytics_reports",+ "analytics_statistics",+ },+ ),+ (+ None,+ {+ "contacts",+ "campaigns",+ "products",+ "categories",+ "automations",+ "segments",+ "analytics_reports",+ "analytics_statistics",+ },+ ),],)def test_schemas_follow_the_pinned_version(self, api_version: str | None, expected_tables: set[str]) -> None: