Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(segment): add 4 missing endpoint tables to the segment source (#129122)

Tom Owers3aa9c35f82cb
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index e006ffe65bf0..0db71cd03682 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -10718,17 +10718,17 @@ Note: The docs page is a Scalar viewer with the full OpenAPI YAML inlined — pa
## Segment — **thin**
-Today (11): `audit_events`, `destinations`, `iam_groups`, `iam_users`, `labels`, `reverse_etl_models`, `sources`, `tracking_plans`, `transformations`, `warehouses`, `workspace`
+Today (17): `audit_events`, `destination_delivery_metrics`, `destinations`, `destinations_catalog`, `iam_groups`, `iam_users`, `labels`, `reverse_etl_models`, `reverse_etl_syncs`, `sources`, `sources_catalog`, `tracking_plan_rules`, `tracking_plans`, `transformations`, `warehouses`, `warehouses_catalog`, `workspace`
Diffed against: <https://docs.segmentapis.com/ (Segment Public API OpenAPI 3.0.3, v73.0.7, extracted from the Redocly state bundle https://docs.segmentapis.com/redocly-state-bf9e006a07a5265a778ac3938ef8f8f891686c5b.js)>
- [ ] `/events/volume (Get Events Volume from Workspace)` — Segment's headline usage metric — event volume by source/day, the number every Segment customer wants in a warehouse (high)
- [ ] `/usage/mtu/daily and /usage/mtu/sources/daily` — Daily MTU usage per workspace and per source — the billing metric (high)
- [ ] `/warehouses/{warehouseId}/syncs and /warehouses/{warehouseId}/connected-sources/{sourceId}/syncs` — Warehouse sync run history/status — the state-transition history for the warehouses table already synced (high)
-- [ ] `/reverse-etl-models/{modelId}/subscriptionId/{subscriptionId}/syncs` — Reverse ETL sync statuses (rows loaded, failures) for the reverse_etl_models already synced (high)
-- [ ] `/destinations/{destinationId}/delivery-metrics` — Per-destination delivery success/failure counts — core deliverability analysis for the destinations table (high)
-- [ ] `/catalog/destinations, /catalog/sources, /catalog/warehouses` — Lookup tables resolving the metadata IDs carried on every synced source, destination, and warehouse row (high)
-- [ ] `/tracking-plans/{trackingPlanId}/rules` — The actual event/property rules inside a tracking plan — tracking_plans alone is just a header row (high)
+- [x] `/reverse-etl-models/{modelId}/subscriptionId/{subscriptionId}/syncs` — Reverse ETL sync statuses (rows loaded, failures) for the reverse_etl_models already synced (high)
+- [x] `/destinations/{destinationId}/delivery-metrics` — Per-destination delivery success/failure counts — core deliverability analysis for the destinations table (high)
+- [x] `/catalog/destinations, /catalog/sources, /catalog/warehouses` — Lookup tables resolving the metadata IDs carried on every synced source, destination, and warehouse row (high)
+- [x] `/tracking-plans/{trackingPlanId}/rules` — The actual event/property rules inside a tracking plan — tracking_plans alone is just a header row (high)
- [ ] `/delivery-overview/successful-delivery, /failed-delivery, /failed-on-ingest, /filtered-at-source, /filtered-at-destination` — Time-series pipeline health metrics broken down by stage; the standard Delivery Overview report (medium)
- [ ] `/destinations/{destinationId}/subscriptions` — Action mappings per destination — explains which events reach which destination (medium)
- [ ] `/spaces and /spaces/{spaceId}/audiences` — Engage spaces and audience definitions plus /audiences/{id}/activations; entirely absent today (medium)
diff --git a/products/warehouse_sources/sources/segment/canonical_descriptions.py b/products/warehouse_sources/sources/segment/canonical_descriptions.py
index a8dfa875c00e..e1aa0ed5094d 100644
--- a/products/warehouse_sources/sources/segment/canonical_descriptions.py
+++ b/products/warehouse_sources/sources/segment/canonical_descriptions.py
@@ -119,4 +119,88 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"resourceName": "Name of the resource the event acted on.",
},
},
+ "destinations_catalog": {
+ "description": "Segment's catalog of destination types. Resolves the `metadata.id` on each destination row.",
+ "docs_url": "https://docs.segmentapis.com/tag/Catalog",
+ "columns": {
+ "id": "Identifier of the destination metadata.",
+ "name": "User-friendly name of the destination type.",
+ "slug": "Slug that identifies the destination type in the Segment app.",
+ "description": "Description of the destination type.",
+ "status": "Support status of the destination type.",
+ "categories": "Categories the destination type belongs to.",
+ "supportedMethods": "Event methods the destination type supports.",
+ "supportedPlatforms": "Platforms from which the destination type receives events.",
+ "actions": "Actions available for the destination type.",
+ },
+ },
+ "sources_catalog": {
+ "description": "Segment's catalog of source types. Resolves the `metadata.id` on each source row.",
+ "docs_url": "https://docs.segmentapis.com/tag/Catalog",
+ "columns": {
+ "id": "Identifier of the source metadata.",
+ "name": "User-friendly name of the source type.",
+ "slug": "Slug that identifies the source type in the Segment app.",
+ "description": "Description of the source type.",
+ "categories": "Categories the source type belongs to.",
+ "isCloudEventSource": "Whether the source type is a cloud event source.",
+ "status": "Support status of the source type.",
+ },
+ },
+ "warehouses_catalog": {
+ "description": "Segment's catalog of warehouse types. Resolves the `metadata.id` on each warehouse row.",
+ "docs_url": "https://docs.segmentapis.com/tag/Catalog",
+ "columns": {
+ "id": "Identifier of the warehouse metadata.",
+ "name": "Name of the warehouse type.",
+ "slug": "Unique, human-readable identifier of the warehouse type.",
+ "description": "Description of the warehouse type.",
+ },
+ },
+ "tracking_plan_rules": {
+ "description": "The event and property rules inside each tracking plan.",
+ "docs_url": "https://docs.segmentapis.com/tag/Tracking-Plans",
+ "columns": {
+ "trackingPlanId": "Identifier of the tracking plan the rule belongs to.",
+ "type": "The type of the rule, for example TRACK or IDENTIFY.",
+ "key": "Key of the rule, such as the event name. Empty for rule types that have no key.",
+ "jsonSchema": "JSON Schema of the rule.",
+ "version": "Version of the rule.",
+ "createdAt": "When the rule was created.",
+ "updatedAt": "When the rule was last changed.",
+ "deprecatedAt": "When the rule was deprecated.",
+ },
+ },
+ "destination_delivery_metrics": {
+ "description": "Event delivery totals per destination and metric over the last 12 full UTC days.",
+ "docs_url": "https://docs.segmentapis.com/tag/Destinations",
+ "columns": {
+ "destinationId": "Identifier of the destination.",
+ "sourceId": "Identifier of the source connected to the destination.",
+ "destinationMetadataId": "Identifier of the destination type in the destinations catalog.",
+ "metricName": "Name of the delivery metric.",
+ "total": "Number of occurrences of the metric in the window.",
+ "breakdown": "Breakdown of the metric into sub-metrics.",
+ "windowStart": "Start of the reporting window (inclusive).",
+ "windowEnd": "End of the reporting window.",
+ },
+ },
+ "reverse_etl_syncs": {
+ "description": "Sync history of each Reverse ETL model subscription.",
+ "docs_url": "https://docs.segmentapis.com/tag/Reverse-ETL",
+ "columns": {
+ "subscriptionId": "Identifier of the destination subscription the sync loaded into.",
+ "syncId": "Identifier of the sync.",
+ "modelId": "Identifier of the Reverse ETL model.",
+ "sourceId": "Identifier of the Reverse ETL source.",
+ "syncStatus": "Status of the sync: IN_PROGRESS, FAIL, SUCCESS or PARTIAL_SUCCESS.",
+ "duration": "Duration of the sync.",
+ "startedAt": "When the sync started.",
+ "finishedAt": "When the sync finished.",
+ "extractPhase": "Details of the extract phase.",
+ "loadPhase": "Details of the load phase.",
+ "error": "Error message, if applicable.",
+ "errorCode": "Code of a fatal sync error, if applicable.",
+ },
+ },
}
diff --git a/products/warehouse_sources/sources/segment/segment.py b/products/warehouse_sources/sources/segment/segment.py
index c97c9ce8dd9c..3cb91e138df1 100644
--- a/products/warehouse_sources/sources/segment/segment.py
+++ b/products/warehouse_sources/sources/segment/segment.py
@@ -1,7 +1,9 @@
+import bisect
import dataclasses
-from collections.abc import Iterator
+from collections.abc import Callable, Iterator
+from datetime import UTC, datetime, timedelta
from typing import Any
-from urllib.parse import quote
+from urllib.parse import quote, urlencode
import requests
from structlog.types import FilteringBoundLogger
@@ -10,10 +12,13 @@ from tenacity import retry_if_exception_type, stop_after_attempt, wait_exponenti
from products.warehouse_sources.sources.sdk import (
ResumableSourceManager,
SourceResponse,
+ frozen,
make_tracked_session,
retry_with_deadline,
)
from products.warehouse_sources.sources.segment.settings import (
+ DELIVERY_METRICS_MAX_LOOKBACK_DAYS,
+ DELIVERY_METRICS_WINDOW_DAYS,
REGION_BASE_URLS,
SEGMENT_ENDPOINTS,
SegmentEndpointConfig,
@@ -23,6 +28,8 @@ from products.warehouse_sources.sources.segment.settings import (
# number of round-trips while staying within documented limits and well clear of the lower per-endpoint
# rate caps (some endpoints are throttled to 250 req/min).
PAGE_SIZE = 200
+# The Reverse ETL sync status endpoint pages with plain `count`/`cursor` params and caps `count` at 100.
+SYNC_STATUS_PAGE_SIZE = 100
class SegmentRetryableError(Exception):
@@ -32,7 +39,21 @@ class SegmentRetryableError(Exception):
@dataclasses.dataclass(frozen=False)
class SegmentResumeConfig:
# Opaque base64 cursor (`data.pagination.next`) for the next page of the endpoint being synced.
- cursor: str
+ cursor: str | None = None
+ # Fan-out tables: the parent to read next. `cursor` is then the next child page of that parent.
+ parent_id: str | None = None
+ # Fan-out tables: every page of `parent_id` is already yielded, so resume after it.
+ parent_done: bool = False
+ # Delivery metrics: the end of the reporting window, so a resumed run reports the same window.
+ window_end: str | None = None
+
+
[email protected](frozen=True)
+class _Parent:
+ id: str
+ path: str
+ # Added to each child row, so the row names the parent it came from.
+ row_fields: dict[str, Any]
def _base_url(region: str) -> str:
@@ -158,6 +179,282 @@ def get_rows(
resumable_source_manager.save_state(SegmentResumeConfig(cursor=cursor))
+def _fetch_unless_not_found(
+ session: requests.Session, url: str, headers: dict[str, str], logger: FilteringBoundLogger
+) -> dict | None:
+ try:
+ return _fetch(session, url, headers, logger)
+ except requests.HTTPError as error:
+ # A parent deleted after the parent listing returns 404. Skip it instead of failing the table.
+ if error.response is not None and error.response.status_code == 404:
+ logger.warning(f"Segment: skipping a parent that no longer exists, url={url}")
+ return None
+ raise
+
+
+def _list_parents(
+ session: requests.Session,
+ base_url: str,
+ path: str,
+ headers: dict[str, str],
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[SegmentResumeConfig],
+) -> list[dict[str, Any]]:
+ parents: list[dict[str, Any]] = []
+ cursor: str | None = None
+ while True:
+ data = _fetch_unless_not_found(session, _build_url(base_url, path, cursor), headers, logger)
+ if data is None:
+ return parents
+ parents.extend(row for row in _extract_rows(data) if isinstance(row, dict))
+ # A parent page yields no rows, so let the pipeline hand the run over between requests.
+ resumable_source_manager.safe_point()
+ cursor = _next_cursor(data)
+ if not cursor:
+ return parents
+
+
+_TIME_FORMAT = "%Y-%m-%dT%H:%M:%S.000Z"
+
+
+@frozen
+class _MetricsWindow:
+ start: str
+ end: str
+
+
+def _delivery_metrics_window(now: datetime, resumed_end: str | None) -> _MetricsWindow:
+ """Whole UTC days, so every sync reports the same span and today's partial day is left out.
+
+ A resumed run keeps the window of its first attempt while Segment still accepts that start time.
+ """
+ now = now.astimezone(UTC)
+ end = now.replace(hour=0, minute=0, second=0, microsecond=0)
+ if resumed_end is not None:
+ resumed = datetime.strptime(resumed_end, _TIME_FORMAT).replace(tzinfo=UTC)
+ if now - (resumed - timedelta(days=DELIVERY_METRICS_WINDOW_DAYS)) < timedelta(
+ days=DELIVERY_METRICS_MAX_LOOKBACK_DAYS
+ ):
+ end = resumed
+ start = end - timedelta(days=DELIVERY_METRICS_WINDOW_DAYS)
+ return _MetricsWindow(start=start.strftime(_TIME_FORMAT), end=end.strftime(_TIME_FORMAT))
+
+
+def _tracking_plan_parents(plans: list[dict[str, Any]], config: SegmentEndpointConfig) -> list[_Parent]:
+ return [
+ _Parent(
+ id=plan["id"],
+ path=config.path.format(tracking_plan_id=quote(plan["id"], safe="")),
+ row_fields={"trackingPlanId": plan["id"]},
+ )
+ for plan in plans
+ if plan.get("id")
+ ]
+
+
+def _delivery_metrics_parents(
+ destinations: list[dict[str, Any]], config: SegmentEndpointConfig, window: _MetricsWindow
+) -> list[_Parent]:
+ parents: list[_Parent] = []
+ for destination in destinations:
+ destination_id, source_id = destination.get("id"), destination.get("sourceId")
+ if not destination_id or not source_id:
+ continue
+ query = urlencode(
+ {"sourceId": source_id, "granularity": "DAY", "startTime": window.start, "endTime": window.end}
+ )
+ parents.append(
+ _Parent(
+ id=destination_id,
+ path=f"{config.path.format(destination_id=quote(destination_id, safe=''))}?{query}",
+ row_fields={
+ "destinationId": destination_id,
+ "sourceId": source_id,
+ "windowStart": window.start,
+ "windowEnd": window.end,
+ },
+ )
+ )
+ return parents
+
+
+def _reverse_etl_parents(
+ session: requests.Session,
+ base_url: str,
+ headers: dict[str, str],
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[SegmentResumeConfig],
+ config: SegmentEndpointConfig,
+) -> list[_Parent]:
+ """Sync history is keyed by (model, subscription), and only a destination's subscriptions carry
+ both. A Reverse ETL destination is connected to the Reverse ETL source of its model, so only
+ those destinations are asked for subscriptions."""
+ models = _list_parents(session, base_url, "/reverse-etl-models", headers, logger, resumable_source_manager)
+ model_ids = {model["id"] for model in models if model.get("id")}
+ model_source_ids = {model["sourceId"] for model in models if model.get("sourceId")}
+ if not model_ids:
+ return []
+
+ destinations = _list_parents(session, base_url, "/destinations", headers, logger, resumable_source_manager)
+ parents: list[_Parent] = []
+ for destination in destinations:
+ destination_id = destination.get("id")
+ if not destination_id or destination.get("sourceId") not in model_source_ids:
+ continue
+ subscriptions = _list_parents(
+ session,
+ base_url,
+ f"/destinations/{quote(destination_id, safe='')}/subscriptions",
+ headers,
+ logger,
+ resumable_source_manager,
+ )
+ for subscription in subscriptions:
+ subscription_id, model_id = subscription.get("id"), subscription.get("modelId")
+ if not subscription_id or not model_id or model_id not in model_ids:
+ continue
+ parents.append(
+ _Parent(
+ id=subscription_id,
+ path=config.path.format(
+ model_id=quote(model_id, safe=""), subscription_id=quote(subscription_id, safe="")
+ ),
+ row_fields={"subscriptionId": subscription_id},
+ )
+ )
+ return parents
+
+
+def _build_sync_status_url(base_url: str, path: str, cursor: str | None) -> str:
+ params: dict[str, str | int] = {"count": SYNC_STATUS_PAGE_SIZE}
+ if cursor:
+ params["cursor"] = cursor
+ return f"{base_url}{path}?{urlencode(params)}"
+
+
+def _build_plain_url(base_url: str, path: str, _cursor: str | None) -> str:
+ return f"{base_url}{path}"
+
+
+def _shape_list_rows(parent: _Parent, data: dict[str, Any]) -> list[dict[str, Any]]:
+ return [{**row, **parent.row_fields} for row in _extract_rows(data) if isinstance(row, dict)]
+
+
+def _shape_rule_rows(parent: _Parent, data: dict[str, Any]) -> list[dict[str, Any]]:
+ # `key` is part of the primary key, and identify, group and common rules have none.
+ return [{**row, "key": row.get("key") or ""} for row in _shape_list_rows(parent, data)]
+
+
+def _shape_delivery_metric_rows(parent: _Parent, data: dict[str, Any]) -> list[dict[str, Any]]:
+ payload = data.get("data")
+ summary = payload.get("deliveryMetricsSummary") if isinstance(payload, dict) else None
+ if not isinstance(summary, dict):
+ return []
+ return [
+ {**metric, "destinationMetadataId": summary.get("destinationMetadataId"), **parent.row_fields}
+ for metric in summary.get("metrics") or []
+ if isinstance(metric, dict)
+ ]
+
+
+def _fan_out(
+ session: requests.Session,
+ base_url: str,
+ headers: dict[str, str],
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[SegmentResumeConfig],
+ resume: SegmentResumeConfig | None,
+ parents: list[_Parent],
+ build_url: Callable[[str, str, str | None], str],
+ shape: Callable[[_Parent, dict[str, Any]], list[dict[str, Any]]],
+ redacted_fields: frozenset[str],
+ window_end: str | None = None,
+) -> Iterator[list[dict[str, Any]]]:
+ # Sorted by id, so a resumed attempt finds its position even when the parent it saved is gone.
+ parents = sorted(parents, key=lambda parent: parent.id)
+ start, cursor = 0, None
+ if resume is not None and resume.parent_id is not None:
+ parent_ids = [parent.id for parent in parents]
+ start = bisect.bisect_left(parent_ids, resume.parent_id)
+ found = start < len(parent_ids) and parent_ids[start] == resume.parent_id
+ if found and resume.parent_done:
+ start += 1
+ elif found:
+ cursor = resume.cursor
+ logger.debug(f"Segment: resuming at parent index={start} of {len(parents)}, cursor={cursor}")
+
+ for index in range(start, len(parents)):
+ parent = parents[index]
+ while True:
+ data = _fetch_unless_not_found(session, build_url(base_url, parent.path, cursor), headers, logger)
+ rows = _redact_rows(shape(parent, data), redacted_fields) if data is not None else []
+ cursor = _next_cursor(data) if data is not None else None
+
+ resumable_source_manager.save_state(
+ SegmentResumeConfig(
+ cursor=cursor, parent_id=parent.id, parent_done=cursor is None, window_end=window_end
+ )
+ )
+
+ if rows:
+ yield rows
+ else:
+ resumable_source_manager.safe_point()
+
+ if not cursor:
+ break
+
+
+def get_fanout_rows(
+ api_token: str,
+ region: str,
+ config: SegmentEndpointConfig,
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[SegmentResumeConfig],
+) -> Iterator[list[dict[str, Any]]]:
+ base_url = _base_url(region)
+ headers = _headers(api_token)
+ # Parent listings return destination `settings`, whose free-form credential keys the sample
+ # scrubber cannot recognise, so these requests stay out of HTTP sample capture.
+ session = make_tracked_session(redact_values=(api_token,), capture=False)
+
+ def list_parents(path: str) -> list[dict[str, Any]]:
+ return _list_parents(session, base_url, path, headers, logger, resumable_source_manager)
+
+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
+ parents: list[_Parent]
+ build_url: Callable[[str, str, str | None], str]
+ shape: Callable[[_Parent, dict[str, Any]], list[dict[str, Any]]]
+ window_end: str | None = None
+ if config.fanout == "tracking_plan_rules":
+ parents = _tracking_plan_parents(list_parents("/tracking-plans"), config)
+ build_url, shape = _build_url, _shape_rule_rows
+ elif config.fanout == "delivery_metrics":
+ window = _delivery_metrics_window(datetime.now(UTC), resume.window_end if resume else None)
+ window_end = window.end
+ parents = _delivery_metrics_parents(list_parents("/destinations"), config, window)
+ build_url, shape = _build_plain_url, _shape_delivery_metric_rows
+ elif config.fanout == "reverse_etl_syncs":
+ parents = _reverse_etl_parents(session, base_url, headers, logger, resumable_source_manager, config)
+ build_url, shape = _build_sync_status_url, _shape_list_rows
+ else:
+ raise ValueError(f"Segment endpoint {config.name} is not a fan-out endpoint")
+
+ yield from _fan_out(
+ session,
+ base_url,
+ headers,
+ logger,
+ resumable_source_manager,
+ resume,
+ parents,
+ build_url,
+ shape,
+ config.redacted_fields,
+ window_end,
+ )
+
+
def segment_source(
api_token: str,
region: str,
@@ -169,7 +466,7 @@ def segment_source(
return SourceResponse(
name=endpoint,
- items=lambda: get_rows(
+ items=lambda: (get_fanout_rows if config.fanout else get_rows)(
api_token=api_token,
region=region,
config=config,
diff --git a/products/warehouse_sources/sources/segment/settings.py b/products/warehouse_sources/sources/segment/settings.py
index 7dbc32639185..6b6530290fc2 100644
--- a/products/warehouse_sources/sources/segment/settings.py
+++ b/products/warehouse_sources/sources/segment/settings.py
@@ -1,5 +1,5 @@
from dataclasses import dataclass, field
-from typing import Optional
+from typing import Literal, Optional
from products.warehouse_sources.sources.sdk import IncrementalField
@@ -23,7 +23,7 @@ class SegmentEndpointConfig:
# rather than a paginated list. Such endpoints yield exactly one row.
single_object_key: Optional[str] = None
# A stable, immutable datetime field to partition on. Segment's config/admin objects carry no
- # timestamps, so only audit events (immutable `timestamp`) set this.
+ # timestamps, so only audit events (`timestamp`) and Reverse ETL syncs (`startedAt`) set this.
partition_key: Optional[str] = None
incremental_fields: list[IncrementalField] = field(default_factory=list)
should_sync_default: bool = True
@@ -33,6 +33,9 @@ class SegmentEndpointConfig:
# no reliable allowlist of safe sub-keys; we drop the whole field rather than risk leaking
# secrets into a queryable warehouse table.
redacted_fields: frozenset[str] = field(default_factory=frozenset)
+ # Child resources read once per parent row. `path` then holds the parent placeholders, and
+ # `segment.py` lists the parents and resumes per parent.
+ fanout: Optional[Literal["tracking_plan_rules", "delivery_metrics", "reverse_etl_syncs"]] = None
# The Segment Public API is a workspace configuration/admin/usage API (not the event/Profile data
@@ -60,8 +63,40 @@ SEGMENT_ENDPOINTS: dict[str, SegmentEndpointConfig] = {
"iam_groups": SegmentEndpointConfig(name="iam_groups", path="/groups"),
"labels": SegmentEndpointConfig(name="labels", path="/labels", primary_keys=["key", "value"]),
"audit_events": SegmentEndpointConfig(name="audit_events", path="/audit-events", partition_key="timestamp"),
+ # Segment-wide catalogs that resolve the `metadata.id` carried on sources, destinations and warehouses.
+ "destinations_catalog": SegmentEndpointConfig(name="destinations_catalog", path="/catalog/destinations"),
+ "sources_catalog": SegmentEndpointConfig(name="sources_catalog", path="/catalog/sources"),
+ "warehouses_catalog": SegmentEndpointConfig(name="warehouses_catalog", path="/catalog/warehouses"),
+ # Rules have no id. A plan holds one rule per (type, key, version), and keyless rule types
+ # (identify, group, common) get an empty `key`.
+ "tracking_plan_rules": SegmentEndpointConfig(
+ name="tracking_plan_rules",
+ path="/tracking-plans/{tracking_plan_id}/rules",
+ primary_keys=["trackingPlanId", "type", "key", "version"],
+ fanout="tracking_plan_rules",
+ ),
+ # One row per destination and metric, with totals over the last DELIVERY_METRICS_WINDOW_DAYS full days.
+ "destination_delivery_metrics": SegmentEndpointConfig(
+ name="destination_delivery_metrics",
+ path="/destinations/{destination_id}/delivery-metrics",
+ primary_keys=["destinationId", "metricName"],
+ fanout="delivery_metrics",
+ ),
+ # A sync can load into several subscriptions of one model, so the sync id is unique only per subscription.
+ "reverse_etl_syncs": SegmentEndpointConfig(
+ name="reverse_etl_syncs",
+ path="/reverse-etl-models/{model_id}/subscriptionId/{subscription_id}/syncs",
+ primary_keys=["subscriptionId", "syncId"],
+ partition_key="startedAt",
+ fanout="reverse_etl_syncs",
+ ),
}
+# Segment accepts a DAY-granularity range of at most 14 days that starts at most 14 days ago. 12 whole
+# days stay inside both limits, also for a resumed run that keeps the window of its first attempt.
+DELIVERY_METRICS_WINDOW_DAYS = 12
+DELIVERY_METRICS_MAX_LOOKBACK_DAYS = 14
+
ENDPOINTS = tuple(SEGMENT_ENDPOINTS.keys())
INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {
diff --git a/products/warehouse_sources/sources/segment/tests/test_segment.py b/products/warehouse_sources/sources/segment/tests/test_segment.py
index e5428dab0b90..85d0620a7d0d 100644
--- a/products/warehouse_sources/sources/segment/tests/test_segment.py
+++ b/products/warehouse_sources/sources/segment/tests/test_segment.py
@@ -1,5 +1,9 @@
+import datetime as dt
from typing import Any, Literal, cast
+import time_machine
+
+import requests
from parameterized import parameterized
from products.warehouse_sources.sources.sdk.testing import ScriptedResponse, SourceDriver
@@ -152,3 +156,186 @@ class TestSegmentSource:
assert result.response is not None
assert result.response.partition_mode == expected_mode
assert result.response.partition_keys == expected_keys
+
+
+def _list(key: str, rows: list[dict[str, Any]], next_cursor: str | None = None) -> ScriptedResponse:
+ return _page({"data": {key: rows, "pagination": {"next": next_cursor}}})
+
+
+_TRACKING_PLAN_PAGES = [
+ _list("trackingPlans", [{"id": "tp_b"}], "plans2"),
+ _list("trackingPlans", [{"id": "tp_a"}]),
+]
+
+
+class TestFanOut:
+ def test_tracking_plan_rules_walk_every_plan_and_child_page(self) -> None:
+ result = _driver().run(
+ "tracking_plan_rules",
+ [
+ *_TRACKING_PLAN_PAGES,
+ _list("rules", [{"type": "TRACK", "key": "Clicked", "version": 1}], "rules2"),
+ _list("rules", [{"type": "IDENTIFY", "version": 1}]),
+ _list("rules", []),
+ ],
+ )
+
+ assert result.paths == [
+ "/tracking-plans",
+ "/tracking-plans",
+ "/tracking-plans/tp_a/rules",
+ "/tracking-plans/tp_a/rules",
+ "/tracking-plans/tp_b/rules",
+ ]
+ assert result.params("pagination[cursor]") == [None, "plans2", None, "rules2", None]
+ assert result.rows == [
+ {"type": "TRACK", "key": "Clicked", "version": 1, "trackingPlanId": "tp_a"},
+ {"type": "IDENTIFY", "key": "", "version": 1, "trackingPlanId": "tp_a"},
+ ]
+ assert result.saved_states == [
+ SegmentResumeConfig(cursor="rules2", parent_id="tp_a"),
+ SegmentResumeConfig(parent_id="tp_a", parent_done=True),
+ SegmentResumeConfig(parent_id="tp_b", parent_done=True),
+ ]
+
+ @parameterized.expand(
+ [
+ (
+ "known_parent_resumes_at_its_child_cursor",
+ SegmentResumeConfig(cursor="rules2", parent_id="tp_b"),
+ ["/tracking-plans/tp_b/rules"],
+ ["rules2"],
+ ),
+ (
+ "deleted_parent_continues_at_the_next_parent",
+ SegmentResumeConfig(cursor="gone_cursor", parent_id="tp_aa"),
+ ["/tracking-plans/tp_b/rules"],
+ [None],
+ ),
+ (
+ "done_parent_continues_at_the_next_parent",
+ SegmentResumeConfig(parent_id="tp_a", parent_done=True),
+ ["/tracking-plans/tp_b/rules"],
+ [None],
+ ),
+ ("done_last_parent_reads_no_child", SegmentResumeConfig(parent_id="tp_b", parent_done=True), [], []),
+ ]
+ )
+ def test_resume(
+ self,
+ _name: str,
+ state: SegmentResumeConfig,
+ expected_child_paths: list[str],
+ expected_child_cursors: list[str | None],
+ ) -> None:
+ child_pages = [_list("rules", [{"type": "TRACK", "key": "k", "version": 1}]) for _ in expected_child_paths]
+ result = _driver().run("tracking_plan_rules", [*_TRACKING_PLAN_PAGES, *child_pages], resume_state=state)
+
+ assert result.paths[2:] == expected_child_paths
+ assert result.params("pagination[cursor]")[2:] == expected_child_cursors
+
+ @parameterized.expand([("not_found_is_skipped", 404, None), ("forbidden_fails", 403, requests.HTTPError)])
+ def test_child_error(self, _name: str, status: int, expected_error: type[BaseException] | None) -> None:
+ result = _driver().run(
+ "tracking_plan_rules",
+ [
+ *_TRACKING_PLAN_PAGES,
+ ScriptedResponse(status=status, json={"errors": []}),
+ _list("rules", [{"type": "TRACK", "key": "k", "version": 1}]),
+ ],
+ )
+
+ if expected_error is None:
+ assert result.raised is None
+ assert result.rows == [{"type": "TRACK", "key": "k", "version": 1, "trackingPlanId": "tp_b"}]
+ else:
+ assert isinstance(result.raised, expected_error)
+ assert result.rows == []
+
+ @parameterized.expand(
+ [
+ ("fresh_run", None, "2026-03-03T00:00:00.000Z", "2026-03-15T00:00:00.000Z"),
+ (
+ "resumed_run_keeps_its_window",
+ SegmentResumeConfig(window_end="2026-03-14T00:00:00.000Z"),
+ "2026-03-02T00:00:00.000Z",
+ "2026-03-14T00:00:00.000Z",
+ ),
+ (
+ "resumed_window_past_the_lookback_limit_is_replaced",
+ SegmentResumeConfig(window_end="2026-03-13T00:00:00.000Z"),
+ "2026-03-03T00:00:00.000Z",
+ "2026-03-15T00:00:00.000Z",
+ ),
+ ]
+ )
+ def test_delivery_metrics_request_a_whole_day_window_per_destination(
+ self, _name: str, resume_state: SegmentResumeConfig | None, window_start: str, window_end: str
+ ) -> None:
+ summary = {
+ "sourceId": "src1",
+ "destinationMetadataId": "meta1",
+ "metrics": [
+ {"metricName": "successes", "total": 10},
+ {"metricName": "discarded", "total": 2, "breakdown": [{"metricName": "dropped", "value": 2}]},
+ ],
+ }
+ with time_machine.travel(dt.datetime(2026, 3, 15, 10, 30, tzinfo=dt.UTC), tick=False):
+ result = _driver().run(
+ "destination_delivery_metrics",
+ [
+ _list("destinations", [{"id": "d1", "sourceId": "src1"}, {"id": "d_no_source"}]),
+ _page({"data": {"deliveryMetricsSummary": summary}}),
+ ],
+ resume_state=resume_state,
+ )
+
+ window = {
+ "destinationId": "d1",
+ "sourceId": "src1",
+ "destinationMetadataId": "meta1",
+ "windowStart": window_start,
+ "windowEnd": window_end,
+ }
+ assert result.paths == ["/destinations", "/destinations/d1/delivery-metrics"]
+ request = result.requests[1]
+ assert (request.param("sourceId"), request.param("granularity")) == ("src1", "DAY")
+ assert (request.param("startTime"), request.param("endTime")) == (window["windowStart"], window["windowEnd"])
+ assert result.rows == [
+ {"metricName": "successes", "total": 10, **window},
+ {"metricName": "discarded", "total": 2, "breakdown": [{"metricName": "dropped", "value": 2}], **window},
+ ]
+
+ def test_reverse_etl_syncs_read_each_model_subscription(self) -> None:
+ result = _driver().run(
+ "reverse_etl_syncs",
+ [
+ _list("models", [{"id": "m1", "sourceId": "retl_src"}]),
+ _list(
+ "destinations",
+ [{"id": "d_retl", "sourceId": "retl_src"}, {"id": "d_events", "sourceId": "web_src"}],
+ ),
+ _list(
+ "subscriptions",
+ [{"id": "sub1", "modelId": "m1"}, {"id": "sub_event", "destinationId": "d_retl"}],
+ ),
+ _list("syncStatuses", [{"syncId": "s1", "modelId": "m1"}], "syncs2"),
+ _list("syncStatuses", [{"syncId": "s2", "modelId": "m1"}]),
+ ],
+ )
+
+ assert result.paths == [
+ "/reverse-etl-models",
+ "/destinations",
+ "/destinations/d_retl/subscriptions",
+ "/reverse-etl-models/m1/subscriptionId/sub1/syncs",
+ "/reverse-etl-models/m1/subscriptionId/sub1/syncs",
+ ]
+ assert [(request.param("count"), request.param("cursor")) for request in result.requests[3:]] == [
+ ("100", None),
+ ("100", "syncs2"),
+ ]
+ assert result.rows == [
+ {"syncId": "s1", "modelId": "m1", "subscriptionId": "sub1"},
+ {"syncId": "s2", "modelId": "m1", "subscriptionId": "sub1"},
+ ]