:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(segment): add 4 missing endpoint tables to the segment source (#129122)
Tom Owers3aa9c35f82cb
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex e006ffe65bf0..0db71cd03682 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -10718,17 +10718,17 @@ Note: The docs page is a Scalar viewer with the full OpenAPI YAML inlined — pa## Segment — **thin**-Today (11): `audit_events`, `destinations`, `iam_groups`, `iam_users`, `labels`, `reverse_etl_models`, `sources`, `tracking_plans`, `transformations`, `warehouses`, `workspace`+Today (17): `audit_events`, `destination_delivery_metrics`, `destinations`, `destinations_catalog`, `iam_groups`, `iam_users`, `labels`, `reverse_etl_models`, `reverse_etl_syncs`, `sources`, `sources_catalog`, `tracking_plan_rules`, `tracking_plans`, `transformations`, `warehouses`, `warehouses_catalog`, `workspace`Diffed against: <https://docs.segmentapis.com/ (Segment Public API OpenAPI 3.0.3, v73.0.7, extracted from the Redocly state bundle https://docs.segmentapis.com/redocly-state-bf9e006a07a5265a778ac3938ef8f8f891686c5b.js)>- [ ] `/events/volume (Get Events Volume from Workspace)` — Segment's headline usage metric — event volume by source/day, the number every Segment customer wants in a warehouse (high)- [ ] `/usage/mtu/daily and /usage/mtu/sources/daily` — Daily MTU usage per workspace and per source — the billing metric (high)- [ ] `/warehouses/{warehouseId}/syncs and /warehouses/{warehouseId}/connected-sources/{sourceId}/syncs` — Warehouse sync run history/status — the state-transition history for the warehouses table already synced (high)-- [ ] `/reverse-etl-models/{modelId}/subscriptionId/{subscriptionId}/syncs` — Reverse ETL sync statuses (rows loaded, failures) for the reverse_etl_models already synced (high)-- [ ] `/destinations/{destinationId}/delivery-metrics` — Per-destination delivery success/failure counts — core deliverability analysis for the destinations table (high)-- [ ] `/catalog/destinations, /catalog/sources, /catalog/warehouses` — Lookup tables resolving the metadata IDs carried on every synced source, destination, and warehouse row (high)-- [ ] `/tracking-plans/{trackingPlanId}/rules` — The actual event/property rules inside a tracking plan — tracking_plans alone is just a header row (high)+- [x] `/reverse-etl-models/{modelId}/subscriptionId/{subscriptionId}/syncs` — Reverse ETL sync statuses (rows loaded, failures) for the reverse_etl_models already synced (high)+- [x] `/destinations/{destinationId}/delivery-metrics` — Per-destination delivery success/failure counts — core deliverability analysis for the destinations table (high)+- [x] `/catalog/destinations, /catalog/sources, /catalog/warehouses` — Lookup tables resolving the metadata IDs carried on every synced source, destination, and warehouse row (high)+- [x] `/tracking-plans/{trackingPlanId}/rules` — The actual event/property rules inside a tracking plan — tracking_plans alone is just a header row (high)- [ ] `/delivery-overview/successful-delivery, /failed-delivery, /failed-on-ingest, /filtered-at-source, /filtered-at-destination` — Time-series pipeline health metrics broken down by stage; the standard Delivery Overview report (medium)- [ ] `/destinations/{destinationId}/subscriptions` — Action mappings per destination — explains which events reach which destination (medium)- [ ] `/spaces and /spaces/{spaceId}/audiences` — Engage spaces and audience definitions plus /audiences/{id}/activations; entirely absent today (medium)diff --git a/products/warehouse_sources/sources/segment/canonical_descriptions.py b/products/warehouse_sources/sources/segment/canonical_descriptions.pyindex a8dfa875c00e..e1aa0ed5094d 100644--- a/products/warehouse_sources/sources/segment/canonical_descriptions.py+++ b/products/warehouse_sources/sources/segment/canonical_descriptions.py@@ -119,4 +119,88 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"resourceName": "Name of the resource the event acted on.",},},+ "destinations_catalog": {+ "description": "Segment's catalog of destination types. Resolves the `metadata.id` on each destination row.",+ "docs_url": "https://docs.segmentapis.com/tag/Catalog",+ "columns": {+ "id": "Identifier of the destination metadata.",+ "name": "User-friendly name of the destination type.",+ "slug": "Slug that identifies the destination type in the Segment app.",+ "description": "Description of the destination type.",+ "status": "Support status of the destination type.",+ "categories": "Categories the destination type belongs to.",+ "supportedMethods": "Event methods the destination type supports.",+ "supportedPlatforms": "Platforms from which the destination type receives events.",+ "actions": "Actions available for the destination type.",+ },+ },+ "sources_catalog": {+ "description": "Segment's catalog of source types. Resolves the `metadata.id` on each source row.",+ "docs_url": "https://docs.segmentapis.com/tag/Catalog",+ "columns": {+ "id": "Identifier of the source metadata.",+ "name": "User-friendly name of the source type.",+ "slug": "Slug that identifies the source type in the Segment app.",+ "description": "Description of the source type.",+ "categories": "Categories the source type belongs to.",+ "isCloudEventSource": "Whether the source type is a cloud event source.",+ "status": "Support status of the source type.",+ },+ },+ "warehouses_catalog": {+ "description": "Segment's catalog of warehouse types. Resolves the `metadata.id` on each warehouse row.",+ "docs_url": "https://docs.segmentapis.com/tag/Catalog",+ "columns": {+ "id": "Identifier of the warehouse metadata.",+ "name": "Name of the warehouse type.",+ "slug": "Unique, human-readable identifier of the warehouse type.",+ "description": "Description of the warehouse type.",+ },+ },+ "tracking_plan_rules": {+ "description": "The event and property rules inside each tracking plan.",+ "docs_url": "https://docs.segmentapis.com/tag/Tracking-Plans",+ "columns": {+ "trackingPlanId": "Identifier of the tracking plan the rule belongs to.",+ "type": "The type of the rule, for example TRACK or IDENTIFY.",+ "key": "Key of the rule, such as the event name. Empty for rule types that have no key.",+ "jsonSchema": "JSON Schema of the rule.",+ "version": "Version of the rule.",+ "createdAt": "When the rule was created.",+ "updatedAt": "When the rule was last changed.",+ "deprecatedAt": "When the rule was deprecated.",+ },+ },+ "destination_delivery_metrics": {+ "description": "Event delivery totals per destination and metric over the last 12 full UTC days.",+ "docs_url": "https://docs.segmentapis.com/tag/Destinations",+ "columns": {+ "destinationId": "Identifier of the destination.",+ "sourceId": "Identifier of the source connected to the destination.",+ "destinationMetadataId": "Identifier of the destination type in the destinations catalog.",+ "metricName": "Name of the delivery metric.",+ "total": "Number of occurrences of the metric in the window.",+ "breakdown": "Breakdown of the metric into sub-metrics.",+ "windowStart": "Start of the reporting window (inclusive).",+ "windowEnd": "End of the reporting window.",+ },+ },+ "reverse_etl_syncs": {+ "description": "Sync history of each Reverse ETL model subscription.",+ "docs_url": "https://docs.segmentapis.com/tag/Reverse-ETL",+ "columns": {+ "subscriptionId": "Identifier of the destination subscription the sync loaded into.",+ "syncId": "Identifier of the sync.",+ "modelId": "Identifier of the Reverse ETL model.",+ "sourceId": "Identifier of the Reverse ETL source.",+ "syncStatus": "Status of the sync: IN_PROGRESS, FAIL, SUCCESS or PARTIAL_SUCCESS.",+ "duration": "Duration of the sync.",+ "startedAt": "When the sync started.",+ "finishedAt": "When the sync finished.",+ "extractPhase": "Details of the extract phase.",+ "loadPhase": "Details of the load phase.",+ "error": "Error message, if applicable.",+ "errorCode": "Code of a fatal sync error, if applicable.",+ },+ },}diff --git a/products/warehouse_sources/sources/segment/segment.py b/products/warehouse_sources/sources/segment/segment.pyindex c97c9ce8dd9c..3cb91e138df1 100644--- a/products/warehouse_sources/sources/segment/segment.py+++ b/products/warehouse_sources/sources/segment/segment.py@@ -1,7 +1,9 @@+import bisectimport dataclasses-from collections.abc import Iterator+from collections.abc import Callable, Iterator+from datetime import UTC, datetime, timedeltafrom typing import Any-from urllib.parse import quote+from urllib.parse import quote, urlencodeimport requestsfrom structlog.types import FilteringBoundLogger@@ -10,10 +12,13 @@ from tenacity import retry_if_exception_type, stop_after_attempt, wait_exponentifrom products.warehouse_sources.sources.sdk import (ResumableSourceManager,SourceResponse,+ frozen,make_tracked_session,retry_with_deadline,)from products.warehouse_sources.sources.segment.settings import (+ DELIVERY_METRICS_MAX_LOOKBACK_DAYS,+ DELIVERY_METRICS_WINDOW_DAYS,REGION_BASE_URLS,SEGMENT_ENDPOINTS,SegmentEndpointConfig,@@ -23,6 +28,8 @@ from products.warehouse_sources.sources.segment.settings import (# number of round-trips while staying within documented limits and well clear of the lower per-endpoint# rate caps (some endpoints are throttled to 250 req/min).PAGE_SIZE = 200+# The Reverse ETL sync status endpoint pages with plain `count`/`cursor` params and caps `count` at 100.+SYNC_STATUS_PAGE_SIZE = 100class SegmentRetryableError(Exception):@@ -32,7 +39,21 @@ class SegmentRetryableError(Exception):@dataclasses.dataclass(frozen=False)class SegmentResumeConfig:# Opaque base64 cursor (`data.pagination.next`) for the next page of the endpoint being synced.- cursor: str+ cursor: str | None = None+ # Fan-out tables: the parent to read next. `cursor` is then the next child page of that parent.+ parent_id: str | None = None+ # Fan-out tables: every page of `parent_id` is already yielded, so resume after it.+ parent_done: bool = False+ # Delivery metrics: the end of the reporting window, so a resumed run reports the same window.+ window_end: str | None = None++[email protected](frozen=True)+class _Parent:+ id: str+ path: str+ # Added to each child row, so the row names the parent it came from.+ row_fields: dict[str, Any]def _base_url(region: str) -> str:@@ -158,6 +179,282 @@ def get_rows(resumable_source_manager.save_state(SegmentResumeConfig(cursor=cursor))+def _fetch_unless_not_found(+ session: requests.Session, url: str, headers: dict[str, str], logger: FilteringBoundLogger+) -> dict | None:+ try:+ return _fetch(session, url, headers, logger)+ except requests.HTTPError as error:+ # A parent deleted after the parent listing returns 404. Skip it instead of failing the table.+ if error.response is not None and error.response.status_code == 404:+ logger.warning(f"Segment: skipping a parent that no longer exists, url={url}")+ return None+ raise+++def _list_parents(+ session: requests.Session,+ base_url: str,+ path: str,+ headers: dict[str, str],+ logger: FilteringBoundLogger,+ resumable_source_manager: ResumableSourceManager[SegmentResumeConfig],+) -> list[dict[str, Any]]:+ parents: list[dict[str, Any]] = []+ cursor: str | None = None+ while True:+ data = _fetch_unless_not_found(session, _build_url(base_url, path, cursor), headers, logger)+ if data is None:+ return parents+ parents.extend(row for row in _extract_rows(data) if isinstance(row, dict))+ # A parent page yields no rows, so let the pipeline hand the run over between requests.+ resumable_source_manager.safe_point()+ cursor = _next_cursor(data)+ if not cursor:+ return parents+++_TIME_FORMAT = "%Y-%m-%dT%H:%M:%S.000Z"+++@frozen+class _MetricsWindow:+ start: str+ end: str+++def _delivery_metrics_window(now: datetime, resumed_end: str | None) -> _MetricsWindow:+ """Whole UTC days, so every sync reports the same span and today's partial day is left out.++ A resumed run keeps the window of its first attempt while Segment still accepts that start time.+ """+ now = now.astimezone(UTC)+ end = now.replace(hour=0, minute=0, second=0, microsecond=0)+ if resumed_end is not None:+ resumed = datetime.strptime(resumed_end, _TIME_FORMAT).replace(tzinfo=UTC)+ if now - (resumed - timedelta(days=DELIVERY_METRICS_WINDOW_DAYS)) < timedelta(+ days=DELIVERY_METRICS_MAX_LOOKBACK_DAYS+ ):+ end = resumed+ start = end - timedelta(days=DELIVERY_METRICS_WINDOW_DAYS)+ return _MetricsWindow(start=start.strftime(_TIME_FORMAT), end=end.strftime(_TIME_FORMAT))+++def _tracking_plan_parents(plans: list[dict[str, Any]], config: SegmentEndpointConfig) -> list[_Parent]:+ return [+ _Parent(+ id=plan["id"],+ path=config.path.format(tracking_plan_id=quote(plan["id"], safe="")),+ row_fields={"trackingPlanId": plan["id"]},+ )+ for plan in plans+ if plan.get("id")+ ]+++def _delivery_metrics_parents(+ destinations: list[dict[str, Any]], config: SegmentEndpointConfig, window: _MetricsWindow+) -> list[_Parent]:+ parents: list[_Parent] = []+ for destination in destinations:+ destination_id, source_id = destination.get("id"), destination.get("sourceId")+ if not destination_id or not source_id:+ continue+ query = urlencode(+ {"sourceId": source_id, "granularity": "DAY", "startTime": window.start, "endTime": window.end}+ )+ parents.append(+ _Parent(+ id=destination_id,+ path=f"{config.path.format(destination_id=quote(destination_id, safe=''))}?{query}",+ row_fields={+ "destinationId": destination_id,+ "sourceId": source_id,+ "windowStart": window.start,+ "windowEnd": window.end,+ },+ )+ )+ return parents+++def _reverse_etl_parents(+ session: requests.Session,+ base_url: str,+ headers: dict[str, str],+ logger: FilteringBoundLogger,+ resumable_source_manager: ResumableSourceManager[SegmentResumeConfig],+ config: SegmentEndpointConfig,+) -> list[_Parent]:+ """Sync history is keyed by (model, subscription), and only a destination's subscriptions carry+ both. A Reverse ETL destination is connected to the Reverse ETL source of its model, so only+ those destinations are asked for subscriptions."""+ models = _list_parents(session, base_url, "/reverse-etl-models", headers, logger, resumable_source_manager)+ model_ids = {model["id"] for model in models if model.get("id")}+ model_source_ids = {model["sourceId"] for model in models if model.get("sourceId")}+ if not model_ids:+ return []++ destinations = _list_parents(session, base_url, "/destinations", headers, logger, resumable_source_manager)+ parents: list[_Parent] = []+ for destination in destinations:+ destination_id = destination.get("id")+ if not destination_id or destination.get("sourceId") not in model_source_ids:+ continue+ subscriptions = _list_parents(+ session,+ base_url,+ f"/destinations/{quote(destination_id, safe='')}/subscriptions",+ headers,+ logger,+ resumable_source_manager,+ )+ for subscription in subscriptions:+ subscription_id, model_id = subscription.get("id"), subscription.get("modelId")+ if not subscription_id or not model_id or model_id not in model_ids:+ continue+ parents.append(+ _Parent(+ id=subscription_id,+ path=config.path.format(+ model_id=quote(model_id, safe=""), subscription_id=quote(subscription_id, safe="")+ ),+ row_fields={"subscriptionId": subscription_id},+ )+ )+ return parents+++def _build_sync_status_url(base_url: str, path: str, cursor: str | None) -> str:+ params: dict[str, str | int] = {"count": SYNC_STATUS_PAGE_SIZE}+ if cursor:+ params["cursor"] = cursor+ return f"{base_url}{path}?{urlencode(params)}"+++def _build_plain_url(base_url: str, path: str, _cursor: str | None) -> str:+ return f"{base_url}{path}"+++def _shape_list_rows(parent: _Parent, data: dict[str, Any]) -> list[dict[str, Any]]:+ return [{**row, **parent.row_fields} for row in _extract_rows(data) if isinstance(row, dict)]+++def _shape_rule_rows(parent: _Parent, data: dict[str, Any]) -> list[dict[str, Any]]:+ # `key` is part of the primary key, and identify, group and common rules have none.+ return [{**row, "key": row.get("key") or ""} for row in _shape_list_rows(parent, data)]+++def _shape_delivery_metric_rows(parent: _Parent, data: dict[str, Any]) -> list[dict[str, Any]]:+ payload = data.get("data")+ summary = payload.get("deliveryMetricsSummary") if isinstance(payload, dict) else None+ if not isinstance(summary, dict):+ return []+ return [+ {**metric, "destinationMetadataId": summary.get("destinationMetadataId"), **parent.row_fields}+ for metric in summary.get("metrics") or []+ if isinstance(metric, dict)+ ]+++def _fan_out(+ session: requests.Session,+ base_url: str,+ headers: dict[str, str],+ logger: FilteringBoundLogger,+ resumable_source_manager: ResumableSourceManager[SegmentResumeConfig],+ resume: SegmentResumeConfig | None,+ parents: list[_Parent],+ build_url: Callable[[str, str, str | None], str],+ shape: Callable[[_Parent, dict[str, Any]], list[dict[str, Any]]],+ redacted_fields: frozenset[str],+ window_end: str | None = None,+) -> Iterator[list[dict[str, Any]]]:+ # Sorted by id, so a resumed attempt finds its position even when the parent it saved is gone.+ parents = sorted(parents, key=lambda parent: parent.id)+ start, cursor = 0, None+ if resume is not None and resume.parent_id is not None:+ parent_ids = [parent.id for parent in parents]+ start = bisect.bisect_left(parent_ids, resume.parent_id)+ found = start < len(parent_ids) and parent_ids[start] == resume.parent_id+ if found and resume.parent_done:+ start += 1+ elif found:+ cursor = resume.cursor+ logger.debug(f"Segment: resuming at parent index={start} of {len(parents)}, cursor={cursor}")++ for index in range(start, len(parents)):+ parent = parents[index]+ while True:+ data = _fetch_unless_not_found(session, build_url(base_url, parent.path, cursor), headers, logger)+ rows = _redact_rows(shape(parent, data), redacted_fields) if data is not None else []+ cursor = _next_cursor(data) if data is not None else None++ resumable_source_manager.save_state(+ SegmentResumeConfig(+ cursor=cursor, parent_id=parent.id, parent_done=cursor is None, window_end=window_end+ )+ )++ if rows:+ yield rows+ else:+ resumable_source_manager.safe_point()++ if not cursor:+ break+++def get_fanout_rows(+ api_token: str,+ region: str,+ config: SegmentEndpointConfig,+ logger: FilteringBoundLogger,+ resumable_source_manager: ResumableSourceManager[SegmentResumeConfig],+) -> Iterator[list[dict[str, Any]]]:+ base_url = _base_url(region)+ headers = _headers(api_token)+ # Parent listings return destination `settings`, whose free-form credential keys the sample+ # scrubber cannot recognise, so these requests stay out of HTTP sample capture.+ session = make_tracked_session(redact_values=(api_token,), capture=False)++ def list_parents(path: str) -> list[dict[str, Any]]:+ return _list_parents(session, base_url, path, headers, logger, resumable_source_manager)++ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None+ parents: list[_Parent]+ build_url: Callable[[str, str, str | None], str]+ shape: Callable[[_Parent, dict[str, Any]], list[dict[str, Any]]]+ window_end: str | None = None+ if config.fanout == "tracking_plan_rules":+ parents = _tracking_plan_parents(list_parents("/tracking-plans"), config)+ build_url, shape = _build_url, _shape_rule_rows+ elif config.fanout == "delivery_metrics":+ window = _delivery_metrics_window(datetime.now(UTC), resume.window_end if resume else None)+ window_end = window.end+ parents = _delivery_metrics_parents(list_parents("/destinations"), config, window)+ build_url, shape = _build_plain_url, _shape_delivery_metric_rows+ elif config.fanout == "reverse_etl_syncs":+ parents = _reverse_etl_parents(session, base_url, headers, logger, resumable_source_manager, config)+ build_url, shape = _build_sync_status_url, _shape_list_rows+ else:+ raise ValueError(f"Segment endpoint {config.name} is not a fan-out endpoint")++ yield from _fan_out(+ session,+ base_url,+ headers,+ logger,+ resumable_source_manager,+ resume,+ parents,+ build_url,+ shape,+ config.redacted_fields,+ window_end,+ )++def segment_source(api_token: str,region: str,@@ -169,7 +466,7 @@ def segment_source(return SourceResponse(name=endpoint,- items=lambda: get_rows(+ items=lambda: (get_fanout_rows if config.fanout else get_rows)(api_token=api_token,region=region,config=config,diff --git a/products/warehouse_sources/sources/segment/settings.py b/products/warehouse_sources/sources/segment/settings.pyindex 7dbc32639185..6b6530290fc2 100644--- a/products/warehouse_sources/sources/segment/settings.py+++ b/products/warehouse_sources/sources/segment/settings.py@@ -1,5 +1,5 @@from dataclasses import dataclass, field-from typing import Optional+from typing import Literal, Optionalfrom products.warehouse_sources.sources.sdk import IncrementalField@@ -23,7 +23,7 @@ class SegmentEndpointConfig:# rather than a paginated list. Such endpoints yield exactly one row.single_object_key: Optional[str] = None# A stable, immutable datetime field to partition on. Segment's config/admin objects carry no- # timestamps, so only audit events (immutable `timestamp`) set this.+ # timestamps, so only audit events (`timestamp`) and Reverse ETL syncs (`startedAt`) set this.partition_key: Optional[str] = Noneincremental_fields: list[IncrementalField] = field(default_factory=list)should_sync_default: bool = True@@ -33,6 +33,9 @@ class SegmentEndpointConfig:# no reliable allowlist of safe sub-keys; we drop the whole field rather than risk leaking# secrets into a queryable warehouse table.redacted_fields: frozenset[str] = field(default_factory=frozenset)+ # Child resources read once per parent row. `path` then holds the parent placeholders, and+ # `segment.py` lists the parents and resumes per parent.+ fanout: Optional[Literal["tracking_plan_rules", "delivery_metrics", "reverse_etl_syncs"]] = None# The Segment Public API is a workspace configuration/admin/usage API (not the event/Profile data@@ -60,8 +63,40 @@ SEGMENT_ENDPOINTS: dict[str, SegmentEndpointConfig] = {"iam_groups": SegmentEndpointConfig(name="iam_groups", path="/groups"),"labels": SegmentEndpointConfig(name="labels", path="/labels", primary_keys=["key", "value"]),"audit_events": SegmentEndpointConfig(name="audit_events", path="/audit-events", partition_key="timestamp"),+ # Segment-wide catalogs that resolve the `metadata.id` carried on sources, destinations and warehouses.+ "destinations_catalog": SegmentEndpointConfig(name="destinations_catalog", path="/catalog/destinations"),+ "sources_catalog": SegmentEndpointConfig(name="sources_catalog", path="/catalog/sources"),+ "warehouses_catalog": SegmentEndpointConfig(name="warehouses_catalog", path="/catalog/warehouses"),+ # Rules have no id. A plan holds one rule per (type, key, version), and keyless rule types+ # (identify, group, common) get an empty `key`.+ "tracking_plan_rules": SegmentEndpointConfig(+ name="tracking_plan_rules",+ path="/tracking-plans/{tracking_plan_id}/rules",+ primary_keys=["trackingPlanId", "type", "key", "version"],+ fanout="tracking_plan_rules",+ ),+ # One row per destination and metric, with totals over the last DELIVERY_METRICS_WINDOW_DAYS full days.+ "destination_delivery_metrics": SegmentEndpointConfig(+ name="destination_delivery_metrics",+ path="/destinations/{destination_id}/delivery-metrics",+ primary_keys=["destinationId", "metricName"],+ fanout="delivery_metrics",+ ),+ # A sync can load into several subscriptions of one model, so the sync id is unique only per subscription.+ "reverse_etl_syncs": SegmentEndpointConfig(+ name="reverse_etl_syncs",+ path="/reverse-etl-models/{model_id}/subscriptionId/{subscription_id}/syncs",+ primary_keys=["subscriptionId", "syncId"],+ partition_key="startedAt",+ fanout="reverse_etl_syncs",+ ),}+# Segment accepts a DAY-granularity range of at most 14 days that starts at most 14 days ago. 12 whole+# days stay inside both limits, also for a resumed run that keeps the window of its first attempt.+DELIVERY_METRICS_WINDOW_DAYS = 12+DELIVERY_METRICS_MAX_LOOKBACK_DAYS = 14+ENDPOINTS = tuple(SEGMENT_ENDPOINTS.keys())INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {diff --git a/products/warehouse_sources/sources/segment/tests/test_segment.py b/products/warehouse_sources/sources/segment/tests/test_segment.pyindex e5428dab0b90..85d0620a7d0d 100644--- a/products/warehouse_sources/sources/segment/tests/test_segment.py+++ b/products/warehouse_sources/sources/segment/tests/test_segment.py@@ -1,5 +1,9 @@+import datetime as dtfrom typing import Any, Literal, cast+import time_machine++import requestsfrom parameterized import parameterizedfrom products.warehouse_sources.sources.sdk.testing import ScriptedResponse, SourceDriver@@ -152,3 +156,186 @@ class TestSegmentSource:assert result.response is not Noneassert result.response.partition_mode == expected_modeassert result.response.partition_keys == expected_keys+++def _list(key: str, rows: list[dict[str, Any]], next_cursor: str | None = None) -> ScriptedResponse:+ return _page({"data": {key: rows, "pagination": {"next": next_cursor}}})+++_TRACKING_PLAN_PAGES = [+ _list("trackingPlans", [{"id": "tp_b"}], "plans2"),+ _list("trackingPlans", [{"id": "tp_a"}]),+]+++class TestFanOut:+ def test_tracking_plan_rules_walk_every_plan_and_child_page(self) -> None:+ result = _driver().run(+ "tracking_plan_rules",+ [+ *_TRACKING_PLAN_PAGES,+ _list("rules", [{"type": "TRACK", "key": "Clicked", "version": 1}], "rules2"),+ _list("rules", [{"type": "IDENTIFY", "version": 1}]),+ _list("rules", []),+ ],+ )++ assert result.paths == [+ "/tracking-plans",+ "/tracking-plans",+ "/tracking-plans/tp_a/rules",+ "/tracking-plans/tp_a/rules",+ "/tracking-plans/tp_b/rules",+ ]+ assert result.params("pagination[cursor]") == [None, "plans2", None, "rules2", None]+ assert result.rows == [+ {"type": "TRACK", "key": "Clicked", "version": 1, "trackingPlanId": "tp_a"},+ {"type": "IDENTIFY", "key": "", "version": 1, "trackingPlanId": "tp_a"},+ ]+ assert result.saved_states == [+ SegmentResumeConfig(cursor="rules2", parent_id="tp_a"),+ SegmentResumeConfig(parent_id="tp_a", parent_done=True),+ SegmentResumeConfig(parent_id="tp_b", parent_done=True),+ ]++ @parameterized.expand(+ [+ (+ "known_parent_resumes_at_its_child_cursor",+ SegmentResumeConfig(cursor="rules2", parent_id="tp_b"),+ ["/tracking-plans/tp_b/rules"],+ ["rules2"],+ ),+ (+ "deleted_parent_continues_at_the_next_parent",+ SegmentResumeConfig(cursor="gone_cursor", parent_id="tp_aa"),+ ["/tracking-plans/tp_b/rules"],+ [None],+ ),+ (+ "done_parent_continues_at_the_next_parent",+ SegmentResumeConfig(parent_id="tp_a", parent_done=True),+ ["/tracking-plans/tp_b/rules"],+ [None],+ ),+ ("done_last_parent_reads_no_child", SegmentResumeConfig(parent_id="tp_b", parent_done=True), [], []),+ ]+ )+ def test_resume(+ self,+ _name: str,+ state: SegmentResumeConfig,+ expected_child_paths: list[str],+ expected_child_cursors: list[str | None],+ ) -> None:+ child_pages = [_list("rules", [{"type": "TRACK", "key": "k", "version": 1}]) for _ in expected_child_paths]+ result = _driver().run("tracking_plan_rules", [*_TRACKING_PLAN_PAGES, *child_pages], resume_state=state)++ assert result.paths[2:] == expected_child_paths+ assert result.params("pagination[cursor]")[2:] == expected_child_cursors++ @parameterized.expand([("not_found_is_skipped", 404, None), ("forbidden_fails", 403, requests.HTTPError)])+ def test_child_error(self, _name: str, status: int, expected_error: type[BaseException] | None) -> None:+ result = _driver().run(+ "tracking_plan_rules",+ [+ *_TRACKING_PLAN_PAGES,+ ScriptedResponse(status=status, json={"errors": []}),+ _list("rules", [{"type": "TRACK", "key": "k", "version": 1}]),+ ],+ )++ if expected_error is None:+ assert result.raised is None+ assert result.rows == [{"type": "TRACK", "key": "k", "version": 1, "trackingPlanId": "tp_b"}]+ else:+ assert isinstance(result.raised, expected_error)+ assert result.rows == []++ @parameterized.expand(+ [+ ("fresh_run", None, "2026-03-03T00:00:00.000Z", "2026-03-15T00:00:00.000Z"),+ (+ "resumed_run_keeps_its_window",+ SegmentResumeConfig(window_end="2026-03-14T00:00:00.000Z"),+ "2026-03-02T00:00:00.000Z",+ "2026-03-14T00:00:00.000Z",+ ),+ (+ "resumed_window_past_the_lookback_limit_is_replaced",+ SegmentResumeConfig(window_end="2026-03-13T00:00:00.000Z"),+ "2026-03-03T00:00:00.000Z",+ "2026-03-15T00:00:00.000Z",+ ),+ ]+ )+ def test_delivery_metrics_request_a_whole_day_window_per_destination(+ self, _name: str, resume_state: SegmentResumeConfig | None, window_start: str, window_end: str+ ) -> None:+ summary = {+ "sourceId": "src1",+ "destinationMetadataId": "meta1",+ "metrics": [+ {"metricName": "successes", "total": 10},+ {"metricName": "discarded", "total": 2, "breakdown": [{"metricName": "dropped", "value": 2}]},+ ],+ }+ with time_machine.travel(dt.datetime(2026, 3, 15, 10, 30, tzinfo=dt.UTC), tick=False):+ result = _driver().run(+ "destination_delivery_metrics",+ [+ _list("destinations", [{"id": "d1", "sourceId": "src1"}, {"id": "d_no_source"}]),+ _page({"data": {"deliveryMetricsSummary": summary}}),+ ],+ resume_state=resume_state,+ )++ window = {+ "destinationId": "d1",+ "sourceId": "src1",+ "destinationMetadataId": "meta1",+ "windowStart": window_start,+ "windowEnd": window_end,+ }+ assert result.paths == ["/destinations", "/destinations/d1/delivery-metrics"]+ request = result.requests[1]+ assert (request.param("sourceId"), request.param("granularity")) == ("src1", "DAY")+ assert (request.param("startTime"), request.param("endTime")) == (window["windowStart"], window["windowEnd"])+ assert result.rows == [+ {"metricName": "successes", "total": 10, **window},+ {"metricName": "discarded", "total": 2, "breakdown": [{"metricName": "dropped", "value": 2}], **window},+ ]++ def test_reverse_etl_syncs_read_each_model_subscription(self) -> None:+ result = _driver().run(+ "reverse_etl_syncs",+ [+ _list("models", [{"id": "m1", "sourceId": "retl_src"}]),+ _list(+ "destinations",+ [{"id": "d_retl", "sourceId": "retl_src"}, {"id": "d_events", "sourceId": "web_src"}],+ ),+ _list(+ "subscriptions",+ [{"id": "sub1", "modelId": "m1"}, {"id": "sub_event", "destinationId": "d_retl"}],+ ),+ _list("syncStatuses", [{"syncId": "s1", "modelId": "m1"}], "syncs2"),+ _list("syncStatuses", [{"syncId": "s2", "modelId": "m1"}]),+ ],+ )++ assert result.paths == [+ "/reverse-etl-models",+ "/destinations",+ "/destinations/d_retl/subscriptions",+ "/reverse-etl-models/m1/subscriptionId/sub1/syncs",+ "/reverse-etl-models/m1/subscriptionId/sub1/syncs",+ ]+ assert [(request.param("count"), request.param("cursor")) for request in result.requests[3:]] == [+ ("100", None),+ ("100", "syncs2"),+ ]+ assert result.rows == [+ {"syncId": "s1", "modelId": "m1", "subscriptionId": "sub1"},+ {"syncId": "s2", "modelId": "m1", "subscriptionId": "sub1"},+ ]