:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(segment): add 4 missing tables to the Segment source (#130167)
Tom Owersd810e39a8560
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 1a28e50599ca..534b4ffa519b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -10718,23 +10718,25 @@ Note: The docs page is a Scalar viewer with the full OpenAPI YAML inlined — pa## Segment — **thin**-Today (17): `audit_events`, `destination_delivery_metrics`, `destinations`, `destinations_catalog`, `iam_groups`, `iam_users`, `labels`, `reverse_etl_models`, `reverse_etl_syncs`, `sources`, `sources_catalog`, `tracking_plan_rules`, `tracking_plans`, `transformations`, `warehouses`, `warehouses_catalog`, `workspace`+Today (22): `audit_events`, `delivery_overview`, `destination_delivery_metrics`, `destinations`, `destinations_catalog`, `event_volume`, `iam_groups`, `iam_users`, `labels`, `reverse_etl_models`, `reverse_etl_syncs`, `source_mtu_usage`, `sources`, `sources_catalog`, `tracking_plan_rules`, `tracking_plans`, `transformations`, `warehouse_syncs`, `warehouses`, `warehouses_catalog`, `workspace`, `workspace_mtu_usage`Diffed against: <https://docs.segmentapis.com/ (Segment Public API OpenAPI 3.0.3, v73.0.7, extracted from the Redocly state bundle https://docs.segmentapis.com/redocly-state-bf9e006a07a5265a778ac3938ef8f8f891686c5b.js)>-- [ ] `/events/volume (Get Events Volume from Workspace)` — Segment's headline usage metric — event volume by source/day, the number every Segment customer wants in a warehouse (high)-- [ ] `/usage/mtu/daily and /usage/mtu/sources/daily` — Daily MTU usage per workspace and per source — the billing metric (high)-- [ ] `/warehouses/{warehouseId}/syncs and /warehouses/{warehouseId}/connected-sources/{sourceId}/syncs` — Warehouse sync run history/status — the state-transition history for the warehouses table already synced (high)+- [x] `/events/volume (Get Events Volume from Workspace)` — Segment's headline usage metric — event volume by source/day, the number every Segment customer wants in a warehouse (high)+- [x] `/usage/mtu/daily and /usage/mtu/sources/daily` — Daily MTU usage per workspace and per source — the billing metric (high)+- [x] `/warehouses/{warehouseId}/syncs and /warehouses/{warehouseId}/connected-sources/{sourceId}/syncs` — Warehouse sync run history/status — the state-transition history for the warehouses table already synced (high)- [x] `/reverse-etl-models/{modelId}/subscriptionId/{subscriptionId}/syncs` — Reverse ETL sync statuses (rows loaded, failures) for the reverse_etl_models already synced (high)- [x] `/destinations/{destinationId}/delivery-metrics` — Per-destination delivery success/failure counts — core deliverability analysis for the destinations table (high)- [x] `/catalog/destinations, /catalog/sources, /catalog/warehouses` — Lookup tables resolving the metadata IDs carried on every synced source, destination, and warehouse row (high)- [x] `/tracking-plans/{trackingPlanId}/rules` — The actual event/property rules inside a tracking plan — tracking_plans alone is just a header row (high)-- [ ] `/delivery-overview/successful-delivery, /failed-delivery, /failed-on-ingest, /filtered-at-source, /filtered-at-destination` — Time-series pipeline health metrics broken down by stage; the standard Delivery Overview report (medium)+- [x] `/delivery-overview/successful-delivery, /failed-delivery, /failed-on-ingest, /filtered-at-source, /filtered-at-destination` — Time-series pipeline health metrics broken down by stage; the standard Delivery Overview report (medium)- [ ] `/destinations/{destinationId}/subscriptions` — Action mappings per destination — explains which events reach which destination (medium)- [ ] `/spaces and /spaces/{spaceId}/audiences` — Engage spaces and audience definitions plus /audiences/{id}/activations; entirely absent today (medium)- [ ] `/usage/api-calls/daily and /usage/api-calls/sources/daily` — Daily API call volume per workspace and source for cost attribution (medium)- [ ] `/roles (and /invites)` — Lookup resolving the role IDs referenced by iam_users and iam_groups permissions (low)+Note: `warehouse_syncs` reads `/warehouses/{warehouseId}/syncs`, which already returns the syncs of every connected source. The per-source `/connected-sources/{sourceId}/syncs` route returns a subset of the same reports under the same 2 requests per minute limit, so it gets no table of its own. `delivery_overview` holds every stage in one table with a `stage` column, and also reads `/delivery-overview/successfully-received`, the first stage of the funnel. Delivery Overview does not cover Reverse ETL, so the sources of Reverse ETL models and their destinations are left out.+Note: Static endpoint list, no dynamic discovery. The synced 11 tables cover ~15 of the ~95 GET operations in the spec; all of Engage (spaces, audiences, computed traits, profile warehouses), all usage/volume metrics, and all sync-history sub-resources are missing.## Select Star — gapsdiff --git a/products/warehouse_sources/sources/segment/canonical_descriptions.py b/products/warehouse_sources/sources/segment/canonical_descriptions.pyindex e1aa0ed5094d..f42a7138321f 100644--- a/products/warehouse_sources/sources/segment/canonical_descriptions.py+++ b/products/warehouse_sources/sources/segment/canonical_descriptions.py@@ -203,4 +203,69 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"errorCode": "Code of a fatal sync error, if applicable.",},},+ "event_volume": {+ "description": "Daily count of events Segment received from each source over the last 28 full UTC days.",+ "docs_url": "https://docs.segmentapis.com/tag/Events",+ "columns": {+ "sourceId": "Identifier of the source the events came from.",+ "sourceName": "Name of the source.",+ "sourceSlug": "Slug of the source.",+ "time": "Start of the day the count covers.",+ "count": "Number of valid events Segment received in the day, after validation.",+ },+ },+ "workspace_mtu_usage": {+ "description": "Daily cumulative monthly tracked users (MTU) of the workspace for the current and previous usage month.",+ "docs_url": "https://docs.segmentapis.com/tag/Monthly-Tracked-Users",+ "columns": {+ "periodStart": "Start of the usage period, in unix seconds.",+ "periodEnd": "End of the usage period, in unix seconds.",+ "anonymous": "Number of anonymous users in this snapshot.",+ "anonymousIdentified": "Number of anonymous identified users in this snapshot.",+ "identified": "Number of identified users in this snapshot.",+ "neverIdentified": "Number of never identified users in this snapshot.",+ "timestamp": "Time of this snapshot within the billing cycle.",+ },+ },+ "source_mtu_usage": {+ "description": "Daily cumulative monthly tracked users (MTU) per source for the current and previous usage month.",+ "docs_url": "https://docs.segmentapis.com/tag/Monthly-Tracked-Users",+ "columns": {+ "sourceId": "Identifier of the source.",+ "periodStart": "Start of the usage period, in unix seconds.",+ "periodEnd": "End of the usage period, in unix seconds.",+ "anonymous": "Number of anonymous users in this snapshot.",+ "anonymousIdentified": "Number of anonymous identified users in this snapshot.",+ "identified": "Number of identified users in this snapshot.",+ "neverIdentified": "Number of never identified users in this snapshot.",+ "timestamp": "Time of this snapshot within the billing cycle.",+ },+ },+ "warehouse_syncs": {+ "description": "Recent sync runs of each warehouse, one row per source loaded in a sync.",+ "docs_url": "https://docs.segmentapis.com/tag/Selective-Sync",+ "columns": {+ "warehouseId": "Identifier of the warehouse.",+ "sourceId": "Identifier of the source loaded in the sync.",+ "start": "When the sync started.",+ "end": "When the sync completed. Null if unfinished.",+ "status": "Status of the sync.",+ "duration": "Duration of the sync in seconds. Partial if the sync has not finished.",+ "humanDuration": "Human-readable duration of the sync.",+ "count": "Number of rows synced into the warehouse.",+ "notices": "Notices about events that occurred during the sync.",+ },+ },+ "delivery_overview": {+ "description": "Daily event counts at each stage of the Delivery Overview pipeline, per source and destination, over the last 28 full UTC days.",+ "docs_url": "https://docs.segmentapis.com/tag/Delivery-Overview",+ "columns": {+ "stage": "Pipeline stage: successfully-received, failed-on-ingest, filtered-at-source, filtered-at-destination, failed-delivery or successful-delivery.",+ "sourceId": "Identifier of the source.",+ "destinationId": "Identifier of the destination. Empty for source-side stages.",+ "time": "Start of the day the count covers.",+ "count": "Number of events at this stage in the day.",+ "retryCount": "Number of retried events that were successfully delivered.",+ },+ },}diff --git a/products/warehouse_sources/sources/segment/segment.py b/products/warehouse_sources/sources/segment/segment.pyindex 3cb91e138df1..6d78759857ea 100644--- a/products/warehouse_sources/sources/segment/segment.py+++ b/products/warehouse_sources/sources/segment/segment.py@@ -19,6 +19,10 @@ from products.warehouse_sources.sources.sdk import (from products.warehouse_sources.sources.segment.settings import (DELIVERY_METRICS_MAX_LOOKBACK_DAYS,DELIVERY_METRICS_WINDOW_DAYS,+ DELIVERY_OVERVIEW_DESTINATION_STAGES,+ DELIVERY_OVERVIEW_SOURCE_STAGES,+ METRICS_SERIES_MAX_LOOKBACK_DAYS,+ METRICS_SERIES_WINDOW_DAYS,REGION_BASE_URLS,SEGMENT_ENDPOINTS,SegmentEndpointConfig,@@ -76,7 +80,8 @@ def _build_url(base_url: str, path: str, cursor: str | None) -> str:params = [f"pagination[count]={PAGE_SIZE}"]if cursor:params.append(f"pagination[cursor]={quote(cursor, safe='')}")- return f"{base_url}{path}?{'&'.join(params)}"+ separator = "&" if "?" in path else "?"+ return f"{base_url}{path}{separator}{'&'.join(params)}"@retry_with_deadline(@@ -223,7 +228,12 @@ class _MetricsWindow:end: str-def _delivery_metrics_window(now: datetime, resumed_end: str | None) -> _MetricsWindow:+def _metrics_window(+ now: datetime,+ resumed_end: str | None,+ window_days: int = DELIVERY_METRICS_WINDOW_DAYS,+ max_lookback_days: int = DELIVERY_METRICS_MAX_LOOKBACK_DAYS,+) -> _MetricsWindow:"""Whole UTC days, so every sync reports the same span and today's partial day is left out.A resumed run keeps the window of its first attempt while Segment still accepts that start time.@@ -232,11 +242,9 @@ def _delivery_metrics_window(now: datetime, resumed_end: str | None) -> _Metricsend = now.replace(hour=0, minute=0, second=0, microsecond=0)if resumed_end is not None:resumed = datetime.strptime(resumed_end, _TIME_FORMAT).replace(tzinfo=UTC)- if now - (resumed - timedelta(days=DELIVERY_METRICS_WINDOW_DAYS)) < timedelta(- days=DELIVERY_METRICS_MAX_LOOKBACK_DAYS- ):+ if now - (resumed - timedelta(days=window_days)) < timedelta(days=max_lookback_days):end = resumed- start = end - timedelta(days=DELIVERY_METRICS_WINDOW_DAYS)+ start = end - timedelta(days=window_days)return _MetricsWindow(start=start.strftime(_TIME_FORMAT), end=end.strftime(_TIME_FORMAT))@@ -278,6 +286,79 @@ def _delivery_metrics_parents(return parents+def _event_volume_parents(config: SegmentEndpointConfig, window: _MetricsWindow) -> list[_Parent]:+ query = urlencode({"granularity": "DAY", "startTime": window.start, "endTime": window.end, "groupBy": "source"})+ return [_Parent(id="workspace", path=f"{config.path}?{query}", row_fields={})]+++def _usage_periods(now: datetime) -> list[str]:+ """The first days of the current and the previous usage month."""+ current = now.astimezone(UTC).date().replace(day=1)+ previous = (current - timedelta(days=1)).replace(day=1)+ return [previous.isoformat(), current.isoformat()]+++def _mtu_usage_parents(config: SegmentEndpointConfig, now: datetime) -> list[_Parent]:+ return [+ _Parent(id=period, path=f"{config.path}?{urlencode({'period': period})}", row_fields={})+ for period in _usage_periods(now)+ ]+++def _warehouse_sync_parents(warehouses: list[dict[str, Any]], config: SegmentEndpointConfig) -> list[_Parent]:+ return [+ _Parent(+ id=warehouse["id"],+ path=config.path.format(warehouse_id=quote(warehouse["id"], safe="")),+ row_fields={"warehouseId": warehouse["id"]},+ )+ for warehouse in warehouses+ if warehouse.get("id")+ ]+++def _delivery_overview_parents(+ sources: list[dict[str, Any]],+ destinations: list[dict[str, Any]],+ reverse_etl_models: list[dict[str, Any]],+ config: SegmentEndpointConfig,+ window: _MetricsWindow,+) -> list[_Parent]:+ """Each source reports the source-side stages, and each destination the destination-side stages.++ Delivery Overview does not cover Reverse ETL, so the sources of Reverse ETL models and their+ destinations are left out."""+ reverse_etl_source_ids = {model["sourceId"] for model in reverse_etl_models if model.get("sourceId")}+ pairs: list[tuple[str, str, str]] = [+ (stage, source["id"], "")+ for source in sources+ if source.get("id") and source["id"] not in reverse_etl_source_ids+ for stage in DELIVERY_OVERVIEW_SOURCE_STAGES+ ]+ pairs.extend(+ (stage, destination["sourceId"], destination["id"])+ for destination in destinations+ if destination.get("id")+ and destination.get("sourceId")+ and destination["sourceId"] not in reverse_etl_source_ids+ for stage in DELIVERY_OVERVIEW_DESTINATION_STAGES+ )++ parents: list[_Parent] = []+ for stage, source_id, destination_id in pairs:+ params = {"sourceId": source_id, "granularity": "DAY", "startTime": window.start, "endTime": window.end}+ if destination_id:+ params["destinationConfigId"] = destination_id+ parents.append(+ _Parent(+ id=f"{source_id}/{destination_id}/{stage}",+ path=f"{config.path.format(stage=stage)}?{urlencode(params)}",+ row_fields={"stage": stage, "sourceId": source_id, "destinationId": destination_id},+ )+ )+ return parents++def _reverse_etl_parents(session: requests.Session,base_url: str,@@ -357,6 +438,43 @@ def _shape_delivery_metric_rows(parent: _Parent, data: dict[str, Any]) -> list[d]+def _shape_event_volume_rows(parent: _Parent, data: dict[str, Any]) -> list[dict[str, Any]]:+ payload = data.get("data")+ series_list = payload.get("result") if isinstance(payload, dict) else None+ rows: list[dict[str, Any]] = []+ for entry in series_list or []:+ if not isinstance(entry, dict):+ continue+ source = entry.get("source")+ if not isinstance(source, dict):+ source = {}+ rows.extend(+ {+ "sourceId": source.get("id"),+ "sourceName": source.get("name"),+ "sourceSlug": source.get("slug"),+ **point,+ **parent.row_fields,+ }+ for point in entry.get("series") or []+ if isinstance(point, dict)+ )+ return rows+++def _shape_delivery_overview_rows(parent: _Parent, data: dict[str, Any]) -> list[dict[str, Any]]:+ # With no groupBy, the dataset holds one series for the whole stage.+ payload = data.get("data")+ dataset = payload.get("dataset") if isinstance(payload, dict) else None+ return [+ {**point, **parent.row_fields}+ for entry in dataset or []+ if isinstance(entry, dict)+ for point in entry.get("series") or []+ if isinstance(point, dict)+ ]++def _fan_out(session: requests.Session,base_url: str,@@ -430,10 +548,38 @@ def get_fanout_rows(parents = _tracking_plan_parents(list_parents("/tracking-plans"), config)build_url, shape = _build_url, _shape_rule_rowselif config.fanout == "delivery_metrics":- window = _delivery_metrics_window(datetime.now(UTC), resume.window_end if resume else None)+ window = _metrics_window(datetime.now(UTC), resume.window_end if resume else None)window_end = window.endparents = _delivery_metrics_parents(list_parents("/destinations"), config, window)build_url, shape = _build_plain_url, _shape_delivery_metric_rows+ elif config.fanout == "event_volume":+ window = _metrics_window(+ datetime.now(UTC),+ resume.window_end if resume else None,+ METRICS_SERIES_WINDOW_DAYS,+ METRICS_SERIES_MAX_LOOKBACK_DAYS,+ )+ window_end = window.end+ parents = _event_volume_parents(config, window)+ build_url, shape = _build_url, _shape_event_volume_rows+ elif config.fanout == "mtu_usage":+ parents = _mtu_usage_parents(config, datetime.now(UTC))+ build_url, shape = _build_url, _shape_list_rows+ elif config.fanout == "warehouse_syncs":+ parents = _warehouse_sync_parents(list_parents("/warehouses"), config)+ build_url, shape = _build_url, _shape_list_rows+ elif config.fanout == "delivery_overview":+ window = _metrics_window(+ datetime.now(UTC),+ resume.window_end if resume else None,+ METRICS_SERIES_WINDOW_DAYS,+ METRICS_SERIES_MAX_LOOKBACK_DAYS,+ )+ window_end = window.end+ parents = _delivery_overview_parents(+ list_parents("/sources"), list_parents("/destinations"), list_parents("/reverse-etl-models"), config, window+ )+ build_url, shape = _build_url, _shape_delivery_overview_rowselif config.fanout == "reverse_etl_syncs":parents = _reverse_etl_parents(session, base_url, headers, logger, resumable_source_manager, config)build_url, shape = _build_sync_status_url, _shape_list_rowsdiff --git a/products/warehouse_sources/sources/segment/settings.py b/products/warehouse_sources/sources/segment/settings.pyindex 6b6530290fc2..e315b393fa52 100644--- a/products/warehouse_sources/sources/segment/settings.py+++ b/products/warehouse_sources/sources/segment/settings.py@@ -23,7 +23,7 @@ class SegmentEndpointConfig:# rather than a paginated list. Such endpoints yield exactly one row.single_object_key: Optional[str] = None# A stable, immutable datetime field to partition on. Segment's config/admin objects carry no- # timestamps, so only audit events (`timestamp`) and Reverse ETL syncs (`startedAt`) set this.+ # timestamps, so only audit events, sync history and the time-series usage tables set this.partition_key: Optional[str] = Noneincremental_fields: list[IncrementalField] = field(default_factory=list)should_sync_default: bool = True@@ -35,7 +35,17 @@ class SegmentEndpointConfig:redacted_fields: frozenset[str] = field(default_factory=frozenset)# Child resources read once per parent row. `path` then holds the parent placeholders, and# `segment.py` lists the parents and resumes per parent.- fanout: Optional[Literal["tracking_plan_rules", "delivery_metrics", "reverse_etl_syncs"]] = None+ fanout: Optional[+ Literal[+ "tracking_plan_rules",+ "delivery_metrics",+ "reverse_etl_syncs",+ "event_volume",+ "mtu_usage",+ "warehouse_syncs",+ "delivery_overview",+ ]+ ] = None# The Segment Public API is a workspace configuration/admin/usage API (not the event/Profile data@@ -90,6 +100,45 @@ SEGMENT_ENDPOINTS: dict[str, SegmentEndpointConfig] = {partition_key="startedAt",fanout="reverse_etl_syncs",),+ # One row per source and day over the last METRICS_SERIES_WINDOW_DAYS full days.+ "event_volume": SegmentEndpointConfig(+ name="event_volume",+ path="/events/volume",+ primary_keys=["sourceId", "time"],+ partition_key="time",+ fanout="event_volume",+ ),+ # Daily cumulative MTU snapshots for the current and the previous usage month.+ "workspace_mtu_usage": SegmentEndpointConfig(+ name="workspace_mtu_usage",+ path="/usage/mtu/daily",+ primary_keys=["periodStart", "timestamp"],+ partition_key="timestamp",+ fanout="mtu_usage",+ ),+ "source_mtu_usage": SegmentEndpointConfig(+ name="source_mtu_usage",+ path="/usage/mtu/sources/daily",+ primary_keys=["sourceId", "periodStart", "timestamp"],+ partition_key="timestamp",+ fanout="mtu_usage",+ ),+ # Sync reports have no id. One warehouse loads each of its sources once per start time.+ "warehouse_syncs": SegmentEndpointConfig(+ name="warehouse_syncs",+ path="/warehouses/{warehouse_id}/syncs",+ primary_keys=["warehouseId", "sourceId", "start"],+ partition_key="start",+ fanout="warehouse_syncs",+ ),+ # One row per pipeline stage, source, destination and day. Source-side stages have an empty destinationId.+ "delivery_overview": SegmentEndpointConfig(+ name="delivery_overview",+ path="/delivery-overview/{stage}",+ primary_keys=["stage", "sourceId", "destinationId", "time"],+ partition_key="time",+ fanout="delivery_overview",+ ),}# Segment accepts a DAY-granularity range of at most 14 days that starts at most 14 days ago. 12 whole@@ -97,6 +146,15 @@ SEGMENT_ENDPOINTS: dict[str, SegmentEndpointConfig] = {DELIVERY_METRICS_WINDOW_DAYS = 12DELIVERY_METRICS_MAX_LOOKBACK_DAYS = 14+# Delivery Overview accepts a DAY-granularity range of at most 30 days that starts at most 30 days ago.+# Events volume documents no limit, so it uses the same window.+METRICS_SERIES_WINDOW_DAYS = 28+METRICS_SERIES_MAX_LOOKBACK_DAYS = 30++# Stages of the Delivery Overview funnel, by the side of the pipeline that reports them.+DELIVERY_OVERVIEW_SOURCE_STAGES = ("successfully-received", "failed-on-ingest", "filtered-at-source")+DELIVERY_OVERVIEW_DESTINATION_STAGES = ("filtered-at-destination", "failed-delivery", "successful-delivery")+ENDPOINTS = tuple(SEGMENT_ENDPOINTS.keys())INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {diff --git a/products/warehouse_sources/sources/segment/tests/test_segment.py b/products/warehouse_sources/sources/segment/tests/test_segment.pyindex 85d0620a7d0d..f7b1f596a0ce 100644--- a/products/warehouse_sources/sources/segment/tests/test_segment.py+++ b/products/warehouse_sources/sources/segment/tests/test_segment.py@@ -339,3 +339,141 @@ class TestFanOut:{"syncId": "s1", "modelId": "m1", "subscriptionId": "sub1"},{"syncId": "s2", "modelId": "m1", "subscriptionId": "sub1"},]++ def test_event_volume_flattens_each_source_series_across_pages(self) -> None:+ with time_machine.travel(dt.datetime(2026, 3, 15, 10, 30, tzinfo=dt.UTC), tick=False):+ result = _driver().run(+ "event_volume",+ [+ _page(+ {+ "data": {+ "result": [+ {+ "source": {"id": "src1", "name": "Web", "slug": "web"},+ "total": 7,+ "series": [+ {"time": "2026-02-15T00:00:00Z", "count": 3},+ {"time": "2026-02-16T00:00:00Z", "count": 4},+ ],+ }+ ],+ "pagination": {"next": "vol2"},+ }+ }+ ),+ _page(+ {+ "data": {+ "result": [+ {"source": {"id": "src2"}, "series": [{"time": "2026-02-15T00:00:00Z", "count": 1}]}+ ]+ }+ }+ ),+ ],+ )++ assert result.paths == ["/events/volume", "/events/volume"]+ first = result.requests[0]+ assert (first.param("granularity"), first.param("groupBy")) == ("DAY", "source")+ assert (first.param("startTime"), first.param("endTime")) == (+ "2026-02-15T00:00:00.000Z",+ "2026-03-15T00:00:00.000Z",+ )+ assert result.params("pagination[cursor]") == [None, "vol2"]+ assert result.requests[1].param("startTime") == "2026-02-15T00:00:00.000Z"+ assert result.rows == [+ {"sourceId": "src1", "sourceName": "Web", "sourceSlug": "web", "time": "2026-02-15T00:00:00Z", "count": 3},+ {"sourceId": "src1", "sourceName": "Web", "sourceSlug": "web", "time": "2026-02-16T00:00:00Z", "count": 4},+ {"sourceId": "src2", "sourceName": None, "sourceSlug": None, "time": "2026-02-15T00:00:00Z", "count": 1},+ ]++ @parameterized.expand(+ [+ ("workspace", "workspace_mtu_usage", "/usage/mtu/daily", "dailyWorkspaceMTUUsage", {}),+ (+ "per_source",+ "source_mtu_usage",+ "/usage/mtu/sources/daily",+ "dailyPerSourceMTUUsage",+ {"sourceId": "s1"},+ ),+ ]+ )+ def test_mtu_usage_reads_the_current_and_previous_usage_month(+ self, _name: str, endpoint: str, path: str, key: str, extra: dict[str, Any]+ ) -> None:+ snapshot = {"periodStart": 1, "periodEnd": 2, "identified": "5", "timestamp": "2026-01-02T00:00:00Z", **extra}+ with time_machine.travel(dt.datetime(2026, 1, 15, tzinfo=dt.UTC), tick=False):+ result = _driver().run(endpoint, [_list(key, [snapshot]), _list(key, [snapshot], "next"), _list(key, [])])++ assert result.paths == [path, path, path]+ assert result.params("period") == ["2025-12-01", "2026-01-01", "2026-01-01"]+ assert result.params("pagination[cursor]") == [None, None, "next"]+ assert result.rows == [snapshot, snapshot]++ def test_warehouse_syncs_read_every_warehouse(self) -> None:+ report = {"sourceId": "src1", "start": "2026-03-01T00:00:00Z", "status": "SUCCESS", "count": 10}+ result = _driver().run(+ "warehouse_syncs",+ [+ _list("warehouses", [{"id": "wh_b"}, {"id": "wh_a", "settings": {"password": "secret"}}]),+ _list("reports", [report], "reports2"),+ _list("reports", []),+ _list("reports", [{**report, "sourceId": "src2"}]),+ ],+ )++ assert result.paths == [+ "/warehouses",+ "/warehouses/wh_a/syncs",+ "/warehouses/wh_a/syncs",+ "/warehouses/wh_b/syncs",+ ]+ assert result.params("pagination[cursor]") == [None, None, "reports2", None]+ assert result.rows == [{**report, "warehouseId": "wh_a"}, {**report, "sourceId": "src2", "warehouseId": "wh_b"}]++ def test_delivery_overview_reads_each_stage_and_skips_reverse_etl(self) -> None:+ def respond(request: Any) -> ScriptedResponse:+ listings = {+ "/sources": _list("sources", [{"id": "src1"}, {"id": "retl_src"}]),+ "/destinations": _list(+ "destinations", [{"id": "d1", "sourceId": "src1"}, {"id": "d_retl", "sourceId": "retl_src"}]+ ),+ "/reverse-etl-models": _list("models", [{"id": "m1", "sourceId": "retl_src"}]),+ }+ if request.path in listings:+ return listings[request.path]+ point = {"time": "2026-03-01T00:00:00Z", "count": 2}+ return _page({"data": {"total": 2, "dataset": [{"total": 2, "series": [point]}], "pagination": {}}})++ with time_machine.travel(dt.datetime(2026, 3, 15, 10, 30, tzinfo=dt.UTC), tick=False):+ result = _driver().run("delivery_overview", respond)++ metric_requests = [request for request in result.requests if request.path.startswith("/delivery-overview/")]+ assert sorted(+ (request.path, request.param("sourceId"), request.param("destinationConfigId"))+ for request in metric_requests+ ) == [+ ("/delivery-overview/failed-delivery", "src1", "d1"),+ ("/delivery-overview/failed-on-ingest", "src1", None),+ ("/delivery-overview/filtered-at-destination", "src1", "d1"),+ ("/delivery-overview/filtered-at-source", "src1", None),+ ("/delivery-overview/successful-delivery", "src1", "d1"),+ ("/delivery-overview/successfully-received", "src1", None),+ ]+ assert {(request.param("granularity"), request.param("startTime")) for request in metric_requests} == {+ ("DAY", "2026-02-15T00:00:00.000Z")+ }+ assert sorted((row["stage"], row["destinationId"]) for row in result.rows) == [+ ("failed-delivery", "d1"),+ ("failed-on-ingest", ""),+ ("filtered-at-destination", "d1"),+ ("filtered-at-source", ""),+ ("successful-delivery", "d1"),+ ("successfully-received", ""),+ ]+ assert {(row["sourceId"], row["time"], row["count"]) for row in result.rows} == {+ ("src1", "2026-03-01T00:00:00Z", 2)+ }