Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(segment): add 4 missing tables to the Segment source (#130167)

Tom Owersd810e39a8560
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 1a28e50599ca..534b4ffa519b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -10718,23 +10718,25 @@ Note: The docs page is a Scalar viewer with the full OpenAPI YAML inlined — pa
## Segment — **thin**
-Today (17): `audit_events`, `destination_delivery_metrics`, `destinations`, `destinations_catalog`, `iam_groups`, `iam_users`, `labels`, `reverse_etl_models`, `reverse_etl_syncs`, `sources`, `sources_catalog`, `tracking_plan_rules`, `tracking_plans`, `transformations`, `warehouses`, `warehouses_catalog`, `workspace`
+Today (22): `audit_events`, `delivery_overview`, `destination_delivery_metrics`, `destinations`, `destinations_catalog`, `event_volume`, `iam_groups`, `iam_users`, `labels`, `reverse_etl_models`, `reverse_etl_syncs`, `source_mtu_usage`, `sources`, `sources_catalog`, `tracking_plan_rules`, `tracking_plans`, `transformations`, `warehouse_syncs`, `warehouses`, `warehouses_catalog`, `workspace`, `workspace_mtu_usage`
Diffed against: <https://docs.segmentapis.com/ (Segment Public API OpenAPI 3.0.3, v73.0.7, extracted from the Redocly state bundle https://docs.segmentapis.com/redocly-state-bf9e006a07a5265a778ac3938ef8f8f891686c5b.js)>
-- [ ] `/events/volume (Get Events Volume from Workspace)` — Segment's headline usage metric — event volume by source/day, the number every Segment customer wants in a warehouse (high)
-- [ ] `/usage/mtu/daily and /usage/mtu/sources/daily` — Daily MTU usage per workspace and per source — the billing metric (high)
-- [ ] `/warehouses/{warehouseId}/syncs and /warehouses/{warehouseId}/connected-sources/{sourceId}/syncs` — Warehouse sync run history/status — the state-transition history for the warehouses table already synced (high)
+- [x] `/events/volume (Get Events Volume from Workspace)` — Segment's headline usage metric — event volume by source/day, the number every Segment customer wants in a warehouse (high)
+- [x] `/usage/mtu/daily and /usage/mtu/sources/daily` — Daily MTU usage per workspace and per source — the billing metric (high)
+- [x] `/warehouses/{warehouseId}/syncs and /warehouses/{warehouseId}/connected-sources/{sourceId}/syncs` — Warehouse sync run history/status — the state-transition history for the warehouses table already synced (high)
- [x] `/reverse-etl-models/{modelId}/subscriptionId/{subscriptionId}/syncs` — Reverse ETL sync statuses (rows loaded, failures) for the reverse_etl_models already synced (high)
- [x] `/destinations/{destinationId}/delivery-metrics` — Per-destination delivery success/failure counts — core deliverability analysis for the destinations table (high)
- [x] `/catalog/destinations, /catalog/sources, /catalog/warehouses` — Lookup tables resolving the metadata IDs carried on every synced source, destination, and warehouse row (high)
- [x] `/tracking-plans/{trackingPlanId}/rules` — The actual event/property rules inside a tracking plan — tracking_plans alone is just a header row (high)
-- [ ] `/delivery-overview/successful-delivery, /failed-delivery, /failed-on-ingest, /filtered-at-source, /filtered-at-destination` — Time-series pipeline health metrics broken down by stage; the standard Delivery Overview report (medium)
+- [x] `/delivery-overview/successful-delivery, /failed-delivery, /failed-on-ingest, /filtered-at-source, /filtered-at-destination` — Time-series pipeline health metrics broken down by stage; the standard Delivery Overview report (medium)
- [ ] `/destinations/{destinationId}/subscriptions` — Action mappings per destination — explains which events reach which destination (medium)
- [ ] `/spaces and /spaces/{spaceId}/audiences` — Engage spaces and audience definitions plus /audiences/{id}/activations; entirely absent today (medium)
- [ ] `/usage/api-calls/daily and /usage/api-calls/sources/daily` — Daily API call volume per workspace and source for cost attribution (medium)
- [ ] `/roles (and /invites)` — Lookup resolving the role IDs referenced by iam_users and iam_groups permissions (low)
+Note: `warehouse_syncs` reads `/warehouses/{warehouseId}/syncs`, which already returns the syncs of every connected source. The per-source `/connected-sources/{sourceId}/syncs` route returns a subset of the same reports under the same 2 requests per minute limit, so it gets no table of its own. `delivery_overview` holds every stage in one table with a `stage` column, and also reads `/delivery-overview/successfully-received`, the first stage of the funnel. Delivery Overview does not cover Reverse ETL, so the sources of Reverse ETL models and their destinations are left out.
+
Note: Static endpoint list, no dynamic discovery. The synced 11 tables cover ~15 of the ~95 GET operations in the spec; all of Engage (spaces, audiences, computed traits, profile warehouses), all usage/volume metrics, and all sync-history sub-resources are missing.
## Select Star — gaps
diff --git a/products/warehouse_sources/sources/segment/canonical_descriptions.py b/products/warehouse_sources/sources/segment/canonical_descriptions.py
index e1aa0ed5094d..f42a7138321f 100644
--- a/products/warehouse_sources/sources/segment/canonical_descriptions.py
+++ b/products/warehouse_sources/sources/segment/canonical_descriptions.py
@@ -203,4 +203,69 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"errorCode": "Code of a fatal sync error, if applicable.",
},
},
+ "event_volume": {
+ "description": "Daily count of events Segment received from each source over the last 28 full UTC days.",
+ "docs_url": "https://docs.segmentapis.com/tag/Events",
+ "columns": {
+ "sourceId": "Identifier of the source the events came from.",
+ "sourceName": "Name of the source.",
+ "sourceSlug": "Slug of the source.",
+ "time": "Start of the day the count covers.",
+ "count": "Number of valid events Segment received in the day, after validation.",
+ },
+ },
+ "workspace_mtu_usage": {
+ "description": "Daily cumulative monthly tracked users (MTU) of the workspace for the current and previous usage month.",
+ "docs_url": "https://docs.segmentapis.com/tag/Monthly-Tracked-Users",
+ "columns": {
+ "periodStart": "Start of the usage period, in unix seconds.",
+ "periodEnd": "End of the usage period, in unix seconds.",
+ "anonymous": "Number of anonymous users in this snapshot.",
+ "anonymousIdentified": "Number of anonymous identified users in this snapshot.",
+ "identified": "Number of identified users in this snapshot.",
+ "neverIdentified": "Number of never identified users in this snapshot.",
+ "timestamp": "Time of this snapshot within the billing cycle.",
+ },
+ },
+ "source_mtu_usage": {
+ "description": "Daily cumulative monthly tracked users (MTU) per source for the current and previous usage month.",
+ "docs_url": "https://docs.segmentapis.com/tag/Monthly-Tracked-Users",
+ "columns": {
+ "sourceId": "Identifier of the source.",
+ "periodStart": "Start of the usage period, in unix seconds.",
+ "periodEnd": "End of the usage period, in unix seconds.",
+ "anonymous": "Number of anonymous users in this snapshot.",
+ "anonymousIdentified": "Number of anonymous identified users in this snapshot.",
+ "identified": "Number of identified users in this snapshot.",
+ "neverIdentified": "Number of never identified users in this snapshot.",
+ "timestamp": "Time of this snapshot within the billing cycle.",
+ },
+ },
+ "warehouse_syncs": {
+ "description": "Recent sync runs of each warehouse, one row per source loaded in a sync.",
+ "docs_url": "https://docs.segmentapis.com/tag/Selective-Sync",
+ "columns": {
+ "warehouseId": "Identifier of the warehouse.",
+ "sourceId": "Identifier of the source loaded in the sync.",
+ "start": "When the sync started.",
+ "end": "When the sync completed. Null if unfinished.",
+ "status": "Status of the sync.",
+ "duration": "Duration of the sync in seconds. Partial if the sync has not finished.",
+ "humanDuration": "Human-readable duration of the sync.",
+ "count": "Number of rows synced into the warehouse.",
+ "notices": "Notices about events that occurred during the sync.",
+ },
+ },
+ "delivery_overview": {
+ "description": "Daily event counts at each stage of the Delivery Overview pipeline, per source and destination, over the last 28 full UTC days.",
+ "docs_url": "https://docs.segmentapis.com/tag/Delivery-Overview",
+ "columns": {
+ "stage": "Pipeline stage: successfully-received, failed-on-ingest, filtered-at-source, filtered-at-destination, failed-delivery or successful-delivery.",
+ "sourceId": "Identifier of the source.",
+ "destinationId": "Identifier of the destination. Empty for source-side stages.",
+ "time": "Start of the day the count covers.",
+ "count": "Number of events at this stage in the day.",
+ "retryCount": "Number of retried events that were successfully delivered.",
+ },
+ },
}
diff --git a/products/warehouse_sources/sources/segment/segment.py b/products/warehouse_sources/sources/segment/segment.py
index 3cb91e138df1..6d78759857ea 100644
--- a/products/warehouse_sources/sources/segment/segment.py
+++ b/products/warehouse_sources/sources/segment/segment.py
@@ -19,6 +19,10 @@ from products.warehouse_sources.sources.sdk import (
from products.warehouse_sources.sources.segment.settings import (
DELIVERY_METRICS_MAX_LOOKBACK_DAYS,
DELIVERY_METRICS_WINDOW_DAYS,
+ DELIVERY_OVERVIEW_DESTINATION_STAGES,
+ DELIVERY_OVERVIEW_SOURCE_STAGES,
+ METRICS_SERIES_MAX_LOOKBACK_DAYS,
+ METRICS_SERIES_WINDOW_DAYS,
REGION_BASE_URLS,
SEGMENT_ENDPOINTS,
SegmentEndpointConfig,
@@ -76,7 +80,8 @@ def _build_url(base_url: str, path: str, cursor: str | None) -> str:
params = [f"pagination[count]={PAGE_SIZE}"]
if cursor:
params.append(f"pagination[cursor]={quote(cursor, safe='')}")
- return f"{base_url}{path}?{'&'.join(params)}"
+ separator = "&" if "?" in path else "?"
+ return f"{base_url}{path}{separator}{'&'.join(params)}"
@retry_with_deadline(
@@ -223,7 +228,12 @@ class _MetricsWindow:
end: str
-def _delivery_metrics_window(now: datetime, resumed_end: str | None) -> _MetricsWindow:
+def _metrics_window(
+ now: datetime,
+ resumed_end: str | None,
+ window_days: int = DELIVERY_METRICS_WINDOW_DAYS,
+ max_lookback_days: int = DELIVERY_METRICS_MAX_LOOKBACK_DAYS,
+) -> _MetricsWindow:
"""Whole UTC days, so every sync reports the same span and today's partial day is left out.
A resumed run keeps the window of its first attempt while Segment still accepts that start time.
@@ -232,11 +242,9 @@ def _delivery_metrics_window(now: datetime, resumed_end: str | None) -> _Metrics
end = now.replace(hour=0, minute=0, second=0, microsecond=0)
if resumed_end is not None:
resumed = datetime.strptime(resumed_end, _TIME_FORMAT).replace(tzinfo=UTC)
- if now - (resumed - timedelta(days=DELIVERY_METRICS_WINDOW_DAYS)) < timedelta(
- days=DELIVERY_METRICS_MAX_LOOKBACK_DAYS
- ):
+ if now - (resumed - timedelta(days=window_days)) < timedelta(days=max_lookback_days):
end = resumed
- start = end - timedelta(days=DELIVERY_METRICS_WINDOW_DAYS)
+ start = end - timedelta(days=window_days)
return _MetricsWindow(start=start.strftime(_TIME_FORMAT), end=end.strftime(_TIME_FORMAT))
@@ -278,6 +286,79 @@ def _delivery_metrics_parents(
return parents
+def _event_volume_parents(config: SegmentEndpointConfig, window: _MetricsWindow) -> list[_Parent]:
+ query = urlencode({"granularity": "DAY", "startTime": window.start, "endTime": window.end, "groupBy": "source"})
+ return [_Parent(id="workspace", path=f"{config.path}?{query}", row_fields={})]
+
+
+def _usage_periods(now: datetime) -> list[str]:
+ """The first days of the current and the previous usage month."""
+ current = now.astimezone(UTC).date().replace(day=1)
+ previous = (current - timedelta(days=1)).replace(day=1)
+ return [previous.isoformat(), current.isoformat()]
+
+
+def _mtu_usage_parents(config: SegmentEndpointConfig, now: datetime) -> list[_Parent]:
+ return [
+ _Parent(id=period, path=f"{config.path}?{urlencode({'period': period})}", row_fields={})
+ for period in _usage_periods(now)
+ ]
+
+
+def _warehouse_sync_parents(warehouses: list[dict[str, Any]], config: SegmentEndpointConfig) -> list[_Parent]:
+ return [
+ _Parent(
+ id=warehouse["id"],
+ path=config.path.format(warehouse_id=quote(warehouse["id"], safe="")),
+ row_fields={"warehouseId": warehouse["id"]},
+ )
+ for warehouse in warehouses
+ if warehouse.get("id")
+ ]
+
+
+def _delivery_overview_parents(
+ sources: list[dict[str, Any]],
+ destinations: list[dict[str, Any]],
+ reverse_etl_models: list[dict[str, Any]],
+ config: SegmentEndpointConfig,
+ window: _MetricsWindow,
+) -> list[_Parent]:
+ """Each source reports the source-side stages, and each destination the destination-side stages.
+
+ Delivery Overview does not cover Reverse ETL, so the sources of Reverse ETL models and their
+ destinations are left out."""
+ reverse_etl_source_ids = {model["sourceId"] for model in reverse_etl_models if model.get("sourceId")}
+ pairs: list[tuple[str, str, str]] = [
+ (stage, source["id"], "")
+ for source in sources
+ if source.get("id") and source["id"] not in reverse_etl_source_ids
+ for stage in DELIVERY_OVERVIEW_SOURCE_STAGES
+ ]
+ pairs.extend(
+ (stage, destination["sourceId"], destination["id"])
+ for destination in destinations
+ if destination.get("id")
+ and destination.get("sourceId")
+ and destination["sourceId"] not in reverse_etl_source_ids
+ for stage in DELIVERY_OVERVIEW_DESTINATION_STAGES
+ )
+
+ parents: list[_Parent] = []
+ for stage, source_id, destination_id in pairs:
+ params = {"sourceId": source_id, "granularity": "DAY", "startTime": window.start, "endTime": window.end}
+ if destination_id:
+ params["destinationConfigId"] = destination_id
+ parents.append(
+ _Parent(
+ id=f"{source_id}/{destination_id}/{stage}",
+ path=f"{config.path.format(stage=stage)}?{urlencode(params)}",
+ row_fields={"stage": stage, "sourceId": source_id, "destinationId": destination_id},
+ )
+ )
+ return parents
+
+
def _reverse_etl_parents(
session: requests.Session,
base_url: str,
@@ -357,6 +438,43 @@ def _shape_delivery_metric_rows(parent: _Parent, data: dict[str, Any]) -> list[d
]
+def _shape_event_volume_rows(parent: _Parent, data: dict[str, Any]) -> list[dict[str, Any]]:
+ payload = data.get("data")
+ series_list = payload.get("result") if isinstance(payload, dict) else None
+ rows: list[dict[str, Any]] = []
+ for entry in series_list or []:
+ if not isinstance(entry, dict):
+ continue
+ source = entry.get("source")
+ if not isinstance(source, dict):
+ source = {}
+ rows.extend(
+ {
+ "sourceId": source.get("id"),
+ "sourceName": source.get("name"),
+ "sourceSlug": source.get("slug"),
+ **point,
+ **parent.row_fields,
+ }
+ for point in entry.get("series") or []
+ if isinstance(point, dict)
+ )
+ return rows
+
+
+def _shape_delivery_overview_rows(parent: _Parent, data: dict[str, Any]) -> list[dict[str, Any]]:
+ # With no groupBy, the dataset holds one series for the whole stage.
+ payload = data.get("data")
+ dataset = payload.get("dataset") if isinstance(payload, dict) else None
+ return [
+ {**point, **parent.row_fields}
+ for entry in dataset or []
+ if isinstance(entry, dict)
+ for point in entry.get("series") or []
+ if isinstance(point, dict)
+ ]
+
+
def _fan_out(
session: requests.Session,
base_url: str,
@@ -430,10 +548,38 @@ def get_fanout_rows(
parents = _tracking_plan_parents(list_parents("/tracking-plans"), config)
build_url, shape = _build_url, _shape_rule_rows
elif config.fanout == "delivery_metrics":
- window = _delivery_metrics_window(datetime.now(UTC), resume.window_end if resume else None)
+ window = _metrics_window(datetime.now(UTC), resume.window_end if resume else None)
window_end = window.end
parents = _delivery_metrics_parents(list_parents("/destinations"), config, window)
build_url, shape = _build_plain_url, _shape_delivery_metric_rows
+ elif config.fanout == "event_volume":
+ window = _metrics_window(
+ datetime.now(UTC),
+ resume.window_end if resume else None,
+ METRICS_SERIES_WINDOW_DAYS,
+ METRICS_SERIES_MAX_LOOKBACK_DAYS,
+ )
+ window_end = window.end
+ parents = _event_volume_parents(config, window)
+ build_url, shape = _build_url, _shape_event_volume_rows
+ elif config.fanout == "mtu_usage":
+ parents = _mtu_usage_parents(config, datetime.now(UTC))
+ build_url, shape = _build_url, _shape_list_rows
+ elif config.fanout == "warehouse_syncs":
+ parents = _warehouse_sync_parents(list_parents("/warehouses"), config)
+ build_url, shape = _build_url, _shape_list_rows
+ elif config.fanout == "delivery_overview":
+ window = _metrics_window(
+ datetime.now(UTC),
+ resume.window_end if resume else None,
+ METRICS_SERIES_WINDOW_DAYS,
+ METRICS_SERIES_MAX_LOOKBACK_DAYS,
+ )
+ window_end = window.end
+ parents = _delivery_overview_parents(
+ list_parents("/sources"), list_parents("/destinations"), list_parents("/reverse-etl-models"), config, window
+ )
+ build_url, shape = _build_url, _shape_delivery_overview_rows
elif config.fanout == "reverse_etl_syncs":
parents = _reverse_etl_parents(session, base_url, headers, logger, resumable_source_manager, config)
build_url, shape = _build_sync_status_url, _shape_list_rows
diff --git a/products/warehouse_sources/sources/segment/settings.py b/products/warehouse_sources/sources/segment/settings.py
index 6b6530290fc2..e315b393fa52 100644
--- a/products/warehouse_sources/sources/segment/settings.py
+++ b/products/warehouse_sources/sources/segment/settings.py
@@ -23,7 +23,7 @@ class SegmentEndpointConfig:
# rather than a paginated list. Such endpoints yield exactly one row.
single_object_key: Optional[str] = None
# A stable, immutable datetime field to partition on. Segment's config/admin objects carry no
- # timestamps, so only audit events (`timestamp`) and Reverse ETL syncs (`startedAt`) set this.
+ # timestamps, so only audit events, sync history and the time-series usage tables set this.
partition_key: Optional[str] = None
incremental_fields: list[IncrementalField] = field(default_factory=list)
should_sync_default: bool = True
@@ -35,7 +35,17 @@ class SegmentEndpointConfig:
redacted_fields: frozenset[str] = field(default_factory=frozenset)
# Child resources read once per parent row. `path` then holds the parent placeholders, and
# `segment.py` lists the parents and resumes per parent.
- fanout: Optional[Literal["tracking_plan_rules", "delivery_metrics", "reverse_etl_syncs"]] = None
+ fanout: Optional[
+ Literal[
+ "tracking_plan_rules",
+ "delivery_metrics",
+ "reverse_etl_syncs",
+ "event_volume",
+ "mtu_usage",
+ "warehouse_syncs",
+ "delivery_overview",
+ ]
+ ] = None
# The Segment Public API is a workspace configuration/admin/usage API (not the event/Profile data
@@ -90,6 +100,45 @@ SEGMENT_ENDPOINTS: dict[str, SegmentEndpointConfig] = {
partition_key="startedAt",
fanout="reverse_etl_syncs",
),
+ # One row per source and day over the last METRICS_SERIES_WINDOW_DAYS full days.
+ "event_volume": SegmentEndpointConfig(
+ name="event_volume",
+ path="/events/volume",
+ primary_keys=["sourceId", "time"],
+ partition_key="time",
+ fanout="event_volume",
+ ),
+ # Daily cumulative MTU snapshots for the current and the previous usage month.
+ "workspace_mtu_usage": SegmentEndpointConfig(
+ name="workspace_mtu_usage",
+ path="/usage/mtu/daily",
+ primary_keys=["periodStart", "timestamp"],
+ partition_key="timestamp",
+ fanout="mtu_usage",
+ ),
+ "source_mtu_usage": SegmentEndpointConfig(
+ name="source_mtu_usage",
+ path="/usage/mtu/sources/daily",
+ primary_keys=["sourceId", "periodStart", "timestamp"],
+ partition_key="timestamp",
+ fanout="mtu_usage",
+ ),
+ # Sync reports have no id. One warehouse loads each of its sources once per start time.
+ "warehouse_syncs": SegmentEndpointConfig(
+ name="warehouse_syncs",
+ path="/warehouses/{warehouse_id}/syncs",
+ primary_keys=["warehouseId", "sourceId", "start"],
+ partition_key="start",
+ fanout="warehouse_syncs",
+ ),
+ # One row per pipeline stage, source, destination and day. Source-side stages have an empty destinationId.
+ "delivery_overview": SegmentEndpointConfig(
+ name="delivery_overview",
+ path="/delivery-overview/{stage}",
+ primary_keys=["stage", "sourceId", "destinationId", "time"],
+ partition_key="time",
+ fanout="delivery_overview",
+ ),
}
# Segment accepts a DAY-granularity range of at most 14 days that starts at most 14 days ago. 12 whole
@@ -97,6 +146,15 @@ SEGMENT_ENDPOINTS: dict[str, SegmentEndpointConfig] = {
DELIVERY_METRICS_WINDOW_DAYS = 12
DELIVERY_METRICS_MAX_LOOKBACK_DAYS = 14
+# Delivery Overview accepts a DAY-granularity range of at most 30 days that starts at most 30 days ago.
+# Events volume documents no limit, so it uses the same window.
+METRICS_SERIES_WINDOW_DAYS = 28
+METRICS_SERIES_MAX_LOOKBACK_DAYS = 30
+
+# Stages of the Delivery Overview funnel, by the side of the pipeline that reports them.
+DELIVERY_OVERVIEW_SOURCE_STAGES = ("successfully-received", "failed-on-ingest", "filtered-at-source")
+DELIVERY_OVERVIEW_DESTINATION_STAGES = ("filtered-at-destination", "failed-delivery", "successful-delivery")
+
ENDPOINTS = tuple(SEGMENT_ENDPOINTS.keys())
INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {
diff --git a/products/warehouse_sources/sources/segment/tests/test_segment.py b/products/warehouse_sources/sources/segment/tests/test_segment.py
index 85d0620a7d0d..f7b1f596a0ce 100644
--- a/products/warehouse_sources/sources/segment/tests/test_segment.py
+++ b/products/warehouse_sources/sources/segment/tests/test_segment.py
@@ -339,3 +339,141 @@ class TestFanOut:
{"syncId": "s1", "modelId": "m1", "subscriptionId": "sub1"},
{"syncId": "s2", "modelId": "m1", "subscriptionId": "sub1"},
]
+
+ def test_event_volume_flattens_each_source_series_across_pages(self) -> None:
+ with time_machine.travel(dt.datetime(2026, 3, 15, 10, 30, tzinfo=dt.UTC), tick=False):
+ result = _driver().run(
+ "event_volume",
+ [
+ _page(
+ {
+ "data": {
+ "result": [
+ {
+ "source": {"id": "src1", "name": "Web", "slug": "web"},
+ "total": 7,
+ "series": [
+ {"time": "2026-02-15T00:00:00Z", "count": 3},
+ {"time": "2026-02-16T00:00:00Z", "count": 4},
+ ],
+ }
+ ],
+ "pagination": {"next": "vol2"},
+ }
+ }
+ ),
+ _page(
+ {
+ "data": {
+ "result": [
+ {"source": {"id": "src2"}, "series": [{"time": "2026-02-15T00:00:00Z", "count": 1}]}
+ ]
+ }
+ }
+ ),
+ ],
+ )
+
+ assert result.paths == ["/events/volume", "/events/volume"]
+ first = result.requests[0]
+ assert (first.param("granularity"), first.param("groupBy")) == ("DAY", "source")
+ assert (first.param("startTime"), first.param("endTime")) == (
+ "2026-02-15T00:00:00.000Z",
+ "2026-03-15T00:00:00.000Z",
+ )
+ assert result.params("pagination[cursor]") == [None, "vol2"]
+ assert result.requests[1].param("startTime") == "2026-02-15T00:00:00.000Z"
+ assert result.rows == [
+ {"sourceId": "src1", "sourceName": "Web", "sourceSlug": "web", "time": "2026-02-15T00:00:00Z", "count": 3},
+ {"sourceId": "src1", "sourceName": "Web", "sourceSlug": "web", "time": "2026-02-16T00:00:00Z", "count": 4},
+ {"sourceId": "src2", "sourceName": None, "sourceSlug": None, "time": "2026-02-15T00:00:00Z", "count": 1},
+ ]
+
+ @parameterized.expand(
+ [
+ ("workspace", "workspace_mtu_usage", "/usage/mtu/daily", "dailyWorkspaceMTUUsage", {}),
+ (
+ "per_source",
+ "source_mtu_usage",
+ "/usage/mtu/sources/daily",
+ "dailyPerSourceMTUUsage",
+ {"sourceId": "s1"},
+ ),
+ ]
+ )
+ def test_mtu_usage_reads_the_current_and_previous_usage_month(
+ self, _name: str, endpoint: str, path: str, key: str, extra: dict[str, Any]
+ ) -> None:
+ snapshot = {"periodStart": 1, "periodEnd": 2, "identified": "5", "timestamp": "2026-01-02T00:00:00Z", **extra}
+ with time_machine.travel(dt.datetime(2026, 1, 15, tzinfo=dt.UTC), tick=False):
+ result = _driver().run(endpoint, [_list(key, [snapshot]), _list(key, [snapshot], "next"), _list(key, [])])
+
+ assert result.paths == [path, path, path]
+ assert result.params("period") == ["2025-12-01", "2026-01-01", "2026-01-01"]
+ assert result.params("pagination[cursor]") == [None, None, "next"]
+ assert result.rows == [snapshot, snapshot]
+
+ def test_warehouse_syncs_read_every_warehouse(self) -> None:
+ report = {"sourceId": "src1", "start": "2026-03-01T00:00:00Z", "status": "SUCCESS", "count": 10}
+ result = _driver().run(
+ "warehouse_syncs",
+ [
+ _list("warehouses", [{"id": "wh_b"}, {"id": "wh_a", "settings": {"password": "secret"}}]),
+ _list("reports", [report], "reports2"),
+ _list("reports", []),
+ _list("reports", [{**report, "sourceId": "src2"}]),
+ ],
+ )
+
+ assert result.paths == [
+ "/warehouses",
+ "/warehouses/wh_a/syncs",
+ "/warehouses/wh_a/syncs",
+ "/warehouses/wh_b/syncs",
+ ]
+ assert result.params("pagination[cursor]") == [None, None, "reports2", None]
+ assert result.rows == [{**report, "warehouseId": "wh_a"}, {**report, "sourceId": "src2", "warehouseId": "wh_b"}]
+
+ def test_delivery_overview_reads_each_stage_and_skips_reverse_etl(self) -> None:
+ def respond(request: Any) -> ScriptedResponse:
+ listings = {
+ "/sources": _list("sources", [{"id": "src1"}, {"id": "retl_src"}]),
+ "/destinations": _list(
+ "destinations", [{"id": "d1", "sourceId": "src1"}, {"id": "d_retl", "sourceId": "retl_src"}]
+ ),
+ "/reverse-etl-models": _list("models", [{"id": "m1", "sourceId": "retl_src"}]),
+ }
+ if request.path in listings:
+ return listings[request.path]
+ point = {"time": "2026-03-01T00:00:00Z", "count": 2}
+ return _page({"data": {"total": 2, "dataset": [{"total": 2, "series": [point]}], "pagination": {}}})
+
+ with time_machine.travel(dt.datetime(2026, 3, 15, 10, 30, tzinfo=dt.UTC), tick=False):
+ result = _driver().run("delivery_overview", respond)
+
+ metric_requests = [request for request in result.requests if request.path.startswith("/delivery-overview/")]
+ assert sorted(
+ (request.path, request.param("sourceId"), request.param("destinationConfigId"))
+ for request in metric_requests
+ ) == [
+ ("/delivery-overview/failed-delivery", "src1", "d1"),
+ ("/delivery-overview/failed-on-ingest", "src1", None),
+ ("/delivery-overview/filtered-at-destination", "src1", "d1"),
+ ("/delivery-overview/filtered-at-source", "src1", None),
+ ("/delivery-overview/successful-delivery", "src1", "d1"),
+ ("/delivery-overview/successfully-received", "src1", None),
+ ]
+ assert {(request.param("granularity"), request.param("startTime")) for request in metric_requests} == {
+ ("DAY", "2026-02-15T00:00:00.000Z")
+ }
+ assert sorted((row["stage"], row["destinationId"]) for row in result.rows) == [
+ ("failed-delivery", "d1"),
+ ("failed-on-ingest", ""),
+ ("filtered-at-destination", "d1"),
+ ("filtered-at-source", ""),
+ ("successful-delivery", "d1"),
+ ("successfully-received", ""),
+ ]
+ assert {(row["sourceId"], row["time"], row["count"]) for row in result.rows} == {
+ ("src1", "2026-03-01T00:00:00Z", 2)
+ }