:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse-sources): add fastly platform metrics table (#126943)
Tom Owers400985671ff8
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 3ad2d6c2816f..6d23f4e499f2 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -4504,7 +4504,7 @@ Note: Developer docs, reference routes, sitemap, and spec candidates returned Cl## Fastly — **thin**-Today (16): `acl_entries`, `ai_usage_metrics`, `billing_usage_metrics`, `current_user`, `dictionary_items`, `historical_stats`, `invoices`, `origin_inspector`, `pops`, `service_acls`, `service_authorizations`, `service_backends`, `service_dictionaries`, `service_domains`, `service_versions`, `services`+Today (17): `acl_entries`, `ai_usage_metrics`, `billing_usage_metrics`, `current_user`, `dictionary_items`, `historical_stats`, `invoices`, `origin_inspector`, `platform_metrics`, `pops`, `service_acls`, `service_authorizations`, `service_backends`, `service_dictionaries`, `service_domains`, `service_versions`, `services`Diffed against: <https://www.fastly.com/documentation/reference/api/>@@ -4520,6 +4520,7 @@ Diffed against: <https://www.fastly.com/documentation/reference/api/>- [x] `account/service-authorization` — user-to-service permission membership table (medium)- [x] `utils/pops` — lookup resolving POP/datacenter codes that appear in stats and inspector breakdowns (medium)- [x] `ai-runtime-control/usage-metrics` — AI gateway request, session and token usage per virtual key, provider and model (medium)+- [x] `metrics-stats/metrics-v2/platform` — Metrics v2 historical delivery and Compute metrics per service and POP, cursor paginated (beta) (medium)Note: Coverage is service configuration objects, the account/billing family, and the historical stats and Origin Inspector halves of metrics-stats; the domain inspector and bots halves are not exposed. Fastly's own OpenAPI YAML is no longer served at the old developer.fastly.com path (404) - the category tree was read from the live documentation reference index and its per-category pages. Logging endpoint types (~25 of them), TLS, purging and VCL objects are config/plumbing and deliberately excluded.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/canonical_descriptions.pyindex 8090db69c66f..c184953eef89 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/canonical_descriptions.py@@ -179,6 +179,20 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"latency_1000_to_5000ms": "Number of responses from origin with latency between 1,000 and 5,000 milliseconds.",},},+ "platform_metrics": {+ "description": "Daily delivery and Compute traffic, cache and error metrics per service and POP, from Fastly's Metrics v2 API.",+ "docs_url": "https://www.fastly.com/documentation/reference/api/metrics-stats/metrics-v2/platform/",+ "columns": {+ "service_id": "Alphanumeric string identifying the service (entity) the sample belongs to.",+ "datacenter": "The POP the sample was measured at.",+ "timestamp": "The start of the reported time period.",+ "start_time": "Start of the day the sample covers, as a Unix timestamp.",+ "requests": "Number of requests processed.",+ "bandwidth": "Total bytes delivered.",+ "hit_ratio": "Ratio of cache hits to cache hits and misses, between 0 and 1.",+ "origin_offload": "Share of bytes served from cache rather than fetched from origin, between 0 and 1.",+ },+ },"service_authorizations": {"description": "Which users can reach which services, and the permission each one holds.","docs_url": "https://www.fastly.com/documentation/reference/api/account/service-authorization/",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/fastly.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/fastly.pyindex 7be6b2448171..793f65f4093e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/fastly.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/fastly.py@@ -60,6 +60,26 @@ ORIGIN_INSPECTOR_UNAVAILABLE_STATUSES = frozenset({403, 404})# AI Runtime Control aggregates usage by `daily` or `hourly`; a day matches the other metrics tables.AI_USAGE_METRICS_GRANULARITY = "daily"+# Metrics v2 pages hold up to 10,000 rows. A day per POP keeps a year of one service to a few pages.+PLATFORM_METRICS_PAGE_LIMIT = 10000++# Metrics v2 names its path granularity `minutely` / `hourly` / `daily`, not the `minute` / `hour` /+# `day` the older metrics APIs take.+PLATFORM_METRICS_GRANULARITY = "daily"++PLATFORM_METRICS_GROUP_BY = "datacenter"++# The delivery and Compute sets cover what `/stats` reports. The response is sparse, so a set a+# service does not use adds no columns to its rows.+PLATFORM_METRICS_METRIC_SETS = (+ "delivery_traffic_flow",+ "delivery_cache_performance",+ "delivery_errors",+ "compute_traffic_flow",+ "compute_performance",+ "compute_errors",+)+class FastlyRetryableError(Exception):pass@@ -564,6 +584,57 @@ def _get_ai_usage_metrics_rows(yield rows+def _get_platform_metrics_rows(+ session: requests.Session,+ config: FastlyEndpointConfig,+ headers: dict[str, str],+ logger: FilteringBoundLogger,+ resumable_source_manager: ResumableSourceManager[FastlyResumeConfig],+ incremental_start: int | None,+) -> Iterator[list[dict[str, Any]]]:+ params: dict[str, Any] = {+ "metric_set": ",".join(PLATFORM_METRICS_METRIC_SETS),+ "group_by": PLATFORM_METRICS_GROUP_BY,+ "limit": PLATFORM_METRICS_PAGE_LIMIT,+ }+ if incremental_start is not None:+ params["from"] = datetime.fromtimestamp(incremental_start, tz=UTC).strftime("%Y-%m-%dT%H:%M:%SZ")++ for service_id in _iter_services_from_bookmark(session, config, headers, logger, resumable_source_manager):+ url_base = (+ f"{FASTLY_BASE_URL}{config.path.format(service_id=service_id, granularity=PLATFORM_METRICS_GRANULARITY)}"+ )+ cursor: str | None = None++ while True:+ page_params = {**params, "cursor": cursor} if cursor else params+ response = _fetch(session, _build_url(url_base, page_params, safe=","), headers, logger)+ payload = response.json()+ if not isinstance(payload, dict):+ break++ rows: list[dict[str, Any]] = []+ for row in payload.get("data") or []:+ if not isinstance(row, dict):+ continue+ start_time = _date_to_epoch_seconds(row.get("timestamp"))+ if start_time is None:+ logger.warning(+ f"Fastly: skipping platform metric with an unreadable timestamp: {row.get('timestamp')!r}"+ )+ continue+ rows.append({**row, "service_id": service_id, "start_time": start_time})+ if rows:+ yield rows++ next_cursor = _next_cursor(payload)+ if not next_cursor:+ break+ if next_cursor == cursor:+ raise FastlyPaginationError(f"Fastly returned an unchanged cursor for {url_base}")+ cursor = next_cursor++def _get_service_list_rows(session: requests.Session,headers: dict[str, str],@@ -786,6 +857,10 @@ def get_rows()elif config.kind == "ai_usage_metrics":yield from _get_ai_usage_metrics_rows(session, config, headers, logger, incremental_start)+ elif config.kind == "platform_metrics":+ yield from _get_platform_metrics_rows(+ session, config, headers, logger, resumable_source_manager, incremental_start+ )elif config.kind == "service_list":yield from _get_service_list_rows(session, headers, logger, resumable_source_manager)elif config.kind == "version_resource_child":diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/settings.pyindex 44d7bee69ae4..e69fe811ee60 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/settings.py@@ -23,6 +23,7 @@ from products.warehouse_sources.backend.types import IncrementalField, Increment# - "stats_list": GET /stats returns one array of time buckets per service, keyed by service id.# - "origin_inspector": fan out over every service, reading its origin metrics timeseries.# - "ai_usage_metrics": GET an unpaginated AI Runtime Control usage report; rows sit under `data`.+# - "platform_metrics": fan out over every service, reading its Metrics v2 platform timeseries.FastlyEndpointKind = Literal["object","service_list",@@ -36,6 +37,7 @@ FastlyEndpointKind = Literal["stats_list","origin_inspector","ai_usage_metrics",+ "platform_metrics",]# Fastly reports every metrics family in whole time buckets keyed by their start. The cursor is the@@ -189,6 +191,19 @@ FASTLY_ENDPOINTS: dict[str, FastlyEndpointConfig] = {should_sync_default=False,description="Daily AI Runtime Control request, session and token usage per virtual key, provider and model.",),+ "platform_metrics": FastlyEndpointConfig(+ name="platform_metrics",+ path="/metrics/v2/platform/entities/{service_id}/{granularity}",+ kind="platform_metrics",+ partition_key="start_time",+ primary_keys=["service_id", "datacenter", "start_time"],+ incremental_fields=_BUCKET_START_INCREMENTAL_FIELDS,+ incremental_lookback_seconds=METRICS_LOOKBACK_SECONDS,+ # Fastly labels Metrics v2 a beta that may still change shape, and `historical_stats` already+ # covers the same traffic, so it is opt-in.+ should_sync_default=False,+ description="Daily delivery and Compute traffic, cache and error metrics per service and POP, from Fastly's Metrics v2 API.",+ ),"service_authorizations": FastlyEndpointConfig(name="service_authorizations",path="/service-authorizations",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/tests/test_fastly.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/tests/test_fastly.pyindex 31253fa27a78..4fb3cf4cc400 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/tests/test_fastly.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/tests/test_fastly.py@@ -20,6 +20,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.fastly impfrom products.warehouse_sources.backend.temporal.data_imports.sources.fastly.fastly import (FASTLY_BASE_URL,ORIGIN_INSPECTOR_PAGE_LIMIT,+ PLATFORM_METRICS_METRIC_SETS,+ PLATFORM_METRICS_PAGE_LIMIT,FastlyPaginationError,FastlyResumeConfig,FastlyRetryableError,@@ -758,3 +760,63 @@ class TestGetRowsOriginInspector:result = _drive("origin_inspector", pages)assert isinstance(result.raised, FastlyPaginationError)assert result.params("cursor") == [None, None, "CUR2"]+++class TestGetRowsPlatformMetrics:+ def _url(self, service_id: str, cursor: str | None = None, start: str | None = None) -> str:+ params: dict[str, Any] = {+ "metric_set": ",".join(PLATFORM_METRICS_METRIC_SETS),+ "group_by": "datacenter",+ "limit": PLATFORM_METRICS_PAGE_LIMIT,+ }+ if start is not None:+ params["from"] = start+ if cursor is not None:+ params["cursor"] = cursor+ return unquote(_build_url(f"{FASTLY_BASE_URL}/metrics/v2/platform/entities/{service_id}/daily", params))++ def test_walks_each_service_cursor_and_keys_rows_by_service_and_day(self) -> None:+ # A row without a readable timestamp has no partition or key and is dropped.+ pages = {+ f"{FASTLY_BASE_URL}/service?per_page=100": ([{"id": "S1"}, {"id": "S2"}], None),+ self._url("S1"): {+ "meta": {"next_cursor": "CUR2"},+ "data": [{"timestamp": "2026-09-01T00:00:00Z", "datacenter": "IAD", "requests": 5}],+ },+ self._url("S1", cursor="CUR2"): {+ "meta": {"next_cursor": ""},+ "data": [+ {"timestamp": "2026-09-02T00:00:00Z", "datacenter": "SYD", "hit_ratio": 0.5},+ {"timestamp": None, "datacenter": "LHR", "requests": 1},+ ],+ },+ self._url("S2"): {"meta": {}, "data": [{"timestamp": "2026-09-01T00:00:00Z", "datacenter": "IAD"}]},+ }+ result = _drive("platform_metrics", pages)+ assert result.raised is None+ assert [(row["service_id"], row["datacenter"], row["start_time"]) for row in result.rows] == [+ ("S1", "IAD", 1_788_220_800),+ ("S1", "SYD", 1_788_307_200),+ ("S2", "IAD", 1_788_220_800),+ ]++ @parameterized.expand(+ [+ ("full_refresh", False, None),+ ("incremental", True, "2023-11-14T22:13:20Z"),+ ]+ )+ def test_watermark_is_sent_as_an_rfc3339_from(self, _name: str, incremental: bool, expected: str | None) -> None:+ pages = {+ f"{FASTLY_BASE_URL}/service?per_page=100": ([{"id": "S1"}], None),+ self._url("S1", start=expected): {"meta": {}, "data": []},+ }+ result = _drive(+ "platform_metrics",+ pages,+ should_use_incremental_field=incremental,+ incremental_field="start_time",+ db_incremental_field_last_value=1_700_000_000,+ )+ assert result.raised is None+ assert result.params("from")[-1] == expected