Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse-sources): add fastly platform metrics table (#126943)

Tom Owers400985671ff8
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 3ad2d6c2816f..6d23f4e499f2 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -4504,7 +4504,7 @@ Note: Developer docs, reference routes, sitemap, and spec candidates returned Cl
## Fastly — **thin**
-Today (16): `acl_entries`, `ai_usage_metrics`, `billing_usage_metrics`, `current_user`, `dictionary_items`, `historical_stats`, `invoices`, `origin_inspector`, `pops`, `service_acls`, `service_authorizations`, `service_backends`, `service_dictionaries`, `service_domains`, `service_versions`, `services`
+Today (17): `acl_entries`, `ai_usage_metrics`, `billing_usage_metrics`, `current_user`, `dictionary_items`, `historical_stats`, `invoices`, `origin_inspector`, `platform_metrics`, `pops`, `service_acls`, `service_authorizations`, `service_backends`, `service_dictionaries`, `service_domains`, `service_versions`, `services`
Diffed against: <https://www.fastly.com/documentation/reference/api/>
@@ -4520,6 +4520,7 @@ Diffed against: <https://www.fastly.com/documentation/reference/api/>
- [x] `account/service-authorization` — user-to-service permission membership table (medium)
- [x] `utils/pops` — lookup resolving POP/datacenter codes that appear in stats and inspector breakdowns (medium)
- [x] `ai-runtime-control/usage-metrics` — AI gateway request, session and token usage per virtual key, provider and model (medium)
+- [x] `metrics-stats/metrics-v2/platform` — Metrics v2 historical delivery and Compute metrics per service and POP, cursor paginated (beta) (medium)
Note: Coverage is service configuration objects, the account/billing family, and the historical stats and Origin Inspector halves of metrics-stats; the domain inspector and bots halves are not exposed. Fastly's own OpenAPI YAML is no longer served at the old developer.fastly.com path (404) - the category tree was read from the live documentation reference index and its per-category pages. Logging endpoint types (~25 of them), TLS, purging and VCL objects are config/plumbing and deliberately excluded.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/canonical_descriptions.py
index 8090db69c66f..c184953eef89 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/canonical_descriptions.py
@@ -179,6 +179,20 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"latency_1000_to_5000ms": "Number of responses from origin with latency between 1,000 and 5,000 milliseconds.",
},
},
+ "platform_metrics": {
+ "description": "Daily delivery and Compute traffic, cache and error metrics per service and POP, from Fastly's Metrics v2 API.",
+ "docs_url": "https://www.fastly.com/documentation/reference/api/metrics-stats/metrics-v2/platform/",
+ "columns": {
+ "service_id": "Alphanumeric string identifying the service (entity) the sample belongs to.",
+ "datacenter": "The POP the sample was measured at.",
+ "timestamp": "The start of the reported time period.",
+ "start_time": "Start of the day the sample covers, as a Unix timestamp.",
+ "requests": "Number of requests processed.",
+ "bandwidth": "Total bytes delivered.",
+ "hit_ratio": "Ratio of cache hits to cache hits and misses, between 0 and 1.",
+ "origin_offload": "Share of bytes served from cache rather than fetched from origin, between 0 and 1.",
+ },
+ },
"service_authorizations": {
"description": "Which users can reach which services, and the permission each one holds.",
"docs_url": "https://www.fastly.com/documentation/reference/api/account/service-authorization/",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/fastly.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/fastly.py
index 7be6b2448171..793f65f4093e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/fastly.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/fastly.py
@@ -60,6 +60,26 @@ ORIGIN_INSPECTOR_UNAVAILABLE_STATUSES = frozenset({403, 404})
# AI Runtime Control aggregates usage by `daily` or `hourly`; a day matches the other metrics tables.
AI_USAGE_METRICS_GRANULARITY = "daily"
+# Metrics v2 pages hold up to 10,000 rows. A day per POP keeps a year of one service to a few pages.
+PLATFORM_METRICS_PAGE_LIMIT = 10000
+
+# Metrics v2 names its path granularity `minutely` / `hourly` / `daily`, not the `minute` / `hour` /
+# `day` the older metrics APIs take.
+PLATFORM_METRICS_GRANULARITY = "daily"
+
+PLATFORM_METRICS_GROUP_BY = "datacenter"
+
+# The delivery and Compute sets cover what `/stats` reports. The response is sparse, so a set a
+# service does not use adds no columns to its rows.
+PLATFORM_METRICS_METRIC_SETS = (
+ "delivery_traffic_flow",
+ "delivery_cache_performance",
+ "delivery_errors",
+ "compute_traffic_flow",
+ "compute_performance",
+ "compute_errors",
+)
+
class FastlyRetryableError(Exception):
pass
@@ -564,6 +584,57 @@ def _get_ai_usage_metrics_rows(
yield rows
+def _get_platform_metrics_rows(
+ session: requests.Session,
+ config: FastlyEndpointConfig,
+ headers: dict[str, str],
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[FastlyResumeConfig],
+ incremental_start: int | None,
+) -> Iterator[list[dict[str, Any]]]:
+ params: dict[str, Any] = {
+ "metric_set": ",".join(PLATFORM_METRICS_METRIC_SETS),
+ "group_by": PLATFORM_METRICS_GROUP_BY,
+ "limit": PLATFORM_METRICS_PAGE_LIMIT,
+ }
+ if incremental_start is not None:
+ params["from"] = datetime.fromtimestamp(incremental_start, tz=UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+ for service_id in _iter_services_from_bookmark(session, config, headers, logger, resumable_source_manager):
+ url_base = (
+ f"{FASTLY_BASE_URL}{config.path.format(service_id=service_id, granularity=PLATFORM_METRICS_GRANULARITY)}"
+ )
+ cursor: str | None = None
+
+ while True:
+ page_params = {**params, "cursor": cursor} if cursor else params
+ response = _fetch(session, _build_url(url_base, page_params, safe=","), headers, logger)
+ payload = response.json()
+ if not isinstance(payload, dict):
+ break
+
+ rows: list[dict[str, Any]] = []
+ for row in payload.get("data") or []:
+ if not isinstance(row, dict):
+ continue
+ start_time = _date_to_epoch_seconds(row.get("timestamp"))
+ if start_time is None:
+ logger.warning(
+ f"Fastly: skipping platform metric with an unreadable timestamp: {row.get('timestamp')!r}"
+ )
+ continue
+ rows.append({**row, "service_id": service_id, "start_time": start_time})
+ if rows:
+ yield rows
+
+ next_cursor = _next_cursor(payload)
+ if not next_cursor:
+ break
+ if next_cursor == cursor:
+ raise FastlyPaginationError(f"Fastly returned an unchanged cursor for {url_base}")
+ cursor = next_cursor
+
+
def _get_service_list_rows(
session: requests.Session,
headers: dict[str, str],
@@ -786,6 +857,10 @@ def get_rows(
)
elif config.kind == "ai_usage_metrics":
yield from _get_ai_usage_metrics_rows(session, config, headers, logger, incremental_start)
+ elif config.kind == "platform_metrics":
+ yield from _get_platform_metrics_rows(
+ session, config, headers, logger, resumable_source_manager, incremental_start
+ )
elif config.kind == "service_list":
yield from _get_service_list_rows(session, headers, logger, resumable_source_manager)
elif config.kind == "version_resource_child":
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/settings.py
index 44d7bee69ae4..e69fe811ee60 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/settings.py
@@ -23,6 +23,7 @@ from products.warehouse_sources.backend.types import IncrementalField, Increment
# - "stats_list": GET /stats returns one array of time buckets per service, keyed by service id.
# - "origin_inspector": fan out over every service, reading its origin metrics timeseries.
# - "ai_usage_metrics": GET an unpaginated AI Runtime Control usage report; rows sit under `data`.
+# - "platform_metrics": fan out over every service, reading its Metrics v2 platform timeseries.
FastlyEndpointKind = Literal[
"object",
"service_list",
@@ -36,6 +37,7 @@ FastlyEndpointKind = Literal[
"stats_list",
"origin_inspector",
"ai_usage_metrics",
+ "platform_metrics",
]
# Fastly reports every metrics family in whole time buckets keyed by their start. The cursor is the
@@ -189,6 +191,19 @@ FASTLY_ENDPOINTS: dict[str, FastlyEndpointConfig] = {
should_sync_default=False,
description="Daily AI Runtime Control request, session and token usage per virtual key, provider and model.",
),
+ "platform_metrics": FastlyEndpointConfig(
+ name="platform_metrics",
+ path="/metrics/v2/platform/entities/{service_id}/{granularity}",
+ kind="platform_metrics",
+ partition_key="start_time",
+ primary_keys=["service_id", "datacenter", "start_time"],
+ incremental_fields=_BUCKET_START_INCREMENTAL_FIELDS,
+ incremental_lookback_seconds=METRICS_LOOKBACK_SECONDS,
+ # Fastly labels Metrics v2 a beta that may still change shape, and `historical_stats` already
+ # covers the same traffic, so it is opt-in.
+ should_sync_default=False,
+ description="Daily delivery and Compute traffic, cache and error metrics per service and POP, from Fastly's Metrics v2 API.",
+ ),
"service_authorizations": FastlyEndpointConfig(
name="service_authorizations",
path="/service-authorizations",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/tests/test_fastly.py b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/tests/test_fastly.py
index 31253fa27a78..4fb3cf4cc400 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/tests/test_fastly.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/fastly/tests/test_fastly.py
@@ -20,6 +20,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.fastly imp
from products.warehouse_sources.backend.temporal.data_imports.sources.fastly.fastly import (
FASTLY_BASE_URL,
ORIGIN_INSPECTOR_PAGE_LIMIT,
+ PLATFORM_METRICS_METRIC_SETS,
+ PLATFORM_METRICS_PAGE_LIMIT,
FastlyPaginationError,
FastlyResumeConfig,
FastlyRetryableError,
@@ -758,3 +760,63 @@ class TestGetRowsOriginInspector:
result = _drive("origin_inspector", pages)
assert isinstance(result.raised, FastlyPaginationError)
assert result.params("cursor") == [None, None, "CUR2"]
+
+
+class TestGetRowsPlatformMetrics:
+ def _url(self, service_id: str, cursor: str | None = None, start: str | None = None) -> str:
+ params: dict[str, Any] = {
+ "metric_set": ",".join(PLATFORM_METRICS_METRIC_SETS),
+ "group_by": "datacenter",
+ "limit": PLATFORM_METRICS_PAGE_LIMIT,
+ }
+ if start is not None:
+ params["from"] = start
+ if cursor is not None:
+ params["cursor"] = cursor
+ return unquote(_build_url(f"{FASTLY_BASE_URL}/metrics/v2/platform/entities/{service_id}/daily", params))
+
+ def test_walks_each_service_cursor_and_keys_rows_by_service_and_day(self) -> None:
+ # A row without a readable timestamp has no partition or key and is dropped.
+ pages = {
+ f"{FASTLY_BASE_URL}/service?per_page=100": ([{"id": "S1"}, {"id": "S2"}], None),
+ self._url("S1"): {
+ "meta": {"next_cursor": "CUR2"},
+ "data": [{"timestamp": "2026-09-01T00:00:00Z", "datacenter": "IAD", "requests": 5}],
+ },
+ self._url("S1", cursor="CUR2"): {
+ "meta": {"next_cursor": ""},
+ "data": [
+ {"timestamp": "2026-09-02T00:00:00Z", "datacenter": "SYD", "hit_ratio": 0.5},
+ {"timestamp": None, "datacenter": "LHR", "requests": 1},
+ ],
+ },
+ self._url("S2"): {"meta": {}, "data": [{"timestamp": "2026-09-01T00:00:00Z", "datacenter": "IAD"}]},
+ }
+ result = _drive("platform_metrics", pages)
+ assert result.raised is None
+ assert [(row["service_id"], row["datacenter"], row["start_time"]) for row in result.rows] == [
+ ("S1", "IAD", 1_788_220_800),
+ ("S1", "SYD", 1_788_307_200),
+ ("S2", "IAD", 1_788_220_800),
+ ]
+
+ @parameterized.expand(
+ [
+ ("full_refresh", False, None),
+ ("incremental", True, "2023-11-14T22:13:20Z"),
+ ]
+ )
+ def test_watermark_is_sent_as_an_rfc3339_from(self, _name: str, incremental: bool, expected: str | None) -> None:
+ pages = {
+ f"{FASTLY_BASE_URL}/service?per_page=100": ([{"id": "S1"}], None),
+ self._url("S1", start=expected): {"meta": {}, "data": []},
+ }
+ result = _drive(
+ "platform_metrics",
+ pages,
+ should_use_incremental_field=incremental,
+ incremental_field="start_time",
+ db_incremental_field_last_value=1_700_000_000,
+ )
+ assert result.raised is None
+ assert result.params("from")[-1] == expected