Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add gcore logs, daily statistics, activity log and purge tables (#120225)

Tom Owers0607479d08b8
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 91c5c384cfcc..ec21b4acda19 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -5171,11 +5171,11 @@ Today (5): `cdn_requests`, `cdn_traffic`, `origin_groups`, `resources`, `ssl_cer
Diffed against: <https://docs.gcore.com/api-reference/services_docs_mintlify_enriched/cdn_api.yaml>
-- [ ] `Get CDN logs (GET /cdn/advanced/v1/logs)` — analyze individual requests, status codes, client locations, and paths beyond hourly request totals (high)
-- [ ] `Aggregated statistics (GET /cdn/statistics/aggregate/stats)` — compare cache efficiency, error rates, and traffic across countries and regions (high)
+- [x] `Get CDN logs (GET /cdn/advanced/v1/logs)` — analyze individual requests, status codes, client locations, and paths beyond hourly request totals (high)
+- [x] `Aggregated statistics (GET /cdn/statistics/aggregate/stats)` — compare cache efficiency, error rates, and traffic across countries and regions (high)
- [ ] `Origin shielding usage statistics (GET /cdn/statistics/shield_usage/series)` — measure shielding usage and its contribution to delivery costs (medium)
-- [ ] `Get CDN activity logs (GET /cdn/activity_log/requests)` — correlate resource changes and operator actions with delivery problems (medium)
-- [ ] `Get purges history list (GET /cdn/purge_statuses)` — analyze cache purge frequency and failures by CDN resource (medium)
+- [x] `Get CDN activity logs (GET /cdn/activity_log/requests)` — correlate resource changes and operator actions with delivery problems (medium)
+- [x] `Get purges history list (GET /cdn/purge_statuses)` — analyze cache purge frequency and failures by CDN resource (medium)
Note: Parsed every collection GET in the official CDN OpenAPI file and inspected statistics parameters and grouping options. Current statistics tables request only requests and sent_bytes, grouped by resource. Request logs cover only the latest three days; aggregate statistics omit the latest two hours. Excluded resource settings, certificates already synced, log uploader configuration, and other Gcore products outside the CDN source.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/canonical_descriptions.py
index 5758dc3abb39..86d60b343283 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/canonical_descriptions.py
@@ -56,4 +56,77 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"value": "Bytes sent from CDN servers to clients during the interval.",
},
},
+ "cdn_logs": {
+ "description": "Individual requests to CDN edge servers. Gcore keeps these logs for three days.",
+ "docs_url": "https://docs.gcore.com/api-reference/cdn/log-viewer/get-cdn-logs",
+ "columns": {
+ "timestamp": "Time of the request, in UTC.",
+ "client_ip": "IP address that sent the request.",
+ "cname": "Custom domain of the CDN resource.",
+ "resource_id": "Identifier of the CDN resource.",
+ "path": "Requested path.",
+ "method": "HTTP method of the request.",
+ "user_agent": "Value of the User-Agent header.",
+ "status": "HTTP status code of the response.",
+ "size": "Response size in bytes.",
+ "cache_status": "Cache status of the response, for example HIT or MISS.",
+ "datacenter": "Data center that processed the request.",
+ "referer": "Value of the Referer header.",
+ "sent_http_content_type": "Content-Type header of the response.",
+ "tcpinfo_rtt": "Round-trip time of the TCP connection.",
+ },
+ },
+ "cdn_daily_statistics": {
+ "description": "Daily traffic, response, and cache statistics, grouped by resource and edge server region.",
+ "docs_url": "https://docs.gcore.com/api-reference/cdn/statistics/aggregated-statistics",
+ "columns": {
+ "resource_id": "Identifier of the CDN resource.",
+ "region": "Region of the CDN edge servers, for example eu or na.",
+ "date": "Start of the day, in UTC.",
+ "requests": "Number of requests to edge servers.",
+ "sent_bytes": "Bytes sent from CDN servers to clients.",
+ "upstream_bytes": "Bytes sent from origin servers to CDN servers or to origin shielding.",
+ "total_bytes": "Origin shielding, upstream, and sent bytes combined.",
+ "responses_2xx": "Number of responses with a 2xx status code.",
+ "responses_3xx": "Number of responses with a 3xx status code.",
+ "responses_4xx": "Number of responses with a 4xx status code.",
+ "responses_5xx": "Number of responses with a 5xx status code.",
+ "responses_hit": "Number of responses served from the cache.",
+ "responses_miss": "Number of responses not served from the cache.",
+ "cache_hit_traffic_ratio": "Share of sent traffic served from the cache: 1 - upstream_bytes / sent_bytes.",
+ "cache_hit_requests_ratio": "Share of requests served from the cache: responses_hit / requests.",
+ "bandwidth_95_percentile": "95th percentile of bandwidth use, in bytes per second.",
+ "max_bandwidth": "Maximum bandwidth use, in bytes per second.",
+ "min_bandwidth": "Minimum bandwidth use, in bytes per second.",
+ },
+ },
+ "activity_logs": {
+ "description": "Requests made to the CDN API, such as changes to CDN resources and cache purges.",
+ "docs_url": "https://docs.gcore.com/api-reference/cdn/cdn-activity-logs/get-cdn-activity-logs",
+ "columns": {
+ "id": "Unique identifier of the activity log record.",
+ "user_id": "Identifier of the user who made the request.",
+ "token_id": "Identifier of the permanent API token used for the request.",
+ "client_id": "Identifier of the account that made the request.",
+ "requested_at": "Time of the request, in UTC.",
+ "path": "Requested URL path.",
+ "remote_ip_address": "IP address that sent the request.",
+ "host": "Host that received the request.",
+ "method": "HTTP method of the request.",
+ "query_params": "Query parameters of the request.",
+ "status_code": "HTTP status code of the response.",
+ },
+ },
+ "purge_statuses": {
+ "description": "Cache purge requests and their results.",
+ "docs_url": "https://docs.gcore.com/api-reference/cdn/purge-history/get-purges-history-list",
+ "columns": {
+ "purge_id": "Unique identifier of the purge.",
+ "status": "Purge status: In progress, Successful, or Failed.",
+ "resource": "CDN resource that the purge applies to, with its ID and domain.",
+ "payload": "URLs or path patterns that the purge removes from the cache.",
+ "created": "Time when the purge was created, in UTC.",
+ "purge_type": "Purge type: purge_by_url, purge_by_pattern, or purge_all.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/gcore.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/gcore.py
index 63ebcd155b9c..3eeb8c2e012b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/gcore.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/gcore.py
@@ -15,14 +15,21 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceInputs, SourceResponse
from products.warehouse_sources.backend.temporal.data_imports.sources.gcore.settings import (
+ AGGREGATE_METRICS,
AUTH_ERRORS,
BASE_URL,
ENDPOINTS,
HISTORY_DAYS,
+ INCREMENTAL_FILTER_PARAMS,
+ LOGS_DELAY,
+ LOGS_HISTORY_DAYS,
+ LOGS_PAGE_SIZE,
+ LOGS_WINDOW,
PAGE_SIZE,
PARTITION_KEYS,
PRIMARY_KEYS,
STATISTICS_METRICS,
+ WINDOWED_ENDPOINTS,
)
@@ -79,7 +86,7 @@ def _resource(
inputs.team_id,
inputs.job_id,
None,
- resume_hook=save_checkpoint if inputs.schema_name not in STATISTICS_METRICS else None,
+ resume_hook=save_checkpoint if inputs.schema_name not in WINDOWED_ENDPOINTS else None,
initial_paginator_state={"offset": resume.offset} if resume is not None else None,
)
@@ -118,6 +125,135 @@ def _origin_metadata(row: dict[str, Any]) -> dict[str, Any]:
return result
+def _activity_log_metadata(row: dict[str, Any]) -> dict[str, Any]:
+ # Request bodies and object states can contain origin credentials and signing keys.
+ return {key: value for key, value in row.items() if key not in ("data", "actions")}
+
+
+def _log_row(row: dict[str, Any]) -> dict[str, Any]:
+ return {**row, "timestamp": datetime.fromtimestamp(row["timestamp"], UTC)}
+
+
+def _aggregate_rows(body: dict[str, Any], day: datetime) -> list[dict[str, Any]]:
+ if body and "resource" not in body:
+ raise ValueError("Gcore aggregated statistics response does not contain resource groups.")
+ rows = []
+ for resource_id, group in body.get("resource", {}).items():
+ for region, values in group.get("region", {}).items():
+ metrics = values["metrics"]
+ row: dict[str, Any] = {"resource_id": resource_id, "region": region, "date": day}
+ for metric in AGGREGATE_METRICS:
+ # A column name that starts with a digit needs quoting in every SQL query.
+ column = "bandwidth_95_percentile" if metric == "95_percentile" else metric
+ row[column] = metrics.get(metric)
+ rows.append(row)
+ return rows
+
+
+def _logs(
+ api_key: str,
+ inputs: SourceInputs,
+ manager: ResumableSourceManager[GcoreCheckpoint],
+ resume: GcoreCheckpoint | None,
+) -> Iterator[list[dict[str, Any]]]:
+ now = datetime.now(UTC)
+ end = (now - LOGS_DELAY).replace(minute=0, second=0, microsecond=0)
+ start = (now - timedelta(days=LOGS_HISTORY_DAYS)).replace(hour=0, minute=0, second=0, microsecond=0)
+ if inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:
+ # Log timestamps have one-second precision, and appended rows are not deduplicated.
+ start = max(start, _timestamp(inputs.db_incremental_field_last_value) + timedelta(seconds=1))
+ offset = 0
+ if resume is not None and resume.window_start is not None and resume.window_end is not None:
+ start = _timestamp(resume.window_start)
+ end = _timestamp(resume.window_end)
+ offset = resume.offset
+ while start < end:
+ window_end = min(start + LOGS_WINDOW, end)
+ resource = _resource(
+ api_key,
+ inputs,
+ {
+ "path": ENDPOINTS[inputs.schema_name],
+ "params": {
+ "from": start.strftime("%Y-%m-%dT%H:%M:%SZ"),
+ "to": window_end.strftime("%Y-%m-%dT%H:%M:%SZ"),
+ "ordering": "timestamp",
+ },
+ "data_selector": "data",
+ "data_selector_required": True,
+ "paginator": {"type": "offset", "limit": LOGS_PAGE_SIZE, "total_path": "meta.count"},
+ },
+ manager,
+ GcoreCheckpoint(offset=offset),
+ ).add_map(_log_row)
+ pages = iter(resource)
+ page = next(pages, None)
+ if page is None:
+ manager.save_state(GcoreCheckpoint(window_start=window_end.isoformat(), window_end=end.isoformat()))
+ while page is not None:
+ # Read one page ahead, so the checkpoint of the last page moves on to the next window.
+ following = next(pages, None)
+ offset += LOGS_PAGE_SIZE
+ if following is None:
+ checkpoint = GcoreCheckpoint(window_start=window_end.isoformat(), window_end=end.isoformat())
+ else:
+ checkpoint = GcoreCheckpoint(offset=offset, window_start=start.isoformat(), window_end=end.isoformat())
+ manager.save_state(checkpoint)
+ # The next window returns the rows at its start time.
+ rows = [row for row in page if start <= row["timestamp"] < window_end]
+ if rows:
+ yield rows
+ page = following
+ offset = 0
+ manager.safe_point()
+ start = window_end
+
+
+def _daily_statistics(
+ api_key: str,
+ inputs: SourceInputs,
+ manager: ResumableSourceManager[GcoreCheckpoint],
+ resume: GcoreCheckpoint | None,
+) -> Iterator[list[dict[str, Any]]]:
+ now = datetime.now(UTC)
+ # Aggregated statistics leave out the last two hours, so only complete days are read.
+ end = (now - timedelta(hours=2)).replace(hour=0, minute=0, second=0, microsecond=0)
+ start = (now - timedelta(days=HISTORY_DAYS)).replace(hour=0, minute=0, second=0, microsecond=0)
+ if inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:
+ start = max(
+ start, _timestamp(inputs.db_incremental_field_last_value).replace(hour=0, minute=0, second=0, microsecond=0)
+ )
+ if resume is not None and resume.window_start is not None and resume.window_end is not None:
+ start = _timestamp(resume.window_start)
+ end = _timestamp(resume.window_end)
+ while start < end:
+ day_end = start + timedelta(days=1)
+ resource = _resource(
+ api_key,
+ inputs,
+ {
+ "path": ENDPOINTS[inputs.schema_name],
+ "paginator": "single_page",
+ "data_selector": "$",
+ "params": {
+ "service": "CDN",
+ "from": start.isoformat(),
+ # The upper bound is inclusive, so stop before the next day starts.
+ "to": (day_end - timedelta(seconds=1)).isoformat(),
+ "metrics": ",".join(AGGREGATE_METRICS),
+ "group_by": ["resource", "region"],
+ },
+ },
+ manager,
+ )
+ rows = [row for page in resource for body in page for row in _aggregate_rows(body, start)]
+ manager.save_state(GcoreCheckpoint(window_start=day_end.isoformat(), window_end=end.isoformat()))
+ if rows:
+ yield rows
+ manager.safe_point()
+ start = day_end
+
+
def _statistics(
api_key: str,
inputs: SourceInputs,
@@ -171,6 +307,26 @@ def gcore_source(
if inputs.schema_name not in ENDPOINTS:
raise ValueError(f"Unknown Gcore table: {inputs.schema_name}")
resume = manager.load_state() if manager.can_resume() else None
+ if inputs.schema_name == "cdn_logs":
+ return SourceResponse(
+ name=inputs.schema_name,
+ items=lambda: _logs(api_key, inputs, manager, resume),
+ primary_keys=PRIMARY_KEYS[inputs.schema_name],
+ partition_keys=PARTITION_KEYS[inputs.schema_name],
+ partition_mode="datetime",
+ partition_format="day",
+ sort_mode="asc",
+ )
+ if inputs.schema_name == "cdn_daily_statistics":
+ return SourceResponse(
+ name=inputs.schema_name,
+ items=lambda: _daily_statistics(api_key, inputs, manager, resume),
+ primary_keys=PRIMARY_KEYS[inputs.schema_name],
+ partition_keys=PARTITION_KEYS[inputs.schema_name],
+ partition_mode="datetime",
+ partition_format="month",
+ sort_mode="desc",
+ )
if inputs.schema_name in STATISTICS_METRICS:
return SourceResponse(
name=inputs.schema_name,
@@ -182,9 +338,10 @@ def gcore_source(
sort_mode="desc",
)
params: dict[str, Any] = {}
- if inputs.schema_name == "resources" and inputs.should_use_incremental_field:
+ filter_param = INCREMENTAL_FILTER_PARAMS.get(inputs.schema_name)
+ if filter_param is not None and inputs.should_use_incremental_field:
if inputs.db_incremental_field_last_value is not None:
- params["min_updated"] = _timestamp(inputs.db_incremental_field_last_value).isoformat()
+ params[filter_param] = _timestamp(inputs.db_incremental_field_last_value).isoformat()
resource = _resource(
api_key,
inputs,
@@ -202,6 +359,8 @@ def gcore_source(
resource.add_map(_resource_metadata)
elif inputs.schema_name == "origin_groups":
resource.add_map(_origin_metadata)
+ elif inputs.schema_name == "activity_logs":
+ resource.add_map(_activity_log_metadata)
return SourceResponse(
name=inputs.schema_name, items=lambda: resource, primary_keys=PRIMARY_KEYS[inputs.schema_name], sort_mode="desc"
)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/settings.py
index b20efcd90bc3..7269ef1cb01b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/settings.py
@@ -1,25 +1,79 @@
+from datetime import timedelta
+
from products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import incremental_field
BASE_URL = "https://api.gcore.com/cdn/"
PAGE_SIZE = 100
HISTORY_DAYS = 365
+# Request logs are kept for three days and a request may span at most six hours.
+LOGS_PAGE_SIZE = 1000
+LOGS_HISTORY_DAYS = 2
+LOGS_WINDOW = timedelta(hours=6)
+LOGS_DELAY = timedelta(hours=1)
+
ENDPOINTS = {
"resources": "resources",
"origin_groups": "origin_groups",
"ssl_certificates": "sslData",
"cdn_requests": "statistics/series",
"cdn_traffic": "statistics/series",
+ "cdn_logs": "advanced/v1/logs",
+ "cdn_daily_statistics": "statistics/aggregate/stats",
+ "activity_logs": "activity_log/requests",
+ "purge_statuses": "purge_statuses",
}
STATISTICS_METRICS = {"cdn_requests": "requests", "cdn_traffic": "sent_bytes"}
-PRIMARY_KEYS = {
- name: ["resource_id", "timestamp", "metric"] if name in STATISTICS_METRICS else ["id"] for name in ENDPOINTS
+AGGREGATE_METRICS = (
+ "requests",
+ "sent_bytes",
+ "upstream_bytes",
+ "total_bytes",
+ "responses_2xx",
+ "responses_3xx",
+ "responses_4xx",
+ "responses_5xx",
+ "responses_hit",
+ "responses_miss",
+ "cache_hit_traffic_ratio",
+ "cache_hit_requests_ratio",
+ "95_percentile",
+ "max_bandwidth",
+ "min_bandwidth",
+)
+WINDOWED_ENDPOINTS = {*STATISTICS_METRICS, "cdn_logs", "cdn_daily_statistics"}
+PRIMARY_KEYS: dict[str, list[str] | None] = {
+ **{name: ["id"] for name in ENDPOINTS},
+ **{name: ["resource_id", "timestamp", "metric"] for name in STATISTICS_METRICS},
+ "cdn_logs": None,
+ "cdn_daily_statistics": ["resource_id", "region", "date"],
+ "purge_statuses": ["purge_id"],
+}
+PARTITION_KEYS = {
+ **{name: ["timestamp"] for name in STATISTICS_METRICS},
+ "cdn_logs": ["timestamp"],
+ "cdn_daily_statistics": ["date"],
}
-PARTITION_KEYS = {name: ["timestamp"] for name in STATISTICS_METRICS}
INCREMENTAL_FIELDS = {
"resources": [incremental_field("updated")],
"cdn_requests": [incremental_field("timestamp")],
"cdn_traffic": [incremental_field("timestamp")],
+ "cdn_logs": [incremental_field("timestamp")],
+ "cdn_daily_statistics": [incremental_field("date")],
+ "activity_logs": [incremental_field("requested_at")],
+ "purge_statuses": [incremental_field("created")],
+}
+INCREMENTAL_FILTER_PARAMS = {
+ "resources": "min_updated",
+ "activity_logs": "min_requested_at",
+ "purge_statuses": "from_created",
+}
+APPEND_ONLY_ENDPOINTS = {"cdn_logs"}
+MERGE_ONLY_ENDPOINTS = {*STATISTICS_METRICS, "cdn_daily_statistics"}
+# Statistics are restated for recent days, and purges change status after they are created.
+INCREMENTAL_LOOKBACK_SECONDS = {
+ **dict.fromkeys(MERGE_ONLY_ENDPOINTS, 2 * 24 * 60 * 60),
+ "purge_statuses": 24 * 60 * 60,
}
AUTH_ERRORS = {
401: "Gcore rejected the API token. Check that the token is valid and has not expired.",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/source.py
index bcf43515e50e..00fa029d3dad 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/source.py
@@ -28,10 +28,12 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.gcore.gcor
validate_credentials,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.gcore.settings import (
+ APPEND_ONLY_ENDPOINTS,
AUTH_ERRORS,
ENDPOINTS,
INCREMENTAL_FIELDS,
- STATISTICS_METRICS,
+ INCREMENTAL_LOOKBACK_SECONDS,
+ MERGE_ONLY_ENDPOINTS,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.gcore import GcoreSourceConfig
from products.warehouse_sources.backend.types import ExternalDataSourceType
@@ -108,11 +110,16 @@ class GcoreSource(ResumableSource[GcoreSourceConfig, GcoreCheckpoint]):
force_refresh: bool = False,
api_version: str | None = None,
) -> list[SourceSchema]:
- schemas = build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names, merge_only=STATISTICS_METRICS)
- for schema in schemas:
- if schema.name in STATISTICS_METRICS:
- schema.default_incremental_lookback_seconds = 2 * 24 * 60 * 60
- return schemas
+ return build_endpoint_schemas(
+ ENDPOINTS,
+ INCREMENTAL_FIELDS,
+ names,
+ append_only=APPEND_ONLY_ENDPOINTS,
+ merge_only=MERGE_ONLY_ENDPOINTS,
+ # Request logs can hold one row per CDN request, so syncing them is opt-in.
+ should_sync_default={"cdn_logs": False},
+ default_incremental_lookback_seconds=INCREMENTAL_LOOKBACK_SECONDS,
+ )
def validate_credentials(
self,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/tests/test_gcore.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/tests/test_gcore.py
index c5bb5c111e4d..2f4455ae7eab 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/tests/test_gcore.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/tests/test_gcore.py
@@ -53,10 +53,16 @@ def test_list_pagination(name: str, path: str, last_page: list[dict[str, int]])
assert result.committed_states == [GcoreCheckpoint(offset=2)]
+ "name,filter_param",
+ [("resources", "min_updated"), ("activity_logs", "min_requested_at"), ("purge_statuses", "from_created")],
+)
@pytest.mark.parametrize("incremental,watermark", [(False, NOW), (True, None), (True, NOW), (True, NOW.isoformat())])
-def test_resource_filter_and_resume(incremental: bool, watermark: datetime | str | None) -> None:
+def test_list_filter_and_resume(
+ name: str, filter_param: str, incremental: bool, watermark: datetime | str | None
+) -> None:
result = driver().run(
- "resources",
+ name,
[ScriptedResponse(json={"count": 101, "next": None, "results": [{"id": 101}]})],
resume_state=GcoreCheckpoint(offset=100),
should_use_incremental_field=incremental,
@@ -66,7 +72,7 @@ def test_resource_filter_and_resume(incremental: bool, watermark: datetime | str
assert result.raised is None
assert result.items == [[{"id": 101}]]
assert result.params("offset") == ["100"]
- assert result.params("min_updated") == [NOW.isoformat() if incremental and watermark else None]
+ assert result.params(filter_param) == [NOW.isoformat() if incremental and watermark else None]
assert result.response is not None
assert result.response.sort_mode == "desc"
@@ -166,21 +172,36 @@ def test_sync_authentication_errors_are_terminal(status: int) -> None:
assert messages and all(messages)
-def test_origin_credentials_are_excluded() -> None:
- body: dict[str, Any] = {
- "count": 1,
- "results": [
+ "name,row,expected",
+ [
+ (
+ "origin_groups",
{
"id": 1,
"auth": {"secret_access_key": "fake-secret"},
"sources": [{"source": "origin.example.com", "enabled": True, "config": {"secret_key": "fake-secret"}}],
- }
- ],
- }
- result = driver().run("origin_groups", [ScriptedResponse(json=body)])
+ },
+ {"id": 1, "sources": [{"source": "origin.example.com", "enabled": True}]},
+ ),
+ (
+ "activity_logs",
+ {
+ "id": 1,
+ "path": "/origin_groups/7",
+ "method": "PUT",
+ "data": {"auth": {"secret_access_key": "fake-secret"}},
+ "actions": [{"state_after_request": {"auth": {"secret_access_key": "fake-secret"}}}],
+ },
+ {"id": 1, "path": "/origin_groups/7", "method": "PUT"},
+ ),
+ ],
+)
+def test_credentials_are_excluded(name: str, row: dict[str, Any], expected: dict[str, Any]) -> None:
+ result = driver().run(name, [ScriptedResponse(json={"count": 1, "results": [row]})])
assert result.raised is None
- assert result.items == [[{"id": 1, "sources": [{"source": "origin.example.com", "enabled": True}]}]]
+ assert result.items == [[expected]]
def test_malformed_statistics_do_not_advance_checkpoint() -> None:
@@ -191,3 +212,109 @@ def test_malformed_statistics_do_not_advance_checkpoint() -> None:
assert "does not contain resource groups" in str(result.raised)
assert result.saved_states == []
assert result.final_state == resume
+
+
+def _window(request: RecordedRequest) -> tuple[datetime, datetime]:
+ start, end = request.param("from"), request.param("to")
+ assert start is not None and end is not None
+ return datetime.fromisoformat(start), datetime.fromisoformat(end)
+
+
+ "mode,expected_windows",
+ [
+ ("full", [(datetime(2026, 1, 5, 0, tzinfo=UTC), 6), (datetime(2026, 1, 5, 6, tzinfo=UTC), 5)]),
+ ("incremental", [(datetime(2026, 1, 5, 4, tzinfo=UTC), 6), (datetime(2026, 1, 5, 10, tzinfo=UTC), 1)]),
+ ("resume", [(datetime(2026, 1, 5, 6, tzinfo=UTC), 5)]),
+ ],
+)
+@patch(f"{TRANSPORT}.datetime", wraps=datetime)
+def test_logs_windows_pages_and_resume(
+ clock: MagicMock, mode: str, expected_windows: list[tuple[datetime, int]]
+) -> None:
+ clock.now.return_value = NOW
+
+ def answer(request: RecordedRequest) -> ScriptedResponse:
+ start, end = _window(request)
+ first = int(start.timestamp())
+ rows = [{"timestamp": first, "status": 200}, {"timestamp": first + 1, "status": 404}]
+ rows.append({"timestamp": int(end.timestamp()), "status": 200})
+ offset = int(request.param("offset") or 0)
+ return ScriptedResponse(json={"data": rows[offset : offset + 2], "meta": {"count": len(rows)}})
+
+ end = datetime(2026, 1, 5, 11, tzinfo=UTC)
+ resume = (
+ GcoreCheckpoint(offset=1, window_start=expected_windows[0][0].isoformat(), window_end=end.isoformat())
+ if mode == "resume"
+ else None
+ )
+ with patch(f"{TRANSPORT}.LOGS_PAGE_SIZE", 2), patch(f"{TRANSPORT}.LOGS_HISTORY_DAYS", 0):
+ result = driver().run(
+ "cdn_logs",
+ answer,
+ resume_state=resume,
+ should_use_incremental_field=mode == "incremental",
+ db_incremental_field_last_value=datetime(2026, 1, 5, 3, 59, 59, tzinfo=UTC),
+ )
+
+ assert result.raised is None
+ assert sorted({_window(request) for request in result.requests}) == [
+ (start, start + timedelta(hours=hours)) for start, hours in expected_windows
+ ]
+ assert result.params("offset") == (["1"] if mode == "resume" else ["0", "2"] * len(expected_windows))
+ assert set(result.params("ordering")) == {"timestamp"}
+ expected_rows = []
+ for start, _ in expected_windows:
+ if mode != "resume":
+ expected_rows.append({"timestamp": start, "status": 200})
+ expected_rows.append({"timestamp": start + timedelta(seconds=1), "status": 404})
+ assert result.rows == expected_rows
+ assert result.final_state == GcoreCheckpoint(window_start=end.isoformat(), window_end=end.isoformat())
+
+
+ "now,incremental,expected_days",
+ [
+ (NOW, False, [datetime(2026, 1, 3, tzinfo=UTC), datetime(2026, 1, 4, tzinfo=UTC)]),
+ (NOW, True, [datetime(2026, 1, 4, tzinfo=UTC)]),
+ (datetime(2026, 1, 5, 1, 30, tzinfo=UTC), False, [datetime(2026, 1, 3, tzinfo=UTC)]),
+ ],
+)
+@patch(f"{TRANSPORT}.datetime", wraps=datetime)
+def test_daily_statistics_rows_and_days(
+ clock: MagicMock, now: datetime, incremental: bool, expected_days: list[datetime]
+) -> None:
+ clock.now.return_value = now
+ body = {
+ "resource": {
+ "42": {
+ "region": {
+ "eu": {"metrics": {"requests": 10, "responses_5xx": 1, "95_percentile": 300}},
+ "na": {"metrics": {"requests": 5}},
+ }
+ }
+ }
+ }
+
+ def answer(request: RecordedRequest) -> ScriptedResponse:
+ start, _ = _window(request)
+ return ScriptedResponse(json=body) if start == expected_days[0] else ScriptedResponse(status=204)
+
+ with patch(f"{TRANSPORT}.HISTORY_DAYS", 2):
+ result = driver().run(
+ "cdn_daily_statistics",
+ answer,
+ should_use_incremental_field=incremental,
+ db_incremental_field_last_value=datetime(2026, 1, 4, 6, tzinfo=UTC),
+ )
+
+ assert result.raised is None
+ assert [_window(request) for request in result.requests] == [
+ (day, day + timedelta(days=1) - timedelta(seconds=1)) for day in expected_days
+ ]
+ assert all(query["group_by"] == ("resource", "region") for query in result.queries)
+ eu, na = result.rows
+ assert (eu["resource_id"], eu["region"], eu["date"]) == ("42", "eu", expected_days[0])
+ assert (eu["requests"], eu["responses_5xx"], eu["bandwidth_95_percentile"]) == (10, 1, 300)
+ assert (na["region"], na["requests"], na["responses_5xx"]) == ("na", 5, None)
+ assert "95_percentile" not in eu