:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add gcore logs, daily statistics, activity log and purge tables (#120225)
Tom Owers0607479d08b8
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 91c5c384cfcc..ec21b4acda19 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -5171,11 +5171,11 @@ Today (5): `cdn_requests`, `cdn_traffic`, `origin_groups`, `resources`, `ssl_cerDiffed against: <https://docs.gcore.com/api-reference/services_docs_mintlify_enriched/cdn_api.yaml>-- [ ] `Get CDN logs (GET /cdn/advanced/v1/logs)` — analyze individual requests, status codes, client locations, and paths beyond hourly request totals (high)-- [ ] `Aggregated statistics (GET /cdn/statistics/aggregate/stats)` — compare cache efficiency, error rates, and traffic across countries and regions (high)+- [x] `Get CDN logs (GET /cdn/advanced/v1/logs)` — analyze individual requests, status codes, client locations, and paths beyond hourly request totals (high)+- [x] `Aggregated statistics (GET /cdn/statistics/aggregate/stats)` — compare cache efficiency, error rates, and traffic across countries and regions (high)- [ ] `Origin shielding usage statistics (GET /cdn/statistics/shield_usage/series)` — measure shielding usage and its contribution to delivery costs (medium)-- [ ] `Get CDN activity logs (GET /cdn/activity_log/requests)` — correlate resource changes and operator actions with delivery problems (medium)-- [ ] `Get purges history list (GET /cdn/purge_statuses)` — analyze cache purge frequency and failures by CDN resource (medium)+- [x] `Get CDN activity logs (GET /cdn/activity_log/requests)` — correlate resource changes and operator actions with delivery problems (medium)+- [x] `Get purges history list (GET /cdn/purge_statuses)` — analyze cache purge frequency and failures by CDN resource (medium)Note: Parsed every collection GET in the official CDN OpenAPI file and inspected statistics parameters and grouping options. Current statistics tables request only requests and sent_bytes, grouped by resource. Request logs cover only the latest three days; aggregate statistics omit the latest two hours. Excluded resource settings, certificates already synced, log uploader configuration, and other Gcore products outside the CDN source.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/canonical_descriptions.pyindex 5758dc3abb39..86d60b343283 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/canonical_descriptions.py@@ -56,4 +56,77 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"value": "Bytes sent from CDN servers to clients during the interval.",},},+ "cdn_logs": {+ "description": "Individual requests to CDN edge servers. Gcore keeps these logs for three days.",+ "docs_url": "https://docs.gcore.com/api-reference/cdn/log-viewer/get-cdn-logs",+ "columns": {+ "timestamp": "Time of the request, in UTC.",+ "client_ip": "IP address that sent the request.",+ "cname": "Custom domain of the CDN resource.",+ "resource_id": "Identifier of the CDN resource.",+ "path": "Requested path.",+ "method": "HTTP method of the request.",+ "user_agent": "Value of the User-Agent header.",+ "status": "HTTP status code of the response.",+ "size": "Response size in bytes.",+ "cache_status": "Cache status of the response, for example HIT or MISS.",+ "datacenter": "Data center that processed the request.",+ "referer": "Value of the Referer header.",+ "sent_http_content_type": "Content-Type header of the response.",+ "tcpinfo_rtt": "Round-trip time of the TCP connection.",+ },+ },+ "cdn_daily_statistics": {+ "description": "Daily traffic, response, and cache statistics, grouped by resource and edge server region.",+ "docs_url": "https://docs.gcore.com/api-reference/cdn/statistics/aggregated-statistics",+ "columns": {+ "resource_id": "Identifier of the CDN resource.",+ "region": "Region of the CDN edge servers, for example eu or na.",+ "date": "Start of the day, in UTC.",+ "requests": "Number of requests to edge servers.",+ "sent_bytes": "Bytes sent from CDN servers to clients.",+ "upstream_bytes": "Bytes sent from origin servers to CDN servers or to origin shielding.",+ "total_bytes": "Origin shielding, upstream, and sent bytes combined.",+ "responses_2xx": "Number of responses with a 2xx status code.",+ "responses_3xx": "Number of responses with a 3xx status code.",+ "responses_4xx": "Number of responses with a 4xx status code.",+ "responses_5xx": "Number of responses with a 5xx status code.",+ "responses_hit": "Number of responses served from the cache.",+ "responses_miss": "Number of responses not served from the cache.",+ "cache_hit_traffic_ratio": "Share of sent traffic served from the cache: 1 - upstream_bytes / sent_bytes.",+ "cache_hit_requests_ratio": "Share of requests served from the cache: responses_hit / requests.",+ "bandwidth_95_percentile": "95th percentile of bandwidth use, in bytes per second.",+ "max_bandwidth": "Maximum bandwidth use, in bytes per second.",+ "min_bandwidth": "Minimum bandwidth use, in bytes per second.",+ },+ },+ "activity_logs": {+ "description": "Requests made to the CDN API, such as changes to CDN resources and cache purges.",+ "docs_url": "https://docs.gcore.com/api-reference/cdn/cdn-activity-logs/get-cdn-activity-logs",+ "columns": {+ "id": "Unique identifier of the activity log record.",+ "user_id": "Identifier of the user who made the request.",+ "token_id": "Identifier of the permanent API token used for the request.",+ "client_id": "Identifier of the account that made the request.",+ "requested_at": "Time of the request, in UTC.",+ "path": "Requested URL path.",+ "remote_ip_address": "IP address that sent the request.",+ "host": "Host that received the request.",+ "method": "HTTP method of the request.",+ "query_params": "Query parameters of the request.",+ "status_code": "HTTP status code of the response.",+ },+ },+ "purge_statuses": {+ "description": "Cache purge requests and their results.",+ "docs_url": "https://docs.gcore.com/api-reference/cdn/purge-history/get-purges-history-list",+ "columns": {+ "purge_id": "Unique identifier of the purge.",+ "status": "Purge status: In progress, Successful, or Failed.",+ "resource": "CDN resource that the purge applies to, with its ID and domain.",+ "payload": "URLs or path patterns that the purge removes from the cache.",+ "created": "Time when the purge was created, in UTC.",+ "purge_type": "Purge type: purge_by_url, purge_by_pattern, or purge_all.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/gcore.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/gcore.pyindex 63ebcd155b9c..3eeb8c2e012b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/gcore.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/gcore.py@@ -15,14 +15,21 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceInputs, SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.gcore.settings import (+ AGGREGATE_METRICS,AUTH_ERRORS,BASE_URL,ENDPOINTS,HISTORY_DAYS,+ INCREMENTAL_FILTER_PARAMS,+ LOGS_DELAY,+ LOGS_HISTORY_DAYS,+ LOGS_PAGE_SIZE,+ LOGS_WINDOW,PAGE_SIZE,PARTITION_KEYS,PRIMARY_KEYS,STATISTICS_METRICS,+ WINDOWED_ENDPOINTS,)@@ -79,7 +86,7 @@ def _resource(inputs.team_id,inputs.job_id,None,- resume_hook=save_checkpoint if inputs.schema_name not in STATISTICS_METRICS else None,+ resume_hook=save_checkpoint if inputs.schema_name not in WINDOWED_ENDPOINTS else None,initial_paginator_state={"offset": resume.offset} if resume is not None else None,)@@ -118,6 +125,135 @@ def _origin_metadata(row: dict[str, Any]) -> dict[str, Any]:return result+def _activity_log_metadata(row: dict[str, Any]) -> dict[str, Any]:+ # Request bodies and object states can contain origin credentials and signing keys.+ return {key: value for key, value in row.items() if key not in ("data", "actions")}+++def _log_row(row: dict[str, Any]) -> dict[str, Any]:+ return {**row, "timestamp": datetime.fromtimestamp(row["timestamp"], UTC)}+++def _aggregate_rows(body: dict[str, Any], day: datetime) -> list[dict[str, Any]]:+ if body and "resource" not in body:+ raise ValueError("Gcore aggregated statistics response does not contain resource groups.")+ rows = []+ for resource_id, group in body.get("resource", {}).items():+ for region, values in group.get("region", {}).items():+ metrics = values["metrics"]+ row: dict[str, Any] = {"resource_id": resource_id, "region": region, "date": day}+ for metric in AGGREGATE_METRICS:+ # A column name that starts with a digit needs quoting in every SQL query.+ column = "bandwidth_95_percentile" if metric == "95_percentile" else metric+ row[column] = metrics.get(metric)+ rows.append(row)+ return rows+++def _logs(+ api_key: str,+ inputs: SourceInputs,+ manager: ResumableSourceManager[GcoreCheckpoint],+ resume: GcoreCheckpoint | None,+) -> Iterator[list[dict[str, Any]]]:+ now = datetime.now(UTC)+ end = (now - LOGS_DELAY).replace(minute=0, second=0, microsecond=0)+ start = (now - timedelta(days=LOGS_HISTORY_DAYS)).replace(hour=0, minute=0, second=0, microsecond=0)+ if inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:+ # Log timestamps have one-second precision, and appended rows are not deduplicated.+ start = max(start, _timestamp(inputs.db_incremental_field_last_value) + timedelta(seconds=1))+ offset = 0+ if resume is not None and resume.window_start is not None and resume.window_end is not None:+ start = _timestamp(resume.window_start)+ end = _timestamp(resume.window_end)+ offset = resume.offset+ while start < end:+ window_end = min(start + LOGS_WINDOW, end)+ resource = _resource(+ api_key,+ inputs,+ {+ "path": ENDPOINTS[inputs.schema_name],+ "params": {+ "from": start.strftime("%Y-%m-%dT%H:%M:%SZ"),+ "to": window_end.strftime("%Y-%m-%dT%H:%M:%SZ"),+ "ordering": "timestamp",+ },+ "data_selector": "data",+ "data_selector_required": True,+ "paginator": {"type": "offset", "limit": LOGS_PAGE_SIZE, "total_path": "meta.count"},+ },+ manager,+ GcoreCheckpoint(offset=offset),+ ).add_map(_log_row)+ pages = iter(resource)+ page = next(pages, None)+ if page is None:+ manager.save_state(GcoreCheckpoint(window_start=window_end.isoformat(), window_end=end.isoformat()))+ while page is not None:+ # Read one page ahead, so the checkpoint of the last page moves on to the next window.+ following = next(pages, None)+ offset += LOGS_PAGE_SIZE+ if following is None:+ checkpoint = GcoreCheckpoint(window_start=window_end.isoformat(), window_end=end.isoformat())+ else:+ checkpoint = GcoreCheckpoint(offset=offset, window_start=start.isoformat(), window_end=end.isoformat())+ manager.save_state(checkpoint)+ # The next window returns the rows at its start time.+ rows = [row for row in page if start <= row["timestamp"] < window_end]+ if rows:+ yield rows+ page = following+ offset = 0+ manager.safe_point()+ start = window_end+++def _daily_statistics(+ api_key: str,+ inputs: SourceInputs,+ manager: ResumableSourceManager[GcoreCheckpoint],+ resume: GcoreCheckpoint | None,+) -> Iterator[list[dict[str, Any]]]:+ now = datetime.now(UTC)+ # Aggregated statistics leave out the last two hours, so only complete days are read.+ end = (now - timedelta(hours=2)).replace(hour=0, minute=0, second=0, microsecond=0)+ start = (now - timedelta(days=HISTORY_DAYS)).replace(hour=0, minute=0, second=0, microsecond=0)+ if inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:+ start = max(+ start, _timestamp(inputs.db_incremental_field_last_value).replace(hour=0, minute=0, second=0, microsecond=0)+ )+ if resume is not None and resume.window_start is not None and resume.window_end is not None:+ start = _timestamp(resume.window_start)+ end = _timestamp(resume.window_end)+ while start < end:+ day_end = start + timedelta(days=1)+ resource = _resource(+ api_key,+ inputs,+ {+ "path": ENDPOINTS[inputs.schema_name],+ "paginator": "single_page",+ "data_selector": "$",+ "params": {+ "service": "CDN",+ "from": start.isoformat(),+ # The upper bound is inclusive, so stop before the next day starts.+ "to": (day_end - timedelta(seconds=1)).isoformat(),+ "metrics": ",".join(AGGREGATE_METRICS),+ "group_by": ["resource", "region"],+ },+ },+ manager,+ )+ rows = [row for page in resource for body in page for row in _aggregate_rows(body, start)]+ manager.save_state(GcoreCheckpoint(window_start=day_end.isoformat(), window_end=end.isoformat()))+ if rows:+ yield rows+ manager.safe_point()+ start = day_end++def _statistics(api_key: str,inputs: SourceInputs,@@ -171,6 +307,26 @@ def gcore_source(if inputs.schema_name not in ENDPOINTS:raise ValueError(f"Unknown Gcore table: {inputs.schema_name}")resume = manager.load_state() if manager.can_resume() else None+ if inputs.schema_name == "cdn_logs":+ return SourceResponse(+ name=inputs.schema_name,+ items=lambda: _logs(api_key, inputs, manager, resume),+ primary_keys=PRIMARY_KEYS[inputs.schema_name],+ partition_keys=PARTITION_KEYS[inputs.schema_name],+ partition_mode="datetime",+ partition_format="day",+ sort_mode="asc",+ )+ if inputs.schema_name == "cdn_daily_statistics":+ return SourceResponse(+ name=inputs.schema_name,+ items=lambda: _daily_statistics(api_key, inputs, manager, resume),+ primary_keys=PRIMARY_KEYS[inputs.schema_name],+ partition_keys=PARTITION_KEYS[inputs.schema_name],+ partition_mode="datetime",+ partition_format="month",+ sort_mode="desc",+ )if inputs.schema_name in STATISTICS_METRICS:return SourceResponse(name=inputs.schema_name,@@ -182,9 +338,10 @@ def gcore_source(sort_mode="desc",)params: dict[str, Any] = {}- if inputs.schema_name == "resources" and inputs.should_use_incremental_field:+ filter_param = INCREMENTAL_FILTER_PARAMS.get(inputs.schema_name)+ if filter_param is not None and inputs.should_use_incremental_field:if inputs.db_incremental_field_last_value is not None:- params["min_updated"] = _timestamp(inputs.db_incremental_field_last_value).isoformat()+ params[filter_param] = _timestamp(inputs.db_incremental_field_last_value).isoformat()resource = _resource(api_key,inputs,@@ -202,6 +359,8 @@ def gcore_source(resource.add_map(_resource_metadata)elif inputs.schema_name == "origin_groups":resource.add_map(_origin_metadata)+ elif inputs.schema_name == "activity_logs":+ resource.add_map(_activity_log_metadata)return SourceResponse(name=inputs.schema_name, items=lambda: resource, primary_keys=PRIMARY_KEYS[inputs.schema_name], sort_mode="desc")diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/settings.pyindex b20efcd90bc3..7269ef1cb01b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/settings.py@@ -1,25 +1,79 @@+from datetime import timedelta+from products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import incremental_fieldBASE_URL = "https://api.gcore.com/cdn/"PAGE_SIZE = 100HISTORY_DAYS = 365+# Request logs are kept for three days and a request may span at most six hours.+LOGS_PAGE_SIZE = 1000+LOGS_HISTORY_DAYS = 2+LOGS_WINDOW = timedelta(hours=6)+LOGS_DELAY = timedelta(hours=1)+ENDPOINTS = {"resources": "resources","origin_groups": "origin_groups","ssl_certificates": "sslData","cdn_requests": "statistics/series","cdn_traffic": "statistics/series",+ "cdn_logs": "advanced/v1/logs",+ "cdn_daily_statistics": "statistics/aggregate/stats",+ "activity_logs": "activity_log/requests",+ "purge_statuses": "purge_statuses",}STATISTICS_METRICS = {"cdn_requests": "requests", "cdn_traffic": "sent_bytes"}-PRIMARY_KEYS = {- name: ["resource_id", "timestamp", "metric"] if name in STATISTICS_METRICS else ["id"] for name in ENDPOINTS+AGGREGATE_METRICS = (+ "requests",+ "sent_bytes",+ "upstream_bytes",+ "total_bytes",+ "responses_2xx",+ "responses_3xx",+ "responses_4xx",+ "responses_5xx",+ "responses_hit",+ "responses_miss",+ "cache_hit_traffic_ratio",+ "cache_hit_requests_ratio",+ "95_percentile",+ "max_bandwidth",+ "min_bandwidth",+)+WINDOWED_ENDPOINTS = {*STATISTICS_METRICS, "cdn_logs", "cdn_daily_statistics"}+PRIMARY_KEYS: dict[str, list[str] | None] = {+ **{name: ["id"] for name in ENDPOINTS},+ **{name: ["resource_id", "timestamp", "metric"] for name in STATISTICS_METRICS},+ "cdn_logs": None,+ "cdn_daily_statistics": ["resource_id", "region", "date"],+ "purge_statuses": ["purge_id"],+}+PARTITION_KEYS = {+ **{name: ["timestamp"] for name in STATISTICS_METRICS},+ "cdn_logs": ["timestamp"],+ "cdn_daily_statistics": ["date"],}-PARTITION_KEYS = {name: ["timestamp"] for name in STATISTICS_METRICS}INCREMENTAL_FIELDS = {"resources": [incremental_field("updated")],"cdn_requests": [incremental_field("timestamp")],"cdn_traffic": [incremental_field("timestamp")],+ "cdn_logs": [incremental_field("timestamp")],+ "cdn_daily_statistics": [incremental_field("date")],+ "activity_logs": [incremental_field("requested_at")],+ "purge_statuses": [incremental_field("created")],+}+INCREMENTAL_FILTER_PARAMS = {+ "resources": "min_updated",+ "activity_logs": "min_requested_at",+ "purge_statuses": "from_created",+}+APPEND_ONLY_ENDPOINTS = {"cdn_logs"}+MERGE_ONLY_ENDPOINTS = {*STATISTICS_METRICS, "cdn_daily_statistics"}+# Statistics are restated for recent days, and purges change status after they are created.+INCREMENTAL_LOOKBACK_SECONDS = {+ **dict.fromkeys(MERGE_ONLY_ENDPOINTS, 2 * 24 * 60 * 60),+ "purge_statuses": 24 * 60 * 60,}AUTH_ERRORS = {401: "Gcore rejected the API token. Check that the token is valid and has not expired.",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/source.pyindex bcf43515e50e..00fa029d3dad 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/source.py@@ -28,10 +28,12 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.gcore.gcorvalidate_credentials,)from products.warehouse_sources.backend.temporal.data_imports.sources.gcore.settings import (+ APPEND_ONLY_ENDPOINTS,AUTH_ERRORS,ENDPOINTS,INCREMENTAL_FIELDS,- STATISTICS_METRICS,+ INCREMENTAL_LOOKBACK_SECONDS,+ MERGE_ONLY_ENDPOINTS,)from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.gcore import GcoreSourceConfigfrom products.warehouse_sources.backend.types import ExternalDataSourceType@@ -108,11 +110,16 @@ class GcoreSource(ResumableSource[GcoreSourceConfig, GcoreCheckpoint]):force_refresh: bool = False,api_version: str | None = None,) -> list[SourceSchema]:- schemas = build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names, merge_only=STATISTICS_METRICS)- for schema in schemas:- if schema.name in STATISTICS_METRICS:- schema.default_incremental_lookback_seconds = 2 * 24 * 60 * 60- return schemas+ return build_endpoint_schemas(+ ENDPOINTS,+ INCREMENTAL_FIELDS,+ names,+ append_only=APPEND_ONLY_ENDPOINTS,+ merge_only=MERGE_ONLY_ENDPOINTS,+ # Request logs can hold one row per CDN request, so syncing them is opt-in.+ should_sync_default={"cdn_logs": False},+ default_incremental_lookback_seconds=INCREMENTAL_LOOKBACK_SECONDS,+ )def validate_credentials(self,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/tests/test_gcore.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/tests/test_gcore.pyindex c5bb5c111e4d..2f4455ae7eab 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/tests/test_gcore.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gcore/tests/test_gcore.py@@ -53,10 +53,16 @@ def test_list_pagination(name: str, path: str, last_page: list[dict[str, int]])assert result.committed_states == [GcoreCheckpoint(offset=2)]+ "name,filter_param",+ [("resources", "min_updated"), ("activity_logs", "min_requested_at"), ("purge_statuses", "from_created")],+)@pytest.mark.parametrize("incremental,watermark", [(False, NOW), (True, None), (True, NOW), (True, NOW.isoformat())])-def test_resource_filter_and_resume(incremental: bool, watermark: datetime | str | None) -> None:+def test_list_filter_and_resume(+ name: str, filter_param: str, incremental: bool, watermark: datetime | str | None+) -> None:result = driver().run(- "resources",+ name,[ScriptedResponse(json={"count": 101, "next": None, "results": [{"id": 101}]})],resume_state=GcoreCheckpoint(offset=100),should_use_incremental_field=incremental,@@ -66,7 +72,7 @@ def test_resource_filter_and_resume(incremental: bool, watermark: datetime | strassert result.raised is Noneassert result.items == [[{"id": 101}]]assert result.params("offset") == ["100"]- assert result.params("min_updated") == [NOW.isoformat() if incremental and watermark else None]+ assert result.params(filter_param) == [NOW.isoformat() if incremental and watermark else None]assert result.response is not Noneassert result.response.sort_mode == "desc"@@ -166,21 +172,36 @@ def test_sync_authentication_errors_are_terminal(status: int) -> None:assert messages and all(messages)-def test_origin_credentials_are_excluded() -> None:- body: dict[str, Any] = {- "count": 1,- "results": [+ "name,row,expected",+ [+ (+ "origin_groups",{"id": 1,"auth": {"secret_access_key": "fake-secret"},"sources": [{"source": "origin.example.com", "enabled": True, "config": {"secret_key": "fake-secret"}}],- }- ],- }- result = driver().run("origin_groups", [ScriptedResponse(json=body)])+ },+ {"id": 1, "sources": [{"source": "origin.example.com", "enabled": True}]},+ ),+ (+ "activity_logs",+ {+ "id": 1,+ "path": "/origin_groups/7",+ "method": "PUT",+ "data": {"auth": {"secret_access_key": "fake-secret"}},+ "actions": [{"state_after_request": {"auth": {"secret_access_key": "fake-secret"}}}],+ },+ {"id": 1, "path": "/origin_groups/7", "method": "PUT"},+ ),+ ],+)+def test_credentials_are_excluded(name: str, row: dict[str, Any], expected: dict[str, Any]) -> None:+ result = driver().run(name, [ScriptedResponse(json={"count": 1, "results": [row]})])assert result.raised is None- assert result.items == [[{"id": 1, "sources": [{"source": "origin.example.com", "enabled": True}]}]]+ assert result.items == [[expected]]def test_malformed_statistics_do_not_advance_checkpoint() -> None:@@ -191,3 +212,109 @@ def test_malformed_statistics_do_not_advance_checkpoint() -> None:assert "does not contain resource groups" in str(result.raised)assert result.saved_states == []assert result.final_state == resume+++def _window(request: RecordedRequest) -> tuple[datetime, datetime]:+ start, end = request.param("from"), request.param("to")+ assert start is not None and end is not None+ return datetime.fromisoformat(start), datetime.fromisoformat(end)+++ "mode,expected_windows",+ [+ ("full", [(datetime(2026, 1, 5, 0, tzinfo=UTC), 6), (datetime(2026, 1, 5, 6, tzinfo=UTC), 5)]),+ ("incremental", [(datetime(2026, 1, 5, 4, tzinfo=UTC), 6), (datetime(2026, 1, 5, 10, tzinfo=UTC), 1)]),+ ("resume", [(datetime(2026, 1, 5, 6, tzinfo=UTC), 5)]),+ ],+)+@patch(f"{TRANSPORT}.datetime", wraps=datetime)+def test_logs_windows_pages_and_resume(+ clock: MagicMock, mode: str, expected_windows: list[tuple[datetime, int]]+) -> None:+ clock.now.return_value = NOW++ def answer(request: RecordedRequest) -> ScriptedResponse:+ start, end = _window(request)+ first = int(start.timestamp())+ rows = [{"timestamp": first, "status": 200}, {"timestamp": first + 1, "status": 404}]+ rows.append({"timestamp": int(end.timestamp()), "status": 200})+ offset = int(request.param("offset") or 0)+ return ScriptedResponse(json={"data": rows[offset : offset + 2], "meta": {"count": len(rows)}})++ end = datetime(2026, 1, 5, 11, tzinfo=UTC)+ resume = (+ GcoreCheckpoint(offset=1, window_start=expected_windows[0][0].isoformat(), window_end=end.isoformat())+ if mode == "resume"+ else None+ )+ with patch(f"{TRANSPORT}.LOGS_PAGE_SIZE", 2), patch(f"{TRANSPORT}.LOGS_HISTORY_DAYS", 0):+ result = driver().run(+ "cdn_logs",+ answer,+ resume_state=resume,+ should_use_incremental_field=mode == "incremental",+ db_incremental_field_last_value=datetime(2026, 1, 5, 3, 59, 59, tzinfo=UTC),+ )++ assert result.raised is None+ assert sorted({_window(request) for request in result.requests}) == [+ (start, start + timedelta(hours=hours)) for start, hours in expected_windows+ ]+ assert result.params("offset") == (["1"] if mode == "resume" else ["0", "2"] * len(expected_windows))+ assert set(result.params("ordering")) == {"timestamp"}+ expected_rows = []+ for start, _ in expected_windows:+ if mode != "resume":+ expected_rows.append({"timestamp": start, "status": 200})+ expected_rows.append({"timestamp": start + timedelta(seconds=1), "status": 404})+ assert result.rows == expected_rows+ assert result.final_state == GcoreCheckpoint(window_start=end.isoformat(), window_end=end.isoformat())+++ "now,incremental,expected_days",+ [+ (NOW, False, [datetime(2026, 1, 3, tzinfo=UTC), datetime(2026, 1, 4, tzinfo=UTC)]),+ (NOW, True, [datetime(2026, 1, 4, tzinfo=UTC)]),+ (datetime(2026, 1, 5, 1, 30, tzinfo=UTC), False, [datetime(2026, 1, 3, tzinfo=UTC)]),+ ],+)+@patch(f"{TRANSPORT}.datetime", wraps=datetime)+def test_daily_statistics_rows_and_days(+ clock: MagicMock, now: datetime, incremental: bool, expected_days: list[datetime]+) -> None:+ clock.now.return_value = now+ body = {+ "resource": {+ "42": {+ "region": {+ "eu": {"metrics": {"requests": 10, "responses_5xx": 1, "95_percentile": 300}},+ "na": {"metrics": {"requests": 5}},+ }+ }+ }+ }++ def answer(request: RecordedRequest) -> ScriptedResponse:+ start, _ = _window(request)+ return ScriptedResponse(json=body) if start == expected_days[0] else ScriptedResponse(status=204)++ with patch(f"{TRANSPORT}.HISTORY_DAYS", 2):+ result = driver().run(+ "cdn_daily_statistics",+ answer,+ should_use_incremental_field=incremental,+ db_incremental_field_last_value=datetime(2026, 1, 4, 6, tzinfo=UTC),+ )++ assert result.raised is None+ assert [_window(request) for request in result.requests] == [+ (day, day + timedelta(days=1) - timedelta(seconds=1)) for day in expected_days+ ]+ assert all(query["group_by"] == ("resource", "region") for query in result.queries)+ eu, na = result.rows+ assert (eu["resource_id"], eu["region"], eu["date"]) == ("42", "eu", expected_days[0])+ assert (eu["requests"], eu["responses_5xx"], eu["bandwidth_95_percentile"]) == (10, 1, 300)+ assert (na["region"], na["requests"], na["responses_5xx"]) == ("na", 5, None)+ assert "95_percentile" not in eu