Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 3 missing tables to the Cisco Meraki source (#124451)

Tom Owersf8c0879d64c2
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 5e657c88ee4e..847b2db287d9 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -2561,10 +2561,10 @@ Diffed against: <https://raw.githubusercontent.com/meraki/openapi/master/openapi
- [x] `Network clients (GET /api/v1/networks/{networkId}/clients)` — measure active clients and join client activity with imported networks and devices (high)
- [x] `Network events (GET /api/v1/networks/{networkId}/events)` — analyze connectivity failures, authentication events, and operational history (high)
-- [ ] `Client usage histories (GET /api/v1/networks/{networkId}/clients/usageHistories)` — measure bandwidth consumption by client over time (high)
-- [ ] `Uplink loss and latency history (GET /api/v1/devices/{serial}/lossAndLatencyHistory)` — measure packet loss and latency beyond the current uplink status (high)
-- [ ] `Device statuses (GET /api/v1/organizations/{organizationId}/devices/statuses)` — monitor device availability and last contact across the imported inventory (medium)
-- [ ] `Uplink usage history (GET /api/v1/networks/{networkId}/appliance/uplinks/usageHistory)` — track sent and received traffic for each network uplink (medium)
+- [x] `Client usage histories (GET /api/v1/networks/{networkId}/clients/usageHistories)` — measure bandwidth consumption by client over time (high) → `client_usage_histories` (client keys come from the network clients listing, in batches of 100)
+- [ ] `Uplink loss and latency history (GET /api/v1/devices/{serial}/lossAndLatencyHistory)` — measure packet loss and latency beyond the current uplink status (high) — skipped: each request needs one device, one uplink, and a required destination IP, so a table needs a device × uplink × destination fan-out
+- [x] `Device statuses (GET /api/v1/organizations/{organizationId}/devices/statuses)` — monitor device availability and last contact across the imported inventory (medium) → `device_availabilities` (implemented against the non-deprecated `GET /organizations/{organizationId}/devices/availabilities`; the spec marks `devices/statuses` deprecated)
+- [x] `Uplink usage history (GET /api/v1/networks/{networkId}/appliance/uplinks/usageHistory)` — track sent and received traffic for each network uplink (medium) → `uplink_usage_history`
- [ ] `Network traffic (GET /api/v1/networks/{networkId}/traffic)` — identify applications that drive network traffic and capacity requirements (medium)
- [ ] `Wireless client connection statistics (GET /api/v1/networks/{networkId}/wireless/clients/connectionStats)` — compare connection failures and wireless reliability across clients (medium)
- [ ] `Organization security events (GET /api/v1/organizations/{organizationId}/appliance/security/events)` — measure security threats and affected networks over time (medium)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/canonical_descriptions.py
index 7bdea351dd69..6113ea150d06 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/canonical_descriptions.py
@@ -100,4 +100,40 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"eventData": "Additional data about the event.",
},
},
+ "device_availabilities": {
+ "description": "Availability status of each device in the selected Meraki organization. Meraki updates this data every 5 minutes.",
+ "docs_url": "https://developer.cisco.com/meraki/api-v1/get-organization-devices-availabilities/",
+ "columns": {
+ "serial": "Serial number that identifies the device.",
+ "mac": "MAC address of the device.",
+ "name": "Name of the device.",
+ "network": "Network that contains the device.",
+ "productType": "Product type of the device.",
+ "status": "Status of the device: online, alerting, offline, or dormant.",
+ "tags": "Tags assigned to the device.",
+ },
+ },
+ "uplink_usage_history": {
+ "description": "Data sent and received by each uplink of the security appliance networks in the selected Meraki organization, for each hour.",
+ "docs_url": "https://developer.cisco.com/meraki/api-v1/get-network-appliance-uplinks-usage-history/",
+ "columns": {
+ "networkId": "Identifier of the network.",
+ "startTime": "Start time of the interval.",
+ "endTime": "End time of the interval.",
+ "byInterface": "Data sent and received by each uplink interface in the interval, in bytes.",
+ },
+ },
+ "client_usage_histories": {
+ "description": "Data sent and received by each client of each network in the selected Meraki organization, for each day.",
+ "docs_url": "https://developer.cisco.com/meraki/api-v1/get-network-clients-usage-histories/",
+ "columns": {
+ "networkId": "Identifier of the network that the client used.",
+ "clientId": "Identifier of the client. It is unique within the network.",
+ "clientIp": "IP address of the client.",
+ "clientMac": "MAC address of the client.",
+ "ts": "Start of the day that the usage covers.",
+ "sent": "Data that the client sent on the day, in kilobytes.",
+ "received": "Data that the client received on the day, in kilobytes.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/cisco_meraki.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/cisco_meraki.py
index 65b31e595bc4..c39872f10a48 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/cisco_meraki.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/cisco_meraki.py
@@ -1,7 +1,9 @@
import re
import json
+import math
import hashlib
-from datetime import datetime
+from collections.abc import Iterator
+from datetime import UTC, datetime
from typing import TYPE_CHECKING, Any, cast
from urllib.parse import parse_qs, urlsplit
@@ -11,6 +13,9 @@ from posthog.dataclasses import frozen
from products.warehouse_sources.backend.temporal.data_imports.sources.cisco_meraki.settings import (
AUTH_ERROR,
+ CLIENT_USAGE_BATCH_SIZE,
+ CLIENT_USAGE_MIN_TIMESPAN_SECONDS,
+ CLIENTS_TIMESPAN_SECONDS,
ENDPOINTS,
EVENT_LOG_FLOOR,
EVENT_PRODUCT_TYPES,
@@ -19,6 +24,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.cisco_mera
PERMISSION_ERROR,
REGION_ERROR,
REGION_HOSTS,
+ UPLINK_USAGE_MAX_TIMESPAN_SECONDS,
+ UPLINK_USAGE_MIN_TIMESPAN_SECONDS,
MerakiEndpoint,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.datetime_utils import parse_datetime_value
@@ -189,6 +196,19 @@ def _network_product_types(network: dict[str, Any]) -> list[dict[str, Any]]:
]
+def _appliance_networks(network: dict[str, Any]) -> list[dict[str, Any]]:
+ # Only security appliance networks have uplink usage. Other networks reject the request.
+ return [network] if "appliance" in (network.get("productTypes") or []) else []
+
+
+def _timespan_since(last_value: Any, minimum: int, maximum: int) -> int:
+ since = parse_datetime_value(last_value) if last_value is not None else None
+ if since is None:
+ return maximum
+ elapsed = math.ceil((datetime.now(UTC) - since).total_seconds())
+ return max(minimum, min(maximum, elapsed))
+
+
def _with_event_id(event: dict[str, Any]) -> dict[str, Any]:
content = json.dumps(event, sort_keys=True, separators=(",", ":"), default=str)
return {"id": hashlib.sha256(content.encode()).hexdigest(), **event}
@@ -223,11 +243,20 @@ def _network_source(
)
is_event_log = endpoint.name == "network_events"
- incremental = is_event_log and should_use_incremental_field
- stop_before = parse_datetime_value(db_incremental_field_last_value) if incremental else None
+ is_uplink_usage = endpoint.name == "uplink_usage_history"
+ incremental = bool(endpoint.incremental_fields) and should_use_incremental_field
+ last_value = db_incremental_field_last_value if incremental else None
child_endpoint_extra: dict[str, Any] = {"data_selector": endpoint.data_selector}
if is_event_log:
- child_endpoint_extra["paginator"] = _EventLogPaginator(stop_before)
+ child_endpoint_extra["paginator"] = _EventLogPaginator(parse_datetime_value(last_value))
+ if not endpoint.paginated:
+ child_endpoint_extra["paginator"] = "single_page"
+ child_params_extra = (
+ {"timespan": _timespan_since(last_value, UPLINK_USAGE_MIN_TIMESPAN_SECONDS, UPLINK_USAGE_MAX_TIMESPAN_SECONDS)}
+ if is_uplink_usage
+ else None
+ )
+ parent_data_map = _network_product_types if is_event_log else _appliance_networks if is_uplink_usage else None
resource = cast(
Resource,
@@ -241,12 +270,13 @@ def _network_source(
job_id=job_id,
db_incremental_field_last_value=None,
should_use_incremental_field=incremental,
- # The event log has no time filter, so the paginator bounds an incremental read instead.
+ # The event log paginator and the uplink usage timespan bound an incremental read instead.
incremental_config_factory=lambda _field: None,
parent_endpoint_extra={"data_selector": "$"},
child_endpoint_extra=cast(Any, child_endpoint_extra),
- parent_data_map=_network_product_types if is_event_log else None,
- page_size_param="perPage",
+ child_params_extra=child_params_extra,
+ parent_data_map=parent_data_map,
+ page_size_param="perPage" if endpoint.paginated else None,
resume_hook=save_checkpoint,
initial_paginator_state=initial_state,
),
@@ -260,10 +290,111 @@ def _network_source(
primary_keys=list(endpoint.primary_keys),
on_complete=manager.clear_state,
# Each network's event log arrives newest first, so the watermark advances only when the sync completes.
- sort_mode="desc" if is_event_log else "asc",
- partition_mode="datetime" if is_event_log else None,
- partition_format="month" if is_event_log else None,
- partition_keys=["occurredAt"] if is_event_log else None,
+ # Uplink usage arrives as one series for each network, so it has no order across the table.
+ sort_mode="desc" if is_event_log else None if is_uplink_usage else "asc",
+ **_partitioning(endpoint),
+ )
+
+
+def _partitioning(endpoint: MerakiEndpoint) -> dict[str, Any]:
+ if endpoint.partition_key is None:
+ return {}
+ return {"partition_mode": "datetime", "partition_format": "month", "partition_keys": [endpoint.partition_key]}
+
+
+def _usage_history_rows(network_id: str, histories: list[Any]) -> list[dict[str, Any]]:
+ return [
+ {
+ "networkId": network_id,
+ "clientId": history.get("clientId"),
+ "clientIp": history.get("clientIp"),
+ "clientMac": history.get("clientMac"),
+ **usage,
+ }
+ for history in histories
+ if isinstance(history, dict)
+ for usage in history.get("usageHistory") or []
+ if isinstance(usage, dict)
+ ]
+
+
+def _client_usage_pages(
+ config: "CiscoMerakiSourceConfig",
+ endpoint: MerakiEndpoint,
+ api_version: str,
+ manager: "ResumableSourceManager[CiscoMerakiResumeConfig]",
+ timespan: int,
+) -> Iterator[list[dict[str, Any]]]:
+ """Read usage histories for the clients of each network, a batch of client keys at a time.
+
+ The endpoint returns usage only for the clients that a request names, so the network clients
+ listing supplies the keys. The resume state is the list of networks that are complete.
+ """
+ settings = client_config(config, api_version)
+ client = RESTClient(
+ base_url=settings["base_url"],
+ auth=APIKeyAuth(api_key=config.api_key, name="X-Cisco-Meraki-API-Key"),
+ paginator=HeaderLinkPaginator(),
+ session=settings["session"],
+ request_timeout=settings["request_timeout"],
+ )
+ resume = manager.load_state() if manager.can_resume() else None
+ completed = list(resume.completed_paths or []) if resume is not None else []
+ networks = ENDPOINTS["networks"]
+
+ for network_page in client.paginate(
+ _organization_path(networks.path, config), params={"perPage": networks.page_size}
+ ):
+ manager.safe_point()
+ for network in network_page:
+ network_id = str(network["id"])
+ path = endpoint.path.replace("{networkId}", network_id)
+ if path in completed:
+ continue
+ manager.save_state(CiscoMerakiResumeConfig(completed_paths=list(completed)))
+ manager.safe_point()
+ for client_page in client.paginate(
+ f"networks/{network_id}/clients", params={"perPage": endpoint.page_size, "timespan": timespan}
+ ):
+ keys = [str(row["id"]) for row in client_page if isinstance(row, dict) and row.get("id")]
+ if not keys:
+ manager.safe_point()
+ for start in range(0, len(keys), CLIENT_USAGE_BATCH_SIZE):
+ params = {
+ "clients": ",".join(keys[start : start + CLIENT_USAGE_BATCH_SIZE]),
+ "perPage": endpoint.page_size,
+ "timespan": timespan,
+ }
+ for history_page in client.paginate(path, params=params):
+ rows = _usage_history_rows(network_id, history_page)
+ if rows:
+ yield rows
+ else:
+ manager.safe_point()
+ completed.append(path)
+
+
+def _client_usage_source(
+ config: "CiscoMerakiSourceConfig",
+ endpoint: MerakiEndpoint,
+ api_version: str,
+ manager: "ResumableSourceManager[CiscoMerakiResumeConfig]",
+ should_use_incremental_field: bool,
+ db_incremental_field_last_value: Any,
+) -> SourceResponse:
+ timespan = _timespan_since(
+ db_incremental_field_last_value if should_use_incremental_field else None,
+ CLIENT_USAGE_MIN_TIMESPAN_SECONDS,
+ CLIENTS_TIMESPAN_SECONDS,
+ )
+ return SourceResponse(
+ name=endpoint.name,
+ items=lambda: _client_usage_pages(config, endpoint, api_version, manager, timespan),
+ primary_keys=list(endpoint.primary_keys),
+ on_complete=manager.clear_state,
+ # Usage arrives as one series for each batch of clients, so it has no order across the table.
+ sort_mode=None,
+ **_partitioning(endpoint),
)
@@ -278,6 +409,10 @@ def cisco_meraki_source(
db_incremental_field_last_value: Any = None,
) -> SourceResponse:
endpoint = schema_for_resource(ENDPOINTS, endpoint_name)
+ if endpoint.name == "client_usage_histories":
+ return _client_usage_source(
+ config, endpoint, api_version, manager, should_use_incremental_field, db_incremental_field_last_value
+ )
if endpoint.fanout is not None:
return _network_source(
config,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/settings.py
index 6a5000cfc0ca..5983b59fd4b5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/settings.py
@@ -19,6 +19,19 @@ REGION_HOSTS = {
# The maximum lookback that the network clients endpoint accepts (31 days).
CLIENTS_TIMESPAN_SECONDS = 31 * 24 * 60 * 60
+# Client usage histories hold one row per client per day. An incremental read always covers at least one day, so
+# it reads the current day again.
+CLIENT_USAGE_MIN_TIMESPAN_SECONDS = 24 * 60 * 60
+
+# The uplink usage history endpoint accepts a lookback of 30 days at most.
+UPLINK_USAGE_MAX_TIMESPAN_SECONDS = 30 * 24 * 60 * 60
+UPLINK_USAGE_RESOLUTION_SECONDS = 60 * 60
+# An incremental read covers at least two intervals, so it reads the current partial hour again.
+UPLINK_USAGE_MIN_TIMESPAN_SECONDS = 2 * UPLINK_USAGE_RESOLUTION_SECONDS
+
+# Client keys go into the query string, so a request names a bounded number of clients.
+CLIENT_USAGE_BATCH_SIZE = 100
+
# Product types that the network events endpoint accepts. A network can contain others, such as sensor.
EVENT_PRODUCT_TYPES = frozenset(
{
@@ -51,6 +64,17 @@ class MerakiEndpoint:
fanout: DependentEndpointConfig | None = None
incremental_fields: list[IncrementalField] = field(default_factory=list)
default_incremental_field: str | None = None
+ paginated: bool = True
+ partition_key: str | None = None
+
+
+def _datetime_field(name: str) -> IncrementalField:
+ return {
+ "label": name,
+ "type": IncrementalFieldType.DateTime,
+ "field": name,
+ "field_type": IncrementalFieldType.DateTime,
+ }
_NETWORK_FANOUT_PARENT = "networks"
@@ -74,6 +98,11 @@ ENDPOINTS = {
primary_keys=("id",),
page_size=300,
),
+ MerakiEndpoint(
+ name="device_availabilities",
+ path="organizations/{organization_id}/devices/availabilities",
+ primary_keys=("serial",),
+ ),
MerakiEndpoint(
name="network_clients",
path="networks/{networkId}/clients",
@@ -105,15 +134,43 @@ ENDPOINTS = {
"productType": {"type": "resolve", "resource": _NETWORK_FANOUT_PARENT, "field": "productType"}
},
),
- incremental_fields=[
- {
- "label": "occurredAt",
- "type": IncrementalFieldType.DateTime,
- "field": "occurredAt",
- "field_type": IncrementalFieldType.DateTime,
- }
- ],
+ incremental_fields=[_datetime_field("occurredAt")],
default_incremental_field="occurredAt",
+ partition_key="occurredAt",
+ ),
+ MerakiEndpoint(
+ name="uplink_usage_history",
+ path="networks/{networkId}/appliance/uplinks/usageHistory",
+ primary_keys=("networkId", "startTime"),
+ paginated=False,
+ fanout=DependentEndpointConfig(
+ parent_name=_NETWORK_FANOUT_PARENT,
+ resolve_param="networkId",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "networkId"},
+ parent_params={"perPage": 1000},
+ child_params={"resolution": UPLINK_USAGE_RESOLUTION_SECONDS},
+ ),
+ incremental_fields=[_datetime_field("startTime")],
+ default_incremental_field="startTime",
+ partition_key="startTime",
+ ),
+ MerakiEndpoint(
+ name="client_usage_histories",
+ path="networks/{networkId}/clients/usageHistories",
+ primary_keys=("networkId", "clientId", "ts"),
+ # Each request names a batch of clients from the network clients table, so the source reads it itself.
+ fanout=DependentEndpointConfig(
+ parent_name=_NETWORK_FANOUT_PARENT,
+ resolve_param="networkId",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "networkId"},
+ ),
+ incremental_fields=[_datetime_field("ts")],
+ default_incremental_field="ts",
+ partition_key="ts",
),
)
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/source.py
index 7d2ea7213c47..bed2344168a3 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/source.py
@@ -136,8 +136,10 @@ class CiscoMerakiSource(ResumableSource[CiscoMerakiSourceConfig, CiscoMerakiResu
caption=(
"In Meraki Dashboard, open **Organization > API & Webhooks > API keys and access**. Create an API key. "
"Enable API access for your organization. Use an administrator account with read access to the selected tables. "
- "Each connection imports one organization. Network events support incremental sync, and other tables use full refresh. "
- "Network clients include clients seen in the last 31 days. Health alerts include active alerts only."
+ "Each connection imports one organization. Network events, uplink usage history, and client usage histories support incremental sync. "
+ "Other tables use full refresh. Network clients include clients seen in the last 31 days. "
+ "The first sync of client usage histories reads the last 31 days, and the first sync of uplink usage history reads the last 30 days. "
+ "Health alerts include active alerts only."
),
fields=cast(
list[FieldType],
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/tests/test_cisco_meraki.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/tests/test_cisco_meraki.py
index 03401131cbb6..8655e89bf4c0 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/tests/test_cisco_meraki.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cisco_meraki/tests/test_cisco_meraki.py
@@ -4,6 +4,7 @@ from typing import Any, cast
from urllib.parse import parse_qs, urlsplit
import pytest
+import time_machine
from unittest.mock import MagicMock
import responses
@@ -72,6 +73,7 @@ def rows(response: SourceResponse) -> list[dict[str, Any]]:
("inventory_devices", "inventory/devices", "serial", 1000),
("uplink_statuses", "uplinks/statuses", "serial", 1000),
("assurance_alerts", "assurance/alerts", "id", 300),
+ ("device_availabilities", "devices/availabilities", "serial", 1000),
],
)
@responses.activate
@@ -390,3 +392,215 @@ def test_network_events_walk_back_per_product_type(
assert {request.param("productType") for request in result.requests[1:]} == {"appliance", "wireless"}
assert {row["productType"] for row in result.rows} == {"appliance"}
assert len({row["id"] for row in result.rows}) == len(result.rows) == 4
+
+
+NOW = datetime(2026, 3, 1, 12, 0, tzinfo=UTC)
+NETWORKS_PATH = "/api/v1/organizations/123456/networks"
+
+
+def _usage_history(client_id: str, ts: str) -> dict[str, Any]:
+ return {
+ "clientId": client_id,
+ "clientIp": "192.0.2.10",
+ "clientMac": "00:00:5e:00:53:01",
+ "usageHistory": [{"ts": ts, "sent": 10.0, "received": 20.0}],
+ }
+
+
+@time_machine.travel(NOW, tick=False)
+ ("endpoint", "child_path", "watermark", "expected_timespan"),
+ [
+ ("uplink_usage_history", "/api/v1/networks/N_1/appliance/uplinks/usageHistory", None, "2592000"),
+ (
+ "uplink_usage_history",
+ "/api/v1/networks/N_1/appliance/uplinks/usageHistory",
+ datetime(2026, 2, 26, 12, 0, tzinfo=UTC),
+ "259200",
+ ),
+ (
+ "uplink_usage_history",
+ "/api/v1/networks/N_1/appliance/uplinks/usageHistory",
+ datetime(2026, 3, 1, 11, 50, tzinfo=UTC),
+ "7200",
+ ),
+ (
+ "uplink_usage_history",
+ "/api/v1/networks/N_1/appliance/uplinks/usageHistory",
+ "2025-12-01T00:00:00Z",
+ "2592000",
+ ),
+ ("client_usage_histories", "/api/v1/networks/N_1/clients/usageHistories", None, "2678400"),
+ (
+ "client_usage_histories",
+ "/api/v1/networks/N_1/clients/usageHistories",
+ datetime(2026, 2, 26, 12, 0, tzinfo=UTC),
+ "259200",
+ ),
+ (
+ "client_usage_histories",
+ "/api/v1/networks/N_1/clients/usageHistories",
+ "2026-03-01T00:00:00Z",
+ "86400",
+ ),
+ (
+ "client_usage_histories",
+ "/api/v1/networks/N_1/clients/usageHistories",
+ datetime(2025, 12, 1, tzinfo=UTC),
+ "2678400",
+ ),
+ ],
+)
+def test_usage_history_timespan_follows_watermark(
+ endpoint: str, child_path: str, watermark: datetime | str | None, expected_timespan: str
+) -> None:
+ result = make_driver().run(
+ endpoint,
+ network_api(
+ {
+ (NETWORKS_PATH, None): ScriptedResponse(json=[{"id": "N_1", "productTypes": ["appliance"]}]),
+ ("/api/v1/networks/N_1/clients", None): ScriptedResponse(json=[{"id": "k1"}]),
+ (child_path, None): ScriptedResponse(json=[]),
+ }
+ ),
+ incremental_field="startTime" if endpoint == "uplink_usage_history" else "ts",
+ should_use_incremental_field=watermark is not None,
+ db_incremental_field_last_value=watermark,
+ )
+
+ assert result.raised is None
+ timespans = {request.path: request.param("timespan") for request in result.requests[1:]}
+ assert timespans[child_path] == expected_timespan
+ if endpoint == "client_usage_histories":
+ assert timespans["/api/v1/networks/N_1/clients"] == expected_timespan
+
+
+def test_full_refresh_ignores_stored_watermark() -> None:
+ result = make_driver().run(
+ "uplink_usage_history",
+ network_api(
+ {
+ (NETWORKS_PATH, None): ScriptedResponse(json=[{"id": "N_1", "productTypes": ["appliance"]}]),
+ ("/api/v1/networks/N_1/appliance/uplinks/usageHistory", None): ScriptedResponse(json=[]),
+ }
+ ),
+ incremental_field="startTime",
+ should_use_incremental_field=False,
+ db_incremental_field_last_value=datetime.now(UTC),
+ )
+
+ assert result.raised is None
+ assert result.requests[-1].param("timespan") == "2592000"
+
+
+def test_uplink_usage_history_reads_only_appliance_networks() -> None:
+ interval = {
+ "startTime": "2026-01-01T00:00:00Z",
+ "endTime": "2026-01-01T01:00:00Z",
+ "byInterface": [{"interface": "wan1", "sent": 100, "received": 200}],
+ }
+ result = make_driver().run(
+ "uplink_usage_history",
+ network_api(
+ {
+ (NETWORKS_PATH, None): ScriptedResponse(
+ json=[
+ {"id": "N_1", "productTypes": ["appliance", "switch"]},
+ {"id": "N_2", "productTypes": ["wireless"]},
+ {"id": "N_3", "productTypes": []},
+ ]
+ ),
+ # A Link header on this endpoint must not start a second request.
+ ("/api/v1/networks/N_1/appliance/uplinks/usageHistory", None): ScriptedResponse(
+ json=[interval],
+ headers={"Link": f'<{API_URL}/networks/N_1/appliance/uplinks/usageHistory?page=2>; rel="next"'},
+ ),
+ }
+ ),
+ )
+
+ assert result.raised is None
+ assert result.rows == [{**interval, "networkId": "N_1"}]
+ assert [request.path for request in result.requests] == [
+ NETWORKS_PATH,
+ "/api/v1/networks/N_1/appliance/uplinks/usageHistory",
+ ]
+ assert result.requests[1].query == {"timespan": ("2592000",), "resolution": ("3600",)}
+ assert result.response is not None
+ assert result.response.sort_mode is None
+ assert result.response.partition_keys == ["startTime"]
+
+
+def test_client_usage_histories_batch_client_keys_for_each_network() -> None:
+ usage_path = "/api/v1/networks/N_1/clients/usageHistories"
+ next_clients = f"{API_URL}/networks/N_1/clients?perPage=1000&timespan=2678400&startingAfter=k120"
+
+ def answer(request: RecordedRequest) -> ScriptedResponse:
+ if request.path == usage_path:
+ keys = (request.param("clients") or "").split(",")
+ return ScriptedResponse(json=[_usage_history(key, "2026-01-01T00:00:00Z") for key in keys[:1]])
+ return network_api(
+ {
+ (NETWORKS_PATH, None): ScriptedResponse(
+ json=[{"id": "N_1"}, {"id": "N_2"}, {"id": "N_3"}],
+ headers={"Link": f'<{API_URL}/organizations/123456/networks?startingAfter=N_3>; rel="next"'},
+ ),
+ (NETWORKS_PATH, "N_3"): ScriptedResponse(json=[]),
+ ("/api/v1/networks/N_1/clients", None): ScriptedResponse(
+ json=[{"id": f"k{index}"} for index in range(120)],
+ headers={"Link": f'<{next_clients}>; rel="next"'},
+ ),
+ ("/api/v1/networks/N_1/clients", "k120"): ScriptedResponse(json=[{"id": "k120"}]),
+ ("/api/v1/networks/N_2/clients", None): ScriptedResponse(json=[]),
+ ("/api/v1/networks/N_3/clients", None): ScriptedResponse(json=[{"id": "k1"}]),
+ ("/api/v1/networks/N_3/clients/usageHistories", None): ScriptedResponse(
+ json=[{"clientId": "k1", "usageHistory": []}]
+ ),
+ }
+ )(request)
+
+ result = make_driver().run("client_usage_histories", answer)
+
+ assert result.raised is None
+ usage_requests = [request for request in result.requests if request.path.endswith("/usageHistories")]
+ assert [len((request.param("clients") or "").split(",")) for request in usage_requests] == [100, 20, 1, 1]
+ assert [request.param("clients") for request in usage_requests][2:] == ["k120", "k1"]
+ assert [(row["networkId"], row["clientId"], row["ts"]) for row in result.rows] == [
+ ("N_1", "k0", "2026-01-01T00:00:00Z"),
+ ("N_1", "k100", "2026-01-01T00:00:00Z"),
+ ("N_1", "k120", "2026-01-01T00:00:00Z"),
+ ]
+ assert result.rows[0] == {
+ "networkId": "N_1",
+ "clientId": "k0",
+ "clientIp": "192.0.2.10",
+ "clientMac": "00:00:5e:00:53:01",
+ "ts": "2026-01-01T00:00:00Z",
+ "sent": 10.0,
+ "received": 20.0,
+ }
+ assert result.completed_state is None
+
+
+def test_client_usage_histories_resume_skips_completed_networks() -> None:
+ result = make_driver().run(
+ "client_usage_histories",
+ network_api(
+ {
+ (NETWORKS_PATH, None): ScriptedResponse(json=[{"id": "N_1"}, {"id": "N_2"}, {"id": "N_3"}]),
+ ("/api/v1/networks/N_2/clients", None): ScriptedResponse(json=[{"id": "k9"}]),
+ ("/api/v1/networks/N_2/clients/usageHistories", None): ScriptedResponse(
+ json=[_usage_history("k9", "2026-01-02T00:00:00Z")]
+ ),
+ ("/api/v1/networks/N_3/clients", None): ScriptedResponse(json=[]),
+ }
+ ),
+ resume_state=CiscoMerakiResumeConfig(completed_paths=["networks/N_1/clients/usageHistories"]),
+ )
+
+ assert result.raised is None
+ assert [row["clientId"] for row in result.rows] == ["k9"]
+ assert not any(request.path.startswith("/api/v1/networks/N_1/") for request in result.requests)
+ assert result.final_state == CiscoMerakiResumeConfig(
+ completed_paths=["networks/N_1/clients/usageHistories", "networks/N_2/clients/usageHistories"]
+ )