Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 6 missing tables to the KongKonnect source (#124584)

Tom Owers0389346b71d3
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 456a3c9a3215..992b2aa781e5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -6988,13 +6988,13 @@ Diffed against: <https://raw.githubusercontent.com/Kong/developer.konghq.com/mai
- [x] `/control-planes/{id}/core-entities/routes` — lookup resolving route_id on api_requests to a path/method (high)
- [x] `/control-planes/{id}/core-entities/consumers` — lookup resolving consumer_id on api_requests to a named API consumer (high)
- [x] `/api-products (API Products v2)` — lookup for the api_product / api_product_version dimensions filterable on api_requests (high)
-- [ ] `/metrics (Analytics Metrics v2)` — vendor's aggregated traffic, latency and error-rate metrics without re-aggregating raw request rows (medium)
-- [ ] `/control-planes/{id}/core-entities/plugins` — which rate-limit/auth plugins were active on a service or route when traffic was served (medium)
+- [x] `/metrics (Analytics Metrics v2)` — vendor's aggregated traffic, latency and error-rate metrics without re-aggregating raw request rows (medium)
+- [x] `/control-planes/{id}/core-entities/plugins` — which rate-limit/auth plugins were active on a service or route when traffic was served (medium)
- [x] `/control-planes/{id}/core-entities/consumer_groups and /consumer_groups/{id}/consumers` — consumer-group membership junction for segmenting request volume by tier (medium)
- [x] `/realms/{realmId}/consumers (Konnect Consumers v1)` — identity-realm consumer registry backing the consumer dimension (medium)
- [x] `/catalog-services and /scorecards (Service Catalog v1)` — service inventory plus scorecard scores, the headline governance metric (medium)
-- [ ] `/control-planes/{id}/core-entities/upstreams and /targets` — upstream/target topology behind each service for latency attribution (low)
-- [ ] `/openmeter/meters and /openmeter/subscriptions (Metering & Billing v3)` — metered usage and subscription state for cost-per-request analysis (low)
+- [x] `/control-planes/{id}/core-entities/upstreams and /targets` — upstream/target topology behind each service for latency attribution (low)
+- [x] `/openmeter/meters and /openmeter/subscriptions (Metering & Billing v3)` — metered usage and subscription state for cost-per-request analysis (low)
Note: Kong publishes 25 separate Konnect OpenAPI specs under https://github.com/Kong/developer.konghq.com/tree/main/api-specs/konnect; PostHog implements one path (/api-requests) from analytics-requests/v2. Other specs diffed this run: control-planes/v2, control-planes-config/v2, analytics-metrics/v2, api-products/v2, consumers/v1, service-catalog/v1, audit-logs/v3, metering-and-billing/v3. The audit-logs v3 spec only exposes webhook/destination config, so there is no queryable audit-log table to add.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/canonical_descriptions.py
index fe582891a377..c39b6a7067ca 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/canonical_descriptions.py
@@ -51,6 +51,22 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"mcp_info": "Model Context Protocol call details for AI routes (nullable), including session id and per-tool call method, latency, and error.",
},
},
+ "daily_service_metrics": {
+ "description": "Aggregated gateway traffic from the Konnect Analytics Metrics API. One row per gateway service "
+ "per UTC day. The row for the current day is replaced on each sync until the day ends.",
+ "docs_url": "https://developer.konghq.com/api/konnect/analytics-metrics/v2/",
+ "columns": {
+ "day": "Start of the UTC day the metrics cover. Used as the incremental cursor and partition key.",
+ "gateway_service": "Gateway service the traffic was routed to, in the form control_plane_id:service_id. "
+ "Empty for traffic that matched no service. ____OTHER____ groups the services above the query limit.",
+ "request_count": "Number of requests the gateway service handled in the day.",
+ "error_rate": "Share of requests in the day that returned an error status code.",
+ "response_latency_average": "Average total time, in milliseconds, from the first byte of the request to the last byte of the response.",
+ "response_latency_p99": "99th percentile total response latency, in milliseconds.",
+ "upstream_latency_p99": "99th percentile time, in milliseconds, that the upstream service took to respond.",
+ "kong_latency_p99": "99th percentile time, in milliseconds, that Kong Gateway spent processing the request.",
+ },
+ },
"control_planes": {
"description": "Control planes in the Konnect organization. Joins to api_requests on control_plane.",
"docs_url": "https://developer.konghq.com/api/konnect/control-planes/v2/",
@@ -120,6 +136,62 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"updated_at": "Unix epoch (seconds) when the consumer was last updated.",
},
},
+ "plugins": {
+ "description": "Gateway plugins, one row per plugin per control plane. A plugin with no service, route, "
+ "or consumer applies globally.",
+ "docs_url": "https://developer.konghq.com/api/konnect/control-planes-config/v2/",
+ "columns": {
+ "control_plane_id": "ID of the control plane the plugin belongs to. Plugin IDs are unique only within a control plane.",
+ "id": "Unique identifier of the plugin within its control plane.",
+ "name": "The name of the plugin, such as rate-limiting or key-auth.",
+ "instance_name": "A unique name for this plugin instance.",
+ "enabled": "Whether the plugin is applied.",
+ "service": "If set, the plugin only runs for requests to this service, as an object with the service id.",
+ "route": "If set, the plugin only runs for requests matched by this route, as an object with the route id.",
+ "consumer": "If set, the plugin only runs for requests authenticated as this consumer, as an object with the consumer id.",
+ "consumer_group": "If set, the plugin only runs for requests from consumers in this group.",
+ "protocols": "The request protocols that trigger the plugin.",
+ "condition": "An expression that decides during the request flow whether the plugin runs.",
+ "tags": "Optional set of strings associated with the plugin for grouping and filtering.",
+ "created_at": "Unix epoch (seconds) when the plugin was created.",
+ "updated_at": "Unix epoch (seconds) when the plugin was last updated.",
+ },
+ },
+ "upstreams": {
+ "description": "Gateway upstreams, one row per upstream per control plane. An upstream is a virtual hostname "
+ "that load balances requests over its targets.",
+ "docs_url": "https://developer.konghq.com/api/konnect/control-planes-config/v2/",
+ "columns": {
+ "control_plane_id": "ID of the control plane the upstream belongs to. Upstream IDs are unique only within a control plane.",
+ "id": "Unique identifier of the upstream within its control plane.",
+ "name": "Hostname of the upstream. It must be equal to the host of a service.",
+ "algorithm": "Load balancing algorithm, such as round-robin or least-connections.",
+ "hash_on": "What to use as hashing input. none gives weighted round-robin with no hashing.",
+ "hash_fallback": "Hashing input to use when hash_on returns no hash.",
+ "healthchecks": "Active and passive health check settings that apply to all targets of the upstream.",
+ "host_header": "Host header to send when proxying requests through Kong.",
+ "slots": "Number of slots in the load balancer algorithm.",
+ "tags": "Optional set of strings associated with the upstream for grouping and filtering.",
+ "created_at": "Unix epoch (seconds) when the upstream was created.",
+ "updated_at": "Unix epoch (seconds) when the upstream was last updated.",
+ },
+ },
+ "targets": {
+ "description": "Upstream targets, one row per target per control plane. A target is a backend host and port "
+ "that an upstream load balances over.",
+ "docs_url": "https://developer.konghq.com/api/konnect/control-planes-config/v2/",
+ "columns": {
+ "control_plane_id": "ID of the control plane the target belongs to. Target IDs are unique only within a control plane.",
+ "id": "Unique identifier of the target within its control plane.",
+ "upstream": "The upstream the target belongs to, as an object with the upstream id.",
+ "target": "The target address (IP or hostname) and port.",
+ "weight": "Weight of the target in the upstream load balancer (0-65535). A weight of 0 disables the target.",
+ "failover": "Whether the target is used only as a backup.",
+ "tags": "Optional set of strings associated with the target for grouping and filtering.",
+ "created_at": "Unix epoch (seconds) when the target was created.",
+ "updated_at": "Unix epoch (seconds) when the target was last updated.",
+ },
+ },
"consumer_groups": {
"description": "Gateway consumer groups, one row per group per control plane.",
"docs_url": "https://developer.konghq.com/api/konnect/control-planes-config/v2/",
@@ -253,4 +325,47 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"updated_at": "Timestamp of when the consumer was last updated.",
},
},
+ "meters": {
+ "description": "Metering & Billing meters. A meter defines which usage events to match and how to aggregate them.",
+ "docs_url": "https://developer.konghq.com/api/konnect/metering-and-billing/v3/",
+ "columns": {
+ "id": "The meter ID (ULID).",
+ "key": "Unique key that identifies the meter.",
+ "name": "Display name of the meter.",
+ "description": "Optional description of the meter.",
+ "aggregation": "How matching events are aggregated: sum, count, unique_count, avg, min, max, or latest.",
+ "event_type": "The event type the meter includes in the aggregation.",
+ "events_from": "Date from which the meter includes events. All historical events are included when not set.",
+ "value_property": "JSONPath expression that extracts the aggregated value from the event data.",
+ "dimensions": "Named JSONPath expressions that extract group-by values from the event data.",
+ "labels": "Key-value metadata for filtering and search.",
+ "created_at": "Timestamp of when the meter was created.",
+ "updated_at": "Timestamp of when the meter was last updated.",
+ "deleted_at": "Timestamp of when the meter was deleted.",
+ },
+ },
+ "subscriptions": {
+ "description": "Metering & Billing customer subscriptions, with their plan, billing cadence, and phases.",
+ "docs_url": "https://developer.konghq.com/api/konnect/metering-and-billing/v3/",
+ "columns": {
+ "id": "The subscription ID (ULID).",
+ "name": "Display name of the subscription. Defaults to the plan name.",
+ "description": "Optional description of the subscription.",
+ "customer_id": "ID of the customer the subscription belongs to.",
+ "plan": "The plan the subscription was created from, with its key and version.",
+ "status": "The status of the subscription.",
+ "active_from": "Timestamp of when the subscription became, or will become, active.",
+ "active_to": "Timestamp of when the subscription stops being active. Open-ended when not set.",
+ "billing_cadence": "How often the customer is billed, as an ISO-8601 duration such as P1M.",
+ "billing_anchor": "Fixed point in time that sets the recurring billing cycle.",
+ "invoice_currency": "The currency the subscription is invoiced in.",
+ "settlement_mode": "How usage is settled: credits first with the remainder invoiced, or credits only.",
+ "current_period": "The current billing period. Present only when the subscription is active and aligned.",
+ "phases": "The phases of the subscription in chronological order, each with the rate cards in effect.",
+ "labels": "Key-value metadata for filtering and search.",
+ "created_at": "Timestamp of when the subscription was created.",
+ "updated_at": "Timestamp of when the subscription was last updated.",
+ "deleted_at": "Timestamp of when the subscription was deleted.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/kong_konnect.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/kong_konnect.py
index f0e4d15ea962..8074bd3554d3 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/kong_konnect.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/kong_konnect.py
@@ -21,7 +21,10 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.kong_konne
KONG_KONNECT_ENDPOINTS,
LIST_PAGE_SIZE,
MAX_PAGE_SIZE,
+ METRICS_MAX_LOOKBACK_DAYS,
+ METRICS_QUERY_LIMIT,
REGION_BASE_URLS,
+ SERVICE_METRICS_GROUPS,
KongKonnectEndpointConfig,
)
@@ -257,6 +260,19 @@ def _iter_offset(session: requests.Session, url: str, logger: FilteringBoundLogg
break
+def _without_fields(row: dict[str, Any], fields: tuple[str, ...]) -> dict[str, Any]:
+ """Drop each field, given as a dotted path into nested objects, without mutating `row`."""
+ for field_path in fields:
+ key, _, rest = field_path.partition(".")
+ if key not in row:
+ continue
+ if not rest:
+ row = {k: v for k, v in row.items() if k != key}
+ elif isinstance(row[key], dict):
+ row = {**row, key: _without_fields(row[key], (rest,))}
+ return row
+
+
def _iter_core_entities(
session: requests.Session,
base_url: str,
@@ -333,14 +349,20 @@ def get_lookup_rows(
resumable_source_manager: ResumableSourceManager[KongKonnectResumeConfig],
) -> Iterator[list[dict[str, Any]]]:
endpoint_config = KONG_KONNECT_ENDPOINTS[endpoint]
- session = make_tracked_session(headers=_get_headers(api_token), redact_values=(api_token,))
+ session = make_tracked_session(
+ headers=_get_headers(api_token),
+ redact_values=(api_token,),
+ # Excluded fields hold credentials, which HTTP samples would capture before the rows are cleaned.
+ capture=not endpoint_config.excluded_fields,
+ )
if endpoint_config.kind == "core_entity":
- yield from _iter_core_entities(
- session, _get_base_url(region), endpoint_config, logger, resumable_source_manager
- )
+ pages = _iter_core_entities(session, _get_base_url(region), endpoint_config, logger, resumable_source_manager)
else:
- yield from _iter_top_level(session, region, endpoint_config, logger, resumable_source_manager)
+ pages = _iter_top_level(session, region, endpoint_config, logger, resumable_source_manager)
+
+ for rows in pages:
+ yield [_without_fields(row, endpoint_config.excluded_fields) for row in rows]
def validate_credentials(api_token: str, region: str) -> bool:
@@ -399,6 +421,83 @@ def get_rows(
resumable_source_manager.save_state(KongKonnectResumeConfig(start=start, end=end, offset=offset))
+def _to_day(value: Any) -> date:
+ if isinstance(value, datetime):
+ return (value.astimezone(UTC) if value.tzinfo is not None else value).date()
+ if isinstance(value, date):
+ return value
+ return datetime.fromisoformat(str(value).replace("Z", "+00:00")).astimezone(UTC).date()
+
+
+def _build_metrics_body(metrics: tuple[str, ...], start: datetime, end: datetime) -> dict[str, Any]:
+ return {
+ "datasource": "api_usage",
+ "metrics": list(metrics),
+ "dimensions": ["gateway_service"],
+ "time_range": {
+ "type": "absolute",
+ "start": _format_datetime(start),
+ "end": _format_datetime(end),
+ "tz": "Etc/UTC",
+ },
+ "limit": METRICS_QUERY_LIMIT,
+ }
+
+
+def get_daily_service_metrics(
+ api_token: str,
+ region: str,
+ logger: FilteringBoundLogger,
+ resumable_source_manager: ResumableSourceManager[KongKonnectResumeConfig],
+ should_use_incremental_field: bool = False,
+ db_incremental_field_last_value: Any = None,
+ lookback_days: int = DEFAULT_INITIAL_LOOKBACK_DAYS,
+) -> Iterator[list[dict[str, Any]]]:
+ url = f"{_get_base_url(region)}{KONG_KONNECT_ENDPOINTS['daily_service_metrics'].path}"
+ headers = _get_headers(api_token)
+ session = make_tracked_session()
+ now = datetime.now(UTC)
+
+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
+ if resume is not None and resume.start:
+ day = _to_day(resume.start)
+ elif should_use_incremental_field and db_incremental_field_last_value:
+ day = _to_day(_clamp_future_value_to_now(db_incremental_field_last_value))
+ else:
+ day = (now - timedelta(days=lookback_days)).date()
+ day = max(day, (now - timedelta(days=METRICS_MAX_LOOKBACK_DAYS)).date())
+
+ day_start = datetime.combine(day, datetime.min.time(), tzinfo=UTC)
+ while day_start < now:
+ next_day_start = day_start + timedelta(days=1)
+ rows_by_service: dict[str, dict[str, Any]] = {}
+
+ for metrics in SERVICE_METRICS_GROUPS:
+ body = _build_metrics_body(metrics, day_start, min(next_day_start, now))
+ data = _fetch_page(session, url, headers, body, logger)
+ if (data.get("meta") or {}).get("truncated"):
+ logger.warning(
+ f"Kong Konnect: metrics for {day_start.date()} exceed {METRICS_QUERY_LIMIT} services, "
+ "the rest are grouped as ____OTHER____"
+ )
+ for record in data.get("data") or []:
+ event = dict(record.get("event") or {})
+ # Traffic that matched no gateway service has no service ID. An empty string keeps the
+ # primary key non-null so the merge can replace the row.
+ service = event.pop("gateway_service", None) or ""
+ rows_by_service.setdefault(service, {"day": day_start, "gateway_service": service}).update(event)
+
+ # A day still in progress is queried again on resume. A completed day, even an empty one, is not.
+ resume_day = next_day_start if next_day_start <= now else day_start
+ resumable_source_manager.save_state(KongKonnectResumeConfig(start=_format_datetime(resume_day)))
+ if rows_by_service:
+ yield list(rows_by_service.values())
+ else:
+ resumable_source_manager.safe_point()
+
+ day_start = next_day_start
+
+
def kong_konnect_source(
api_token: str,
region: str,
@@ -412,6 +511,16 @@ def kong_konnect_source(
endpoint_config = KONG_KONNECT_ENDPOINTS[endpoint]
def items() -> Iterator[list[dict[str, Any]]]:
+ if endpoint_config.kind == "metrics":
+ return get_daily_service_metrics(
+ api_token=api_token,
+ region=region,
+ logger=logger,
+ resumable_source_manager=resumable_source_manager,
+ should_use_incremental_field=should_use_incremental_field,
+ db_incremental_field_last_value=db_incremental_field_last_value,
+ lookback_days=lookback_days,
+ )
if endpoint_config.kind != "analytics":
return get_lookup_rows(
api_token=api_token,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/settings.py
index db78543986a9..6f139f2a9901 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/settings.py
@@ -34,6 +34,19 @@ CORE_ENTITY_PAGE_SIZE = 1000
# out over) individually, so listing entities on the group too would duplicate them.
CONTROL_PLANE_GROUP_CLUSTER_TYPE = "CLUSTER_TYPE_CONTROL_PLANE_GROUP"
+# A metrics query returns at most 3 metrics, so each day is queried once per group and the groups are
+# merged on the gateway service.
+SERVICE_METRICS_GROUPS: tuple[tuple[str, ...], ...] = (
+ ("request_count", "error_rate", "response_latency_average"),
+ ("response_latency_p99", "upstream_latency_p99", "kong_latency_p99"),
+)
+
+# Above this many services in one day, Konnect folds the rest into a single `____OTHER____` row.
+METRICS_QUERY_LIMIT = 1000
+
+# The metrics API rejects time ranges older than 32 days, whatever the configured initial sync window.
+METRICS_MAX_LOOKBACK_DAYS = 31
+
@dataclass(frozen=True)
class KongKonnectEndpointConfig:
@@ -45,16 +58,19 @@ class KongKonnectEndpointConfig:
partition_key: Optional[str] = None
primary_keys: list[str] = field(default_factory=lambda: ["request_id"])
should_sync_default: bool = True
- # `analytics` is the time-windowed POST query; `page_number` is a `page[number]` list; `cursor` is a
- # `page[after]` list; `core_entity` fans out over every control plane.
- kind: Literal["analytics", "page_number", "cursor", "core_entity"] = "analytics"
+ # `analytics` is the time-windowed POST query; `metrics` is the per-day aggregate POST query;
+ # `page_number` is a `page[number]` list; `cursor` is a `page[after]` list; `core_entity` fans out
+ # over every control plane.
+ kind: Literal["analytics", "metrics", "page_number", "cursor", "core_entity"] = "analytics"
# Konnect versions each API separately, so the path prefix is per endpoint.
- api_version: Literal["v1", "v2"] = "v2"
+ api_version: Literal["v1", "v2", "v3"] = "v2"
sort: Optional[str] = None
# A child endpoint fans out over the rows of `parent` (an endpoint of the same kind). `{id}` in `path`
# is the parent row ID, which is copied onto each child row as `parent_id_field`.
parent: Optional[str] = None
parent_id_field: Optional[str] = None
+ # Dotted paths dropped from every row before it is written, because they can hold credentials.
+ excluded_fields: tuple[str, ...] = ()
description: Optional[str] = None
@@ -77,6 +93,24 @@ KONG_KONNECT_ENDPOINTS: dict[str, KongKonnectEndpointConfig] = {
description="Detailed records for every request proxied through the gateway (Advanced Analytics). "
"Historical depth on initial sync is limited by your Konnect plan's data retention.",
),
+ # Aggregates from the Analytics Metrics API, one row per gateway service per UTC day. Each run re-reads
+ # the last synced day, because that day can still be in progress, and the merge replaces its rows.
+ "daily_service_metrics": KongKonnectEndpointConfig(
+ name="daily_service_metrics",
+ path="/metrics",
+ kind="metrics",
+ partition_key="day",
+ primary_keys=["day", "gateway_service"],
+ incremental_fields=[
+ {
+ "label": "day",
+ "type": IncrementalFieldType.DateTime,
+ "field": "day",
+ "field_type": IncrementalFieldType.DateTime,
+ },
+ ],
+ description="Daily traffic, error rate, and latency for each gateway service, from the Konnect Analytics Metrics API.",
+ ),
# Lookups that resolve the IDs carried on api_requests rows. None of these list endpoints accept a
# timestamp filter, so they sync as full refresh.
"control_planes": KongKonnectEndpointConfig(
@@ -115,6 +149,32 @@ KONG_KONNECT_ENDPOINTS: dict[str, KongKonnectEndpointConfig] = {
incremental_fields=[],
description="Gateway consumers across all control planes. Resolves the consumer ID on api_requests.",
),
+ "plugins": KongKonnectEndpointConfig(
+ name="plugins",
+ path="/core-entities/plugins",
+ kind="core_entity",
+ primary_keys=["control_plane_id", "id"],
+ excluded_fields=("config",),
+ incremental_fields=[],
+ description="Gateway plugins across all control planes, with the service, route, or consumer each one is scoped to.",
+ ),
+ "upstreams": KongKonnectEndpointConfig(
+ name="upstreams",
+ path="/core-entities/upstreams",
+ kind="core_entity",
+ primary_keys=["control_plane_id", "id"],
+ excluded_fields=("healthchecks.active.headers",),
+ incremental_fields=[],
+ description="Gateway upstreams across all control planes, with their load balancing and health check settings.",
+ ),
+ "targets": KongKonnectEndpointConfig(
+ name="targets",
+ path="/core-entities/targets",
+ kind="core_entity",
+ primary_keys=["control_plane_id", "id"],
+ incremental_fields=[],
+ description="Upstream targets across all control planes: the backend host and port each upstream balances over.",
+ ),
"consumer_groups": KongKonnectEndpointConfig(
name="consumer_groups",
path="/core-entities/consumer_groups",
@@ -213,6 +273,30 @@ KONG_KONNECT_ENDPOINTS: dict[str, KongKonnectEndpointConfig] = {
incremental_fields=[],
description="Consumers registered in each Konnect identity realm, shared across the control planes the realm allows.",
),
+ # Metering & Billing v3. Subscriptions sort by ID because their ULIDs are stable and time-ordered,
+ # while the default `active_from` can change.
+ "meters": KongKonnectEndpointConfig(
+ name="meters",
+ path="/openmeter/meters",
+ kind="page_number",
+ api_version="v3",
+ sort="created_at",
+ partition_key="created_at",
+ primary_keys=["id"],
+ incremental_fields=[],
+ description="Metering & Billing meters: the event type and aggregation that each usage meter counts.",
+ ),
+ "subscriptions": KongKonnectEndpointConfig(
+ name="subscriptions",
+ path="/openmeter/subscriptions",
+ kind="page_number",
+ api_version="v3",
+ sort="id",
+ partition_key="created_at",
+ primary_keys=["id"],
+ incremental_fields=[],
+ description="Metering & Billing customer subscriptions, with their plan, status, and current billing period.",
+ ),
}
ENDPOINTS = tuple(KONG_KONNECT_ENDPOINTS.keys())
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/source.py
index 388fc9bd170f..e3e716ac165b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/source.py
@@ -69,7 +69,7 @@ class KongKonnectSource(ResumableSource[KongKonnectSourceConfig, KongKonnectResu
category=DataWarehouseSourceCategory.ENGINEERING___MONITORING,
label="Kong Inc. (Kong Konnect)",
releaseStatus=ReleaseStatus.BETA,
- caption="""Enter a Kong Konnect access token to pull your gateway's Advanced Analytics API request logs, plus the control planes, gateway entities, consumer groups, API products, Service Catalog scorecards, and identity realm consumers behind them, into the PostHog Data warehouse.
+ caption="""Enter a Kong Konnect access token to pull your gateway's Advanced Analytics API request logs and daily service metrics, plus the control planes, gateway entities, consumer groups, API products, Service Catalog scorecards, identity realm consumers, and Metering & Billing meters and subscriptions behind them, into the PostHog Data warehouse.
Create a **Personal Access Token** under **Konnect → Personal access tokens**, or a **System Account access token** for a service identity. Either is sent as a bearer token.
@@ -126,7 +126,7 @@ How far back the initial sync can reach depends on your Konnect plan's Advanced
def get_non_retryable_errors(self) -> dict[str, str | None]:
return {
"401 Client Error: Unauthorized": "Your Kong Konnect access token is invalid or has expired. Create a new token in Konnect and reconnect.",
- "403 Client Error: Forbidden": "Your Kong Konnect access token is missing the permissions needed to read this data. Grant the token or account read access to the Konnect area this table comes from (Advanced Analytics, control planes, API Products, Service Catalog, or Consumers), then reconnect.",
+ "403 Client Error: Forbidden": "Your Kong Konnect access token is missing the permissions needed to read this data. Grant the token or account read access to the Konnect area this table comes from (Advanced Analytics, control planes, API Products, Service Catalog, Consumers, or Metering & Billing), then reconnect.",
}
def get_schemas(
@@ -138,7 +138,14 @@ How far back the initial sync can reach depends on your Konnect plan's Advanced
force_refresh: bool = False,
api_version: str | None = None,
) -> list[SourceSchema]:
- return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names, descriptions=DESCRIPTIONS)
+ return build_endpoint_schemas(
+ ENDPOINTS,
+ INCREMENTAL_FIELDS,
+ names,
+ # Each run re-reads the last synced day, so append would duplicate its rows.
+ merge_only=("daily_service_metrics",),
+ descriptions=DESCRIPTIONS,
+ )
def validate_credentials(
self,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/test_kong_konnect.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/test_kong_konnect.py
index 88c2ccbf43f6..6a5131b63c6a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/test_kong_konnect.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/test_kong_konnect.py
@@ -121,6 +121,109 @@ class TestGetRowsPagination:
assert body["offset"] == 5000
+def _metrics_page(*events: dict[str, Any]) -> ScriptedResponse:
+ return ScriptedResponse(
+ json={"data": [{"event": event, "timestamp": "2026-06-14T00:00:00Z"} for event in events], "meta": {}}
+ )
+
+
+class TestDailyServiceMetrics:
+ @time_machine.travel("2026-06-15T12:00:00Z", tick=False)
+ def test_walks_days_and_merges_metric_groups_per_service(self) -> None:
+ result = _driver().run(
+ "daily_service_metrics",
+ [
+ _metrics_page(
+ {"gateway_service": "cp:svc-a", "request_count": 10, "error_rate": 0.1},
+ {"gateway_service": None, "request_count": 2, "error_rate": 0},
+ ),
+ _metrics_page({"gateway_service": "cp:svc-a", "response_latency_p99": 40}),
+ _metrics_page(),
+ _metrics_page(),
+ ],
+ incremental_field="day",
+ db_incremental_field_last_value=datetime(2026, 6, 14, tzinfo=UTC),
+ )
+
+ assert result.raised is None
+ day = datetime(2026, 6, 14, tzinfo=UTC)
+ assert result.rows == [
+ {
+ "day": day,
+ "gateway_service": "cp:svc-a",
+ "request_count": 10,
+ "error_rate": 0.1,
+ "response_latency_p99": 40,
+ },
+ {"day": day, "gateway_service": "", "request_count": 2, "error_rate": 0},
+ ]
+ bodies = [request.json() for request in result.requests]
+ assert all(request.method == "POST" and request.path == "/v2/metrics" for request in result.requests)
+ assert [body["dimensions"] for body in bodies] == [["gateway_service"]] * 4
+ assert [body["metrics"] for body in bodies[:2]] == [
+ ["request_count", "error_rate", "response_latency_average"],
+ ["response_latency_p99", "upstream_latency_p99", "kong_latency_p99"],
+ ]
+ # The current day ends at now, so a later sync re-reads it and the merge replaces its rows.
+ assert [(body["time_range"]["start"], body["time_range"]["end"]) for body in bodies] == [
+ ("2026-06-14T00:00:00Z", "2026-06-15T00:00:00Z"),
+ ("2026-06-14T00:00:00Z", "2026-06-15T00:00:00Z"),
+ ("2026-06-15T00:00:00Z", "2026-06-15T12:00:00Z"),
+ ("2026-06-15T00:00:00Z", "2026-06-15T12:00:00Z"),
+ ]
+ # The completed day moves the cursor on; the day still in progress keeps it there.
+ assert [state.start for state in result.saved_states] == ["2026-06-15T00:00:00Z", "2026-06-15T00:00:00Z"]
+ assert result.response is not None
+ assert result.response.sort_mode == "asc"
+
+ @parameterized.expand(
+ [
+ ("full_refresh_walks_back_lookback", {"should_use_incremental_field": False}, None, "2026-06-12T00:00:00Z"),
+ (
+ "incremental_starts_at_watermark_day",
+ {
+ "incremental_field": "day",
+ "db_incremental_field_last_value": datetime(2026, 6, 14, 9, 30, tzinfo=UTC),
+ },
+ None,
+ "2026-06-14T00:00:00Z",
+ ),
+ (
+ "future_watermark_clamped_to_today",
+ {"incremental_field": "day", "db_incremental_field_last_value": datetime(2027, 1, 1, tzinfo=UTC)},
+ None,
+ "2026-06-15T00:00:00Z",
+ ),
+ (
+ "stale_watermark_clamped_to_metrics_api_limit",
+ {"incremental_field": "day", "db_incremental_field_last_value": datetime(2026, 1, 1, tzinfo=UTC)},
+ None,
+ "2026-05-15T00:00:00Z",
+ ),
+ (
+ "resume_state_wins_over_watermark",
+ {"incremental_field": "day", "db_incremental_field_last_value": datetime(2026, 6, 12, tzinfo=UTC)},
+ KongKonnectResumeConfig(start="2026-06-14T00:00:00Z"),
+ "2026-06-14T00:00:00Z",
+ ),
+ ]
+ )
+ @time_machine.travel("2026-06-15T12:00:00Z", tick=False)
+ def test_first_day_queried(
+ self, _name: str, inputs: dict[str, Any], resume: KongKonnectResumeConfig | None, expected_start: str
+ ) -> None:
+ start = datetime.fromisoformat(expected_start.replace("Z", "+00:00"))
+ days = (datetime(2026, 6, 15, tzinfo=UTC) - start).days + 1
+ result = _driver(lookback_days=3).run(
+ "daily_service_metrics", [_metrics_page()] * (2 * days), resume_state=resume, **inputs
+ )
+
+ assert result.raised is None
+ assert result.rows == []
+ assert len(result.requests) == 2 * days
+ assert result.requests[0].json()["time_range"]["start"] == expected_start
+
+
class TestValidateCredentials:
@parameterized.expand([("ok", 200, True), ("unauthorized", 401, False), ("forbidden", 403, False)])
def test_status_mapping(self, _name: str, status: int, expected: bool) -> None:
@@ -227,6 +330,49 @@ class TestLookupRows:
]
assert result.params("offset") == [None, None, "next-token", None, None]
+ @parameterized.expand(
+ [
+ (
+ "plugin_config",
+ "plugins",
+ {"id": "p-1", "name": "aws-lambda", "config": {"aws_secret": "shh"}},
+ {"id": "p-1", "name": "aws-lambda", "control_plane_id": "cp-a"},
+ ),
+ (
+ "upstream_health_check_headers",
+ "upstreams",
+ {
+ "id": "u-1",
+ "healthchecks": {
+ "active": {"headers": {"Authorization": ["Bearer shh"]}, "http_path": "/health"},
+ "passive": {"type": "http"},
+ },
+ },
+ {
+ "id": "u-1",
+ "healthchecks": {"active": {"http_path": "/health"}, "passive": {"type": "http"}},
+ "control_plane_id": "cp-a",
+ },
+ ),
+ ]
+ )
+ def test_core_entities_drop_credential_fields(
+ self, _name: str, endpoint: str, row: dict[str, Any], expected: dict[str, Any]
+ ) -> None:
+ with patch(f"{kong_konnect.__name__}.make_tracked_session", wraps=kong_konnect.make_tracked_session) as spy:
+ result = _driver().run(
+ endpoint,
+ [
+ ScriptedResponse(json={"data": [_control_plane("cp-a")], "meta": {"page": {"total": 1}}}),
+ ScriptedResponse(json={"data": [row]}),
+ ],
+ )
+
+ assert result.raised is None
+ assert result.rows == [expected]
+ # HTTP samples would capture the raw response before the fields are dropped.
+ assert spy.call_args.kwargs["capture"] is False
+
def test_core_entities_raise_on_auth_error(self) -> None:
result = _driver().run(
"consumers",
@@ -305,6 +451,39 @@ class TestLookupRows:
]
assert result.params("sort") == ["name", "name", "name"]
+ @parameterized.expand(
+ [
+ ("meters", "meters", "/v3/openmeter/meters", "created_at"),
+ ("subscriptions", "subscriptions", "/v3/openmeter/subscriptions", "id"),
+ ]
+ )
+ def test_metering_lists_use_v3_page_number_pagination(
+ self, _name: str, endpoint: str, path: str, sort: str
+ ) -> None:
+ result = _driver().run(
+ endpoint,
+ [
+ ScriptedResponse(
+ json={
+ "data": [{"id": f"m-{i}"} for i in range(LIST_PAGE_SIZE)],
+ "meta": {"page": {"number": 1, "size": LIST_PAGE_SIZE, "total": LIST_PAGE_SIZE + 1}},
+ }
+ ),
+ ScriptedResponse(
+ json={
+ "data": [{"id": "m-last"}],
+ "meta": {"page": {"number": 2, "size": LIST_PAGE_SIZE, "total": LIST_PAGE_SIZE + 1}},
+ }
+ ),
+ ],
+ )
+
+ assert result.raised is None
+ assert len(result.rows) == LIST_PAGE_SIZE + 1
+ assert result.paths == [path, path]
+ assert result.params("page[number]") == ["1", "2"]
+ assert result.params("sort") == [sort, sort]
+
def test_realm_consumers_follow_page_after_cursor_from_meta_next(self) -> None:
result = _driver().run(
"realm_consumers",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/test_kong_konnect_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/test_kong_konnect_source.py
index 11dd05553931..a8423a0f6d36 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/test_kong_konnect_source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kong_konnect/test_kong_konnect_source.py
@@ -36,12 +36,14 @@ class TestSourceConfig:
class TestGetSchemas:
- def test_only_api_requests_supports_incremental(self) -> None:
+ def test_only_time_windowed_endpoints_support_incremental(self) -> None:
schemas = {s.name: s for s in KongKonnectSource().get_schemas(_config(), team_id=1)}
assert schemas["api_requests"].supports_incremental is True
assert [f["field"] for f in schemas["api_requests"].incremental_fields] == ["request_start"]
+ assert schemas["daily_service_metrics"].supports_incremental is True
+ assert [f["field"] for f in schemas["daily_service_metrics"].incremental_fields] == ["day"]
# The lookup list endpoints have no server-side timestamp filter, so they must stay full refresh.
- for name in set(schemas) - {"api_requests"}:
+ for name in set(schemas) - {"api_requests", "daily_service_metrics"}:
assert schemas[name].supports_incremental is False
assert schemas[name].supports_append is False