:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add komodor issue, event and cost tables (#122650)
Tom Owersca3a66106689
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex fc3ecc15b3e7..fb90738d0829 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -6965,10 +6965,10 @@ Today (4): `clusters`, `jobs`, `monitors`, `services`Diffed against: <https://api.komodor.com/api/docs/doc.json>-- [ ] `Cost allocation (GET /api/v2/cost/allocation)` — measure Kubernetes spending by cluster, workspace, and workload (high)-- [ ] `Service issues (POST /api/v2/services/issues/search)` — measure service incidents, resolution times, and recurring failures (high)-- [ ] `Cluster issues (POST /api/v2/clusters/issues/search)` — track infrastructure incidents that service inventory cannot show (high)-- [ ] `Kubernetes events (POST /api/v2/services/k8s-events/search; POST /api/v2/clusters/k8s-events/search)` — connect deployments and Kubernetes changes to service and cluster incidents (high)+- [x] `Cost allocation (GET /api/v2/cost/allocation)` — measure Kubernetes spending by cluster, workspace, and workload (high). Added as `cost_allocation`: per-service cost over the past 30 days, refreshed each sync.+- [ ] `Service issues (POST /api/v2/services/issues/search)` — measure service incidents, resolution times, and recurring failures (high); skipped: the search needs one request per service, issue type and 2-day window, and `cluster_issues` returns the same issue types for every namespace in a cluster+- [x] `Cluster issues (POST /api/v2/clusters/issues/search)` — track infrastructure incidents that service inventory cannot show (high). Added as `cluster_issues` (fan-out over `clusters`, incremental on `startTime`).+- [x] `Kubernetes events (POST /api/v2/services/k8s-events/search; POST /api/v2/clusters/k8s-events/search)` — connect deployments and Kubernetes changes to service and cluster incidents (high). Added as `k8s_events` from the cluster-scoped search (fan-out over `clusters`, incremental on `startTime`); the service-scoped search returns a subset of the same events.- [ ] `Health risks (GET /api/v2/health/risks)` — track workload reliability risks and their severity (medium)- [ ] `Audit logs (GET /api/v2/audit-log)` — analyze account activity and operational changes over time (medium)- [ ] `Service right-sizing recommendations (GET /api/v2/cost/right-sizing/service)` — compare requested resources with recommendations to identify potential savings (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/canonical_descriptions.pyindex 4addaa9cd60c..225a3d11f2e1 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/canonical_descriptions.py@@ -54,4 +54,52 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"createdFrom": "Application that created the configuration.",},},+ "cluster_issues": {+ "description": "Issues Komodor detected in each cluster over its 7-day retention window, such as availability problems and failed deploys.",+ "docs_url": f"{API_DOCS_URL}#/Issues/post_api_v2_clusters_issues_search",+ "columns": {+ "cluster": "Cluster the issue was found in.",+ "type": "Issue type: availability, failed-deploy, node-issue, pvc-issue, or workflow-issue.",+ "status": "Open while the issue is ongoing, closed once it is resolved or finished.",+ "startTime": "Time the issue started as a Unix timestamp in seconds.",+ "endTime": "Time the issue ended as a Unix timestamp in seconds.",+ "summary": "Brief summary of the issue.",+ "details": "Detailed information about the issue.",+ "link": "URL of the issue in Komodor.",+ },+ },+ "k8s_events": {+ "description": "Kubernetes deploy and node events in each cluster over Komodor's 7-day retention window.",+ "docs_url": f"{API_DOCS_URL}#/Events/post_api_v2_clusters_k8s_events_search",+ "columns": {+ "cluster": "Cluster the event happened in.",+ "type": "Event type: deploy, node-terminated, or node-created.",+ "status": "Status of the event.",+ "startTime": "Time the event started as a Unix timestamp in seconds.",+ "endTime": "Time the event ended as a Unix timestamp in seconds.",+ "summary": "Brief summary of the event.",+ "details": "Detailed information about the event.",+ "link": "URL of the event in Komodor.",+ "resource": "Kubernetes resource the event is about, with its name, cluster, namespace, and kind.",+ },+ },+ "cost_allocation": {+ "description": "Kubernetes cost per service over the past 30 days, refreshed on each sync.",+ "docs_url": f"{API_DOCS_URL}#/KomodorCost/getCostAllocation",+ "columns": {+ "komodorServiceId": "Komodor service identifier.",+ "komodorServiceName": "Service name.",+ "komodorServiceKind": "Service kind.",+ "clusterName": "Cluster that runs the service.",+ "namespace": "Namespace that contains the service.",+ "totalCost": "Total cost of the service over the past 30 days.",+ "shareOfCost": "Share of the total cost that the service accounts for.",+ "cpuRequestCoreHours": "Requested CPU in core hours.",+ "memoryRequestGBHours": "Requested memory in GB hours.",+ "idleResourcesCost": "Cost of requested resources the service did not use.",+ "efficiency": "Resource efficiency of the service.",+ "optimizationScore": "Optimization score of the service.",+ "potentialSaving": "Potential saving from right-sizing the service.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/komodor.py b/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/komodor.pyindex 85f77ac9c7b3..ea5fbdf55709 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/komodor.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/komodor.py@@ -1,6 +1,9 @@+from collections.abc import Iterator+from datetime import UTC, datetime+from functools import partialfrom typing import Any-from requests import Request+from requests import Request, Responsefrom requests.exceptions import HTTPErrorfrom posthog.dataclasses import frozen@@ -26,12 +29,32 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.komodor.sePAGE_SIZE,PRIMARY_KEYS,REGION_URLS,+ SEARCH_INCREMENTAL_OVERLAP_SECONDS,+ SEARCH_RETENTION_MARGIN_SECONDS,+ SEARCH_RETENTION_SECONDS,+ SEARCH_WINDOW_SECONDS,+ WINDOWED_SEARCHES,)+REQUEST_TIMEOUT = (10, 60)+@frozenclass KomodorResumeConfig:- cursor: str | int+ cursor: str | int | None = None+ cluster: str | None = None+ search_type: str | None = None+ window_start: int | None = None+ start: int | None = None+ until: int | None = None+++@frozen+class SearchUnit:+ cluster: str+ search_type: str+ window_start: int+ window_end: intclass KomodorPaginator(JSONResponseCursorPaginator):@@ -50,24 +73,39 @@ class KomodorPaginator(JSONResponseCursorPaginator):self.update_request(request)+class CostAllocationPaginator(JSONResponseCursorPaginator):+ def __init__(self) -> None:+ super().__init__(cursor_path="paginationToken", cursor_param="paginationToken", raise_on_repeated_cursor=True)++ def update_state(self, response: Response, data: list[Any] | None = None) -> None:+ if not response.json().get("hasMorePages"):+ self._has_next_page = False+ return+ super().update_state(response, data)++def get_base_url(region: str, api_version: str) -> str:if region not in REGION_URLS:raise ValueError("Select a valid Komodor region: US or EU.")return f"{REGION_URLS[region]}/api/{api_version}/"+def _client(api_key: str, base_url: str) -> RESTClient:+ return RESTClient(+ base_url=base_url,+ auth=APIKeyAuth(api_key=api_key, name="X-API-KEY"),+ allow_redirects=False,+ request_timeout=REQUEST_TIMEOUT,+ )++def validate_credentials(config: KomodorSourceConfig, api_version: str) -> tuple[bool, str | None]:try:base_url = get_base_url(config.region, api_version)except ValueError as error:return False, str(error)- client = RESTClient(- base_url=base_url,- auth=APIKeyAuth(api_key=config.api_key, name="X-API-KEY"),- allow_redirects=False,- request_timeout=(10, 60),- )+ client = _client(config.api_key, base_url)try:next(client.paginate(@@ -87,6 +125,140 @@ def validate_credentials(config: KomodorSourceConfig, api_version: str) -> tuplereturn True, None+def _search_units(clusters: list[str], types: tuple[str, ...], start: int, until: int) -> list[SearchUnit]:+ windows: list[tuple[int, int]] = []+ window_start = start+ while window_start < until:+ window_end = min(window_start + SEARCH_WINDOW_SECONDS, until)+ windows.append((window_start, window_end))+ window_start = window_end+ return [+ SearchUnit(cluster=cluster, search_type=search_type, window_start=window_start, window_end=window_end)+ for cluster in clusters+ for search_type in types+ for window_start, window_end in windows+ ]+++def _retention_floor(now: int) -> int:+ return now - SEARCH_RETENTION_SECONDS + SEARCH_RETENTION_MARGIN_SECONDS+++def _search_start(until: int, db_incremental_field_last_value: Any) -> int:+ start = _retention_floor(until)+ try:+ last_value = int(db_incremental_field_last_value)+ except (TypeError, ValueError):+ return start+ # A watermark ahead of our clock still reads the latest window.+ return min(max(start, last_value - SEARCH_INCREMENTAL_OVERLAP_SECONDS), until - SEARCH_WINDOW_SECONDS)+++def _owns(unit: SearchUnit, start: int, until: int, row: dict[str, Any]) -> bool:+ start_time = row.get("startTime")+ if not isinstance(start_time, int | float):+ return True+ after_start = unit.window_start == start or start_time >= unit.window_start+ before_end = unit.window_end == until or start_time < unit.window_end+ return after_start and before_end+++def iter_windowed_search(+ endpoint: str,+ api_key: str,+ base_url: str,+ manager: ResumableSourceManager[KomodorResumeConfig],+ db_incremental_field_last_value: Any,+) -> Iterator[list[dict[str, Any]]]:+ search = WINDOWED_SEARCHES[endpoint]+ resume = manager.load_state() if manager.can_resume() else None+ if resume is not None and resume.start is not None and resume.until is not None:+ start, until = resume.start, resume.until+ else:+ until = int(datetime.now(UTC).timestamp())+ start = _search_start(until, db_incremental_field_last_value)++ client = _client(api_key, base_url)+ try:+ clusters = sorted(+ {+ row["name"]+ for page in client.paginate(+ "clusters",+ data_selector="data.clusters",+ data_selector_required=True,+ paginator=SinglePagePaginator(),+ )+ for row in page+ }+ )+ units = _search_units(clusters, search.types, start, until)++ first, cursor = 0, None+ if resume is not None:+ for index, unit in enumerate(units):+ if (unit.cluster, unit.search_type, unit.window_start) == (+ resume.cluster,+ resume.search_type,+ resume.window_start,+ ):+ first, cursor = index, resume.cursor+ break++ def stage(index: int, state: dict[str, Any] | None, has_next_page: bool) -> None:+ if has_next_page and state is not None:+ unit, next_cursor = units[index], state["cursor"]+ elif index + 1 < len(units):+ unit, next_cursor = units[index + 1], None+ else:+ return+ manager.save_state(+ KomodorResumeConfig(+ cursor=next_cursor,+ cluster=unit.cluster,+ search_type=unit.search_type,+ window_start=unit.window_start,+ start=start,+ until=until,+ )+ )++ for index in range(first, len(units)):+ unit = units[index]+ # A long or resumed run can age its oldest windows past the 7-day limit, which Komodor rejects.+ floor = _retention_floor(int(datetime.now(UTC).timestamp()))+ if unit.window_end <= floor:+ continue+ manager.safe_point()+ props: dict[str, Any] = {+ "type": unit.search_type,+ "fromEpoch": max(unit.window_start, floor),+ # Both bounds are inclusive, so the next window starts one second later.+ "toEpoch": unit.window_end - 1,+ }+ if search.statuses is not None:+ props["statuses"] = list(search.statuses)+ for page in client.paginate(+ search.path,+ method="POST",+ json={"scope": {"cluster": unit.cluster}, "props": props, "pagination": {"pageSize": PAGE_SIZE}},+ paginator=KomodorPaginator(cursor_path=search.cursor_path, pagination_field=search.pagination_field),+ data_selector=search.data_selector,+ data_selector_required=True,+ initial_paginator_state={"cursor": cursor} if cursor is not None else None,+ page_state_hook=partial(stage, index),+ ):+ # A search can return an issue or event in two windows, so only the window that holds its start keeps it.+ rows = [{**row, "cluster": unit.cluster} for row in page if _owns(unit, start, until, row)]+ if rows:+ yield rows+ else:+ manager.safe_point()+ cursor = None+ finally:+ client.session.close()++def komodor_source(config: KomodorSourceConfig,endpoint: str,@@ -94,13 +266,29 @@ def komodor_source(job_id: str,api_version: str,resumable_source_manager: ResumableSourceManager[KomodorResumeConfig],+ db_incremental_field_last_value: Any = None,) -> SourceResponse:if endpoint not in ENDPOINTS:raise ValueError(f"Unknown Komodor table: {endpoint}")+ base_url = get_base_url(config.region, api_version)++ if endpoint in WINDOWED_SEARCHES:+ return SourceResponse(+ name=endpoint,+ items=lambda: iter_windowed_search(+ endpoint, config.api_key, base_url, resumable_source_manager, db_incremental_field_last_value+ ),+ primary_keys=PRIMARY_KEYS[endpoint],+ sort_mode=None,+ supports_resume=True,+ )+endpoint_config = ENDPOINTS[endpoint].copy()- paginated = endpoint in ("services", "jobs")- if paginated:+ paginated = endpoint in ("services", "jobs", "cost_allocation")+ if endpoint == "cost_allocation":+ endpoint_config["paginator"] = CostAllocationPaginator()+ elif paginated:endpoint_config["json"] = {"pagination": {"pageSize": PAGE_SIZE}}endpoint_config["paginator"] = KomodorPaginator(cursor_path="meta.token" if endpoint == "services" else "meta.nextPage",@@ -113,7 +301,7 @@ def komodor_source(initial_state: dict[str, Any] | None = Noneif paginated and resumable_source_manager.can_resume():saved = resumable_source_manager.load_state()- if saved is not None:+ if saved is not None and saved.cursor is not None:initial_state = {"cursor": saved.cursor}def save_checkpoint(state: dict[str, Any] | None) -> None:@@ -122,10 +310,10 @@ def komodor_source(rest_config: RESTAPIConfig = {"client": {- "base_url": get_base_url(config.region, api_version),+ "base_url": base_url,"auth": {"type": "api_key", "api_key": config.api_key, "name": "X-API-KEY", "location": "header"},"allow_redirects": False,- "request_timeout": (10, 60),+ "request_timeout": REQUEST_TIMEOUT,},"resources": [{"name": endpoint, "endpoint": endpoint_config, "write_disposition": "replace"}],}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/settings.pyindex b3834ad0c3ca..8a2e1f781064 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/settings.py@@ -1,14 +1,69 @@+from posthog.dataclasses import frozen+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import Endpoint+from products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import incremental_field+from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldTypeAPI_DOCS_URL = "https://api.komodor.com/api/docs/index.html"REGION_URLS = {"us": "https://api.komodor.com", "eu": "https://api.eu.komodor.com"}PAGE_SIZE = 100+COST_ALLOCATION_PAGE_SIZE = 1000++# Issue and event searches accept at most 2 days per request and reach at most 7 days back.+SEARCH_WINDOW_SECONDS = 2 * 24 * 60 * 60+SEARCH_RETENTION_SECONDS = 7 * 24 * 60 * 60+# Keeps the first window inside the 7-day limit when our clock runs ahead of Komodor's.+SEARCH_RETENTION_MARGIN_SECONDS = 10 * 60+# Issues and events change status after they start, so each incremental run reads the last window again.+SEARCH_INCREMENTAL_OVERLAP_SECONDS = SEARCH_WINDOW_SECONDS+++@frozen+class WindowedSearch:+ path: str+ data_selector: str+ types: tuple[str, ...]+ cursor_path: str+ pagination_field: str+ statuses: tuple[str, ...] | None = None+++WINDOWED_SEARCHES: dict[str, WindowedSearch] = {+ "cluster_issues": WindowedSearch(+ path="clusters/issues/search",+ data_selector="data.issues",+ types=("availability", "failed-deploy", "node-issue", "pvc-issue", "workflow-issue"),+ cursor_path="meta.nextPage",+ pagination_field="page",+ statuses=("open", "closed"),+ ),+ "k8s_events": WindowedSearch(+ path="clusters/k8s-events/search",+ data_selector="data.events",+ types=("deploy", "node-terminated", "node-created"),+ cursor_path="meta.token",+ pagination_field="token",+ ),+}ENDPOINTS: dict[str, Endpoint] = {"services": {"path": "services/search", "method": "POST", "data_selector": "data.services"},"jobs": {"path": "jobs/search", "method": "POST", "data_selector": "data.jobs"},"clusters": {"path": "clusters", "data_selector": "data.clusters"},"monitors": {"path": "realtime-monitors/config", "data_selector": "data.monitors"},+ **{+ name: {"path": search.path, "method": "POST", "data_selector": search.data_selector}+ for name, search in WINDOWED_SEARCHES.items()+ },+ "cost_allocation": {+ "path": "cost/allocation",+ "data_selector": "groupedCosts",+ "params": {"timeFrame": "Past_30_days", "groupBy": "komodorServiceName", "pageSize": COST_ALLOCATION_PAGE_SIZE},+ },+}++INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {+ name: [incremental_field("startTime", IncrementalFieldType.Integer)] for name in WINDOWED_SEARCHES}PRIMARY_KEYS = {@@ -16,6 +71,9 @@ PRIMARY_KEYS = {"jobs": ["cluster", "namespace", "kind", "name"],"clusters": ["name"],"monitors": ["id"],+ "cluster_issues": ["cluster", "type", "startTime", "link"],+ "k8s_events": ["cluster", "type", "startTime", "link"],+ "cost_allocation": ["komodorServiceId"],}AUTH_ERRORS = {diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/source.pyindex 83951a4b147c..c55cf40703b9 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/source.py@@ -36,6 +36,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.komodor.seAPI_DOCS_URL,AUTH_ERRORS,ENDPOINTS,+ INCREMENTAL_FIELDS,)from products.warehouse_sources.backend.types import ExternalDataSourceType@@ -72,7 +73,7 @@ class KomodorSource(ResumableSource[KomodorSourceConfig, KomodorResumeConfig]):force_refresh: bool = False,api_version: str | None = None,) -> list[SourceSchema]:- return build_endpoint_schemas(ENDPOINTS, {}, names)+ return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names, merge_only=INCREMENTAL_FIELDS)def validate_credentials(self,@@ -99,6 +100,7 @@ class KomodorSource(ResumableSource[KomodorSourceConfig, KomodorResumeConfig]):inputs.job_id,self.resolve_api_version(inputs.api_version),resumable_source_manager,+ inputs.db_incremental_field_last_value if inputs.should_use_incremental_field else None,)@propertydiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/tests/test_komodor.py b/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/tests/test_komodor.pyindex 7f128d40057c..d6f81c233c4a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/tests/test_komodor.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/komodor/tests/test_komodor.py@@ -1,4 +1,5 @@import json+import timefrom collections.abc import Iterable, Iteratorfrom http.client import responses as status_reasonsfrom typing import Any, Literal, cast@@ -11,6 +12,7 @@ from requests.exceptions import HTTPErrorfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.testing import (+ RecordedRequest,ScriptedResponse,SourceDriver,scripted_network,@@ -24,6 +26,13 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.komodor.kokomodor_source,validate_credentials,)+from products.warehouse_sources.backend.temporal.data_imports.sources.komodor.settings import (+ SEARCH_INCREMENTAL_OVERLAP_SECONDS,+ SEARCH_RETENTION_MARGIN_SECONDS,+ SEARCH_RETENTION_SECONDS,+ SEARCH_WINDOW_SECONDS,+ WINDOWED_SEARCHES,+)from products.warehouse_sources.backend.temporal.data_imports.sources.komodor.source import KomodorSource@@ -178,3 +187,187 @@ def test_repeated_resume_cursor_fails(table: str, cursor_field: str, cursor: strassert isinstance(result.raised, ValueError)assert "not advancing" in str(result.raised)assert len(result.requests) == 1+++ENDPOINT_PATHS = {"cluster_issues": "clusters/issues/search", "k8s_events": "clusters/k8s-events/search"}+SEARCHES = [+ ("cluster_issues", "issues", "nextPage", "page", 2),+ ("k8s_events", "events", "token", "token", "next-event"),+]+++def search_script(collection: str, cursor_field: str, cursor: str | int, pages: dict[tuple[str, int], list[dict]]):+ def answer(request: RecordedRequest) -> ScriptedResponse:+ if request.path.endswith("/clusters"):+ return ScriptedResponse(json={"data": {"clusters": [{"name": "b"}, {"name": "a"}]}})+ body = request.json()+ props = body["props"]+ if body["scope"]["cluster"] != "a":+ return ScriptedResponse(json={"data": {collection: []}, "meta": {}})+ if body["pagination"].get("token" if cursor_field == "token" else "page") is None:+ rows = pages.get((props["type"], props["fromEpoch"]), [])+ meta = {cursor_field: cursor} if rows else {}+ else:+ rows, meta = pages.get(("next", props["fromEpoch"]), []), {}+ return ScriptedResponse(json={"data": {collection: rows}, "meta": meta})++ return answer++[email protected](("table", "collection", "cursor_field", "body_field", "cursor"), SEARCHES)+def test_windowed_search_walks_clusters_types_windows_and_pages(+ table: str, collection: str, cursor_field: str, body_field: str, cursor: str | int+) -> None:+ search = WINDOWED_SEARCHES[table]+ first_type = search.types[0]+ start = int(time.time()) - 3 * 24 * 60 * 60+ repeated = {"type": first_type, "startTime": start + 5, "link": "https://app.komodor.com/x"}+ script = search_script(+ collection,+ cursor_field,+ cursor,+ {+ (first_type, start): [repeated],+ ("next", start): [{"type": first_type, "startTime": start + 6, "link": "https://app.komodor.com/y"}],+ (first_type, start + SEARCH_WINDOW_SECONDS): [repeated],+ },+ )+ until = start + SEARCH_WINDOW_SECONDS + 100+ result = driver().run(table, script, resume_state=KomodorResumeConfig(start=start, until=until))++ assert result.raised is None+ assert result.rows == [+ {**repeated, "cluster": "a"},+ {"type": first_type, "startTime": start + 6, "link": "https://app.komodor.com/y", "cluster": "a"},+ ]+ searches = [request.json() for request in result.requests[1:]]+ assert len(searches) == 2 * len(search.types) * 2 + 2+ assert searches[:3] == [+ {+ "scope": {"cluster": "a"},+ "props": {+ "type": first_type,+ "fromEpoch": start,+ "toEpoch": start + SEARCH_WINDOW_SECONDS - 1,+ **({"statuses": list(search.statuses)} if search.statuses else {}),+ },+ "pagination": {"pageSize": 100, **extra},+ }+ for extra in ({}, {body_field: cursor})+ ] + [+ {+ "scope": {"cluster": "a"},+ "props": {+ "type": first_type,+ "fromEpoch": start + SEARCH_WINDOW_SECONDS,+ "toEpoch": until - 1,+ **({"statuses": list(search.statuses)} if search.statuses else {}),+ },+ "pagination": {"pageSize": 100},+ }+ ]+ assert [body["scope"]["cluster"] for body in searches[-len(search.types) * 2 :]] == ["b"] * len(search.types) * 2+ assert result.committed_states[1:3] == [+ KomodorResumeConfig(+ cursor=cursor, cluster="a", search_type=first_type, window_start=start, start=start, until=until+ ),+ KomodorResumeConfig(+ cluster="a", search_type=first_type, window_start=start + SEARCH_WINDOW_SECONDS, start=start, until=until+ ),+ ]++[email protected](("table", "collection", "cursor_field", "body_field", "cursor"), SEARCHES)+def test_windowed_search_resumes_at_the_saved_cluster_type_window_and_cursor(+ table: str, collection: str, cursor_field: str, body_field: str, cursor: str | int+) -> None:+ search = WINDOWED_SEARCHES[table]+ start = int(time.time()) - 3 * 24 * 60 * 60+ state = KomodorResumeConfig(+ cursor=cursor,+ cluster="b",+ search_type=search.types[-1],+ window_start=start + SEARCH_WINDOW_SECONDS,+ start=start,+ until=start + SEARCH_WINDOW_SECONDS + 100,+ )+ result = driver().run(table, search_script(collection, cursor_field, cursor, {}), resume_state=state)++ assert result.raised is None+ assert [request.path for request in result.requests] == ["/api/v2/clusters", f"/api/v2/{ENDPOINT_PATHS[table]}"]+ body = result.requests[1].json()+ assert body["scope"] == {"cluster": "b"}+ assert (body["props"]["type"], body["props"]["fromEpoch"]) == (search.types[-1], start + SEARCH_WINDOW_SECONDS)+ assert body["pagination"] == {"pageSize": 100, body_field: cursor}++[email protected]("table", WINDOWED_SEARCHES)[email protected]("last_value_age", [None, 3600])+def test_windowed_search_reads_retention_or_from_the_watermark_in_two_day_windows(+ table: str, last_value_age: int | None+) -> None:+ now = int(time.time())+ last_value = None if last_value_age is None else now - last_value_age+ result = driver().run(+ table,+ lambda request: ScriptedResponse(+ json={"data": {"clusters": [{"name": "a"}]}}+ if request.path.endswith("/clusters")+ else {"data": {"issues": [], "events": []}, "meta": {}}+ ),+ incremental_field="startTime",+ db_incremental_field_last_value=last_value,+ )++ assert result.raised is None+ windows = sorted({(r.json()["props"]["fromEpoch"], r.json()["props"]["toEpoch"]) for r in result.requests[1:]})+ expected_start = (+ now - SEARCH_RETENTION_SECONDS + SEARCH_RETENTION_MARGIN_SECONDS+ if last_value is None+ else last_value - SEARCH_INCREMENTAL_OVERLAP_SECONDS+ )+ assert abs(windows[0][0] - expected_start) <= 5+ assert all(to - start < SEARCH_WINDOW_SECONDS for start, to in windows)+ assert all(later[0] == earlier[1] + 1 for earlier, later in zip(windows, windows[1:]))+ assert abs(windows[-1][1] - now) <= 5+++def test_windowed_search_skips_and_clamps_windows_past_retention() -> None:+ now = int(time.time())+ start = now - SEARCH_RETENTION_SECONDS - SEARCH_WINDOW_SECONDS - 100+ result = driver().run(+ "k8s_events",+ lambda request: ScriptedResponse(+ json={"data": {"clusters": [{"name": "a"}]}}+ if request.path.endswith("/clusters")+ else {"data": {"events": []}, "meta": {}}+ ),+ resume_state=KomodorResumeConfig(start=start, until=now),+ )++ assert result.raised is None+ starts = sorted({request.json()["props"]["fromEpoch"] for request in result.requests[1:]})+ floor = now - SEARCH_RETENTION_SECONDS + SEARCH_RETENTION_MARGIN_SECONDS+ assert floor <= starts[0] <= floor + 5+ assert starts[1:] == [start + n * SEARCH_WINDOW_SECONDS for n in range(2, 5)]+++def test_cost_allocation_stops_when_no_more_pages_despite_a_token() -> None:+ result = driver().run(+ "cost_allocation",+ [+ ScriptedResponse(+ json={"groupedCosts": [{"komodorServiceId": "s1"}], "paginationToken": "t1", "hasMorePages": True}+ ),+ ScriptedResponse(+ json={"groupedCosts": [{"komodorServiceId": "s2"}], "paginationToken": "t2", "hasMorePages": False}+ ),+ ],+ )++ assert result.raised is None+ assert result.rows == [{"komodorServiceId": "s1"}, {"komodorServiceId": "s2"}]+ assert result.params("paginationToken") == [None, "t1"]+ assert result.params("timeFrame") == ["Past_30_days", "Past_30_days"]+ assert result.params("groupBy") == ["komodorServiceName", "komodorServiceName"]+ assert result.committed_states == [KomodorResumeConfig(cursor="t1")]