:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add kisi roles and events tables (#122651)
Tom Owers8c5f17797f80
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex fb90738d0829..09b1a039a4dd 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -6871,17 +6871,17 @@ Note: Parsed the embedded OpenAPI document used by the official Swagger UI and e## Kisi — gaps-Today (7): `controllers`, `groups`, `locks`, `places`, `readers`, `role_assignments`, `users`+Today (9): `controllers`, `events`, `groups`, `locks`, `places`, `readers`, `role_assignments`, `roles`, `users`Diffed against: <https://api.getkisi.com/openapi/apps.yaml>-- [ ] `Roles (GET /roles)` — resolve role_id in role assignments and audit granted permissions (high)-- [ ] `Event sets (POST /event_sets; GET /event_sets/{id})` — analyze door access attempts, failures, and security activity missing from device and user inventories (high)-- [ ] `Presences (GET /presences)` — measure attendance and occupancy from first and last daily unlock activity (high)+- [x] `Roles (GET /roles)` — resolve role_id in role assignments and audit granted permissions (high) → `roles`+- [x] `Event sets (POST /event_sets; GET /event_sets/{id})` — analyze door access attempts, failures, and security activity missing from device and user inventories (high) → `events` (incremental on `created_at`, read in 90-day event set windows)+- [ ] `Presences (GET /presences)` — measure attendance and occupancy from first and last daily unlock activity (high); skipped: needs a `place_id` and a single `date` per request, and the first and last daily unlocks it reports can be derived from the `events` table- [ ] `Group locks (GET /group_locks)` — connect access groups to doors and audit effective access rights (medium)- [ ] `Team memberships (GET /team_memberships)` — expand access assigned to teams into individual users and guests (medium)- [ ] `Teams (GET /teams)` — analyze teams used as role assignees and join them to membership records (medium)-- [ ] `Members (GET /members)` — analyze organization membership, access status, and links between members and user accounts (medium)+- [ ] `Members (GET /members)` — analyze organization membership, access status, and links between members and user accounts (medium); skipped: Kisi marks the endpoint deprecated, and `users` already carries the same access fieldsNote: Parsed the official OpenAPI document linked by the vendor API reference, including nested response schemas. Event exports require POST /event_sets, polling, and cursor pagination; each requested interval is limited to 90 days. Presences require a place and date, while role assignments already cover direct access assignments. Deprecated shares, login endpoints, device wiring, and settings were excluded.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/canonical_descriptions.pyindex 5cbe111e67f4..09cdf621a202 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/canonical_descriptions.py@@ -82,4 +82,44 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"updated_at": "Time when the reader was last updated.",},},+ "roles": {+ "description": "Roles that can be assigned to users in Kisi, with the permissions each role grants.",+ "docs_url": "https://api.getkisi.com/docs#tag/roles/GET/roles",+ "columns": {+ "id": "Unique role identifier, referenced by role assignments.",+ "applies_to": "Scope the role applies to: Organization, Place or Group.",+ "name": "Role name.",+ "description": "Role description.",+ "custom": "Whether the role is a custom role.",+ "permissions": "Map of each permission to whether the role grants it.",+ "custom_role_id": "Identifier of the custom role, for custom roles.",+ },+ },+ "events": {+ "description": "Access and audit events, such as unlock attempts, read through Kisi event sets.",+ "docs_url": "https://api.getkisi.com/docs#tag/event-sets/POST/event_sets",+ "columns": {+ "uuid": "Unique event identifier.",+ "type": "Event type.",+ "action": "Action of the event.",+ "success": "Whether the action was successful.",+ "actor_type": "Type of the actor of the event, such as User or Guest.",+ "actor_id": "Identifier of the actor of the event.",+ "actor_name": "Name of the actor of the event.",+ "actor_email": "Email of the actor of the event.",+ "authenticated_by_type": "Type of the credential used, such as Card or Login.",+ "authenticated_by_id": "Identifier of the credential used.",+ "object_type": "Type of the object of the event, such as Lock.",+ "object_id": "Identifier of the object of the event.",+ "object_name": "Name of the object of the event.",+ "message": "Text representation of the event.",+ "code": "Code of the event.",+ "sequence": "Sequence that groups related events of the same action.",+ "organization_id": "Identifier of the organization of the event.",+ "place_id": "Identifier of the place of the event.",+ "error_code": "Error code of the event.",+ "error_message": "Error message of the event.",+ "created_at": "Time when the event was created.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/kisi.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/kisi.pyindex 6994418191be..1ab65a95a186 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/kisi.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/kisi.py@@ -1,11 +1,17 @@import re+from collections.abc import Iterator+from datetime import UTC, datetimefrom typing import Anyfrom requests import Responsefrom posthog.dataclasses import frozen+from products.warehouse_sources.backend.temporal.data_imports.sources.common.datetime_utils import parse_datetime_valuefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session+from products.warehouse_sources.backend.temporal.data_imports.sources.common.interruptible_wait import (+ interruptible_wait,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source import (RESTAPIConfig,rest_api_resource,@@ -13,23 +19,50 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.auth import APIKeyAuthfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (OffsetPaginator,+ SinglePagePaginator,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceInputs, SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.kisi.settings import (BASE_URL,+ DEFAULT_PARTITION_KEYS,+ DEFAULT_PRIMARY_KEYS,ENDPOINTS,+ EVENT_SET_CURSOR_ERROR,+ EVENT_SET_EXPIRED_ERROR,+ EVENT_SET_FAILED_ERROR,+ EVENT_SET_TIMEOUT_ERROR,+ EVENTS,+ EVENTS_CREATE_INTERVAL_SECONDS,+ EVENTS_INITIAL_LOOKBACK,+ EVENTS_MAX_PAGES_PER_WINDOW,+ EVENTS_POLL_INTERVAL_SECONDS,+ EVENTS_POLL_MAX_ATTEMPTS,+ EVENTS_WINDOW,MAX_OFFSET,OFFSET_ERROR,PAGE_SIZE,PARTITION_KEYS,PRIMARY_KEYS,+ UNPAGINATED_ENDPOINTS,)+HEADERS = {"Accept": "application/json", "Content-Type": "application/json"}+REQUEST_TIMEOUT = (10, 30)+@frozenclass KisiResumeConfig:- offset: int+ offset: int = 0+ events_window_start: str | None = None+ events_window_end: str | None = None+ event_set_id: int | None = None+ events_cursor: str | None = None+ events_written: int = 0+++class KisiEventSetError(Exception):+ passclass KisiPaginator(OffsetPaginator):@@ -55,8 +88,8 @@ def validate_credentials(api_key: str) -> None:response = session.get(f"{BASE_URL}/user",auth=kisi_auth(api_key),- headers={"Accept": "application/json", "Content-Type": "application/json"},- timeout=(10, 30),+ headers=HEADERS,+ timeout=REQUEST_TIMEOUT,allow_redirects=False,)response.raise_for_status()@@ -72,6 +105,24 @@ def kisi_source(if inputs.schema_name not in ENDPOINTS:raise ValueError(f"Unsupported Kisi table: {inputs.schema_name}")+ primary_keys = PRIMARY_KEYS.get(inputs.schema_name, DEFAULT_PRIMARY_KEYS)+ partition_keys = PARTITION_KEYS.get(inputs.schema_name, DEFAULT_PARTITION_KEYS)+ partitioning: dict[str, Any] = (+ {"partition_keys": partition_keys, "partition_mode": "datetime", "partition_format": "month"}+ if partition_keys+ else {}+ )++ if inputs.schema_name == EVENTS:+ return SourceResponse(+ name=inputs.schema_name,+ items=lambda: get_events(api_key, inputs, resumable_source_manager),+ primary_keys=primary_keys,+ # Windows run oldest first, but Kisi does not document the order of events inside an event set.+ sort_mode=None,+ **partitioning,+ )+resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else Nonedef save_checkpoint(state: dict[str, Any] | None) -> None:@@ -82,9 +133,9 @@ def kisi_source("client": {"base_url": BASE_URL,"auth": kisi_auth(api_key),- "headers": {"Accept": "application/json", "Content-Type": "application/json"},- "paginator": KisiPaginator(),- "request_timeout": (10, 30),+ "headers": HEADERS,+ "paginator": SinglePagePaginator() if inputs.schema_name in UNPAGINATED_ENDPOINTS else KisiPaginator(),+ "request_timeout": REQUEST_TIMEOUT,"allow_redirects": False,},"resources": [@@ -106,9 +157,154 @@ def kisi_source(return SourceResponse(name=inputs.schema_name,items=lambda: resource,- primary_keys=PRIMARY_KEYS,- partition_keys=PARTITION_KEYS,- partition_mode="datetime",- partition_format="month",+ primary_keys=primary_keys,sort_mode=None,+ **partitioning,+ )+++def _format_timestamp(value: datetime) -> str:+ return value.astimezone(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")+++def _create_event_set(session: Any, auth: APIKeyAuth, start: datetime, end: datetime) -> dict[str, Any]:+ response = session.post(+ f"{BASE_URL}/event_sets",+ json={"event_set": {"interval": f"{_format_timestamp(start)}/{_format_timestamp(end)}"}},+ auth=auth,+ headers=HEADERS,+ timeout=REQUEST_TIMEOUT,+ allow_redirects=False,+ )+ response.raise_for_status()+ return response.json()+++def _fetch_event_set(+ session: Any, auth: APIKeyAuth, event_set_id: int, cursor: str | None = None+) -> dict[str, Any] | None:+ params: dict[str, Any] = {"limit": PAGE_SIZE}+ if cursor:+ params["cursor"] = cursor+ response = session.get(+ f"{BASE_URL}/event_sets/{event_set_id}",+ params=params,+ auth=auth,+ headers=HEADERS,+ timeout=REQUEST_TIMEOUT,+ allow_redirects=False,)+ # Kisi keeps an event set for about 24 hours, so a set saved by an earlier attempt can be gone.+ if response.status_code == 404:+ return None+ response.raise_for_status()+ return response.json()+++def get_events(+ api_key: str,+ inputs: SourceInputs,+ manager: ResumableSourceManager[KisiResumeConfig],+) -> Iterator[list[dict[str, Any]]]:+ auth = kisi_auth(api_key)+ now = datetime.now(UTC)++ resume = manager.load_state() if manager.can_resume() else None+ resumed_start = parse_datetime_value(resume.events_window_start) if resume else None+ resumed_end = parse_datetime_value(resume.events_window_end) if resume else None+ event_set_id: int | None = None+ cursor: str | None = None+ # Events of the current window that earlier attempts already handed to the pipeline.+ written = 0+ if resume is not None and resumed_start is not None:+ window_start = resumed_start+ event_set_id, cursor, written = resume.event_set_id, resume.events_cursor, resume.events_written+ # A window that has a set or written rows keeps its interval. A window with neither runs up to now.+ keeps_interval = event_set_id is not None or written > 0+ window_end = resumed_end if keeps_interval and resumed_end is not None else now+ else:+ watermark = (+ parse_datetime_value(inputs.db_incremental_field_last_value)+ if inputs.should_use_incremental_field+ else None+ )+ window_start = watermark or now - EVENTS_INITIAL_LOOKBACK+ window_end = now+ window_end = min(window_end, window_start + EVENTS_WINDOW)++ def stage(start: datetime, end: datetime, set_id: int | None, next_cursor: str | None, rows: int) -> None:+ manager.save_state(+ KisiResumeConfig(+ events_window_start=_format_timestamp(start),+ events_window_end=_format_timestamp(end),+ event_set_id=set_id,+ events_cursor=next_cursor,+ events_written=rows,+ )+ )++ created_any = False+ with make_tracked_session(redact_values=(api_key,)) as session:+ while window_start < window_end:+ event_set = _fetch_event_set(session, auth, event_set_id, cursor) if event_set_id is not None else None+ if event_set is None:+ if created_any:+ interruptible_wait(EVENTS_CREATE_INTERVAL_SECONDS, safe_point=manager.safe_point)+ event_set = _create_event_set(session, auth, window_start, window_end)+ created_any = True+ cursor = None+ stage(window_start, window_end, int(event_set["id"]), None, written)+ manager.safe_point()+ set_id = int(event_set["id"])++ for _attempt in range(EVENTS_POLL_MAX_ATTEMPTS):+ if event_set.get("status") != "in_progress":+ break+ interruptible_wait(EVENTS_POLL_INTERVAL_SECONDS, safe_point=manager.safe_point)+ polled = _fetch_event_set(session, auth, set_id)+ if polled is None:+ raise KisiEventSetError(EVENT_SET_EXPIRED_ERROR)+ event_set = polled+ manager.safe_point()+ if event_set.get("status") != "finished":+ # A failed set fails on every fetch, so the next attempt builds the window again.+ stage(window_start, window_end, None, None, written)+ manager.safe_point()+ raise KisiEventSetError(+ EVENT_SET_FAILED_ERROR if event_set.get("status") == "failed" else EVENT_SET_TIMEOUT_ERROR+ )++ # A set read from its start replays rows an earlier set of this window already wrote. Kisi builds+ # a set for a past interval from the same stored events, so the first `written` rows are skipped.+ position = written if cursor else 0+ next_window_start = window_end+ next_window_end = min(next_window_start + EVENTS_WINDOW, now)+ seen_cursors: set[str] = set()+ while True:+ page = event_set.get("events") or []+ fresh = page[max(0, written - position) :]+ position += len(page)+ written += len(fresh)+ next_cursor = event_set.get("cursor")+ if not next_cursor:+ stage(next_window_start, next_window_end, None, None, 0)+ elif position >= written:+ stage(window_start, window_end, set_id, next_cursor, written)+ if fresh:+ yield fresh+ else:+ manager.safe_point()+ if not next_cursor:+ break+ if next_cursor in seen_cursors or len(seen_cursors) >= EVENTS_MAX_PAGES_PER_WINDOW:+ raise KisiEventSetError(EVENT_SET_CURSOR_ERROR)+ seen_cursors.add(next_cursor)+ next_page = _fetch_event_set(session, auth, set_id, next_cursor)+ if next_page is None:+ # The staged cursor still names this set, so the next attempt builds the window again.+ manager.safe_point()+ raise KisiEventSetError(EVENT_SET_EXPIRED_ERROR)+ event_set = next_page++ window_start, window_end = next_window_start, next_window_end+ event_set_id, cursor, written = None, None, 0diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/settings.pyindex 287811d601d0..f33066ca29c6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/settings.py@@ -1,11 +1,50 @@+from datetime import timedelta++from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType+BASE_URL = "https://api.kisi.io"PAGE_SIZE = 100MAX_OFFSET = 20_000-ENDPOINTS = ("locks", "places", "users", "groups", "role_assignments", "controllers", "readers")-PRIMARY_KEYS = ["id"]-PARTITION_KEYS = ["created_at"]+EVENTS = "events"+ROLES = "roles"++ENDPOINTS = ("locks", "places", "users", "groups", "role_assignments", "controllers", "readers", ROLES, EVENTS)+PRIMARY_KEYS: dict[str, list[str]] = {EVENTS: ["uuid"]}+DEFAULT_PRIMARY_KEYS = ["id"]+# Roles have no creation timestamp to partition on.+PARTITION_KEYS: dict[str, list[str] | None] = {ROLES: None}+DEFAULT_PARTITION_KEYS = ["created_at"]+# Kisi returns every role in one response, with no pagination parameters.+UNPAGINATED_ENDPOINTS = (ROLES,)++INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {+ EVENTS: [+ IncrementalField(+ label="created_at",+ type=IncrementalFieldType.DateTime,+ field="created_at",+ field_type=IncrementalFieldType.DateTime,+ )+ ]+}+# Re-read the last hour on each incremental sync, so events that Kisi stores late still land.+EVENTS_INCREMENTAL_LOOKBACK_SECONDS = 3600++# Kisi rejects an event set interval wider than 90 days.+EVENTS_WINDOW = timedelta(days=90)+# Kisi keeps events according to the organization's retention policy. A window older than that is empty and costs one request.+EVENTS_INITIAL_LOOKBACK = timedelta(days=730)+EVENTS_POLL_INTERVAL_SECONDS = 2.0+EVENTS_POLL_MAX_ATTEMPTS = 150+# Kisi allows one event set creation per second.+EVENTS_CREATE_INTERVAL_SECONDS = 1.0+EVENTS_MAX_PAGES_PER_WINDOW = 100_000AUTH_ERROR = "Kisi rejected the API key. Create a new key in My Account > API and reconnect."PERMISSION_ERROR = "The Kisi API key cannot read this resource. Check the account permissions in Kisi."OFFSET_ERROR = "Kisi's pagination limit prevents a complete import. Contact Kisi support for an export of this table."+EVENT_SET_FAILED_ERROR = "Kisi could not prepare the events for export. Try the sync again later."+EVENT_SET_TIMEOUT_ERROR = "Kisi did not finish preparing the events for export in time."+EVENT_SET_EXPIRED_ERROR = "Kisi expired the event export before the sync read it."+EVENT_SET_CURSOR_ERROR = "Kisi repeated an event page cursor, so the events sync stopped to avoid a loop."diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/source.pyindex 308befd71ddb..7294eebaf6d3 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/source.py@@ -33,6 +33,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.kisi.kisifrom products.warehouse_sources.backend.temporal.data_imports.sources.kisi.settings import (AUTH_ERROR,ENDPOINTS,+ EVENTS,+ EVENTS_INCREMENTAL_LOOKBACK_SECONDS,+ INCREMENTAL_FIELDS,OFFSET_ERROR,PERMISSION_ERROR,)@@ -94,7 +97,12 @@ class KisiSource(ResumableSource[KisiSourceConfig, KisiResumeConfig]):force_refresh: bool = False,api_version: str | None = None,) -> list[SourceSchema]:- return build_endpoint_schemas(ENDPOINTS, {}, names)+ return build_endpoint_schemas(+ ENDPOINTS,+ INCREMENTAL_FIELDS,+ names,+ default_incremental_lookback_seconds={EVENTS: EVENTS_INCREMENTAL_LOOKBACK_SECONDS},+ )def validate_credentials(self,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/tests/test_kisi.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/tests/test_kisi.pyindex ae0523eb5dbf..04234fa59a4b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/tests/test_kisi.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/tests/test_kisi.py@@ -1,3 +1,5 @@+from datetime import UTC, datetime, timedelta+import pytestfrom requests.exceptions import ConnectionError, HTTPError@@ -9,9 +11,15 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.tesscripted_network,)from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.kisi import KisiSourceConfig-from products.warehouse_sources.backend.temporal.data_imports.sources.kisi.kisi import KisiResumeConfig+from products.warehouse_sources.backend.temporal.data_imports.sources.kisi.kisi import (+ KisiEventSetError,+ KisiResumeConfig,+)from products.warehouse_sources.backend.temporal.data_imports.sources.kisi.settings import (AUTH_ERROR,+ EVENT_SET_CURSOR_ERROR,+ EVENT_SET_FAILED_ERROR,+ EVENTS_WINDOW,OFFSET_ERROR,PERMISSION_ERROR,)@@ -127,8 +135,117 @@ def test_validate_connection_error() -> None:def test_unsupported_table(driver: SourceDriver) -> None:- result = driver.run("events", [])+ result = driver.run("presences", [])assert isinstance(result.raised, ValueError)assert "Unsupported Kisi table" in str(result.raised)assert result.requests == []+++def _timestamp(value: datetime) -> str:+ return value.strftime("%Y-%m-%dT%H:%M:%SZ")+++def _event_set(set_id: int, status: str, events: list[dict[str, object]], cursor: str | None) -> ScriptedResponse:+ return ScriptedResponse(json={"id": set_id, "status": status, "events": events, "cursor": cursor})+++def test_roles_are_read_in_one_request(driver: SourceDriver) -> None:+ roles: list[dict[str, object]] = [{"id": f"role-{i}", "name": f"Role {i}"} for i in range(100)]+ result = driver.run("roles", [ScriptedResponse(json=roles)])++ assert result.raised is None+ assert result.rows == roles+ assert result.paths == ["/roles"]+ assert result.params("offset") == [None]+++def test_events_walk_windows_from_the_watermark(driver: SourceDriver) -> None:+ watermark = (datetime.now(UTC) - timedelta(days=100)).replace(microsecond=0)+ first_page: list[dict[str, object]] = [{"uuid": "a"}, {"uuid": "b"}]+ second_page: list[dict[str, object]] = [{"uuid": "c"}]+ result = driver.run(+ "events",+ [+ _event_set(1, "in_progress", [], None),+ _event_set(1, "finished", first_page, "page-2"),+ _event_set(1, "finished", second_page, None),+ _event_set(2, "finished", [], None),+ ],+ incremental_field="created_at",+ db_incremental_field_last_value=watermark.isoformat(),+ )++ assert result.raised is None+ assert result.rows == first_page + second_page+ assert [(request.method, request.path) for request in result.requests] == [+ ("POST", "/event_sets"),+ ("GET", "/event_sets/1"),+ ("GET", "/event_sets/1"),+ ("POST", "/event_sets"),+ ]+ assert result.params("cursor") == [None, None, "page-2", None]+ intervals = [request.json()["event_set"]["interval"] for request in result.requests if request.method == "POST"]+ assert intervals[0] == f"{_timestamp(watermark)}/{_timestamp(watermark + EVENTS_WINDOW)}"+ assert intervals[1].startswith(f"{_timestamp(watermark + EVENTS_WINDOW)}/")+ assert (+ KisiResumeConfig(+ events_window_start=_timestamp(watermark),+ events_window_end=_timestamp(watermark + EVENTS_WINDOW),+ event_set_id=1,+ events_cursor="page-2",+ events_written=2,+ )+ in result.committed_states+ )++[email protected]("expired", [False, True])+def test_events_resume_from_saved_event_set(expired: bool, driver: SourceDriver) -> None:+ window_start = (datetime.now(UTC) - timedelta(days=10)).replace(microsecond=0)+ window_end = window_start + timedelta(days=5)+ pages = [_event_set(7, "finished", [{"uuid": "b"}], None)]+ if expired:+ pages = [+ ScriptedResponse(status=404, json={"error": "Not found"}),+ _event_set(8, "finished", [{"uuid": "a"}, {"uuid": "b"}], None),+ ]+ result = driver.run(+ "events",+ [*pages, _event_set(9, "finished", [], None)],+ resume_state=KisiResumeConfig(+ events_window_start=_timestamp(window_start),+ events_window_end=_timestamp(window_end),+ event_set_id=7,+ events_cursor="page-3",+ events_written=1,+ ),+ )++ assert result.raised is None+ assert result.rows == [{"uuid": "b"}]+ assert result.requests[0].path == "/event_sets/7"+ assert result.requests[0].param("cursor") == "page-3"+ posts = [request.json()["event_set"]["interval"] for request in result.requests if request.method == "POST"]+ expected_posts = [f"{_timestamp(window_start)}/{_timestamp(window_end)}"] if expired else []+ assert posts[: len(expected_posts)] == expected_posts+ assert posts[len(expected_posts)].startswith(f"{_timestamp(window_end)}/")+++ "pages,message",+ [+ ([_event_set(1, "in_progress", [], None), _event_set(1, "failed", [], None)], EVENT_SET_FAILED_ERROR),+ (+ [_event_set(1, "finished", [{"uuid": "a"}], "same"), _event_set(1, "finished", [], "same")],+ EVENT_SET_CURSOR_ERROR,+ ),+ ],+)+def test_events_errors_stop_the_window(pages: list[ScriptedResponse], message: str, driver: SourceDriver) -> None:+ result = driver.run("events", pages)++ assert isinstance(result.raised, KisiEventSetError)+ assert str(result.raised) == message+ if message == EVENT_SET_FAILED_ERROR:+ assert result.final_state.event_set_id is None