Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add kisi roles and events tables (#122651)

Tom Owers8c5f17797f80
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index fb90738d0829..09b1a039a4dd 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -6871,17 +6871,17 @@ Note: Parsed the embedded OpenAPI document used by the official Swagger UI and e
## Kisi — gaps
-Today (7): `controllers`, `groups`, `locks`, `places`, `readers`, `role_assignments`, `users`
+Today (9): `controllers`, `events`, `groups`, `locks`, `places`, `readers`, `role_assignments`, `roles`, `users`
Diffed against: <https://api.getkisi.com/openapi/apps.yaml>
-- [ ] `Roles (GET /roles)` — resolve role_id in role assignments and audit granted permissions (high)
-- [ ] `Event sets (POST /event_sets; GET /event_sets/{id})` — analyze door access attempts, failures, and security activity missing from device and user inventories (high)
-- [ ] `Presences (GET /presences)` — measure attendance and occupancy from first and last daily unlock activity (high)
+- [x] `Roles (GET /roles)` — resolve role_id in role assignments and audit granted permissions (high) → `roles`
+- [x] `Event sets (POST /event_sets; GET /event_sets/{id})` — analyze door access attempts, failures, and security activity missing from device and user inventories (high) → `events` (incremental on `created_at`, read in 90-day event set windows)
+- [ ] `Presences (GET /presences)` — measure attendance and occupancy from first and last daily unlock activity (high); skipped: needs a `place_id` and a single `date` per request, and the first and last daily unlocks it reports can be derived from the `events` table
- [ ] `Group locks (GET /group_locks)` — connect access groups to doors and audit effective access rights (medium)
- [ ] `Team memberships (GET /team_memberships)` — expand access assigned to teams into individual users and guests (medium)
- [ ] `Teams (GET /teams)` — analyze teams used as role assignees and join them to membership records (medium)
-- [ ] `Members (GET /members)` — analyze organization membership, access status, and links between members and user accounts (medium)
+- [ ] `Members (GET /members)` — analyze organization membership, access status, and links between members and user accounts (medium); skipped: Kisi marks the endpoint deprecated, and `users` already carries the same access fields
Note: Parsed the official OpenAPI document linked by the vendor API reference, including nested response schemas. Event exports require POST /event_sets, polling, and cursor pagination; each requested interval is limited to 90 days. Presences require a place and date, while role assignments already cover direct access assignments. Deprecated shares, login endpoints, device wiring, and settings were excluded.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/canonical_descriptions.py
index 5cbe111e67f4..09cdf621a202 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/canonical_descriptions.py
@@ -82,4 +82,44 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"updated_at": "Time when the reader was last updated.",
},
},
+ "roles": {
+ "description": "Roles that can be assigned to users in Kisi, with the permissions each role grants.",
+ "docs_url": "https://api.getkisi.com/docs#tag/roles/GET/roles",
+ "columns": {
+ "id": "Unique role identifier, referenced by role assignments.",
+ "applies_to": "Scope the role applies to: Organization, Place or Group.",
+ "name": "Role name.",
+ "description": "Role description.",
+ "custom": "Whether the role is a custom role.",
+ "permissions": "Map of each permission to whether the role grants it.",
+ "custom_role_id": "Identifier of the custom role, for custom roles.",
+ },
+ },
+ "events": {
+ "description": "Access and audit events, such as unlock attempts, read through Kisi event sets.",
+ "docs_url": "https://api.getkisi.com/docs#tag/event-sets/POST/event_sets",
+ "columns": {
+ "uuid": "Unique event identifier.",
+ "type": "Event type.",
+ "action": "Action of the event.",
+ "success": "Whether the action was successful.",
+ "actor_type": "Type of the actor of the event, such as User or Guest.",
+ "actor_id": "Identifier of the actor of the event.",
+ "actor_name": "Name of the actor of the event.",
+ "actor_email": "Email of the actor of the event.",
+ "authenticated_by_type": "Type of the credential used, such as Card or Login.",
+ "authenticated_by_id": "Identifier of the credential used.",
+ "object_type": "Type of the object of the event, such as Lock.",
+ "object_id": "Identifier of the object of the event.",
+ "object_name": "Name of the object of the event.",
+ "message": "Text representation of the event.",
+ "code": "Code of the event.",
+ "sequence": "Sequence that groups related events of the same action.",
+ "organization_id": "Identifier of the organization of the event.",
+ "place_id": "Identifier of the place of the event.",
+ "error_code": "Error code of the event.",
+ "error_message": "Error message of the event.",
+ "created_at": "Time when the event was created.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/kisi.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/kisi.py
index 6994418191be..1ab65a95a186 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/kisi.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/kisi.py
@@ -1,11 +1,17 @@
import re
+from collections.abc import Iterator
+from datetime import UTC, datetime
from typing import Any
from requests import Response
from posthog.dataclasses import frozen
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.datetime_utils import parse_datetime_value
from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_session
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.interruptible_wait import (
+ interruptible_wait,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source import (
RESTAPIConfig,
rest_api_resource,
@@ -13,23 +19,50 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.auth import APIKeyAuth
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (
OffsetPaginator,
+ SinglePagePaginator,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceInputs, SourceResponse
from products.warehouse_sources.backend.temporal.data_imports.sources.kisi.settings import (
BASE_URL,
+ DEFAULT_PARTITION_KEYS,
+ DEFAULT_PRIMARY_KEYS,
ENDPOINTS,
+ EVENT_SET_CURSOR_ERROR,
+ EVENT_SET_EXPIRED_ERROR,
+ EVENT_SET_FAILED_ERROR,
+ EVENT_SET_TIMEOUT_ERROR,
+ EVENTS,
+ EVENTS_CREATE_INTERVAL_SECONDS,
+ EVENTS_INITIAL_LOOKBACK,
+ EVENTS_MAX_PAGES_PER_WINDOW,
+ EVENTS_POLL_INTERVAL_SECONDS,
+ EVENTS_POLL_MAX_ATTEMPTS,
+ EVENTS_WINDOW,
MAX_OFFSET,
OFFSET_ERROR,
PAGE_SIZE,
PARTITION_KEYS,
PRIMARY_KEYS,
+ UNPAGINATED_ENDPOINTS,
)
+HEADERS = {"Accept": "application/json", "Content-Type": "application/json"}
+REQUEST_TIMEOUT = (10, 30)
+
@frozen
class KisiResumeConfig:
- offset: int
+ offset: int = 0
+ events_window_start: str | None = None
+ events_window_end: str | None = None
+ event_set_id: int | None = None
+ events_cursor: str | None = None
+ events_written: int = 0
+
+
+class KisiEventSetError(Exception):
+ pass
class KisiPaginator(OffsetPaginator):
@@ -55,8 +88,8 @@ def validate_credentials(api_key: str) -> None:
response = session.get(
f"{BASE_URL}/user",
auth=kisi_auth(api_key),
- headers={"Accept": "application/json", "Content-Type": "application/json"},
- timeout=(10, 30),
+ headers=HEADERS,
+ timeout=REQUEST_TIMEOUT,
allow_redirects=False,
)
response.raise_for_status()
@@ -72,6 +105,24 @@ def kisi_source(
if inputs.schema_name not in ENDPOINTS:
raise ValueError(f"Unsupported Kisi table: {inputs.schema_name}")
+ primary_keys = PRIMARY_KEYS.get(inputs.schema_name, DEFAULT_PRIMARY_KEYS)
+ partition_keys = PARTITION_KEYS.get(inputs.schema_name, DEFAULT_PARTITION_KEYS)
+ partitioning: dict[str, Any] = (
+ {"partition_keys": partition_keys, "partition_mode": "datetime", "partition_format": "month"}
+ if partition_keys
+ else {}
+ )
+
+ if inputs.schema_name == EVENTS:
+ return SourceResponse(
+ name=inputs.schema_name,
+ items=lambda: get_events(api_key, inputs, resumable_source_manager),
+ primary_keys=primary_keys,
+ # Windows run oldest first, but Kisi does not document the order of events inside an event set.
+ sort_mode=None,
+ **partitioning,
+ )
+
resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
def save_checkpoint(state: dict[str, Any] | None) -> None:
@@ -82,9 +133,9 @@ def kisi_source(
"client": {
"base_url": BASE_URL,
"auth": kisi_auth(api_key),
- "headers": {"Accept": "application/json", "Content-Type": "application/json"},
- "paginator": KisiPaginator(),
- "request_timeout": (10, 30),
+ "headers": HEADERS,
+ "paginator": SinglePagePaginator() if inputs.schema_name in UNPAGINATED_ENDPOINTS else KisiPaginator(),
+ "request_timeout": REQUEST_TIMEOUT,
"allow_redirects": False,
},
"resources": [
@@ -106,9 +157,154 @@ def kisi_source(
return SourceResponse(
name=inputs.schema_name,
items=lambda: resource,
- primary_keys=PRIMARY_KEYS,
- partition_keys=PARTITION_KEYS,
- partition_mode="datetime",
- partition_format="month",
+ primary_keys=primary_keys,
sort_mode=None,
+ **partitioning,
+ )
+
+
+def _format_timestamp(value: datetime) -> str:
+ return value.astimezone(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def _create_event_set(session: Any, auth: APIKeyAuth, start: datetime, end: datetime) -> dict[str, Any]:
+ response = session.post(
+ f"{BASE_URL}/event_sets",
+ json={"event_set": {"interval": f"{_format_timestamp(start)}/{_format_timestamp(end)}"}},
+ auth=auth,
+ headers=HEADERS,
+ timeout=REQUEST_TIMEOUT,
+ allow_redirects=False,
+ )
+ response.raise_for_status()
+ return response.json()
+
+
+def _fetch_event_set(
+ session: Any, auth: APIKeyAuth, event_set_id: int, cursor: str | None = None
+) -> dict[str, Any] | None:
+ params: dict[str, Any] = {"limit": PAGE_SIZE}
+ if cursor:
+ params["cursor"] = cursor
+ response = session.get(
+ f"{BASE_URL}/event_sets/{event_set_id}",
+ params=params,
+ auth=auth,
+ headers=HEADERS,
+ timeout=REQUEST_TIMEOUT,
+ allow_redirects=False,
)
+ # Kisi keeps an event set for about 24 hours, so a set saved by an earlier attempt can be gone.
+ if response.status_code == 404:
+ return None
+ response.raise_for_status()
+ return response.json()
+
+
+def get_events(
+ api_key: str,
+ inputs: SourceInputs,
+ manager: ResumableSourceManager[KisiResumeConfig],
+) -> Iterator[list[dict[str, Any]]]:
+ auth = kisi_auth(api_key)
+ now = datetime.now(UTC)
+
+ resume = manager.load_state() if manager.can_resume() else None
+ resumed_start = parse_datetime_value(resume.events_window_start) if resume else None
+ resumed_end = parse_datetime_value(resume.events_window_end) if resume else None
+ event_set_id: int | None = None
+ cursor: str | None = None
+ # Events of the current window that earlier attempts already handed to the pipeline.
+ written = 0
+ if resume is not None and resumed_start is not None:
+ window_start = resumed_start
+ event_set_id, cursor, written = resume.event_set_id, resume.events_cursor, resume.events_written
+ # A window that has a set or written rows keeps its interval. A window with neither runs up to now.
+ keeps_interval = event_set_id is not None or written > 0
+ window_end = resumed_end if keeps_interval and resumed_end is not None else now
+ else:
+ watermark = (
+ parse_datetime_value(inputs.db_incremental_field_last_value)
+ if inputs.should_use_incremental_field
+ else None
+ )
+ window_start = watermark or now - EVENTS_INITIAL_LOOKBACK
+ window_end = now
+ window_end = min(window_end, window_start + EVENTS_WINDOW)
+
+ def stage(start: datetime, end: datetime, set_id: int | None, next_cursor: str | None, rows: int) -> None:
+ manager.save_state(
+ KisiResumeConfig(
+ events_window_start=_format_timestamp(start),
+ events_window_end=_format_timestamp(end),
+ event_set_id=set_id,
+ events_cursor=next_cursor,
+ events_written=rows,
+ )
+ )
+
+ created_any = False
+ with make_tracked_session(redact_values=(api_key,)) as session:
+ while window_start < window_end:
+ event_set = _fetch_event_set(session, auth, event_set_id, cursor) if event_set_id is not None else None
+ if event_set is None:
+ if created_any:
+ interruptible_wait(EVENTS_CREATE_INTERVAL_SECONDS, safe_point=manager.safe_point)
+ event_set = _create_event_set(session, auth, window_start, window_end)
+ created_any = True
+ cursor = None
+ stage(window_start, window_end, int(event_set["id"]), None, written)
+ manager.safe_point()
+ set_id = int(event_set["id"])
+
+ for _attempt in range(EVENTS_POLL_MAX_ATTEMPTS):
+ if event_set.get("status") != "in_progress":
+ break
+ interruptible_wait(EVENTS_POLL_INTERVAL_SECONDS, safe_point=manager.safe_point)
+ polled = _fetch_event_set(session, auth, set_id)
+ if polled is None:
+ raise KisiEventSetError(EVENT_SET_EXPIRED_ERROR)
+ event_set = polled
+ manager.safe_point()
+ if event_set.get("status") != "finished":
+ # A failed set fails on every fetch, so the next attempt builds the window again.
+ stage(window_start, window_end, None, None, written)
+ manager.safe_point()
+ raise KisiEventSetError(
+ EVENT_SET_FAILED_ERROR if event_set.get("status") == "failed" else EVENT_SET_TIMEOUT_ERROR
+ )
+
+ # A set read from its start replays rows an earlier set of this window already wrote. Kisi builds
+ # a set for a past interval from the same stored events, so the first `written` rows are skipped.
+ position = written if cursor else 0
+ next_window_start = window_end
+ next_window_end = min(next_window_start + EVENTS_WINDOW, now)
+ seen_cursors: set[str] = set()
+ while True:
+ page = event_set.get("events") or []
+ fresh = page[max(0, written - position) :]
+ position += len(page)
+ written += len(fresh)
+ next_cursor = event_set.get("cursor")
+ if not next_cursor:
+ stage(next_window_start, next_window_end, None, None, 0)
+ elif position >= written:
+ stage(window_start, window_end, set_id, next_cursor, written)
+ if fresh:
+ yield fresh
+ else:
+ manager.safe_point()
+ if not next_cursor:
+ break
+ if next_cursor in seen_cursors or len(seen_cursors) >= EVENTS_MAX_PAGES_PER_WINDOW:
+ raise KisiEventSetError(EVENT_SET_CURSOR_ERROR)
+ seen_cursors.add(next_cursor)
+ next_page = _fetch_event_set(session, auth, set_id, next_cursor)
+ if next_page is None:
+ # The staged cursor still names this set, so the next attempt builds the window again.
+ manager.safe_point()
+ raise KisiEventSetError(EVENT_SET_EXPIRED_ERROR)
+ event_set = next_page
+
+ window_start, window_end = next_window_start, next_window_end
+ event_set_id, cursor, written = None, None, 0
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/settings.py
index 287811d601d0..f33066ca29c6 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/settings.py
@@ -1,11 +1,50 @@
+from datetime import timedelta
+
+from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType
+
BASE_URL = "https://api.kisi.io"
PAGE_SIZE = 100
MAX_OFFSET = 20_000
-ENDPOINTS = ("locks", "places", "users", "groups", "role_assignments", "controllers", "readers")
-PRIMARY_KEYS = ["id"]
-PARTITION_KEYS = ["created_at"]
+EVENTS = "events"
+ROLES = "roles"
+
+ENDPOINTS = ("locks", "places", "users", "groups", "role_assignments", "controllers", "readers", ROLES, EVENTS)
+PRIMARY_KEYS: dict[str, list[str]] = {EVENTS: ["uuid"]}
+DEFAULT_PRIMARY_KEYS = ["id"]
+# Roles have no creation timestamp to partition on.
+PARTITION_KEYS: dict[str, list[str] | None] = {ROLES: None}
+DEFAULT_PARTITION_KEYS = ["created_at"]
+# Kisi returns every role in one response, with no pagination parameters.
+UNPAGINATED_ENDPOINTS = (ROLES,)
+
+INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {
+ EVENTS: [
+ IncrementalField(
+ label="created_at",
+ type=IncrementalFieldType.DateTime,
+ field="created_at",
+ field_type=IncrementalFieldType.DateTime,
+ )
+ ]
+}
+# Re-read the last hour on each incremental sync, so events that Kisi stores late still land.
+EVENTS_INCREMENTAL_LOOKBACK_SECONDS = 3600
+
+# Kisi rejects an event set interval wider than 90 days.
+EVENTS_WINDOW = timedelta(days=90)
+# Kisi keeps events according to the organization's retention policy. A window older than that is empty and costs one request.
+EVENTS_INITIAL_LOOKBACK = timedelta(days=730)
+EVENTS_POLL_INTERVAL_SECONDS = 2.0
+EVENTS_POLL_MAX_ATTEMPTS = 150
+# Kisi allows one event set creation per second.
+EVENTS_CREATE_INTERVAL_SECONDS = 1.0
+EVENTS_MAX_PAGES_PER_WINDOW = 100_000
AUTH_ERROR = "Kisi rejected the API key. Create a new key in My Account > API and reconnect."
PERMISSION_ERROR = "The Kisi API key cannot read this resource. Check the account permissions in Kisi."
OFFSET_ERROR = "Kisi's pagination limit prevents a complete import. Contact Kisi support for an export of this table."
+EVENT_SET_FAILED_ERROR = "Kisi could not prepare the events for export. Try the sync again later."
+EVENT_SET_TIMEOUT_ERROR = "Kisi did not finish preparing the events for export in time."
+EVENT_SET_EXPIRED_ERROR = "Kisi expired the event export before the sync read it."
+EVENT_SET_CURSOR_ERROR = "Kisi repeated an event page cursor, so the events sync stopped to avoid a loop."
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/source.py
index 308befd71ddb..7294eebaf6d3 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/source.py
@@ -33,6 +33,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.kisi.kisi
from products.warehouse_sources.backend.temporal.data_imports.sources.kisi.settings import (
AUTH_ERROR,
ENDPOINTS,
+ EVENTS,
+ EVENTS_INCREMENTAL_LOOKBACK_SECONDS,
+ INCREMENTAL_FIELDS,
OFFSET_ERROR,
PERMISSION_ERROR,
)
@@ -94,7 +97,12 @@ class KisiSource(ResumableSource[KisiSourceConfig, KisiResumeConfig]):
force_refresh: bool = False,
api_version: str | None = None,
) -> list[SourceSchema]:
- return build_endpoint_schemas(ENDPOINTS, {}, names)
+ return build_endpoint_schemas(
+ ENDPOINTS,
+ INCREMENTAL_FIELDS,
+ names,
+ default_incremental_lookback_seconds={EVENTS: EVENTS_INCREMENTAL_LOOKBACK_SECONDS},
+ )
def validate_credentials(
self,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/tests/test_kisi.py b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/tests/test_kisi.py
index ae0523eb5dbf..04234fa59a4b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/tests/test_kisi.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/kisi/tests/test_kisi.py
@@ -1,3 +1,5 @@
+from datetime import UTC, datetime, timedelta
+
import pytest
from requests.exceptions import ConnectionError, HTTPError
@@ -9,9 +11,15 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.tes
scripted_network,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.kisi import KisiSourceConfig
-from products.warehouse_sources.backend.temporal.data_imports.sources.kisi.kisi import KisiResumeConfig
+from products.warehouse_sources.backend.temporal.data_imports.sources.kisi.kisi import (
+ KisiEventSetError,
+ KisiResumeConfig,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.kisi.settings import (
AUTH_ERROR,
+ EVENT_SET_CURSOR_ERROR,
+ EVENT_SET_FAILED_ERROR,
+ EVENTS_WINDOW,
OFFSET_ERROR,
PERMISSION_ERROR,
)
@@ -127,8 +135,117 @@ def test_validate_connection_error() -> None:
def test_unsupported_table(driver: SourceDriver) -> None:
- result = driver.run("events", [])
+ result = driver.run("presences", [])
assert isinstance(result.raised, ValueError)
assert "Unsupported Kisi table" in str(result.raised)
assert result.requests == []
+
+
+def _timestamp(value: datetime) -> str:
+ return value.strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def _event_set(set_id: int, status: str, events: list[dict[str, object]], cursor: str | None) -> ScriptedResponse:
+ return ScriptedResponse(json={"id": set_id, "status": status, "events": events, "cursor": cursor})
+
+
+def test_roles_are_read_in_one_request(driver: SourceDriver) -> None:
+ roles: list[dict[str, object]] = [{"id": f"role-{i}", "name": f"Role {i}"} for i in range(100)]
+ result = driver.run("roles", [ScriptedResponse(json=roles)])
+
+ assert result.raised is None
+ assert result.rows == roles
+ assert result.paths == ["/roles"]
+ assert result.params("offset") == [None]
+
+
+def test_events_walk_windows_from_the_watermark(driver: SourceDriver) -> None:
+ watermark = (datetime.now(UTC) - timedelta(days=100)).replace(microsecond=0)
+ first_page: list[dict[str, object]] = [{"uuid": "a"}, {"uuid": "b"}]
+ second_page: list[dict[str, object]] = [{"uuid": "c"}]
+ result = driver.run(
+ "events",
+ [
+ _event_set(1, "in_progress", [], None),
+ _event_set(1, "finished", first_page, "page-2"),
+ _event_set(1, "finished", second_page, None),
+ _event_set(2, "finished", [], None),
+ ],
+ incremental_field="created_at",
+ db_incremental_field_last_value=watermark.isoformat(),
+ )
+
+ assert result.raised is None
+ assert result.rows == first_page + second_page
+ assert [(request.method, request.path) for request in result.requests] == [
+ ("POST", "/event_sets"),
+ ("GET", "/event_sets/1"),
+ ("GET", "/event_sets/1"),
+ ("POST", "/event_sets"),
+ ]
+ assert result.params("cursor") == [None, None, "page-2", None]
+ intervals = [request.json()["event_set"]["interval"] for request in result.requests if request.method == "POST"]
+ assert intervals[0] == f"{_timestamp(watermark)}/{_timestamp(watermark + EVENTS_WINDOW)}"
+ assert intervals[1].startswith(f"{_timestamp(watermark + EVENTS_WINDOW)}/")
+ assert (
+ KisiResumeConfig(
+ events_window_start=_timestamp(watermark),
+ events_window_end=_timestamp(watermark + EVENTS_WINDOW),
+ event_set_id=1,
+ events_cursor="page-2",
+ events_written=2,
+ )
+ in result.committed_states
+ )
+
+
[email protected]("expired", [False, True])
+def test_events_resume_from_saved_event_set(expired: bool, driver: SourceDriver) -> None:
+ window_start = (datetime.now(UTC) - timedelta(days=10)).replace(microsecond=0)
+ window_end = window_start + timedelta(days=5)
+ pages = [_event_set(7, "finished", [{"uuid": "b"}], None)]
+ if expired:
+ pages = [
+ ScriptedResponse(status=404, json={"error": "Not found"}),
+ _event_set(8, "finished", [{"uuid": "a"}, {"uuid": "b"}], None),
+ ]
+ result = driver.run(
+ "events",
+ [*pages, _event_set(9, "finished", [], None)],
+ resume_state=KisiResumeConfig(
+ events_window_start=_timestamp(window_start),
+ events_window_end=_timestamp(window_end),
+ event_set_id=7,
+ events_cursor="page-3",
+ events_written=1,
+ ),
+ )
+
+ assert result.raised is None
+ assert result.rows == [{"uuid": "b"}]
+ assert result.requests[0].path == "/event_sets/7"
+ assert result.requests[0].param("cursor") == "page-3"
+ posts = [request.json()["event_set"]["interval"] for request in result.requests if request.method == "POST"]
+ expected_posts = [f"{_timestamp(window_start)}/{_timestamp(window_end)}"] if expired else []
+ assert posts[: len(expected_posts)] == expected_posts
+ assert posts[len(expected_posts)].startswith(f"{_timestamp(window_end)}/")
+
+
+ "pages,message",
+ [
+ ([_event_set(1, "in_progress", [], None), _event_set(1, "failed", [], None)], EVENT_SET_FAILED_ERROR),
+ (
+ [_event_set(1, "finished", [{"uuid": "a"}], "same"), _event_set(1, "finished", [], "same")],
+ EVENT_SET_CURSOR_ERROR,
+ ),
+ ],
+)
+def test_events_errors_stop_the_window(pages: list[ScriptedResponse], message: str, driver: SourceDriver) -> None:
+ result = driver.run("events", pages)
+
+ assert isinstance(result.raised, KisiEventSetError)
+ assert str(result.raised) == message
+ if message == EVENT_SET_FAILED_ERROR:
+ assert result.final_state.event_set_id is None