:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add prefect cloud logs, events, artifacts and state tables (#126785)
Tom Owersedd20ee71104
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex aaaa194689ba..5a9ae7a4a16b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -9690,14 +9690,14 @@ Note: The Microsoft SDK Swagger spec lists tenant inventory and nested admin GET## PrefectCloud — gaps-Today (6): `deployments`, `flow_runs`, `flows`, `task_runs`, `work_pools`, `work_queues`+Today (11): `artifacts`, `deployments`, `events`, `flow_run_states`, `flow_runs`, `flows`, `logs`, `task_run_states`, `task_runs`, `work_pools`, `work_queues`Diffed against: <https://api.prefect.cloud/api/openapi.json>-- [ ] `POST /events/filter` — the workspace event feed — every state change, deployment and automation trigger; the richest analytical stream Prefect has (high)-- [ ] `POST /logs/filter` — run logs, needed to diagnose failures alongside the flow_runs already synced (high)-- [ ] `POST /artifacts/filter` — artifacts are how flows publish metrics, tables and links; the actual outputs of the runs already synced (high)-- [ ] `GET /flow_run_states/ and GET /task_run_states/` — state transition history — queue time, retry count and time-in-state cannot be derived from the terminal state on the run row (high)+- [x] `POST /events/filter` — the workspace event feed — every state change, deployment and automation trigger; the richest analytical stream Prefect has (high). Added as `events`.+- [x] `POST /logs/filter` — run logs, needed to diagnose failures alongside the flow_runs already synced (high). Added as `logs`.+- [x] `POST /artifacts/filter` — artifacts are how flows publish metrics, tables and links; the actual outputs of the runs already synced (high). Added as `artifacts`.+- [x] `GET /flow_run_states/ and GET /task_run_states/` — state transition history — queue time, retry count and time-in-state cannot be derived from the terminal state on the run row (high). Added as `flow_run_states` and `task_run_states` (fan-out over `flow_runs` and `task_runs`).- [ ] `GET /deployments/{id}/schedules` — lookup for the schedules driving each deployment, needed to measure lateness against intent (medium)- [ ] `POST /automations/filter` — automation definitions that explain why runs were triggered or cancelled (medium)- [ ] `POST /work_pools/{work_pool_name}/workers/filter` — worker inventory and heartbeats per pool, for capacity analysis against the pools already synced (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/canonical_descriptions.pyindex fa892d0b534b..85125e873c0a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/canonical_descriptions.py@@ -137,4 +137,80 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"last_polled": "The last time a worker polled this queue for work.",},},+ "artifacts": {+ "description": "An artifact a flow or task run published, such as a table, markdown report, link, or result.",+ "docs_url": "https://docs.prefect.io/v3/api-ref/rest-api/server/artifacts",+ "columns": {+ "id": "Unique identifier for the artifact.",+ "created": "When the artifact was created.",+ "updated": "When the artifact was last updated.",+ "key": "An optional unique reference key for the artifact.",+ "type": "The shape of the data field, e.g. result, table, markdown, link, or progress.",+ "description": "A markdown-enabled description of the artifact.",+ "data": "Data associated with the artifact. Its structure depends on the artifact type.",+ "metadata_": "User-defined artifact metadata.",+ "flow_run_id": "The flow run that published the artifact.",+ "task_run_id": "The task run that published the artifact.",+ },+ },+ "logs": {+ "description": "A log line emitted by a flow run, task run, or worker.",+ "docs_url": "https://docs.prefect.io/v3/api-ref/rest-api/server/logs",+ "columns": {+ "id": "Unique identifier for the log.",+ "created": "When the log record was created.",+ "updated": "When the log record was last updated.",+ "name": "The logger name.",+ "level": "The numeric log level (e.g. 20 for INFO, 40 for ERROR).",+ "message": "The log message.",+ "timestamp": "When the log was emitted.",+ "flow_run_id": "The flow run the log belongs to.",+ "task_run_id": "The task run the log belongs to.",+ "worker_id": "The worker that emitted the log.",+ },+ },+ "events": {+ "description": "An event from the workspace event feed, such as a run state change, a deployment change, or an automation action.",+ "docs_url": "https://docs.prefect.io/v3/api-ref/rest-api/server/events",+ "columns": {+ "id": "The client-provided identifier of the event.",+ "occurred": "When the event happened, from the sender's perspective.",+ "received": "When Prefect received the event.",+ "event": "The name of the event (e.g. prefect.flow-run.Completed).",+ "resource": "The primary resource the event concerns.",+ "related": "Other resources involved in the event.",+ "payload": "An open-ended set of data describing what happened.",+ "follows": "The id of an event known to have occurred before this one.",+ "account": "The account the event belongs to.",+ "workspace": "The workspace the event belongs to.",+ },+ },+ "flow_run_states": {+ "description": "One state in a flow run's history. Each row is a state the flow run entered.",+ "docs_url": "https://docs.prefect.io/v3/api-ref/rest-api/server/flow-run-states",+ "columns": {+ "id": "Unique identifier for the state.",+ "flow_run_id": "The id of the flow run the state belongs to.",+ "type": "The state type (e.g. SCHEDULED, PENDING, RUNNING, COMPLETED, FAILED, CRASHED, CANCELLED).",+ "name": "The state name (e.g. Late, AwaitingRetry, Retrying).",+ "timestamp": "When the flow run entered the state.",+ "message": "A message describing the state transition.",+ "data": "Data associated with the state, e.g. a result.",+ "state_details": "Details of the state, such as the scheduled time, cache key, and pause or retry settings.",+ },+ },+ "task_run_states": {+ "description": "One state in a task run's history. Each row is a state the task run entered.",+ "docs_url": "https://docs.prefect.io/v3/api-ref/rest-api/server/task-run-states",+ "columns": {+ "id": "Unique identifier for the state.",+ "task_run_id": "The id of the task run the state belongs to.",+ "type": "The state type (e.g. SCHEDULED, PENDING, RUNNING, COMPLETED, FAILED, CRASHED, CANCELLED).",+ "name": "The state name (e.g. Late, AwaitingRetry, Retrying).",+ "timestamp": "When the task run entered the state.",+ "message": "A message describing the state transition.",+ "data": "Data associated with the state, e.g. a result.",+ "state_details": "Details of the state, such as the scheduled time, cache key, and pause or retry settings.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/prefect_cloud.py b/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/prefect_cloud.pyindex 19a9bd053dc6..2ee38c55eee3 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/prefect_cloud.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/prefect_cloud.py@@ -1,35 +1,195 @@import re+import copyimport dataclasses+from collections.abc import Callablefrom datetime import UTC, date, datetimefrom typing import Any, Optional+from requests import Request, Response+from products.warehouse_sources.backend.temporal.data_imports.sources.common.http import make_tracked_sessionfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source import (RESTAPIConfig,rest_api_resource,+ rest_api_resources,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (+ rename_parent_fields,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (+ BasePaginator,+ JSONResponsePaginator,OffsetPaginator,+ set_json_param,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.resource import Resource+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (+ ClientConfig,+ Endpoint,+ EndpointResource,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.prefect_cloud.settings import (+ EVENTS_FULL_REFRESH_SINCE,+ LOGS_MAX_OFFSET,PAGE_LIMIT,PREFECT_CLOUD_ENDPOINTS,PrefectCloudEndpointConfig,)-PREFECT_CLOUD_API_BASE = "https://api.prefect.cloud/api"+PREFECT_CLOUD_API_HOST = "api.prefect.cloud"+PREFECT_CLOUD_API_BASE = f"https://{PREFECT_CLOUD_API_HOST}/api"# Account and workspace IDs are UUIDs embedded in the request path. Rejecting anything else keeps# user input from rewriting the path (e.g. `../` traversal into another account's routes)._UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")[email protected](frozen=True)class PrefectCloudResumeConfig:# Row offset of the next page to fetch. 0 means "start from the first page".offset: int = 0+ # Timestamp bounds of the logs walk; `offset` then counts rows read under those bounds.+ after: Optional[str] = None+ before: Optional[str] = None+ # Next `/events/filter/next` link.+ next_url: Optional[str] = None+ # Latest logs timestamp read, and the ids already emitted at it, so a restart skips them.+ tail_timestamp: Optional[str] = None+ tail_ids: Optional[list[str]] = None+ # The framework's per-parent state for the state-history fan-outs, stored unchanged.+ fanout: Optional[dict[str, Any]] = None+++class TimestampKeysetPaginator(BasePaginator):+ """Walk an ascending timestamp sort with offset pages, moving the lower bound before the offset cap.++ `POST /logs/filter` caps `offset`, so a plain offset walk stops partway through a large+ workspace. Near the cap the walk restarts at offset 0 from the last timestamp read. The sort+ has no tie-breaker, so the restart reads the rows at that timestamp again rather than skipping+ a count of them. The ids already emitted at the latest timestamp are tracked and dropped from+ the reread pages, so full-refresh and append syncs don't write the boundary rows twice. The+ fixed upper bound keeps a busy workspace from extending the walk while it runs.+ """++ def __init__(+ self,+ limit: int,+ timestamp_field: str,+ after_param: str,+ before_param: str,+ before: str,+ max_offset: int,+ ) -> None:+ super().__init__()+ self.limit = limit+ self.timestamp_field = timestamp_field+ self.after_param = after_param+ self.before_param = before_param+ self.max_offset = max_offset+ self.after: Optional[str] = None+ self.before = before+ self.offset = 0+ self.tail_timestamp: Optional[str] = None+ self.tail_ids: set[str] = set()++ def _apply(self, request: Request) -> None:+ if request.json is None:+ request.json = {}+ if self.after is not None:+ set_json_param(request.json, self.after_param, self.after)+ set_json_param(request.json, self.before_param, self.before)+ request.json["offset"] = self.offset+ request.json["limit"] = self.limit++ def init_request(self, request: Request) -> None:+ self._apply(request)++ def update_request(self, request: Request) -> None:+ self._apply(request)++ def _drop_emitted_rows(self, data: list[Any]) -> None:+ fresh: list[Any] = []+ for row in data:+ timestamp, row_id = row.get(self.timestamp_field), row.get("id")+ if timestamp is not None and timestamp == self.tail_timestamp and row_id in self.tail_ids:+ continue+ fresh.append(row)+ if timestamp is None or row_id is None:+ continue+ if timestamp != self.tail_timestamp:+ self.tail_timestamp = timestamp+ self.tail_ids = set()+ self.tail_ids.add(row_id)+ # The client yields this same list after `update_state`, so filter it in place.+ data[:] = fresh++ def update_state(self, response: Response, data: Optional[list[Any]] = None) -> None:+ if not data or len(data) < self.limit:+ if data:+ self._drop_emitted_rows(data)+ self._has_next_page = False+ return++ last = data[-1].get(self.timestamp_field)+ self._drop_emitted_rows(data)+ if self.offset + self.limit <= self.max_offset:+ self.offset += self.limit+ else:+ if last is None:+ raise ValueError(f"Prefect Cloud returned a row without '{self.timestamp_field}'; can't page past it")+ if last == self.after:+ raise ValueError(+ f"More than {self.max_offset} Prefect Cloud rows share the timestamp {last}; can't page past them"+ )+ self.after = last+ self.offset = 0+ self._has_next_page = True++ def get_resume_state(self) -> Optional[dict[str, Any]]:+ if not self._has_next_page:+ return None+ return {+ "after": self.after,+ "before": self.before,+ "offset": self.offset,+ "tail_timestamp": self.tail_timestamp,+ "tail_ids": sorted(self.tail_ids),+ }++ def set_resume_state(self, state: dict[str, Any]) -> None:+ if state.get("before") is not None:+ self.after = state.get("after")+ self.before = state["before"]+ self.offset = int(state.get("offset") or 0)+ self.tail_timestamp = state.get("tail_timestamp")+ self.tail_ids = set(state.get("tail_ids") or [])+ self._has_next_page = True+++class EventsNextPagePaginator(JSONResponsePaginator):+ """Follow `next_page` from `POST /events/filter`.++ The link points at `GET /events/filter/next`, whose page token carries the filter and offset,+ so the follow-up requests drop the POST body.+ """++ def __init__(self) -> None:+ super().__init__(next_url_path="next_page")++ def _to_next_page(self, request: Request) -> None:+ if self._next_url is not None:+ request.method = "GET"+ request.json = None++ def init_request(self, request: Request) -> None:+ super().init_request(request)+ self._to_next_page(request)++ def update_request(self, request: Request) -> None:+ super().update_request(request)+ self._to_next_page(request)def normalize_uuid(value: str, label: str) -> str:@@ -87,23 +247,102 @@ def _build_json_body(params). The paginator injects `limit`/`offset` per page; this body carries the watermarkfilter and stable sort, and because the paginator mutates the same dict in place the filterrides every page (server-side incremental)."""- body: dict[str, Any] = {}+ body: dict[str, Any] = copy.deepcopy(config.body)sort = config.sort+ watermark = (+ _format_after(db_incremental_field_last_value)+ if should_use_incremental_field and db_incremental_field_last_value is not None+ else None+ )++ if config.pagination == "timestamp_keyset" and watermark is not None:+ body["logs"] = {"timestamp": {"after_": watermark}}+ if config.pagination == "next_page":+ # Always bound `since`: the server default only looks back a limited window.+ set_json_param(body, "filter.occurred.since", watermark or EVENTS_FULL_REFRESH_SINCE)- if (- config.filter_key- and config.incremental_sorts- and should_use_incremental_field- and db_incremental_field_last_value is not None- ):+ if config.filter_key and config.incremental_sorts and watermark is not None:field_name, sort = _resolve_incremental_sort(config, incremental_field)- body[config.filter_key] = {field_name: {"after_": _format_after(db_incremental_field_last_value)}}+ body[config.filter_key] = {field_name: {"after_": watermark}}if sort:body["sort"] = sortreturn body+def _paginator(config: PrefectCloudEndpointConfig) -> BasePaginator:+ if config.pagination == "timestamp_keyset":+ return TimestampKeysetPaginator(+ limit=PAGE_LIMIT,+ timestamp_field="timestamp",+ after_param="logs.timestamp.after_",+ before_param="logs.timestamp.before_",+ before=_format_after(datetime.now(UTC)),+ max_offset=LOGS_MAX_OFFSET,+ )+ if config.pagination == "next_page":+ return EventsNextPagePaginator()+ # Prefect paginates via `limit`/`offset` in the POST body; no top-level total, so+ # termination is a short/empty page (OffsetPaginator default).+ return OffsetPaginator(limit=PAGE_LIMIT, total_path=None, param_location="json")+++def _state_history_resource(+ config: PrefectCloudEndpointConfig,+ client: ClientConfig,+ team_id: int,+ job_id: str,+ should_use_incremental_field: bool,+ db_incremental_field_last_value: Any,+ resume_hook: Callable[[Optional[dict[str, Any]]], None],+ initial_paginator_state: Optional[dict[str, Any]],+) -> Resource:+ """List the parent runs, then read each run's state history.++ The states endpoints take one run id and have no time filter, so an incremental sync bounds the+ requests through the parent instead: only runs expected to start after the watermark (less the+ lookback) are listed.+ """+ assert config.fanout is not None+ fanout = config.fanout+ parent_config = PREFECT_CLOUD_ENDPOINTS[fanout.parent]+ parent_body = _build_json_body(+ parent_config, should_use_incremental_field, db_incremental_field_last_value, "expected_start_time"+ )+ parent: EndpointResource = {+ "name": fanout.parent,+ "endpoint": {+ "path": parent_config.path,+ "method": "POST",+ "json": parent_body,+ "paginator": _paginator(parent_config),+ },+ }+ child: EndpointResource = {+ "name": config.name,+ "endpoint": {+ "path": f"{config.path}?{fanout.resolve_param}={{{fanout.resolve_param}}}",+ "method": "GET",+ "params": {fanout.resolve_param: {"type": "resolve", "resource": fanout.parent, "field": "id"}},+ # Each run's full history comes back as one bare array.+ "paginator": "single_page",+ },+ "include_from_parent": ["id"],+ "resume_parent_pagination": True,+ }+ rest_config: RESTAPIConfig = {"client": client, "resource_defaults": {}, "resources": [parent, child]}+ resources = rest_api_resources(+ rest_config,+ team_id,+ job_id,+ None,+ resume_hook=resume_hook,+ initial_paginator_state=initial_paginator_state,+ )+ resource = next(r for r in resources if r.name == config.name)+ return resource.add_map(rename_parent_fields(fanout.parent, {"id": fanout.resolve_param}))++def prefect_cloud_source(account_id: str,workspace_id: str,@@ -117,57 +356,85 @@ def prefect_cloud_source(incremental_field: str | None = None,) -> SourceResponse:config = PREFECT_CLOUD_ENDPOINTS[endpoint]- base_url = _workspace_url(account_id, workspace_id)- json_body = _build_json_body(- config, should_use_incremental_field, db_incremental_field_last_value, incremental_field- )-- rest_config: RESTAPIConfig = {- "client": {- "base_url": base_url,- # Non-secret header only; the API key rides the framework Bearer auth so it's redacted.- "headers": {"Accept": "application/json"},- "auth": {"type": "bearer", "token": api_key},- # Prefect paginates via `limit`/`offset` in the POST body; no top-level total, so- # termination is a short/empty page (OffsetPaginator default).- "paginator": OffsetPaginator(limit=PAGE_LIMIT, total_path=None, param_location="json"),- },- "resource_defaults": {},- "resources": [- {- "name": endpoint,- "endpoint": {- "path": config.path,- "method": "POST",- "json": json_body,- # Bare-array body: the whole response is the row list (no data_selector).- },- }- ],+ client: ClientConfig = {+ "base_url": _workspace_url(account_id, workspace_id),+ # Non-secret header only; the API key rides the framework Bearer auth so it's redacted.+ "headers": {"Accept": "application/json"},+ "auth": {"type": "bearer", "token": api_key},+ # Keeps the key on Prefect's host when following an events `next_page` link.+ "allowed_hosts": [PREFECT_CLOUD_API_HOST],+ "allow_redirects": False,}initial_paginator_state: Optional[dict[str, Any]] = Noneif resumable_source_manager.can_resume():resume = resumable_source_manager.load_state()if resume is not None:- initial_paginator_state = {"offset": resume.offset}+ if config.fanout is not None:+ initial_paginator_state = resume.fanout+ else:+ initial_paginator_state = {+ "offset": resume.offset,+ "after": resume.after,+ "before": resume.before,+ "next_url": resume.next_url,+ "tail_timestamp": resume.tail_timestamp,+ "tail_ids": resume.tail_ids,+ }def save_checkpoint(state: Optional[dict[str, Any]]) -> None:- # Persist only when a next page remains; save AFTER a page is yielded so a crash re-yields- # the last page (merge dedupes on the primary key) rather than skipping it.- if state and state.get("offset") is not None:- resumable_source_manager.save_state(PrefectCloudResumeConfig(offset=int(state["offset"])))+ # Persist only when a next page remains; the pipeline commits it once the page is written.+ if not state:+ return+ if config.fanout is not None:+ resumable_source_manager.save_state(PrefectCloudResumeConfig(fanout=state))+ else:+ resumable_source_manager.save_state(+ PrefectCloudResumeConfig(+ offset=int(state.get("offset") or 0),+ after=state.get("after"),+ before=state.get("before"),+ next_url=state.get("next_url"),+ tail_timestamp=state.get("tail_timestamp"),+ tail_ids=state.get("tail_ids"),+ )+ )- resource = rest_api_resource(- rest_config,- team_id,- job_id,- # The watermark is baked into the request body above, so the framework's own incremental- # param injection is unused.- None,- resume_hook=save_checkpoint,- initial_paginator_state=initial_paginator_state,- )+ if config.fanout is not None:+ resource = _state_history_resource(+ config,+ client,+ team_id,+ job_id,+ should_use_incremental_field,+ db_incremental_field_last_value,+ save_checkpoint,+ initial_paginator_state,+ )+ else:+ json_body = _build_json_body(+ config, should_use_incremental_field, db_incremental_field_last_value, incremental_field+ )+ endpoint_config: Endpoint = {+ "path": config.path,+ "method": "POST",+ "json": json_body,+ "paginator": _paginator(config),+ }+ if config.data_selector:+ endpoint_config["data_selector"] = config.data_selector+ resource_config: EndpointResource = {"name": endpoint, "endpoint": endpoint_config}+ rest_config: RESTAPIConfig = {"client": client, "resource_defaults": {}, "resources": [resource_config]}+ resource = rest_api_resource(+ rest_config,+ team_id,+ job_id,+ # The watermark is baked into the request body above, so the framework's own incremental+ # param injection is unused.+ None,+ resume_hook=save_checkpoint,+ initial_paginator_state=initial_paginator_state,+ )return SourceResponse(name=endpoint,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/settings.pyindex 323dda8a187a..3a00a954ac6c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/settings.py@@ -1,5 +1,5 @@from dataclasses import dataclass, field-from typing import Optional+from typing import Any, Literal, Optionalfrom products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType@@ -10,6 +10,16 @@ PAGE_LIMIT = 200# incremental syncs re-read a trailing day; merge dedupes the re-pulled rows on `id`.RUN_LOOKBACK_SECONDS = 24 * 60 * 60+# Workers ship logs and events in batches, so a row can land after newer rows were already synced.+LATE_ARRIVAL_LOOKBACK_SECONDS = 60 * 60++# `POST /logs/filter` rejects an `offset` above this, so logs are walked by timestamp instead.+LOGS_MAX_OFFSET = 20000++# Without `occurred.since` the events filter only looks back a limited window. Prefect clamps an+# older bound to the workspace's event retention, so this floor reads everything still retained.+EVENTS_FULL_REFRESH_SINCE = "2020-01-01T00:00:00Z"+def _datetime_incremental_field(name: str) -> IncrementalField:return {@@ -20,7 +30,14 @@ def _datetime_incremental_field(name: str) -> IncrementalField:}-@dataclass+@dataclass(frozen=True)+class StateHistoryFanout:+ # The run table to list, and the query param the per-run states endpoint takes its id in.+ parent: str+ resolve_param: str+++@dataclass(frozen=True)class PrefectCloudEndpointConfig:name: strpath: str # POST path under the workspace base URL, e.g. "/flow_runs/filter"@@ -40,6 +57,13 @@ class PrefectCloudEndpointConfig:primary_keys: list[str] = field(default_factory=lambda: ["id"])default_incremental_lookback_seconds: Optional[int] = Noneshould_sync_default: bool = True+ # Static JSON body fields sent on every request, merged under the filter and sort.+ body: dict[str, Any] = field(default_factory=dict)+ data_selector: Optional[str] = None+ # How the endpoint pages: `offset` in the POST body, a timestamp keyset (logs), the+ # `next_page` link (events), or one GET per parent run (state history).+ pagination: Literal["offset", "timestamp_keyset", "next_page", "fanout"] = "offset"+ fanout: Optional[StateHistoryFanout] = NonePREFECT_CLOUD_ENDPOINTS: dict[str, PrefectCloudEndpointConfig] = {@@ -97,6 +121,59 @@ PREFECT_CLOUD_ENDPOINTS: dict[str, PrefectCloudEndpointConfig] = {# offset paging on the implicit order is acceptable for full refresh.sort=None,),+ "artifacts": PrefectCloudEndpointConfig(+ name="artifacts",+ path="/artifacts/filter",+ # ArtifactFilter has no time-range filters, and ArtifactSort has no ascending time order.+ # Newest-first keeps offset paging stable: rows created mid-sync push later pages back,+ # which repeats a row instead of skipping one.+ sort="CREATED_DESC",+ partition_key="created",+ ),+ "logs": PrefectCloudEndpointConfig(+ name="logs",+ path="/logs/filter",+ incremental_fields=[_datetime_incremental_field("timestamp")],+ sort="TIMESTAMP_ASC",+ partition_key="timestamp",+ default_incremental_lookback_seconds=LATE_ARRIVAL_LOOKBACK_SECONDS,+ should_sync_default=False,+ pagination="timestamp_keyset",+ ),+ "events": PrefectCloudEndpointConfig(+ name="events",+ path="/events/filter",+ incremental_fields=[_datetime_incremental_field("occurred")],+ partition_key="occurred",+ default_incremental_lookback_seconds=LATE_ARRIVAL_LOOKBACK_SECONDS,+ body={"filter": {"order": "ASC"}},+ data_selector="events",+ pagination="next_page",+ ),+ "flow_run_states": PrefectCloudEndpointConfig(+ name="flow_run_states",+ path="/flow_run_states/",+ incremental_fields=[_datetime_incremental_field("timestamp")],+ partition_key="timestamp",+ primary_keys=["flow_run_id", "id"],+ default_incremental_lookback_seconds=RUN_LOOKBACK_SECONDS,+ # One request per flow run, so it is opt-in.+ should_sync_default=False,+ pagination="fanout",+ fanout=StateHistoryFanout(parent="flow_runs", resolve_param="flow_run_id"),+ ),+ "task_run_states": PrefectCloudEndpointConfig(+ name="task_run_states",+ path="/task_run_states/",+ incremental_fields=[_datetime_incremental_field("timestamp")],+ partition_key="timestamp",+ primary_keys=["task_run_id", "id"],+ default_incremental_lookback_seconds=RUN_LOOKBACK_SECONDS,+ # One request per task run, so it is opt-in.+ should_sync_default=False,+ pagination="fanout",+ fanout=StateHistoryFanout(parent="task_runs", resolve_param="task_run_id"),+ ),}ENDPOINTS = tuple(PREFECT_CLOUD_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/source.pyindex 58ec74cfb0de..fd86900cf262 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/source.py@@ -58,7 +58,7 @@ class PrefectCloudSource(ResumableSource[PrefectCloudSourceConfig, PrefectCloudRcategory=DataWarehouseSourceCategory.ENGINEERING___MONITORING,label="Prefect Cloud",releaseStatus=ReleaseStatus.BETA,- caption="""Enter your Prefect Cloud account ID, workspace ID, and an API key to pull your flow and task run history into the PostHog Data warehouse.+ caption="""Enter your Prefect Cloud account ID, workspace ID, and an API key to pull your flows, runs, logs, artifacts, and events into the PostHog Data warehouse.Both IDs are in your workspace URL: `https://app.prefect.cloud/account/<account ID>/workspace/<workspace ID>`.@@ -127,11 +127,18 @@ Create an API key under [API keys in your profile settings](https://app.prefect."Incremental syncs re-read a 24-hour trailing window so recently started runs pick up ""state changes; runs that change state later than that are only refreshed by a full refresh")+ if endpoint in ("flow_run_states", "task_run_states"):+ return (+ "Makes one request per run. Incremental syncs only read runs expected to start within 24 hours "+ "of the latest synced state, so a run that changes state later than that is only refreshed by a full refresh"+ )+ if endpoint == "events":+ return "Prefect Cloud keeps events for your plan's retention period, so a sync can only read events still retained"return Nonedef _build_schema(endpoint: str) -> SourceSchema:endpoint_config = PREFECT_CLOUD_ENDPOINTS[endpoint]- has_incremental = bool(endpoint_config.incremental_sorts)+ has_incremental = bool(endpoint_config.incremental_fields)return SourceSchema(name=endpoint,supports_incremental=has_incremental,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/tests/test_prefect_cloud.py b/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/tests/test_prefect_cloud.pyindex efe63117c6b9..0baae4a2eef6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/tests/test_prefect_cloud.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/prefect_cloud/tests/test_prefect_cloud.py@@ -2,6 +2,7 @@ from datetime import UTC, date, datetimefrom typing import Anyimport pytest+from unittest import mockfrom parameterized import parameterized@@ -13,6 +14,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.tesfrom products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.prefectcloud import (PrefectCloudSourceConfig,)+from products.warehouse_sources.backend.temporal.data_imports.sources.prefect_cloud import prefect_cloudfrom products.warehouse_sources.backend.temporal.data_imports.sources.prefect_cloud.prefect_cloud import (PrefectCloudResumeConfig,_build_json_body,@@ -136,6 +138,118 @@ class TestPagination:assert [b["offset"] for b in bodies] == [0, PAGE_LIMIT]+def _logs(*timestamps: str) -> list[dict[str, Any]]:+ return [{"id": f"log-{n}", "timestamp": ts} for n, ts in enumerate(timestamps)]+++class TestLogsKeysetPagination:+ def test_moves_the_lower_bound_before_the_offset_cap_and_rereads_the_boundary_timestamp(self) -> None:+ # The sort has no tie-breaker, so the restart must read every row at T2 again (offset 0)+ # rather than skip a count of them.+ pages = [+ _page(_logs(*["T1"] * PAGE_LIMIT)),+ _page(_logs(*["T1"] * PAGE_LIMIT)),+ _page(_logs(*["T1"] * (PAGE_LIMIT - 2), "T2", "T2")),+ _page(_logs("T2", "T2", "T3")),+ ]++ with mock.patch.object(prefect_cloud, "LOGS_MAX_OFFSET", 2 * PAGE_LIMIT):+ result = _driver().run("logs", pages)++ bodies = [r.json() for r in result.requests]+ assert [b["logs"]["timestamp"].get("after_") for b in bodies] == [None, None, None, "T2"]+ assert [b["offset"] for b in bodies] == [0, PAGE_LIMIT, 2 * PAGE_LIMIT, 0]+ assert len({b["logs"]["timestamp"]["before_"] for b in bodies}) == 1+ assert all(b["sort"] == "TIMESTAMP_ASC" and b["limit"] == PAGE_LIMIT for b in bodies)++ def test_the_restart_does_not_emit_the_boundary_rows_twice(self) -> None:+ first = [{"id": f"a{n}", "timestamp": "T1"} for n in range(PAGE_LIMIT - 2)]+ boundary = [{"id": "b1", "timestamp": "T2"}, {"id": "b2", "timestamp": "T2"}]+ reread = [*boundary, {"id": "b3", "timestamp": "T2"}, {"id": "c1", "timestamp": "T3"}]++ with mock.patch.object(prefect_cloud, "LOGS_MAX_OFFSET", 0):+ result = _driver().run("logs", [_page(first + boundary), _page(reread)])++ assert [r.json()["logs"]["timestamp"].get("after_") for r in result.requests] == [None, "T2"]+ assert [r["id"] for r in result.rows] == [*(r["id"] for r in first), "b1", "b2", "b3", "c1"]++ def test_a_resumed_restart_skips_the_saved_boundary_ids(self) -> None:+ result = _driver().run(+ "logs",+ [_page([{"id": "b1", "timestamp": "T2"}, {"id": "b2", "timestamp": "T2"}])],+ resume_state=PrefectCloudResumeConfig(+ after="T2", before="T8", offset=0, tail_timestamp="T2", tail_ids=["b1"]+ ),+ )++ assert [r["id"] for r in result.rows] == ["b2"]++ def test_resume_keeps_the_saved_bounds_over_the_watermark(self) -> None:+ result = _driver().run(+ "logs",+ [_page(_logs("T9"))],+ resume_state=PrefectCloudResumeConfig(after="T5", before="T8", offset=3),+ incremental_field="timestamp",+ db_incremental_field_last_value=datetime(2026, 3, 4, tzinfo=UTC),+ )++ assert [r.json()["logs"] for r in result.requests] == [{"timestamp": {"after_": "T5", "before_": "T8"}}]+ assert [r.json()["offset"] for r in result.requests] == [3]+++class TestEventsPagination:+ @parameterized.expand(+ [+ ("full_refresh", None, "2020-01-01T00:00:00Z"),+ ("incremental", datetime(2026, 3, 4, 2, 58, 14, tzinfo=UTC), "2026-03-04T02:58:14Z"),+ ]+ )+ def test_posts_the_filter_then_follows_next_page_with_get(+ self, _name: str, watermark: datetime | None, expected_since: str+ ) -> None:+ next_page = f"{_WORKSPACE_URL}/events/filter/next?page-token=abc"+ result = _driver().run(+ "events",+ [+ ScriptedResponse(json={"events": [{"id": "e1"}], "total": 2, "next_page": next_page}),+ ScriptedResponse(json={"events": [{"id": "e2"}], "total": 2, "next_page": None}),+ ],+ incremental_field="occurred" if watermark else None,+ db_incremental_field_last_value=watermark,+ )++ assert [r["id"] for r in result.rows] == ["e1", "e2"]+ first, second = result.requests+ assert first.method == "POST"+ assert first.json()["filter"] == {"order": "ASC", "occurred": {"since": expected_since}}+ assert (second.method, second.url, second.body) == ("GET", next_page, None)+++class TestStateHistoryFanout:+ @parameterized.expand(+ [+ ("flow_run_states", "flow_runs", "flow_run_id"),+ ("task_run_states", "task_runs", "task_run_id"),+ ]+ )+ def test_reads_each_runs_states_and_tags_rows_with_the_run(self, endpoint: str, parent: str, param: str) -> None:+ result = _driver().run(+ endpoint,+ [+ _page([{"id": "run-a"}, {"id": "run-b"}]),+ _page([{"id": "s1", "type": "SCHEDULED"}, {"id": "s2", "type": "RUNNING"}]),+ _page([{"id": "s3", "type": "SCHEDULED"}]),+ ],+ incremental_field="timestamp",+ db_incremental_field_last_value=datetime(2026, 3, 4, tzinfo=UTC),+ )++ parent_request, *state_requests = result.requests+ assert parent_request.json()[parent] == {"expected_start_time": {"after_": "2026-03-04T00:00:00Z"}}+ assert [(r.method, r.param(param)) for r in state_requests] == [("GET", "run-a"), ("GET", "run-b")]+ assert [(r[param], r["id"]) for r in result.rows] == [("run-a", "s1"), ("run-a", "s2"), ("run-b", "s3")]++class TestValidateCredentials:@pytest.mark.parametrize("status_code, expected",