:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse): add opsgenie alert log and schedule override tables (#124777)
Tom Owersf4ac3b148295
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex ffe145a6c7a0..ce326f0798b6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -8893,21 +8893,21 @@ Note: Parsed official resource references for schedules, occurrences, receipts,## Opsgenie — gaps-Today (12): `alerts`, `escalations`, `incident_timeline_entries`, `incidents`, `integrations`, `schedule_on_calls`, `schedule_rotations`, `schedules`, `services`, `team_members`, `teams`, `users`+Today (14): `alert_logs`, `alerts`, `escalations`, `incident_timeline_entries`, `incidents`, `integrations`, `schedule_on_calls`, `schedule_overrides`, `schedule_rotations`, `schedules`, `services`, `team_members`, `teams`, `users`Diffed against: <https://docs.opsgenie.com/docs/api-overview>-- [ ] `GET /v2/alerts/{identifier}/logs` — per-alert state transition history (created, acked, escalated, closed) — the basis of MTTA/MTTR (high)+- [x] `GET /v2/alerts/{identifier}/logs` — per-alert state transition history (created, acked, escalated, closed) — the basis of MTTA/MTTR (high) — shipped as `alert_logs`- [x] `Team member API (GET /v2/teams/{id}/members)` — membership table joining the users and teams already synced (high) — shipped as `team_members`; Opsgenie has no member list GET, so members come from `GET /v2/teams/{id}`- [x] `Schedule rotation API (GET /v2/schedules/{id}/rotations)` — lookup resolving the rotation structure behind each synced schedule; without it schedules are opaque (high)- [x] `Who is on call API (GET /v2/schedules/{id}/on-calls, /v2/schedules/on-calls)` — resolves schedule + rotation into the actual on-call person, the headline operational fact (high) — shipped as `schedule_on_calls` from the per-schedule path; the all-schedules list is deprecated- [x] `Incident timeline API (GET /v1/incidents/{id}/timeline)` — state/transition history for the incidents table already synced (high) — shipped as `incident_timeline_entries` from `GET /v2/incident-timelines/{id}/entries`- [ ] `GET /v2/alerts/{identifier}/notes` — responder commentary attached to alerts, needed for incident review analysis (medium)- [ ] `GET /v2/alerts/{identifier}/recipients` — who was notified per alert and in what state — notification-reach analysis (medium)-- [ ] `Schedule override API (GET /v2/schedules/{id}/overrides)` — explains gaps between rotation and actual on-call coverage (medium)-- [ ] `Postmortem API (GET /v1/incidents/{id}/postmortems)` — post-incident write-ups joined to synced incidents (medium)+- [x] `Schedule override API (GET /v2/schedules/{id}/overrides)` — explains gaps between rotation and actual on-call coverage (medium) — shipped as `schedule_overrides`+- [ ] `Postmortem API (GET /v1/incidents/{id}/postmortems)` — post-incident write-ups joined to synced incidents (medium) — skipped: the Incident API has no postmortem read endpoint- [ ] `Heartbeat API (GET /v2/heartbeats)` — monitoring heartbeat state, a distinct alerting signal from alerts (medium)-- [ ] `Logs API (GET /v2/logs/list/{marker})` — account-level audit/activity log for configuration and alert changes over time (medium)+- [ ] `Logs API (GET /v2/logs/list/{marker})` — account-level audit/activity log for configuration and alert changes over time (medium) — skipped: the endpoint lists downloadable log files (name, date, size), not log records, and is Enterprise-only- [ ] `Maintenance API (GET /v1/maintenance)` — maintenance windows that suppress alerts — needed to exclude planned downtime (low)Note: Alert sub-resource paths confirmed against the vendor SDK reference at https://raw.githubusercontent.com/opsgenie/opsgenie-python-sdk/master/README.md. Notification-rule, policy, forwarding-rule, integration-action, service-incident-template and custom-role endpoints were excluded as configuration.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/canonical_descriptions.pyindex 776188b25906..10de2965fb9f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/canonical_descriptions.py@@ -141,6 +141,18 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"lastEdit": "Time and actor of the last edit to the entry.",},},+ "alert_logs": {+ "description": "The activity log of an alert: each state change (created, acknowledged, escalated, closed) and responder action, in order.",+ "docs_url": "https://docs.opsgenie.com/docs/alert-api-continued#list-alert-logs",+ "columns": {+ "alert_id": "Id of the alert the log entry belongs to.",+ "offset": "Cursor of the log entry, unique within the alert and ordered by time.",+ "log": "Text of the log entry.",+ "type": "Kind of log entry, e.g. system or alertRecipient.",+ "owner": "User or system that made the change.",+ "createdAt": "Time the log entry was created.",+ },+ },"schedule_on_calls": {"description": "The users on call for each schedule at the time of the sync, with teams and escalations resolved to users.","docs_url": "https://docs.opsgenie.com/docs/who-is-on-call-api",@@ -165,6 +177,18 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"timeRestriction": "Days and hours the rotation is active, if restricted.",},},+ "schedule_overrides": {+ "description": "A temporary override that puts a user on call for a schedule in place of the rotation for a time range.",+ "docs_url": "https://docs.opsgenie.com/docs/schedule-override-api",+ "columns": {+ "schedule_id": "Id of the schedule the override belongs to.",+ "alias": "Identifier of the override, unique within the schedule.",+ "user": "User, team, or escalation that is on call during the override.",+ "startDate": "Time the override starts.",+ "endDate": "Time the override ends.",+ "rotations": "Rotations the override applies to; empty when it applies to the whole schedule.",+ },+ },"team_members": {"description": "A user's membership of a team, with the user's role in that team.","docs_url": "https://docs.opsgenie.com/docs/team-api",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/opsgenie.py b/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/opsgenie.pyindex 9dbebfd23774..1c86dda1cc8a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/opsgenie.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/opsgenie.py@@ -17,6 +17,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (BasePaginator,JSONResponseCursorPaginator,+ JSONResponsePaginator,SinglePagePaginator,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.resource import Resource@@ -244,6 +245,8 @@ def opsgenie_source("name": "Authorization","location": "header",},+ # Alert logs follow the response's `paging.next` URL; pin it to the regional API host.+ "allowed_hosts": [],}if config.fanout is not None:@@ -340,15 +343,17 @@ def _fanout_source(if parent_config.paginatedelse SinglePagePaginator())- child_paginator: BasePaginator = (- JSONResponseCursorPaginator(cursor_path=config.next_cursor_path, cursor_param="offset")- if config.next_cursor_path- else SinglePagePaginator()- )+ child_paginator: BasePaginator+ if config.next_cursor_path:+ child_paginator = JSONResponseCursorPaginator(cursor_path=config.next_cursor_path, cursor_param="offset")+ elif config.next_url_path:+ child_paginator = JSONResponsePaginator(next_url_path=config.next_url_path)+ else:+ child_paginator = SinglePagePaginator()# The shared fan-out checkpoint lists every finished parent, so its size grows with each parent.# Only the small, unpaginated parents (schedules, teams) are resumable; the incident timeline- # starts again from the first incident after an interruption.+ # and alert logs start again from the first parent after an interruption.resumable = not parent_config.paginatedresume = resumable_source_manager.load_state() if resumable and resumable_source_manager.can_resume() else None@@ -385,5 +390,8 @@ def _fanout_source(primary_keys=config.primary_keys,partition_count=1,partition_size=1,+ partition_mode="datetime" if config.partition_key else None,+ partition_format="week" if config.partition_key else None,+ partition_keys=[config.partition_key] if config.partition_key else None,column_hints=resource.column_hints,)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/settings.pyindex 8c4efdc2a807..f41604ff0983 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/settings.py@@ -37,6 +37,8 @@ class OpsgenieEndpointConfig:data_selector: str = "data"# JSON path of the next-page cursor, for fan-out children that page with a cursor, not an offset.next_cursor_path: Optional[str] = None+ # JSON path of the full next-page URL, for fan-out children whose cursor sits on each row.+ next_url_path: Optional[str] = None_CREATED_AT_INCREMENTAL: list[IncrementalField] = [@@ -114,6 +116,25 @@ OPSGENIE_ENDPOINTS: dict[str, OpsgenieEndpointConfig] = {child_params={"order": "asc"},),),+ "alert_logs": OpsgenieEndpointConfig(+ name="alert_logs",+ path="/v2/alerts/{alert_id}/logs",+ # A log entry has no id; its `offset` cursor (epoch millis + nanos) is unique within the alert.+ primary_keys=["alert_id", "offset"],+ partition_key="createdAt",+ next_url_path="paging.next",+ fanout=DependentEndpointConfig(+ parent_name="alerts",+ resolve_param="alert_id",+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": "alert_id"},+ parent_params={"sort": "createdAt", "order": "asc"},+ child_params={"order": "asc", "direction": "next"},+ # An alert deleted after the alert listing returns 404; skip it rather than fail the run.+ child_response_actions=[{"status_code": 404, "action": "ignore"}],+ ),+ ),"schedule_on_calls": OpsgenieEndpointConfig(name="schedule_on_calls",path="/v2/schedules/{schedule_id}/on-calls",@@ -141,6 +162,20 @@ OPSGENIE_ENDPOINTS: dict[str, OpsgenieEndpointConfig] = {parent_field_renames={"id": "schedule_id"},),),+ "schedule_overrides": OpsgenieEndpointConfig(+ name="schedule_overrides",+ path="/v2/schedules/{schedule_id}/overrides",+ # Opsgenie allows only one override with a given alias per schedule.+ primary_keys=["schedule_id", "alias"],+ fanout=DependentEndpointConfig(+ parent_name="schedules",+ resolve_param="schedule_id",+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": "schedule_id"},+ child_response_actions=[{"status_code": 404, "action": "ignore"}],+ ),+ ),# Opsgenie has no member listing endpoint; the members come embedded in the single-team read."team_members": OpsgenieEndpointConfig(name="team_members",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/tests/test_opsgenie.py b/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/tests/test_opsgenie.pyindex bf9b7d25e2a7..7990509166ad 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/tests/test_opsgenie.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/opsgenie/tests/test_opsgenie.py@@ -140,6 +140,51 @@ class TestFanout:assert [(row["incident_id"], row["id"]) for row in result.rows] == [("inc-1", "e1"), ("inc-1", "e2")]assert result.saved_states == []+ def test_alert_logs_follow_each_alerts_next_link(self) -> None:+ next_url = "https://api.opsgenie.com/v2/alerts/a1/logs?offset=o1&direction=next&limit=100&order=asc"+ result = _driver().run(+ "alert_logs",+ [+ _page([{"id": "a1"}]),+ ScriptedResponse(json={"data": [{"offset": "o1", "log": "created"}], "paging": {"next": next_url}}),+ ScriptedResponse(json={"data": [{"offset": "o2", "log": "closed"}], "paging": {"first": next_url}}),+ ],+ )++ assert result.raised is None+ assert result.paths == ["/v2/alerts", "/v2/alerts/a1/logs", "/v2/alerts/a1/logs"]+ assert result.requests[0].param("sort") == "createdAt"+ assert result.requests[1].param("order") == "asc"+ assert result.params("offset") == ["0", None, "o1"]+ assert [(row["alert_id"], row["offset"]) for row in result.rows] == [("a1", "o1"), ("a1", "o2")]++ def test_alert_logs_skip_alert_deleted_after_listing(self) -> None:+ result = _driver().run(+ "alert_logs",+ [+ _page([{"id": "gone"}, {"id": "a2"}]),+ ScriptedResponse(status=404, json={"message": "Alert does not exist"}),+ ScriptedResponse(json={"data": [{"offset": "o1", "log": "created"}]}),+ ],+ )++ assert result.raised is None+ assert [(row["alert_id"], row["offset"]) for row in result.rows] == [("a2", "o1")]++ def test_alert_logs_refuse_next_link_to_another_host(self) -> None:+ result = _driver().run(+ "alert_logs",+ [+ _page([{"id": "a1"}]),+ ScriptedResponse(+ json={"data": [{"offset": "o1"}], "paging": {"next": "https://evil.example.com/logs?offset=o1"}}+ ),+ ],+ )++ assert result.raised is not None+ assert [request.origin for request in result.requests] == ["https://api.opsgenie.com"] * 2+@pytest.mark.parametrize("endpoint,parent_path,child_body,expected",[