:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 4 missing tables to the LogicMonitor source (#125202)
Tom Owerse0181e011a48
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex c97e487e4e3e..160c8cf4b6d8 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -7442,14 +7442,14 @@ Today (6): `alerts`, `collectors`, `device_groups`, `devices`, `sdts`, `websitesDiffed against: <https://www.logicmonitor.com/swagger-ui-master/api-v3/dist/swagger.json>-- [ ] `Collector groups (GET /setting/collector/groups)` — resolve collectorGroupId in existing collector rows for operational ownership reports (high)+- [x] `Collector groups (GET /setting/collector/groups)` — resolve collectorGroupId in existing collector rows for operational ownership reports (high). Added as `collector_groups` (full refresh).- [x] `Device datasources (GET /device/devices/{deviceId}/devicedatasources)` — resolve alert resourceTemplateId references and identify monitored datasource assignments (high). Added as `device_datasources` (fan-out over `devices`).- [x] `Device datasource instances (GET /device/devices/{deviceId}/devicedatasources/{hdsId}/instances)` — resolve alert instanceId references to monitored components (high). Added as `device_datasource_instances`, read from `GET /device/devices/{id}/instances`, which returns the same instance rows for all of a device's datasources in one fan-out over `devices`.- [ ] `Device instance data (GET /device/devices/{deviceId}/devicedatasources/{hdsId}/instances/{id}/data)` — measure performance, utilization, and availability instead of relying only on alert counts (high) — skipped: raw per-instance metric series, one paginated request per instance per sync after a device and instance fan-out, so a sync grows with every monitored instance and lands a row per poll interval- [ ] `Website graph data (GET /website/websites/{id}/graphs/{graphName}/data)` — measure website uptime and response times over time (high) — skipped: it returns a chart render payload (colors, legends, scaled lines) for a graph name the spec does not list. `GET /website/websites/{websiteId}/checkpoints/{checkpointId}/data` is the raw-data alternative-- [ ] `Device SDT history (GET /device/devices/{id}/historysdts)` — explain historical alert suppression and separate maintenance from service failures (medium)-- [ ] `Opsnotes (GET /setting/opsnotes)` — connect incidents with operational notes and change annotations (medium)-- [ ] `Audit logs (GET /setting/accesslogs)` — track administrative activity and changes associated with monitoring incidents (medium)+- [x] `Device SDT history (GET /device/devices/{id}/historysdts)` — explain historical alert suppression and separate maintenance from service failures (medium). Added as `device_sdt_history` (fan-out over `devices`, full refresh).+- [x] `Opsnotes (GET /setting/opsnotes)` — connect incidents with operational notes and change annotations (medium). Added as `ops_notes` (full refresh).+- [x] `Audit logs (GET /setting/accesslogs)` — track administrative activity and changes associated with monitoring incidents (medium). Added as `audit_logs` (incremental on `happenedOn`).- [ ] `FOCUS Cost and Usage export (GET /usage/focusCostUsage)` — attribute monitoring usage and cost to resources and tenants (medium)Note: Fetched the v3 Swagger JSON linked by the official reference and enumerated GET operations at every nesting depth. Metric imports need device, datasource, and instance fan-out plus bounded time ranges. Existing collector fields include collectorGroupId, but no group table resolves it. Excluded credentials, configuration payloads, installers, diagnostic commands, and alert settings.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.pyindex 9d4b44864e04..4e83f0033b9c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py@@ -108,4 +108,55 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"disableAlerting": "Whether alerting is disabled for the instance.",},},+ "collector_groups": {+ "description": "Collector groups that organize collectors and balance monitoring load across them.",+ "docs_url": API_DOCS_URL,+ "columns": {+ "id": "Unique collector group identifier. Collectors reference it as collectorGroupId.",+ "name": "Name of the collector group.",+ "createOn": "Time when the group was created, in Unix seconds.",+ "numOfCollectors": "Number of collectors in the group.",+ "numOfHosts": "Number of hosts that belong to the group.",+ "autoBalance": "Whether the group balances devices across its collectors automatically.",+ "mismatchVersion": "Whether the collectors in the group run different versions.",+ },+ },+ "ops_notes": {+ "description": "Ops notes that annotate graphs and dashboards with operational events, such as deployments and changes.",+ "docs_url": API_DOCS_URL,+ "columns": {+ "id": "Unique ops note identifier.",+ "note": "Text of the note.",+ "createdBy": "User that created the note.",+ "happenOnInSec": "Time associated with the note, in Unix seconds.",+ "scopes": "Devices, services, and groups the note applies to. A note with no scope applies to all of them.",+ "tags": "Tags associated with the note.",+ },+ },+ "audit_logs": {+ "description": "Audit log entries that record user and API actions in the portal.",+ "docs_url": API_DOCS_URL,+ "columns": {+ "id": "Unique audit log entry identifier.",+ "happenedOn": "Time when the action occurred, in Unix seconds.",+ "username": "User that performed the action.",+ "ip": "IP address that the action came from.",+ "description": "Description of the action.",+ },+ },+ "device_sdt_history": {+ "description": "Past scheduled downtime (SDT) periods that suppressed alerts for each device.",+ "docs_url": API_DOCS_URL,+ "columns": {+ "id": "Identifier of the SDT.",+ "deviceId": "Identifier of the device.",+ "itemId": "Identifier of the resource in SDT, such as the device or one of its groups.",+ "type": "SDT type.",+ "admin": "User that added the SDT.",+ "comment": "Comment associated with the SDT.",+ "approximateStartEpoch": "Time when the SDT started.",+ "approximateEndEpoch": "Time when the SDT ended.",+ "duration": "Duration of the SDT, in minutes.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.pyindex 2290cf784bf3..8c3bcd82ec89 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py@@ -103,6 +103,12 @@ def validate_portal_host(value: str, team_id: int) -> tuple[bool, str | None]:return ValidateDatabaseHostMixin().is_database_host_valid(urlsplit(url).hostname or "", team_id)+def audit_log_start(inputs: SourceInputs) -> int:+ if inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:+ return int(inputs.db_incremental_field_last_value)+ return 0++class AlertWindowTooLarge(Exception):pass@@ -220,6 +226,10 @@ class LogicMonitorClient:if is_alerts:params["filter"] = f'cleared:"*",startEpoch>:{state.window_start},startEpoch<{state.window_end}'params["sort"] = "startEpoch"+ elif inputs.schema_name == "audit_logs":+ # The upper bound keeps entries written during the sync from shifting the offset pages.+ params["filter"] = f"happenedOn>:{audit_log_start(inputs)},happenedOn<{state.sync_end}"+ params["sort"] = "happenedOn"paginator_class = AlertPaginator if is_alerts else OffsetPaginatorconfig: RESTAPIConfig = {"client": self.client_config(paginator_class(limit=PAGE_SIZE, limit_param="size", total_path="total")),diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.pyindex efe351ade432..be121141ff59 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py@@ -1,6 +1,7 @@from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (DependentEndpointConfig,)+from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldTypeAPI_VERSION = "3"API_DOCS_URL = "https://www.logicmonitor.com/support/rest-api-v3-swagger-documentation"@@ -12,11 +13,15 @@ ENDPOINTS: dict[str, str] = {"devices": "device/devices","device_groups": "device/groups","collectors": "setting/collector/collectors",+ "collector_groups": "setting/collector/groups","sdts": "sdt/sdts","websites": "website/websites",+ "ops_notes": "setting/opsnotes",+ "audit_logs": "setting/accesslogs","device_datasources": "device/devices/{deviceId}/devicedatasources",# Lists every datasource instance of a device, so one request per device covers all of its datasources."device_datasource_instances": "device/devices/{deviceId}/instances",+ "device_sdt_history": "device/devices/{deviceId}/historysdts",}# Select inventory fields because device properties and collector configuration can contain credentials.@@ -24,6 +29,10 @@ FIELDS: dict[str, str] = {"devices": "id,name,displayName,description,createdOn,updatedOn,preferredCollectorId,hostGroupIds,disableAlerting","device_groups": "id,name,description,fullPath,parentId,createdOn,disableAlerting","collectors": "id,hostname,description,createdOn,updatedOn,collectorGroupId,numberOfHosts,isDown,build",+ "collector_groups": (+ "id,name,description,createOn,platform,numOfCollectors,numOfHosts,numOfInstances,autoBalance,"+ "autoBalanceStrategy,mismatchVersion"+ ),"websites": "id,name,description,type,overallAlertLevel,disableAlerting","device_datasources": ("id,deviceId,deviceName,deviceDisplayName,dataSourceId,dataSourceName,dataSourceDisplayName,"@@ -39,18 +48,38 @@ FIELDS: dict[str, str] = {}PRIMARY_KEYS = ["id"]-DEVICE_FANOUTS: dict[str, DependentEndpointConfig] = {- name: DependentEndpointConfig(++def device_fanout(child_fields: str | None, *, include_device_id: bool = False) -> DependentEndpointConfig:+ return DependentEndpointConfig(parent_name="devices",resolve_param="deviceId",resolve_field="id",- include_from_parent=[],+ include_from_parent=["id"] if include_device_id else [],+ parent_field_renames={"id": "deviceId"} if include_device_id else {},parent_params={"fields": "id"},- child_params={"fields": FIELDS[name]},+ child_params={"fields": child_fields} if child_fields else {},# A device deleted after the device list was read has no children left to sync.child_response_actions=[{"status_code": 404, "action": "ignore"}],)- for name in ("device_datasources", "device_datasource_instances")+++DEVICE_FANOUTS: dict[str, DependentEndpointConfig] = {+ "device_datasources": device_fanout(FIELDS["device_datasources"]),+ "device_datasource_instances": device_fanout(FIELDS["device_datasource_instances"]),+ # SDT history rows name the SDT target in itemId, which can be a device group, so the device id comes from the parent.+ "device_sdt_history": device_fanout(None, include_device_id=True),}-# Device datasource and instance ids are only documented per device.+# Device datasource, instance, and SDT history ids are only documented per device.FANOUT_PRIMARY_KEYS = ["deviceId", "id"]++# Audit log entries never change after they are written, so new entries can be read by their time.+INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {+ "audit_logs": [+ {+ "label": "happenedOn",+ "type": IncrementalFieldType.Integer,+ "field": "happenedOn",+ "field_type": IncrementalFieldType.Integer,+ }+ ],+}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/source.pyindex 79c28f2a0986..f83102591848 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/source.py@@ -37,6 +37,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.logicmonitAPI_DOCS_URL,API_VERSION,ENDPOINTS,+ INCREMENTAL_FIELDS,)from products.warehouse_sources.backend.types import ExternalDataSourceType@@ -82,7 +83,7 @@ class LogicmonitorSource(ResumableSource[LogicmonitorSourceConfig, LogicMonitorRapi_version: str | None = None,) -> list[SourceSchema]:# A start-time watermark misses later acknowledgments and clear events for older alerts.- return build_endpoint_schemas(ENDPOINTS, {}, names)+ return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names)def validate_credentials(self,@@ -118,7 +119,8 @@ class LogicmonitorSource(ResumableSource[LogicmonitorSourceConfig, LogicMonitorRlabel="LogicMonitor",caption=("Create a bearer token in Settings > Users & Roles > API Tokens > Bearer. "- "Give its user view access to Resources, Dashboards, Collectors, and Websites for the tables you select. "+ "Give its user view access to Resources, Dashboards, Collectors, Websites, Ops Notes, and Audit Logs "+ "for the tables you select. ""Alerts use full refresh to capture acknowledgments and cleared status."),iconPath="/static/services/logicmonitor.png",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.pyindex 68b9c26e4ebb..706c99125b33 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py@@ -102,17 +102,26 @@ def test_alert_growth_after_a_page_fails_without_replaying() -> None:@pytest.mark.parametrize(- "table, child_path",- [("device_datasources", "devicedatasources"), ("device_datasource_instances", "instances")],+ "table, child_path, rows_carry_device_id",+ [+ ("device_datasources", "devicedatasources", True),+ ("device_datasource_instances", "instances", True),+ ("device_sdt_history", "historysdts", False),+ ],)@pytest.mark.parametrize("resume", [False, True])-def test_device_fanout_pages_each_device_and_resumes(table: str, child_path: str, resume: bool) -> None:+def test_device_fanout_pages_each_device_and_resumes(+ table: str, child_path: str, rows_carry_device_id: bool, resume: bool+) -> None:+ def child_row(row_id: int, device_id: int) -> dict[str, Any]:+ return {"id": row_id, "deviceId": device_id} if rows_carry_device_id else {"id": row_id, "itemId": 3}+devices = ScriptedResponse(json={"items": [{"id": 7}, {"id": 9}], "total": 2})first_device = [- ScriptedResponse(json={"items": [{"id": i, "deviceId": 7} for i in range(1000)], "total": 1001}),- ScriptedResponse(json={"items": [{"id": 1000, "deviceId": 7}], "total": 1001}),+ ScriptedResponse(json={"items": [child_row(i, 7) for i in range(1000)], "total": 1001}),+ ScriptedResponse(json={"items": [child_row(1000, 7)], "total": 1001}),]- second_device = ScriptedResponse(json={"items": [{"id": 1, "deviceId": 9}], "total": 1})+ second_device = ScriptedResponse(json={"items": [child_row(1, 9)], "total": 1})first_path = f"device/devices/7/{child_path}"result = driver().run(@@ -131,7 +140,7 @@ def test_device_fanout_pages_each_device_and_resumes(table: str, child_path: strif not resume:child_paths = [f"/santaba/rest/{first_path}"] * 2 + child_pathsassert result.paths == [BASE_PATH, *child_paths]- assert result.params("fields") == ["id"] + [FIELDS[table]] * len(child_paths)+ assert result.params("fields") == ["id"] + [FIELDS.get(table)] * len(child_paths)assert result.params("offset") == ["0"] + (["0"] if resume else ["0", "1000", "0"])assert result.final_state is not None and result.final_state.fanout_state["completed"] == [first_path,@@ -140,7 +149,7 @@ def test_device_fanout_pages_each_device_and_resumes(table: str, child_path: strassert result.completed_state is None[email protected]("table", ["device_datasources", "device_datasource_instances"])[email protected]("table", ["device_datasources", "device_datasource_instances", "device_sdt_history"])def test_device_fanout_skips_device_deleted_mid_sync(table: str) -> None:result = driver().run(table,@@ -156,6 +165,62 @@ def test_device_fanout_skips_device_deleted_mid_sync(table: str) -> None:assert len(result.requests) == 3+ "inputs, resume_state, expected_filter",+ [+ ({}, None, "happenedOn>:0,happenedOn<{sync_end}"),+ (+ {"incremental_field": "happenedOn", "db_incremental_field_last_value": 1700000000},+ None,+ "happenedOn>:1700000000,happenedOn<{sync_end}",+ ),+ (+ {"incremental_field": "happenedOn", "db_incremental_field_last_value": "1700000000"},+ None,+ "happenedOn>:1700000000,happenedOn<{sync_end}",+ ),+ (+ {+ "incremental_field": "happenedOn",+ "db_incremental_field_last_value": 1700000000,+ "should_use_incremental_field": False,+ },+ None,+ "happenedOn>:0,happenedOn<{sync_end}",+ ),+ (+ {"incremental_field": "happenedOn", "db_incremental_field_last_value": 1700000000},+ LogicMonitorResumeConfig(offset=1000, window_end=1800000000, sync_end=1800000000),+ "happenedOn>:1700000000,happenedOn<1800000000",+ ),+ ],+)+def test_audit_logs_bound_each_sync(+ inputs: dict[str, Any], resume_state: LogicMonitorResumeConfig | None, expected_filter: str+) -> None:+ first_offset = resume_state.offset if resume_state else 0+ result = driver().run(+ "audit_logs",+ [+ ScriptedResponse(json={"items": [{"id": "a", "happenedOn": 1}] * 1000, "total": first_offset + 1001}),+ ScriptedResponse(json={"items": [{"id": "b", "happenedOn": 2}], "total": first_offset + 1001}),+ ],+ resume_state=resume_state,+ **inputs,+ )++ assert result.raised is None+ assert len(result.rows) == 1001+ assert result.paths == ["/santaba/rest/setting/accesslogs"] * 2+ filters = result.params("filter")+ assert filters[0] is not None+ sync_end = filters[0].rsplit("<", 1)[1]+ assert sync_end.isdigit()+ assert filters == [expected_filter.format(sync_end=sync_end)] * 2+ assert result.params("sort") == ["happenedOn"] * 2+ assert result.params("offset") == [str(first_offset), str(first_offset + 1000)]++@pytest.mark.parametrize("status", [401, 403, 404])def test_error_mapping(status: int) -> None:answer = ScriptedResponse(status=status, json={"errorCode": status, "errorMessage": "fake error"})