:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add logicmonitor device datasource and instance tables (#122649)
Tom Owersa2f765810903
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 9f68414749e8..83615f7d36e2 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -7442,10 +7442,10 @@ Today (6): `alerts`, `collectors`, `device_groups`, `devices`, `sdts`, `websitesDiffed against: <https://www.logicmonitor.com/swagger-ui-master/api-v3/dist/swagger.json>- [ ] `Collector groups (GET /setting/collector/groups)` — resolve collectorGroupId in existing collector rows for operational ownership reports (high)-- [ ] `Device datasources (GET /device/devices/{deviceId}/devicedatasources)` — resolve alert resourceTemplateId references and identify monitored datasource assignments (high)-- [ ] `Device datasource instances (GET /device/devices/{deviceId}/devicedatasources/{hdsId}/instances)` — resolve alert instanceId references to monitored components (high)-- [ ] `Device instance data (GET /device/devices/{deviceId}/devicedatasources/{hdsId}/instances/{id}/data)` — measure performance, utilization, and availability instead of relying only on alert counts (high)-- [ ] `Website graph data (GET /website/websites/{id}/graphs/{graphName}/data)` — measure website uptime and response times over time (high)+- [x] `Device datasources (GET /device/devices/{deviceId}/devicedatasources)` — resolve alert resourceTemplateId references and identify monitored datasource assignments (high). Added as `device_datasources` (fan-out over `devices`).+- [x] `Device datasource instances (GET /device/devices/{deviceId}/devicedatasources/{hdsId}/instances)` — resolve alert instanceId references to monitored components (high). Added as `device_datasource_instances`, read from `GET /device/devices/{id}/instances`, which returns the same instance rows for all of a device's datasources in one fan-out over `devices`.+- [ ] `Device instance data (GET /device/devices/{deviceId}/devicedatasources/{hdsId}/instances/{id}/data)` — measure performance, utilization, and availability instead of relying only on alert counts (high) — skipped: raw per-instance metric series, one paginated request per instance per sync after a device and instance fan-out, so a sync grows with every monitored instance and lands a row per poll interval+- [ ] `Website graph data (GET /website/websites/{id}/graphs/{graphName}/data)` — measure website uptime and response times over time (high) — skipped: it returns a chart render payload (colors, legends, scaled lines) for a graph name the spec does not list. `GET /website/websites/{websiteId}/checkpoints/{checkpointId}/data` is the raw-data alternative- [ ] `Device SDT history (GET /device/devices/{id}/historysdts)` — explain historical alert suppression and separate maintenance from service failures (medium)- [ ] `Opsnotes (GET /setting/opsnotes)` — connect incidents with operational notes and change annotations (medium)- [ ] `Audit logs (GET /setting/accesslogs)` — track administrative activity and changes associated with monitoring incidents (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.pyindex 21d09c71ca7d..9d4b44864e04 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py@@ -79,4 +79,33 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"disableAlerting": "Whether alerting is disabled for this website.",},},+ "device_datasources": {+ "description": "DataSources applied to each monitored device, with their monitoring and alert status.",+ "docs_url": API_DOCS_URL,+ "columns": {+ "id": "Identifier of the device DataSource, the hdsId used in instance and data requests.",+ "deviceId": "Identifier of the device.",+ "dataSourceId": "Identifier of the DataSource definition. Alerts reference it as resourceTemplateId.",+ "dataSourceName": "Name of the DataSource.",+ "instanceNumber": "Number of instances discovered for the DataSource on this device.",+ "monitoringInstanceNumber": "Number of instances that are monitored.",+ "stopMonitoring": "Whether monitoring is stopped for this DataSource on the device.",+ "createdOn": "Time when the DataSource was applied to the device, in Unix seconds.",+ },+ },+ "device_datasource_instances": {+ "description": "Instances discovered for each device DataSource, such as disks, interfaces, or processes.",+ "docs_url": API_DOCS_URL,+ "columns": {+ "id": "Unique instance identifier. Alerts reference it as instanceId.",+ "name": "Instance name, in the format DataSourceName-instanceAlias.",+ "displayName": "Instance alias shown in LogicMonitor.",+ "deviceId": "Identifier of the device.",+ "deviceDataSourceId": "Identifier of the device DataSource that owns the instance.",+ "dataSourceId": "Identifier of the DataSource definition.",+ "wildValue": "Value that identifies the instance when the collector queries the device.",+ "stopMonitoring": "Whether monitoring is disabled for the instance.",+ "disableAlerting": "Whether alerting is disabled for the instance.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.pyindex 3caa89cd2535..2290cf784bf3 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py@@ -1,7 +1,7 @@import reimport time-from collections.abc import Iterator-from dataclasses import replace+from collections.abc import Iterable, Iterator+from dataclasses import field, replacefrom typing import Anyfrom urllib.parse import urlsplit@@ -15,11 +15,15 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resrest_api_resource,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.auth import BearerTokenAuth+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (+ build_dependent_resource,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (OffsetPaginator,SinglePagePaginator,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.rest_client import RESTClient+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import ClientConfigfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceInputs, SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.logicmonitor import (@@ -27,7 +31,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_)from products.warehouse_sources.backend.temporal.data_imports.sources.logicmonitor.settings import (API_VERSION,+ DEVICE_FANOUTS,ENDPOINTS,+ FANOUT_PRIMARY_KEYS,FIELDS,MAX_ALERTS_PER_WINDOW,PAGE_SIZE,@@ -48,6 +54,19 @@ class LogicMonitorResumeConfig:window_end: int = 0sync_end: int = 0complete: bool = False+ fanout_state: dict[str, Any] | None = None+++@frozen+class FanoutEndpoint:+ name: str+ path: str+ page_size: int = PAGE_SIZE+ incremental_fields: list[Any] = field(default_factory=list)+ default_incremental_field: str | None = None+++FANOUT_ENDPOINTS = {name: FanoutEndpoint(name=name, path=ENDPOINTS[name]) for name in ("devices", *DEVICE_FANOUTS)}def portal_url(value: str) -> str:@@ -105,6 +124,17 @@ class LogicMonitorClient:self.base_url = f"{portal_url(config.portal_url)}/santaba/rest/"self.token = config.bearer_token+ def client_config(self, paginator: OffsetPaginator) -> ClientConfig:+ return {+ "base_url": self.base_url,+ "auth": {"type": "bearer", "token": self.token},+ "headers": {"X-Version": API_VERSION},+ "allowed_hosts": [],+ "allow_redirects": False,+ "request_timeout": 60,+ "paginator": paginator,+ }+def validate_credentials(self) -> tuple[bool, str | None]:client = RESTClient(base_url=self.base_url,@@ -135,6 +165,14 @@ class LogicMonitorClient:) -> SourceResponse:if inputs.schema_name not in ENDPOINTS:raise ValueError(f"Unknown LogicMonitor table: {inputs.schema_name}")+ if inputs.schema_name in DEVICE_FANOUTS:+ return SourceResponse(+ name=inputs.schema_name,+ items=lambda: self.device_fanout_rows(inputs, manager),+ primary_keys=FANOUT_PRIMARY_KEYS,+ sort_mode=None,+ on_complete=manager.clear_state,+ )return SourceResponse(name=inputs.schema_name,items=lambda: self.iter_rows(inputs, manager),@@ -143,6 +181,31 @@ class LogicMonitorClient:on_complete=manager.clear_state,)+ def device_fanout_rows(+ self, inputs: SourceInputs, manager: ResumableSourceManager[LogicMonitorResumeConfig]+ ) -> Iterable[Any]:+ saved = manager.load_state() if manager.can_resume() else None++ def save_checkpoint(fanout_state: dict[str, Any] | None) -> None:+ if fanout_state:+ manager.save_state(LogicMonitorResumeConfig(fanout_state=fanout_state))++ return build_dependent_resource(+ endpoint_configs=FANOUT_ENDPOINTS,+ child_endpoint=inputs.schema_name,+ fanout=DEVICE_FANOUTS[inputs.schema_name],+ client_config=self.client_config(OffsetPaginator(limit=PAGE_SIZE, limit_param="size", total_path="total")),+ path_format_values={},+ team_id=inputs.team_id,+ job_id=inputs.job_id,+ db_incremental_field_last_value=None,+ page_size_param=None,+ parent_endpoint_extra={"data_selector": "items", "data_selector_required": True},+ child_endpoint_extra={"data_selector": "items", "data_selector_required": True},+ resume_hook=save_checkpoint,+ initial_paginator_state=saved.fanout_state if saved else None,+ )+def iter_rows(self, inputs: SourceInputs, manager: ResumableSourceManager[LogicMonitorResumeConfig]) -> Iterator[list[dict[str, Any]]]:@@ -159,15 +222,7 @@ class LogicMonitorClient:params["sort"] = "startEpoch"paginator_class = AlertPaginator if is_alerts else OffsetPaginatorconfig: RESTAPIConfig = {- "client": {- "base_url": self.base_url,- "auth": {"type": "bearer", "token": self.token},- "headers": {"X-Version": API_VERSION},- "allowed_hosts": [],- "allow_redirects": False,- "request_timeout": 60,- "paginator": paginator_class(limit=PAGE_SIZE, limit_param="size", total_path="total"),- },+ "client": self.client_config(paginator_class(limit=PAGE_SIZE, limit_param="size", total_path="total")),"resources": [{"name": inputs.schema_name,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.pyindex b83a34cad16e..efe351ade432 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py@@ -1,3 +1,7 @@+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (+ DependentEndpointConfig,+)+API_VERSION = "3"API_DOCS_URL = "https://www.logicmonitor.com/support/rest-api-v3-swagger-documentation"PAGE_SIZE = 1000@@ -10,6 +14,9 @@ ENDPOINTS: dict[str, str] = {"collectors": "setting/collector/collectors","sdts": "sdt/sdts","websites": "website/websites",+ "device_datasources": "device/devices/{deviceId}/devicedatasources",+ # Lists every datasource instance of a device, so one request per device covers all of its datasources.+ "device_datasource_instances": "device/devices/{deviceId}/instances",}# Select inventory fields because device properties and collector configuration can contain credentials.@@ -18,5 +25,32 @@ FIELDS: dict[str, str] = {"device_groups": "id,name,description,fullPath,parentId,createdOn,disableAlerting","collectors": "id,hostname,description,createdOn,updatedOn,collectorGroupId,numberOfHosts,isDown,build","websites": "id,name,description,type,overallAlertLevel,disableAlerting",+ "device_datasources": (+ "id,deviceId,deviceName,deviceDisplayName,dataSourceId,dataSourceName,dataSourceDisplayName,"+ "dataSourceDescription,dataSourceType,collectMethod,groupName,isMultiple,instanceNumber,"+ "monitoringInstanceNumber,alertStatus,alertStatusPriority,alertDisableStatus,sdtStatus,"+ "stopMonitoring,autoDiscovery,status,createdOn,updatedOn,assignedOn"+ ),+ "device_datasource_instances": (+ "id,name,displayName,description,deviceId,deviceDisplayName,deviceDataSourceId,dataSourceId,"+ "dataSourceType,groupId,groupName,collectorId,wildValue,wildValue2,stopMonitoring,disableAlerting,"+ "isUNCInstance,lockDescription"+ ),}PRIMARY_KEYS = ["id"]++DEVICE_FANOUTS: dict[str, DependentEndpointConfig] = {+ name: DependentEndpointConfig(+ parent_name="devices",+ resolve_param="deviceId",+ resolve_field="id",+ include_from_parent=[],+ parent_params={"fields": "id"},+ child_params={"fields": FIELDS[name]},+ # A device deleted after the device list was read has no children left to sync.+ child_response_actions=[{"status_code": 404, "action": "ignore"}],+ )+ for name in ("device_datasources", "device_datasource_instances")+}+# Device datasource and instance ids are only documented per device.+FANOUT_PRIMARY_KEYS = ["deviceId", "id"]diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.pyindex ad43b3cf9dcf..68b9c26e4ebb 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py@@ -19,6 +19,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.logicmonitLogicMonitorResumeConfig,portal_url,)+from products.warehouse_sources.backend.temporal.data_imports.sources.logicmonitor.settings import FIELDSfrom products.warehouse_sources.backend.temporal.data_imports.sources.logicmonitor.source import LogicmonitorSourceBASE_PATH = "/santaba/rest/device/devices"@@ -100,6 +101,61 @@ def test_alert_growth_after_a_page_fails_without_replaying() -> None:assert len(result.requests) == 2+ "table, child_path",+ [("device_datasources", "devicedatasources"), ("device_datasource_instances", "instances")],+)[email protected]("resume", [False, True])+def test_device_fanout_pages_each_device_and_resumes(table: str, child_path: str, resume: bool) -> None:+ devices = ScriptedResponse(json={"items": [{"id": 7}, {"id": 9}], "total": 2})+ first_device = [+ ScriptedResponse(json={"items": [{"id": i, "deviceId": 7} for i in range(1000)], "total": 1001}),+ ScriptedResponse(json={"items": [{"id": 1000, "deviceId": 7}], "total": 1001}),+ ]+ second_device = ScriptedResponse(json={"items": [{"id": 1, "deviceId": 9}], "total": 1})+ first_path = f"device/devices/7/{child_path}"++ result = driver().run(+ table,+ [devices, second_device] if resume else [devices, *first_device, second_device],+ resume_state=LogicMonitorResumeConfig(fanout_state={"completed": [first_path], "current": None})+ if resume+ else None,+ )++ assert result.raised is None+ assert result.response is not None and result.response.primary_keys == ["deviceId", "id"]+ expected_rows = [(9, 1)] if resume else [*((7, i) for i in range(1001)), (9, 1)]+ assert [(row["deviceId"], row["id"]) for row in result.rows] == expected_rows+ child_paths = [f"/santaba/rest/device/devices/9/{child_path}"]+ if not resume:+ child_paths = [f"/santaba/rest/{first_path}"] * 2 + child_paths+ assert result.paths == [BASE_PATH, *child_paths]+ assert result.params("fields") == ["id"] + [FIELDS[table]] * len(child_paths)+ assert result.params("offset") == ["0"] + (["0"] if resume else ["0", "1000", "0"])+ assert result.final_state is not None and result.final_state.fanout_state["completed"] == [+ first_path,+ f"device/devices/9/{child_path}",+ ]+ assert result.completed_state is None++[email protected]("table", ["device_datasources", "device_datasource_instances"])+def test_device_fanout_skips_device_deleted_mid_sync(table: str) -> None:+ result = driver().run(+ table,+ [+ ScriptedResponse(json={"items": [{"id": 7}, {"id": 9}], "total": 2}),+ ScriptedResponse(status=404, json={"errorCode": 1404, "errorMessage": "fake not found"}),+ ScriptedResponse(json={"items": [{"id": 1, "deviceId": 9}], "total": 1}),+ ],+ )++ assert result.raised is None+ assert [(row["deviceId"], row["id"]) for row in result.rows] == [(9, 1)]+ assert len(result.requests) == 3++@pytest.mark.parametrize("status", [401, 403, 404])def test_error_mapping(status: int) -> None:answer = ScriptedResponse(status=status, json={"errorCode": status, "errorMessage": "fake error"})