Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add logicmonitor device datasource and instance tables (#122649)

Tom Owersa2f765810903
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 9f68414749e8..83615f7d36e2 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -7442,10 +7442,10 @@ Today (6): `alerts`, `collectors`, `device_groups`, `devices`, `sdts`, `websites
Diffed against: <https://www.logicmonitor.com/swagger-ui-master/api-v3/dist/swagger.json>
- [ ] `Collector groups (GET /setting/collector/groups)` — resolve collectorGroupId in existing collector rows for operational ownership reports (high)
-- [ ] `Device datasources (GET /device/devices/{deviceId}/devicedatasources)` — resolve alert resourceTemplateId references and identify monitored datasource assignments (high)
-- [ ] `Device datasource instances (GET /device/devices/{deviceId}/devicedatasources/{hdsId}/instances)` — resolve alert instanceId references to monitored components (high)
-- [ ] `Device instance data (GET /device/devices/{deviceId}/devicedatasources/{hdsId}/instances/{id}/data)` — measure performance, utilization, and availability instead of relying only on alert counts (high)
-- [ ] `Website graph data (GET /website/websites/{id}/graphs/{graphName}/data)` — measure website uptime and response times over time (high)
+- [x] `Device datasources (GET /device/devices/{deviceId}/devicedatasources)` — resolve alert resourceTemplateId references and identify monitored datasource assignments (high). Added as `device_datasources` (fan-out over `devices`).
+- [x] `Device datasource instances (GET /device/devices/{deviceId}/devicedatasources/{hdsId}/instances)` — resolve alert instanceId references to monitored components (high). Added as `device_datasource_instances`, read from `GET /device/devices/{id}/instances`, which returns the same instance rows for all of a device's datasources in one fan-out over `devices`.
+- [ ] `Device instance data (GET /device/devices/{deviceId}/devicedatasources/{hdsId}/instances/{id}/data)` — measure performance, utilization, and availability instead of relying only on alert counts (high) — skipped: raw per-instance metric series, one paginated request per instance per sync after a device and instance fan-out, so a sync grows with every monitored instance and lands a row per poll interval
+- [ ] `Website graph data (GET /website/websites/{id}/graphs/{graphName}/data)` — measure website uptime and response times over time (high) — skipped: it returns a chart render payload (colors, legends, scaled lines) for a graph name the spec does not list. `GET /website/websites/{websiteId}/checkpoints/{checkpointId}/data` is the raw-data alternative
- [ ] `Device SDT history (GET /device/devices/{id}/historysdts)` — explain historical alert suppression and separate maintenance from service failures (medium)
- [ ] `Opsnotes (GET /setting/opsnotes)` — connect incidents with operational notes and change annotations (medium)
- [ ] `Audit logs (GET /setting/accesslogs)` — track administrative activity and changes associated with monitoring incidents (medium)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py
index 21d09c71ca7d..9d4b44864e04 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py
@@ -79,4 +79,33 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"disableAlerting": "Whether alerting is disabled for this website.",
},
},
+ "device_datasources": {
+ "description": "DataSources applied to each monitored device, with their monitoring and alert status.",
+ "docs_url": API_DOCS_URL,
+ "columns": {
+ "id": "Identifier of the device DataSource, the hdsId used in instance and data requests.",
+ "deviceId": "Identifier of the device.",
+ "dataSourceId": "Identifier of the DataSource definition. Alerts reference it as resourceTemplateId.",
+ "dataSourceName": "Name of the DataSource.",
+ "instanceNumber": "Number of instances discovered for the DataSource on this device.",
+ "monitoringInstanceNumber": "Number of instances that are monitored.",
+ "stopMonitoring": "Whether monitoring is stopped for this DataSource on the device.",
+ "createdOn": "Time when the DataSource was applied to the device, in Unix seconds.",
+ },
+ },
+ "device_datasource_instances": {
+ "description": "Instances discovered for each device DataSource, such as disks, interfaces, or processes.",
+ "docs_url": API_DOCS_URL,
+ "columns": {
+ "id": "Unique instance identifier. Alerts reference it as instanceId.",
+ "name": "Instance name, in the format DataSourceName-instanceAlias.",
+ "displayName": "Instance alias shown in LogicMonitor.",
+ "deviceId": "Identifier of the device.",
+ "deviceDataSourceId": "Identifier of the device DataSource that owns the instance.",
+ "dataSourceId": "Identifier of the DataSource definition.",
+ "wildValue": "Value that identifies the instance when the collector queries the device.",
+ "stopMonitoring": "Whether monitoring is disabled for the instance.",
+ "disableAlerting": "Whether alerting is disabled for the instance.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py
index 3caa89cd2535..2290cf784bf3 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py
@@ -1,7 +1,7 @@
import re
import time
-from collections.abc import Iterator
-from dataclasses import replace
+from collections.abc import Iterable, Iterator
+from dataclasses import field, replace
from typing import Any
from urllib.parse import urlsplit
@@ -15,11 +15,15 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
rest_api_resource,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.auth import BearerTokenAuth
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
+ build_dependent_resource,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (
OffsetPaginator,
SinglePagePaginator,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.rest_client import RESTClient
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import ClientConfig
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceInputs, SourceResponse
from products.warehouse_sources.backend.temporal.data_imports.sources.generated_configs.logicmonitor import (
@@ -27,7 +31,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_
)
from products.warehouse_sources.backend.temporal.data_imports.sources.logicmonitor.settings import (
API_VERSION,
+ DEVICE_FANOUTS,
ENDPOINTS,
+ FANOUT_PRIMARY_KEYS,
FIELDS,
MAX_ALERTS_PER_WINDOW,
PAGE_SIZE,
@@ -48,6 +54,19 @@ class LogicMonitorResumeConfig:
window_end: int = 0
sync_end: int = 0
complete: bool = False
+ fanout_state: dict[str, Any] | None = None
+
+
+@frozen
+class FanoutEndpoint:
+ name: str
+ path: str
+ page_size: int = PAGE_SIZE
+ incremental_fields: list[Any] = field(default_factory=list)
+ default_incremental_field: str | None = None
+
+
+FANOUT_ENDPOINTS = {name: FanoutEndpoint(name=name, path=ENDPOINTS[name]) for name in ("devices", *DEVICE_FANOUTS)}
def portal_url(value: str) -> str:
@@ -105,6 +124,17 @@ class LogicMonitorClient:
self.base_url = f"{portal_url(config.portal_url)}/santaba/rest/"
self.token = config.bearer_token
+ def client_config(self, paginator: OffsetPaginator) -> ClientConfig:
+ return {
+ "base_url": self.base_url,
+ "auth": {"type": "bearer", "token": self.token},
+ "headers": {"X-Version": API_VERSION},
+ "allowed_hosts": [],
+ "allow_redirects": False,
+ "request_timeout": 60,
+ "paginator": paginator,
+ }
+
def validate_credentials(self) -> tuple[bool, str | None]:
client = RESTClient(
base_url=self.base_url,
@@ -135,6 +165,14 @@ class LogicMonitorClient:
) -> SourceResponse:
if inputs.schema_name not in ENDPOINTS:
raise ValueError(f"Unknown LogicMonitor table: {inputs.schema_name}")
+ if inputs.schema_name in DEVICE_FANOUTS:
+ return SourceResponse(
+ name=inputs.schema_name,
+ items=lambda: self.device_fanout_rows(inputs, manager),
+ primary_keys=FANOUT_PRIMARY_KEYS,
+ sort_mode=None,
+ on_complete=manager.clear_state,
+ )
return SourceResponse(
name=inputs.schema_name,
items=lambda: self.iter_rows(inputs, manager),
@@ -143,6 +181,31 @@ class LogicMonitorClient:
on_complete=manager.clear_state,
)
+ def device_fanout_rows(
+ self, inputs: SourceInputs, manager: ResumableSourceManager[LogicMonitorResumeConfig]
+ ) -> Iterable[Any]:
+ saved = manager.load_state() if manager.can_resume() else None
+
+ def save_checkpoint(fanout_state: dict[str, Any] | None) -> None:
+ if fanout_state:
+ manager.save_state(LogicMonitorResumeConfig(fanout_state=fanout_state))
+
+ return build_dependent_resource(
+ endpoint_configs=FANOUT_ENDPOINTS,
+ child_endpoint=inputs.schema_name,
+ fanout=DEVICE_FANOUTS[inputs.schema_name],
+ client_config=self.client_config(OffsetPaginator(limit=PAGE_SIZE, limit_param="size", total_path="total")),
+ path_format_values={},
+ team_id=inputs.team_id,
+ job_id=inputs.job_id,
+ db_incremental_field_last_value=None,
+ page_size_param=None,
+ parent_endpoint_extra={"data_selector": "items", "data_selector_required": True},
+ child_endpoint_extra={"data_selector": "items", "data_selector_required": True},
+ resume_hook=save_checkpoint,
+ initial_paginator_state=saved.fanout_state if saved else None,
+ )
+
def iter_rows(
self, inputs: SourceInputs, manager: ResumableSourceManager[LogicMonitorResumeConfig]
) -> Iterator[list[dict[str, Any]]]:
@@ -159,15 +222,7 @@ class LogicMonitorClient:
params["sort"] = "startEpoch"
paginator_class = AlertPaginator if is_alerts else OffsetPaginator
config: RESTAPIConfig = {
- "client": {
- "base_url": self.base_url,
- "auth": {"type": "bearer", "token": self.token},
- "headers": {"X-Version": API_VERSION},
- "allowed_hosts": [],
- "allow_redirects": False,
- "request_timeout": 60,
- "paginator": paginator_class(limit=PAGE_SIZE, limit_param="size", total_path="total"),
- },
+ "client": self.client_config(paginator_class(limit=PAGE_SIZE, limit_param="size", total_path="total")),
"resources": [
{
"name": inputs.schema_name,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py
index b83a34cad16e..efe351ade432 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py
@@ -1,3 +1,7 @@
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
+ DependentEndpointConfig,
+)
+
API_VERSION = "3"
API_DOCS_URL = "https://www.logicmonitor.com/support/rest-api-v3-swagger-documentation"
PAGE_SIZE = 1000
@@ -10,6 +14,9 @@ ENDPOINTS: dict[str, str] = {
"collectors": "setting/collector/collectors",
"sdts": "sdt/sdts",
"websites": "website/websites",
+ "device_datasources": "device/devices/{deviceId}/devicedatasources",
+ # Lists every datasource instance of a device, so one request per device covers all of its datasources.
+ "device_datasource_instances": "device/devices/{deviceId}/instances",
}
# Select inventory fields because device properties and collector configuration can contain credentials.
@@ -18,5 +25,32 @@ FIELDS: dict[str, str] = {
"device_groups": "id,name,description,fullPath,parentId,createdOn,disableAlerting",
"collectors": "id,hostname,description,createdOn,updatedOn,collectorGroupId,numberOfHosts,isDown,build",
"websites": "id,name,description,type,overallAlertLevel,disableAlerting",
+ "device_datasources": (
+ "id,deviceId,deviceName,deviceDisplayName,dataSourceId,dataSourceName,dataSourceDisplayName,"
+ "dataSourceDescription,dataSourceType,collectMethod,groupName,isMultiple,instanceNumber,"
+ "monitoringInstanceNumber,alertStatus,alertStatusPriority,alertDisableStatus,sdtStatus,"
+ "stopMonitoring,autoDiscovery,status,createdOn,updatedOn,assignedOn"
+ ),
+ "device_datasource_instances": (
+ "id,name,displayName,description,deviceId,deviceDisplayName,deviceDataSourceId,dataSourceId,"
+ "dataSourceType,groupId,groupName,collectorId,wildValue,wildValue2,stopMonitoring,disableAlerting,"
+ "isUNCInstance,lockDescription"
+ ),
}
PRIMARY_KEYS = ["id"]
+
+DEVICE_FANOUTS: dict[str, DependentEndpointConfig] = {
+ name: DependentEndpointConfig(
+ parent_name="devices",
+ resolve_param="deviceId",
+ resolve_field="id",
+ include_from_parent=[],
+ parent_params={"fields": "id"},
+ child_params={"fields": FIELDS[name]},
+ # A device deleted after the device list was read has no children left to sync.
+ child_response_actions=[{"status_code": 404, "action": "ignore"}],
+ )
+ for name in ("device_datasources", "device_datasource_instances")
+}
+# Device datasource and instance ids are only documented per device.
+FANOUT_PRIMARY_KEYS = ["deviceId", "id"]
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py
index ad43b3cf9dcf..68b9c26e4ebb 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py
@@ -19,6 +19,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.logicmonit
LogicMonitorResumeConfig,
portal_url,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.logicmonitor.settings import FIELDS
from products.warehouse_sources.backend.temporal.data_imports.sources.logicmonitor.source import LogicmonitorSource
BASE_PATH = "/santaba/rest/device/devices"
@@ -100,6 +101,61 @@ def test_alert_growth_after_a_page_fails_without_replaying() -> None:
assert len(result.requests) == 2
+ "table, child_path",
+ [("device_datasources", "devicedatasources"), ("device_datasource_instances", "instances")],
+)
[email protected]("resume", [False, True])
+def test_device_fanout_pages_each_device_and_resumes(table: str, child_path: str, resume: bool) -> None:
+ devices = ScriptedResponse(json={"items": [{"id": 7}, {"id": 9}], "total": 2})
+ first_device = [
+ ScriptedResponse(json={"items": [{"id": i, "deviceId": 7} for i in range(1000)], "total": 1001}),
+ ScriptedResponse(json={"items": [{"id": 1000, "deviceId": 7}], "total": 1001}),
+ ]
+ second_device = ScriptedResponse(json={"items": [{"id": 1, "deviceId": 9}], "total": 1})
+ first_path = f"device/devices/7/{child_path}"
+
+ result = driver().run(
+ table,
+ [devices, second_device] if resume else [devices, *first_device, second_device],
+ resume_state=LogicMonitorResumeConfig(fanout_state={"completed": [first_path], "current": None})
+ if resume
+ else None,
+ )
+
+ assert result.raised is None
+ assert result.response is not None and result.response.primary_keys == ["deviceId", "id"]
+ expected_rows = [(9, 1)] if resume else [*((7, i) for i in range(1001)), (9, 1)]
+ assert [(row["deviceId"], row["id"]) for row in result.rows] == expected_rows
+ child_paths = [f"/santaba/rest/device/devices/9/{child_path}"]
+ if not resume:
+ child_paths = [f"/santaba/rest/{first_path}"] * 2 + child_paths
+ assert result.paths == [BASE_PATH, *child_paths]
+ assert result.params("fields") == ["id"] + [FIELDS[table]] * len(child_paths)
+ assert result.params("offset") == ["0"] + (["0"] if resume else ["0", "1000", "0"])
+ assert result.final_state is not None and result.final_state.fanout_state["completed"] == [
+ first_path,
+ f"device/devices/9/{child_path}",
+ ]
+ assert result.completed_state is None
+
+
[email protected]("table", ["device_datasources", "device_datasource_instances"])
+def test_device_fanout_skips_device_deleted_mid_sync(table: str) -> None:
+ result = driver().run(
+ table,
+ [
+ ScriptedResponse(json={"items": [{"id": 7}, {"id": 9}], "total": 2}),
+ ScriptedResponse(status=404, json={"errorCode": 1404, "errorMessage": "fake not found"}),
+ ScriptedResponse(json={"items": [{"id": 1, "deviceId": 9}], "total": 1}),
+ ],
+ )
+
+ assert result.raised is None
+ assert [(row["deviceId"], row["id"]) for row in result.rows] == [(9, 1)]
+ assert len(result.requests) == 3
+
+
@pytest.mark.parametrize("status", [401, 403, 404])
def test_error_mapping(status: int) -> None:
answer = ScriptedResponse(status=status, json={"errorCode": status, "errorMessage": "fake error"})