Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 4 missing tables to the LogicMonitor source (#125202)

Tom Owerse0181e011a48
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index c97e487e4e3e..160c8cf4b6d8 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -7442,14 +7442,14 @@ Today (6): `alerts`, `collectors`, `device_groups`, `devices`, `sdts`, `websites
Diffed against: <https://www.logicmonitor.com/swagger-ui-master/api-v3/dist/swagger.json>
-- [ ] `Collector groups (GET /setting/collector/groups)` — resolve collectorGroupId in existing collector rows for operational ownership reports (high)
+- [x] `Collector groups (GET /setting/collector/groups)` — resolve collectorGroupId in existing collector rows for operational ownership reports (high). Added as `collector_groups` (full refresh).
- [x] `Device datasources (GET /device/devices/{deviceId}/devicedatasources)` — resolve alert resourceTemplateId references and identify monitored datasource assignments (high). Added as `device_datasources` (fan-out over `devices`).
- [x] `Device datasource instances (GET /device/devices/{deviceId}/devicedatasources/{hdsId}/instances)` — resolve alert instanceId references to monitored components (high). Added as `device_datasource_instances`, read from `GET /device/devices/{id}/instances`, which returns the same instance rows for all of a device's datasources in one fan-out over `devices`.
- [ ] `Device instance data (GET /device/devices/{deviceId}/devicedatasources/{hdsId}/instances/{id}/data)` — measure performance, utilization, and availability instead of relying only on alert counts (high) — skipped: raw per-instance metric series, one paginated request per instance per sync after a device and instance fan-out, so a sync grows with every monitored instance and lands a row per poll interval
- [ ] `Website graph data (GET /website/websites/{id}/graphs/{graphName}/data)` — measure website uptime and response times over time (high) — skipped: it returns a chart render payload (colors, legends, scaled lines) for a graph name the spec does not list. `GET /website/websites/{websiteId}/checkpoints/{checkpointId}/data` is the raw-data alternative
-- [ ] `Device SDT history (GET /device/devices/{id}/historysdts)` — explain historical alert suppression and separate maintenance from service failures (medium)
-- [ ] `Opsnotes (GET /setting/opsnotes)` — connect incidents with operational notes and change annotations (medium)
-- [ ] `Audit logs (GET /setting/accesslogs)` — track administrative activity and changes associated with monitoring incidents (medium)
+- [x] `Device SDT history (GET /device/devices/{id}/historysdts)` — explain historical alert suppression and separate maintenance from service failures (medium). Added as `device_sdt_history` (fan-out over `devices`, full refresh).
+- [x] `Opsnotes (GET /setting/opsnotes)` — connect incidents with operational notes and change annotations (medium). Added as `ops_notes` (full refresh).
+- [x] `Audit logs (GET /setting/accesslogs)` — track administrative activity and changes associated with monitoring incidents (medium). Added as `audit_logs` (incremental on `happenedOn`).
- [ ] `FOCUS Cost and Usage export (GET /usage/focusCostUsage)` — attribute monitoring usage and cost to resources and tenants (medium)
Note: Fetched the v3 Swagger JSON linked by the official reference and enumerated GET operations at every nesting depth. Metric imports need device, datasource, and instance fan-out plus bounded time ranges. Existing collector fields include collectorGroupId, but no group table resolves it. Excluded credentials, configuration payloads, installers, diagnostic commands, and alert settings.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py
index 9d4b44864e04..4e83f0033b9c 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/canonical_descriptions.py
@@ -108,4 +108,55 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"disableAlerting": "Whether alerting is disabled for the instance.",
},
},
+ "collector_groups": {
+ "description": "Collector groups that organize collectors and balance monitoring load across them.",
+ "docs_url": API_DOCS_URL,
+ "columns": {
+ "id": "Unique collector group identifier. Collectors reference it as collectorGroupId.",
+ "name": "Name of the collector group.",
+ "createOn": "Time when the group was created, in Unix seconds.",
+ "numOfCollectors": "Number of collectors in the group.",
+ "numOfHosts": "Number of hosts that belong to the group.",
+ "autoBalance": "Whether the group balances devices across its collectors automatically.",
+ "mismatchVersion": "Whether the collectors in the group run different versions.",
+ },
+ },
+ "ops_notes": {
+ "description": "Ops notes that annotate graphs and dashboards with operational events, such as deployments and changes.",
+ "docs_url": API_DOCS_URL,
+ "columns": {
+ "id": "Unique ops note identifier.",
+ "note": "Text of the note.",
+ "createdBy": "User that created the note.",
+ "happenOnInSec": "Time associated with the note, in Unix seconds.",
+ "scopes": "Devices, services, and groups the note applies to. A note with no scope applies to all of them.",
+ "tags": "Tags associated with the note.",
+ },
+ },
+ "audit_logs": {
+ "description": "Audit log entries that record user and API actions in the portal.",
+ "docs_url": API_DOCS_URL,
+ "columns": {
+ "id": "Unique audit log entry identifier.",
+ "happenedOn": "Time when the action occurred, in Unix seconds.",
+ "username": "User that performed the action.",
+ "ip": "IP address that the action came from.",
+ "description": "Description of the action.",
+ },
+ },
+ "device_sdt_history": {
+ "description": "Past scheduled downtime (SDT) periods that suppressed alerts for each device.",
+ "docs_url": API_DOCS_URL,
+ "columns": {
+ "id": "Identifier of the SDT.",
+ "deviceId": "Identifier of the device.",
+ "itemId": "Identifier of the resource in SDT, such as the device or one of its groups.",
+ "type": "SDT type.",
+ "admin": "User that added the SDT.",
+ "comment": "Comment associated with the SDT.",
+ "approximateStartEpoch": "Time when the SDT started.",
+ "approximateEndEpoch": "Time when the SDT ended.",
+ "duration": "Duration of the SDT, in minutes.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py
index 2290cf784bf3..8c3bcd82ec89 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/logicmonitor.py
@@ -103,6 +103,12 @@ def validate_portal_host(value: str, team_id: int) -> tuple[bool, str | None]:
return ValidateDatabaseHostMixin().is_database_host_valid(urlsplit(url).hostname or "", team_id)
+def audit_log_start(inputs: SourceInputs) -> int:
+ if inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:
+ return int(inputs.db_incremental_field_last_value)
+ return 0
+
+
class AlertWindowTooLarge(Exception):
pass
@@ -220,6 +226,10 @@ class LogicMonitorClient:
if is_alerts:
params["filter"] = f'cleared:"*",startEpoch>:{state.window_start},startEpoch<{state.window_end}'
params["sort"] = "startEpoch"
+ elif inputs.schema_name == "audit_logs":
+ # The upper bound keeps entries written during the sync from shifting the offset pages.
+ params["filter"] = f"happenedOn>:{audit_log_start(inputs)},happenedOn<{state.sync_end}"
+ params["sort"] = "happenedOn"
paginator_class = AlertPaginator if is_alerts else OffsetPaginator
config: RESTAPIConfig = {
"client": self.client_config(paginator_class(limit=PAGE_SIZE, limit_param="size", total_path="total")),
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py
index efe351ade432..be121141ff59 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/settings.py
@@ -1,6 +1,7 @@
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
DependentEndpointConfig,
)
+from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType
API_VERSION = "3"
API_DOCS_URL = "https://www.logicmonitor.com/support/rest-api-v3-swagger-documentation"
@@ -12,11 +13,15 @@ ENDPOINTS: dict[str, str] = {
"devices": "device/devices",
"device_groups": "device/groups",
"collectors": "setting/collector/collectors",
+ "collector_groups": "setting/collector/groups",
"sdts": "sdt/sdts",
"websites": "website/websites",
+ "ops_notes": "setting/opsnotes",
+ "audit_logs": "setting/accesslogs",
"device_datasources": "device/devices/{deviceId}/devicedatasources",
# Lists every datasource instance of a device, so one request per device covers all of its datasources.
"device_datasource_instances": "device/devices/{deviceId}/instances",
+ "device_sdt_history": "device/devices/{deviceId}/historysdts",
}
# Select inventory fields because device properties and collector configuration can contain credentials.
@@ -24,6 +29,10 @@ FIELDS: dict[str, str] = {
"devices": "id,name,displayName,description,createdOn,updatedOn,preferredCollectorId,hostGroupIds,disableAlerting",
"device_groups": "id,name,description,fullPath,parentId,createdOn,disableAlerting",
"collectors": "id,hostname,description,createdOn,updatedOn,collectorGroupId,numberOfHosts,isDown,build",
+ "collector_groups": (
+ "id,name,description,createOn,platform,numOfCollectors,numOfHosts,numOfInstances,autoBalance,"
+ "autoBalanceStrategy,mismatchVersion"
+ ),
"websites": "id,name,description,type,overallAlertLevel,disableAlerting",
"device_datasources": (
"id,deviceId,deviceName,deviceDisplayName,dataSourceId,dataSourceName,dataSourceDisplayName,"
@@ -39,18 +48,38 @@ FIELDS: dict[str, str] = {
}
PRIMARY_KEYS = ["id"]
-DEVICE_FANOUTS: dict[str, DependentEndpointConfig] = {
- name: DependentEndpointConfig(
+
+def device_fanout(child_fields: str | None, *, include_device_id: bool = False) -> DependentEndpointConfig:
+ return DependentEndpointConfig(
parent_name="devices",
resolve_param="deviceId",
resolve_field="id",
- include_from_parent=[],
+ include_from_parent=["id"] if include_device_id else [],
+ parent_field_renames={"id": "deviceId"} if include_device_id else {},
parent_params={"fields": "id"},
- child_params={"fields": FIELDS[name]},
+ child_params={"fields": child_fields} if child_fields else {},
# A device deleted after the device list was read has no children left to sync.
child_response_actions=[{"status_code": 404, "action": "ignore"}],
)
- for name in ("device_datasources", "device_datasource_instances")
+
+
+DEVICE_FANOUTS: dict[str, DependentEndpointConfig] = {
+ "device_datasources": device_fanout(FIELDS["device_datasources"]),
+ "device_datasource_instances": device_fanout(FIELDS["device_datasource_instances"]),
+ # SDT history rows name the SDT target in itemId, which can be a device group, so the device id comes from the parent.
+ "device_sdt_history": device_fanout(None, include_device_id=True),
}
-# Device datasource and instance ids are only documented per device.
+# Device datasource, instance, and SDT history ids are only documented per device.
FANOUT_PRIMARY_KEYS = ["deviceId", "id"]
+
+# Audit log entries never change after they are written, so new entries can be read by their time.
+INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {
+ "audit_logs": [
+ {
+ "label": "happenedOn",
+ "type": IncrementalFieldType.Integer,
+ "field": "happenedOn",
+ "field_type": IncrementalFieldType.Integer,
+ }
+ ],
+}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/source.py
index 79c28f2a0986..f83102591848 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/source.py
@@ -37,6 +37,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.logicmonit
API_DOCS_URL,
API_VERSION,
ENDPOINTS,
+ INCREMENTAL_FIELDS,
)
from products.warehouse_sources.backend.types import ExternalDataSourceType
@@ -82,7 +83,7 @@ class LogicmonitorSource(ResumableSource[LogicmonitorSourceConfig, LogicMonitorR
api_version: str | None = None,
) -> list[SourceSchema]:
# A start-time watermark misses later acknowledgments and clear events for older alerts.
- return build_endpoint_schemas(ENDPOINTS, {}, names)
+ return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names)
def validate_credentials(
self,
@@ -118,7 +119,8 @@ class LogicmonitorSource(ResumableSource[LogicmonitorSourceConfig, LogicMonitorR
label="LogicMonitor",
caption=(
"Create a bearer token in Settings > Users & Roles > API Tokens > Bearer. "
- "Give its user view access to Resources, Dashboards, Collectors, and Websites for the tables you select. "
+ "Give its user view access to Resources, Dashboards, Collectors, Websites, Ops Notes, and Audit Logs "
+ "for the tables you select. "
"Alerts use full refresh to capture acknowledgments and cleared status."
),
iconPath="/static/services/logicmonitor.png",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py
index 68b9c26e4ebb..706c99125b33 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/logicmonitor/tests/test_logicmonitor.py
@@ -102,17 +102,26 @@ def test_alert_growth_after_a_page_fails_without_replaying() -> None:
@pytest.mark.parametrize(
- "table, child_path",
- [("device_datasources", "devicedatasources"), ("device_datasource_instances", "instances")],
+ "table, child_path, rows_carry_device_id",
+ [
+ ("device_datasources", "devicedatasources", True),
+ ("device_datasource_instances", "instances", True),
+ ("device_sdt_history", "historysdts", False),
+ ],
)
@pytest.mark.parametrize("resume", [False, True])
-def test_device_fanout_pages_each_device_and_resumes(table: str, child_path: str, resume: bool) -> None:
+def test_device_fanout_pages_each_device_and_resumes(
+ table: str, child_path: str, rows_carry_device_id: bool, resume: bool
+) -> None:
+ def child_row(row_id: int, device_id: int) -> dict[str, Any]:
+ return {"id": row_id, "deviceId": device_id} if rows_carry_device_id else {"id": row_id, "itemId": 3}
+
devices = ScriptedResponse(json={"items": [{"id": 7}, {"id": 9}], "total": 2})
first_device = [
- ScriptedResponse(json={"items": [{"id": i, "deviceId": 7} for i in range(1000)], "total": 1001}),
- ScriptedResponse(json={"items": [{"id": 1000, "deviceId": 7}], "total": 1001}),
+ ScriptedResponse(json={"items": [child_row(i, 7) for i in range(1000)], "total": 1001}),
+ ScriptedResponse(json={"items": [child_row(1000, 7)], "total": 1001}),
]
- second_device = ScriptedResponse(json={"items": [{"id": 1, "deviceId": 9}], "total": 1})
+ second_device = ScriptedResponse(json={"items": [child_row(1, 9)], "total": 1})
first_path = f"device/devices/7/{child_path}"
result = driver().run(
@@ -131,7 +140,7 @@ def test_device_fanout_pages_each_device_and_resumes(table: str, child_path: str
if not resume:
child_paths = [f"/santaba/rest/{first_path}"] * 2 + child_paths
assert result.paths == [BASE_PATH, *child_paths]
- assert result.params("fields") == ["id"] + [FIELDS[table]] * len(child_paths)
+ assert result.params("fields") == ["id"] + [FIELDS.get(table)] * len(child_paths)
assert result.params("offset") == ["0"] + (["0"] if resume else ["0", "1000", "0"])
assert result.final_state is not None and result.final_state.fanout_state["completed"] == [
first_path,
@@ -140,7 +149,7 @@ def test_device_fanout_pages_each_device_and_resumes(table: str, child_path: str
assert result.completed_state is None
[email protected]("table", ["device_datasources", "device_datasource_instances"])
[email protected]("table", ["device_datasources", "device_datasource_instances", "device_sdt_history"])
def test_device_fanout_skips_device_deleted_mid_sync(table: str) -> None:
result = driver().run(
table,
@@ -156,6 +165,62 @@ def test_device_fanout_skips_device_deleted_mid_sync(table: str) -> None:
assert len(result.requests) == 3
+ "inputs, resume_state, expected_filter",
+ [
+ ({}, None, "happenedOn>:0,happenedOn<{sync_end}"),
+ (
+ {"incremental_field": "happenedOn", "db_incremental_field_last_value": 1700000000},
+ None,
+ "happenedOn>:1700000000,happenedOn<{sync_end}",
+ ),
+ (
+ {"incremental_field": "happenedOn", "db_incremental_field_last_value": "1700000000"},
+ None,
+ "happenedOn>:1700000000,happenedOn<{sync_end}",
+ ),
+ (
+ {
+ "incremental_field": "happenedOn",
+ "db_incremental_field_last_value": 1700000000,
+ "should_use_incremental_field": False,
+ },
+ None,
+ "happenedOn>:0,happenedOn<{sync_end}",
+ ),
+ (
+ {"incremental_field": "happenedOn", "db_incremental_field_last_value": 1700000000},
+ LogicMonitorResumeConfig(offset=1000, window_end=1800000000, sync_end=1800000000),
+ "happenedOn>:1700000000,happenedOn<1800000000",
+ ),
+ ],
+)
+def test_audit_logs_bound_each_sync(
+ inputs: dict[str, Any], resume_state: LogicMonitorResumeConfig | None, expected_filter: str
+) -> None:
+ first_offset = resume_state.offset if resume_state else 0
+ result = driver().run(
+ "audit_logs",
+ [
+ ScriptedResponse(json={"items": [{"id": "a", "happenedOn": 1}] * 1000, "total": first_offset + 1001}),
+ ScriptedResponse(json={"items": [{"id": "b", "happenedOn": 2}], "total": first_offset + 1001}),
+ ],
+ resume_state=resume_state,
+ **inputs,
+ )
+
+ assert result.raised is None
+ assert len(result.rows) == 1001
+ assert result.paths == ["/santaba/rest/setting/accesslogs"] * 2
+ filters = result.params("filter")
+ assert filters[0] is not None
+ sync_end = filters[0].rsplit("<", 1)[1]
+ assert sync_end.isdigit()
+ assert filters == [expected_filter.format(sync_end=sync_end)] * 2
+ assert result.params("sort") == ["happenedOn"] * 2
+ assert result.params("offset") == [str(first_offset), str(first_offset + 1000)]
+
+
@pytest.mark.parametrize("status", [401, 403, 404])
def test_error_mapping(status: int) -> None:
answer = ScriptedResponse(status=status, json={"errorCode": status, "errorMessage": "fake error"})