Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

chore(monitoring): stop declaring the deleted get_channel_messages draft monitor (#9093)

Wolf Mermelstein3953a658d877
diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.md
index caef04f3ab..7ed4374cbe 100644
--- a/infra/stacks/monitoring/README.md
+++ b/infra/stacks/monitoring/README.md
@@ -2,7 +2,7 @@
Datadog monitors and synthetic tests as code.
-36 resources here (35 monitors and one synthetic test) already exist in
+35 resources here (34 monitors and one synthetic test) already exist in
Datadog. This stack **adopts** them — it does not recreate them. `import.json`
lists every one by id. Everything else is declared here first and is created
on deploy: the prod threshold and anomaly monitors (`monitors/macrodb.ts`,
@@ -62,9 +62,9 @@ need the key, and paging monitors reference them:
pulumi config set --secret goalertIntegrationKey '<Generic API key of the Datadog service in GoAlert>'
```
-**Run the import before the first `pulumi up`.** The program declares 35
+**Run the import before the first `pulumi up`.** The program declares 34
monitors that already exist in Datadog. Until `pulumi import` has told Pulumi
-they exist, a deploy would create 35 duplicates — duplicate pages
+they exist, a deploy would create 34 duplicates — duplicate pages
included.
Adoption is a one-time local run. `pulumi import` reads Datadog and writes
@@ -92,7 +92,7 @@ and records options omitted during import.
The gate is that preview:
-- **35 monitors + 1 synthetic test updated, in `message` and
+- **34 monitors + 1 synthetic test updated, in `message` and
`renotifyInterval` only** — the routing move. The two AI-editing monitors,
already deployed from this stack, update the same way.
- **73 resources to create**: 64 monitors (25 prod threshold and anomaly, 19 GraphQL soup, 9
diff --git a/infra/stacks/monitoring/import.json b/infra/stacks/monitoring/import.json
index d2577b4933..ab77143370 100644
--- a/infra/stacks/monitoring/import.json
+++ b/infra/stacks/monitoring/import.json
@@ -33,7 +33,6 @@
{ "type": "datadog:index/monitor:Monitor", "name": "dynamodb-read-capacity", "id": "16917574" },
{ "type": "datadog:index/monitor:Monitor", "name": "rds-low-freeable-memory", "id": "16918778" },
{ "type": "datadog:index/monitor:Monitor", "name": "rds-high-iops", "id": "16918920" },
- { "type": "datadog:index/monitor:Monitor", "name": "apm-get-channel-messages-p95", "id": "19271629" },
{ "type": "datadog:index/monitor:Monitor", "name": "opensearch-storage-low", "id": "21836655" },
{ "type": "datadog:index/syntheticsTest:SyntheticsTest", "name": "app-welcome-health-check", "id": "3xv-xcg-7kp" }
]
diff --git a/infra/stacks/monitoring/index.ts b/infra/stacks/monitoring/index.ts
index 440efbe8bc..5b3f400dae 100644
--- a/infra/stacks/monitoring/index.ts
+++ b/infra/stacks/monitoring/index.ts
@@ -1,11 +1,10 @@
// Datadog monitors and synthetic tests.
//
// Every monitor in the org is declared here, and every one notifies through
-// notify.ts. See README.md before deploying — the 35 monitors under `monitors/`
+// notify.ts. See README.md before deploying — the 34 monitors under `monitors/`
// are adopted, not created, and a program that stops declaring one deletes it.
// Adopted from the Datadog UI.
-import './monitors/apm';
import './monitors/cache';
import './monitors/compute';
import './monitors/dynamodb';
diff --git a/infra/stacks/monitoring/monitors/apm.ts b/infra/stacks/monitoring/monitors/apm.ts
deleted file mode 100644
index 3ae9f75cf7..0000000000
--- a/infra/stacks/monitoring/monitors/apm.ts
+++ /dev/null
@@ -1,53 +0,0 @@
-// APM latency.
-//
-// Adopted from monitors created by hand in the Datadog UI. Do not edit a
-// monitor here and in the UI at the same time — this file wins on deploy.
-import { adopted } from '../adopted';
-import { notify } from '../notify';
-
-// https://us5.datadoghq.com/monitors/19271629
-adopted('apm-get-channel-messages-p95', {
- draftStatus: 'draft',
- name: 'High p95 Latency – get_channel_messages (cloud-storage-service prod)',
- type: 'query alert',
- query:
- 'percentile(last_15m):p95:trace.Internal{service:cloud-storage-service,resource_name:get_channel_messages,env:prod} > 0.1',
- message: `## ⚠️ High p95 Latency on get_channel_messages
-
-**Service:** cloud-storage-service
-**Resource:** get_channel_messages
-**Environment:** prod
-
-### What's happening?
-p95 response time has exceeded **100ms** (threshold). During the Apr 17 incident this reached 322ms.
-
-### What to check?
-1. Check for recent ECS deployments on \`cloud-storage-service\` — the Apr 17 spike was caused by a bad task version
-2. Look for \`idle_ns >> busy_ns\` in slow spans — indicates DB wait / connection pool exhaustion
-3. Check channels \`0195ceb6-ec2e-7023-80e4-6e084fa6cccd\` and \`0195cea4-5491-77f9-9224-1291fdb8f150\` (repeatedly affected in the last incident)
-4. Review the \`Sort(CreatedAt)\` query path on backward pagination (\`direction: Older\`)
-
-### Useful Links
-- [Slow spans explorer](https://us5.datadoghq.com/apm/traces?query=service%3Acloud-storage-service+resource_name%3Aget_channel_messages+env%3Aprod+%40duration%3A%3E100000000)
-- [Investigation notebook](https://us5.datadoghq.com/notebook/277323)
-
-${notify({ pages: false })}`,
- priority: '5',
- tags: [
- 'service:cloud-storage-service',
- 'env:prod',
- 'resource:get_channel_messages',
- 'product:apm',
- 'team:hutch',
- ],
- monitorThresholds: {
- critical: '0.1',
- warning: '0.05',
- },
- onMissingData: 'default',
- notifyAudit: false,
- includeTags: true,
- newHostDelay: 300,
- evaluationDelay: 60,
- requireFullWindow: false,
-});