Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
chore(monitoring): stop declaring the deleted get_channel_messages draft monitor (#9093)
Wolf Mermelstein3953a658d877
diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.mdindex caef04f3ab..7ed4374cbe 100644--- a/infra/stacks/monitoring/README.md+++ b/infra/stacks/monitoring/README.md@@ -2,7 +2,7 @@Datadog monitors and synthetic tests as code.-36 resources here (35 monitors and one synthetic test) already exist in+35 resources here (34 monitors and one synthetic test) already exist inDatadog. This stack **adopts** them — it does not recreate them. `import.json`lists every one by id. Everything else is declared here first and is createdon deploy: the prod threshold and anomaly monitors (`monitors/macrodb.ts`,@@ -62,9 +62,9 @@ need the key, and paging monitors reference them:pulumi config set --secret goalertIntegrationKey '<Generic API key of the Datadog service in GoAlert>'```-**Run the import before the first `pulumi up`.** The program declares 35+**Run the import before the first `pulumi up`.** The program declares 34monitors that already exist in Datadog. Until `pulumi import` has told Pulumi-they exist, a deploy would create 35 duplicates — duplicate pages+they exist, a deploy would create 34 duplicates — duplicate pagesincluded.Adoption is a one-time local run. `pulumi import` reads Datadog and writes@@ -92,7 +92,7 @@ and records options omitted during import.The gate is that preview:-- **35 monitors + 1 synthetic test updated, in `message` and+- **34 monitors + 1 synthetic test updated, in `message` and`renotifyInterval` only** — the routing move. The two AI-editing monitors,already deployed from this stack, update the same way.- **73 resources to create**: 64 monitors (25 prod threshold and anomaly, 19 GraphQL soup, 9diff --git a/infra/stacks/monitoring/import.json b/infra/stacks/monitoring/import.jsonindex d2577b4933..ab77143370 100644--- a/infra/stacks/monitoring/import.json+++ b/infra/stacks/monitoring/import.json@@ -33,7 +33,6 @@{ "type": "datadog:index/monitor:Monitor", "name": "dynamodb-read-capacity", "id": "16917574" },{ "type": "datadog:index/monitor:Monitor", "name": "rds-low-freeable-memory", "id": "16918778" },{ "type": "datadog:index/monitor:Monitor", "name": "rds-high-iops", "id": "16918920" },- { "type": "datadog:index/monitor:Monitor", "name": "apm-get-channel-messages-p95", "id": "19271629" },{ "type": "datadog:index/monitor:Monitor", "name": "opensearch-storage-low", "id": "21836655" },{ "type": "datadog:index/syntheticsTest:SyntheticsTest", "name": "app-welcome-health-check", "id": "3xv-xcg-7kp" }]diff --git a/infra/stacks/monitoring/index.ts b/infra/stacks/monitoring/index.tsindex 440efbe8bc..5b3f400dae 100644--- a/infra/stacks/monitoring/index.ts+++ b/infra/stacks/monitoring/index.ts@@ -1,11 +1,10 @@// Datadog monitors and synthetic tests.//// Every monitor in the org is declared here, and every one notifies through-// notify.ts. See README.md before deploying — the 35 monitors under `monitors/`+// notify.ts. See README.md before deploying — the 34 monitors under `monitors/`// are adopted, not created, and a program that stops declaring one deletes it.// Adopted from the Datadog UI.-import './monitors/apm';import './monitors/cache';import './monitors/compute';import './monitors/dynamodb';diff --git a/infra/stacks/monitoring/monitors/apm.ts b/infra/stacks/monitoring/monitors/apm.tsdeleted file mode 100644index 3ae9f75cf7..0000000000--- a/infra/stacks/monitoring/monitors/apm.ts+++ /dev/null@@ -1,53 +0,0 @@-// APM latency.-//-// Adopted from monitors created by hand in the Datadog UI. Do not edit a-// monitor here and in the UI at the same time — this file wins on deploy.-import { adopted } from '../adopted';-import { notify } from '../notify';--// https://us5.datadoghq.com/monitors/19271629-adopted('apm-get-channel-messages-p95', {- draftStatus: 'draft',- name: 'High p95 Latency – get_channel_messages (cloud-storage-service prod)',- type: 'query alert',- query:- 'percentile(last_15m):p95:trace.Internal{service:cloud-storage-service,resource_name:get_channel_messages,env:prod} > 0.1',- message: `## ⚠️ High p95 Latency on get_channel_messages--**Service:** cloud-storage-service-**Resource:** get_channel_messages-**Environment:** prod--### What's happening?-p95 response time has exceeded **100ms** (threshold). During the Apr 17 incident this reached 322ms.--### What to check?-1. Check for recent ECS deployments on \`cloud-storage-service\` — the Apr 17 spike was caused by a bad task version-2. Look for \`idle_ns >> busy_ns\` in slow spans — indicates DB wait / connection pool exhaustion-3. Check channels \`0195ceb6-ec2e-7023-80e4-6e084fa6cccd\` and \`0195cea4-5491-77f9-9224-1291fdb8f150\` (repeatedly affected in the last incident)-4. Review the \`Sort(CreatedAt)\` query path on backward pagination (\`direction: Older\`)--### Useful Links-- [Slow spans explorer](https://us5.datadoghq.com/apm/traces?query=service%3Acloud-storage-service+resource_name%3Aget_channel_messages+env%3Aprod+%40duration%3A%3E100000000)-- [Investigation notebook](https://us5.datadoghq.com/notebook/277323)--${notify({ pages: false })}`,- priority: '5',- tags: [- 'service:cloud-storage-service',- 'env:prod',- 'resource:get_channel_messages',- 'product:apm',- 'team:hutch',- ],- monitorThresholds: {- critical: '0.1',- warning: '0.05',- },- onMissingData: 'default',- notifyAudit: false,- includeTags: true,- newHostDelay: 300,- evaluationDelay: 60,- requireFullWindow: false,-});