Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
fix(monitoring): stop alerting on normal load and keep dev monitors out of on-call (#9122)
Wolf Mermelsteinfa28afecb167
diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.mdindex 763cdd2b18..3042d786ff 100644--- a/infra/stacks/monitoring/README.md+++ b/infra/stacks/monitoring/README.md@@ -23,11 +23,13 @@ and has two services, each with its pair of webhooks declared in `goalert.ts`:warnings and no-data go to Notify.- **Every recovery** closes on each service the monitor could have opened.Closing an alert GoAlert does not have open does nothing.-- **The hourly re-post comes from GoAlert**: its escalation posts to the channel- and repeats every hour until someone acknowledges. Monitors still renotify+- **Re-posts come from GoAlert**: a page posts to the channel again every hour+ until someone acknowledges it; a Notify alert posts once. Monitors still renotifyhourly (`RENOTIFY_MINUTES`); the repeat dedups into the open GoAlert alert(`$ALERT_CYCLE_KEY`) and reopens it if someone closed it while the monitorstill fires.+- **Dev and staging monitors notify no one** (`notify({ pages, environment })`).+ They stay visible in Datadog; on-call is for prod.- **Datadog never posts to the channel itself**, so each alert appears once.The only exception is the GoAlert health check (`goalert-health`), whichposts straight to `@webhook-macro-alert-hook` (a webhook integrationdiff --git a/infra/stacks/monitoring/monitors/compute.ts b/infra/stacks/monitoring/monitors/compute.tsindex 6e47a98dc0..5f3fa7bc3e 100644--- a/infra/stacks/monitoring/monitors/compute.ts+++ b/infra/stacks/monitoring/monitors/compute.ts@@ -6,17 +6,20 @@ import { adopted } from '../adopted';import { notify } from '../notify';// https://us5.datadoghq.com/monitors/1062688+// Average per task, not the sum over a family's tasks: the sum crossed the old+// 400/500 thresholds whenever a service scaled out. Per-task averages peaked+// around 50% over the 3 days to 2026-10-29.adopted('ecs-high-cpu', {name: '[PROD] High cpu usage across ECS Clusters',type: 'query alert',query:- 'avg(last_5m):sum:ecs.fargate.cpu.percent{(ecs_cluster:macro-cluster-prod OR ecs_cluster:cloud-storage-cluster-prod OR ecs_cluster:macroai-cluster-prod-4cbe721)} by {task_name} > 500',+ 'avg(last_5m):avg:ecs.fargate.cpu.percent{(ecs_cluster:macro-cluster-prod OR ecs_cluster:cloud-storage-cluster-prod OR ecs_cluster:macroai-cluster-prod-4cbe721)} by {task_name} > 90',message: `{{task_name.name}} is experiencing high cpu usage in ECS${notify({ pages: false })}`,monitorThresholds: {- critical: '500',- warning: '400',+ critical: '90',+ warning: '80',},onMissingData: 'show_no_data',notifyAudit: false,@@ -72,7 +75,7 @@ adopted('document-text-extractor-dev', {'sum(last_4h):(sum:aws.lambda.errors{resource:document-text-extractor-dev}.as_count() / sum:aws.lambda.invocations{resource:document-text-extractor-dev}.as_count()) * 100 > 80',message: `Document text extractor failing: {{value}}% of invocations errored over 4 hours.-${notify({ pages: false })}`,+${notify({ pages: false, environment: 'dev' })}`,monitorThresholds: {critical: '80',warning: '50',diff --git a/infra/stacks/monitoring/monitors/dynamodb.ts b/infra/stacks/monitoring/monitors/dynamodb.tsindex 6c7c5ed744..cdc1583d9a 100644--- a/infra/stacks/monitoring/monitors/dynamodb.ts+++ b/infra/stacks/monitoring/monitors/dynamodb.ts@@ -26,17 +26,19 @@ ${notify({ pages: false })}`,});// https://us5.datadoghq.com/monitors/16917574+// connection-gateway-table-prod reads normally peak around 27 units an hour+// (week to 2026-10-29) without throttling, so 10/20 fired on ordinary traffic.adopted('dynamodb-read-capacity', {name: 'Dynamodb ConsumedReadCapacityUnits High for {{tablename.name}}',type: 'query alert',query:- 'avg(last_1h):avg:aws.dynamodb.consumed_read_capacity_units{*} by {tablename} > 20',+ 'avg(last_1h):avg:aws.dynamodb.consumed_read_capacity_units{*} by {tablename} > 60',message: `DynamoDB table {{tablename.name}} is consuming {{value}} read capacity units.${notify({ pages: false })}`,monitorThresholds: {- critical: '20',- warning: '10',+ critical: '60',+ warning: '40',},onMissingData: 'default',notifyAudit: false,diff --git a/infra/stacks/monitoring/monitors/graphql-soup.ts b/infra/stacks/monitoring/monitors/graphql-soup.tsindex fc4368593c..819ee1a090 100644--- a/infra/stacks/monitoring/monitors/graphql-soup.ts+++ b/infra/stacks/monitoring/monitors/graphql-soup.ts@@ -282,11 +282,11 @@ rolloutMonitor('graphql-soup-db-cpu', {rolloutMonitor('graphql-soup-alb-bytes', {name: '[PROD] GraphQL soup: prod-gateway-alb processed bytes surge',type: 'query alert',- query: `sum(last_15m):sum:aws.applicationelb.processed_bytes{${GATEWAY_ALB}}.as_count() > 3000000000`,+ query: `sum(last_15m):sum:aws.applicationelb.processed_bytes{${GATEWAY_ALB}}.as_count() > 8000000000`,description:- 'prod-gateway-alb processed {{value}} bytes in 15 minutes (pre-rollout max was 1.9 GB). GraphQL request bodies are unbounded.',- critical: '3000000000',- warning: '1500000000',+ 'prod-gateway-alb processed {{value}} bytes in 15 minutes (the 7 days to 2026-10-29 averaged 0.7 GB and peaked at 4.1 GB). GraphQL request bodies are unbounded.',+ critical: '8000000000',+ warning: '6000000000',service: 'prod-gateway-alb',});diff --git a/infra/stacks/monitoring/monitors/logs.ts b/infra/stacks/monitoring/monitors/logs.tsindex 1d8d4d419e..ea850e0c61 100644--- a/infra/stacks/monitoring/monitors/logs.ts+++ b/infra/stacks/monitoring/monitors/logs.ts@@ -62,7 +62,7 @@ adopted('dss-errors-dev', {query: `logs("service:cloud-storage-service env:dev status:error ${DSS_EXPIRED_JWT_EXCLUSION}").index("*").rollup("count").last("5m") > 200`,message: `Dev cloud-storage-service logged {{value}} errors in 5 minutes.-${notify({ pages: false })}`,+${notify({ pages: false, environment: 'dev' })}`,monitorThresholds: {critical: '200',warning: '100',@@ -208,7 +208,7 @@ adopted('channel-invite-spam-dev', {'logs("service:notification-service-dev message:\\"processing message\\" @notification_event_type:channel_invite").index("*").rollup("count").by("@sender_id").last("1h") > 10',message: `A user is generating a high number of channel invites. This could mean they are spamming users with invite emails.-${notify({ pages: false })}`,+${notify({ pages: false, environment: 'dev' })}`,monitorThresholds: {critical: '10',warning: '5',diff --git a/infra/stacks/monitoring/monitors/notification-retention.ts b/infra/stacks/monitoring/monitors/notification-retention.tsindex 3d8027c8a4..1ca6a2fde8 100644--- a/infra/stacks/monitoring/monitors/notification-retention.ts+++ b/infra/stacks/monitoring/monitors/notification-retention.ts@@ -5,13 +5,15 @@ import { goalertWebhooks } from '../goalert';import { notify, RENOTIFY_MINUTES } from '../notify';const HEALTH = '@telemetry_event:notification_consumer_health';-const CONTEXT = `Notification-retention rollout: pause scheduled cleanup before investigating.+const context = (+ environment: 'dev' | 'prod'+) => `Notification-retention rollout: pause scheduled cleanup before investigating.Check consumer health/backlog, notification.delete.publish spans, database latency,and replica lag. A zero application backlog is not an end-to-end delivery ACK:Postgres NOTIFY publication and WebSocket delivery can still fail.-${notify({ pages: false })}`;+${notify({ pages: false, environment })}`;-for (const environment of ['dev', 'prod']) {+for (const environment of ['dev', 'prod'] as const) {const query = `${HEALTH} env:${environment}`;const common = {tags: [@@ -31,7 +33,7 @@ for (const environment of ['dev', 'prod']) {name: `[${environment.toUpperCase()}] Notification consumer: application backlog high`,type: 'log alert',query: `logs("${query} @consumer_health:healthy").index("*").rollup("avg", "@max_offset_backlog").by("@consumer_instance").last("5m") > 1000`,- message: `One consumer's maximum partition offset backlog averages over {{threshold}} for five minutes. This is not committed-offset lag. The initial threshold is conservative; tune after observing dry-run and canary baselines.\n${CONTEXT}`,+ message: `One consumer's maximum partition offset backlog averages over {{threshold}} for five minutes. This is not committed-offset lag. The initial threshold is conservative; tune after observing dry-run and canary baselines.\n${context(environment)}`,monitorThresholds: { critical: '1000', warning: '100' },// A retired process must not leave its old lag alert firing forever.// Stalls in live processes produce explicit health events below.@@ -46,7 +48,7 @@ for (const environment of ['dev', 'prod']) {name: `[${environment.toUpperCase()}] Notification consumer: telemetry stale or unavailable`,type: 'log alert',query: `logs("${query} @consumer_health:(stale OR unavailable OR disconnected)").index("*").rollup("count").by("@consumer_instance").last("5m") > 0`,- message: `A live consumer has stale statistics, unknown offsets, or no connected brokers after its startup grace period. Do not interpret missing/partial telemetry as zero lag.\n${CONTEXT}`,+ message: `A live consumer has stale statistics, unknown offsets, or no connected brokers after its startup grace period. Do not interpret missing/partial telemetry as zero lag.\n${context(environment)}`,monitorThresholds: { critical: '0' },onMissingData: 'resolve',},@@ -59,7 +61,7 @@ for (const environment of ['dev', 'prod']) {name: `[${environment.toUpperCase()}] Notification consumer: no telemetry`,type: 'log alert',query: `logs("${query}").index("*").rollup("count").last("5m") < 1`,- message: `No notification-consumer health events arrived for five minutes. Verify DSS consumers are running and log ingestion is healthy. Deploy the consumer instrumentation before enabling this monitor.\n${CONTEXT}`,+ message: `No notification-consumer health events arrived for five minutes. Verify DSS consumers are running and log ingestion is healthy. Deploy the consumer instrumentation before enabling this monitor.\n${context(environment)}`,monitorThresholds: { critical: '1' },onMissingData: 'default',},@@ -87,7 +89,7 @@ for (const environment of ['dev', 'prod']) {name: `[${environment.toUpperCase()}] Notification deletion: realtime publication failed`,type: 'log alert',query: `logs("env:${environment} \\"failed to publish notification delete realtime update\\"").index("*").rollup("count").last("5m") > 0`,- message: `A database deletion could not be forwarded to realtime. Deletion has already committed; PostgreSQL NOTIFY is best-effort, not a durable retry queue.\n${CONTEXT}`,+ message: `A database deletion could not be forwarded to realtime. Deletion has already committed; PostgreSQL NOTIFY is best-effort, not a durable retry queue.\n${context(environment)}`,monitorThresholds: { critical: '0' },onMissingData: 'default',},@@ -104,7 +106,7 @@ new datadog.Monitor(type: 'log alert',query:'logs("\\"notification retention telemetry flush\\"").index("*").rollup("count").last("15m") > 0',- message: `Cleanup may have succeeded while its traces were lost. Check the Datadog extension, its scoped secret access, and OTLP receiver. Do not retry deletion merely to retry telemetry.\n${CONTEXT}`,+ message: `Cleanup may have succeeded while its traces were lost. Check the Datadog extension, its scoped secret access, and OTLP receiver. Do not retry deletion merely to retry telemetry.\n${context('prod')}`,tags: ['service:notification-retention', 'rollout:notification-retention'],priority: '3',monitorThresholds: { critical: '0' },diff --git a/infra/stacks/monitoring/monitors/regressions.ts b/infra/stacks/monitoring/monitors/regressions.tsindex 75d8579a7b..a8e408345b 100644--- a/infra/stacks/monitoring/monitors/regressions.ts+++ b/infra/stacks/monitoring/monitors/regressions.ts@@ -33,7 +33,7 @@ const dashboard = new datadog.Dashboard('regression-alerts', {thresholdGraph({title: 'Internal span errors abnormally high',query:- "anomalies(sum:trace.Internal.errors{env:prod} by {service}.as_count(), 'agile', 3)",+ "anomalies(sum:trace.Internal.errors{env:prod AND NOT service:web-app} by {service}.as_count(), 'agile', 3)",}),thresholdGraph({title: 'Queue consumer errors abnormally high',@@ -77,16 +77,18 @@ prodMonitor('apm-server-error-anomaly', {runbook: 'docs/runbooks/regressions.md#errors',});+// Excludes web-app: browser spans fail on users' networks and devices, so its+// band swings with traffic. Browser errors have their own RUM monitors.prodMonitor('apm-internal-error-anomaly', {title: 'Internal span errors abnormally high',service: 'apm',query:- "avg(last_4h):anomalies(sum:trace.Internal.errors{env:prod} by {service}.as_count(), 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",+ "avg(last_4h):anomalies(sum:trace.Internal.errors{env:prod AND NOT service:web-app} by {service}.as_count(), 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",monitorThresholds: ANOMALY_THRESHOLDS,monitorThresholdWindows: ANOMALY_WINDOWS,pages: false,description:- '{{service.name}} is recording more failed internal spans (background jobs, workers, browser operations) than its usual band. Check the latest deploy of that service first.',+ '{{service.name}} is recording more failed internal spans (background jobs, workers) than its usual band. Check the latest deploy of that service first.',dashboard: dashboard.url,runbook: 'docs/runbooks/regressions.md#errors',});diff --git a/infra/stacks/monitoring/monitors/rum.ts b/infra/stacks/monitoring/monitors/rum.tsindex f636ac39d5..bd8686c967 100644--- a/infra/stacks/monitoring/monitors/rum.ts+++ b/infra/stacks/monitoring/monitors/rum.ts@@ -13,7 +13,7 @@ adopted('web-app-errors-staging', {'rum("@type:error @application.name:\\"Web App\\" env:production @view.url_host:staging.macro.com").rollup("count").last("5m") > 90',message: `The staging web app recorded {{value}} errors in 5 minutes.-${notify({ pages: false })}`,+${notify({ pages: false, environment: 'staging' })}`,monitorThresholds: {critical: '90',warning: '50',diff --git a/infra/stacks/monitoring/monitors/ses.ts b/infra/stacks/monitoring/monitors/ses.tsindex 593e046c9b..9b77d69a06 100644--- a/infra/stacks/monitoring/monitors/ses.ts+++ b/infra/stacks/monitoring/monitors/ses.ts@@ -26,17 +26,18 @@ ${notify({ pages: true })}`,});// https://us5.datadoghq.com/monitors/4178105+// No warning: SES has reported a flat 0.47% complaint rate since at least+// 2026-10-15 while recording no complaints, so a 0.1% warning never cleared.adopted('ses-complaint-rate', {name: '[SES] Complaint Rate Increasing',type: 'query alert',query:'avg(last_1h):avg:aws.ses.reputation_complaint_rate{aws_account:569036502058} > 0.005',- message: `SES Bounce Rate Increasing. Action needs to be taken to prevent account suspension.+ message: `SES complaint rate is {{value}}. SES may pause sending at 0.5%.${notify({ pages: true })}`,monitorThresholds: {critical: '0.005',- warning: '0.001',},onMissingData: 'default',notifyAudit: false,diff --git a/infra/stacks/monitoring/monitors/whatsapp-session.ts b/infra/stacks/monitoring/monitors/whatsapp-session.tsindex 40290e2924..2d65995053 100644--- a/infra/stacks/monitoring/monitors/whatsapp-session.ts+++ b/infra/stacks/monitoring/monitors/whatsapp-session.ts@@ -18,15 +18,17 @@ import * as datadog from '@pulumi/datadog';import { goalertWebhooks } from '../goalert';import { notify, RENOTIFY_MINUTES } from '../notify';-const CONTEXT = `Runbook: services/whatsapp_session_service/AGENTS.md. To stop all+const context = (+ environment: 'dev' | 'prod'+) => `Runbook: services/whatsapp_session_service/AGENTS.md. To stop allWhatsApp traffic, scale whatsapp-session-service to 0 tasks; sends recordedmeanwhile expire unsent after ten minutes. Turning off the enable-whatsapp-linkedflag only hides the app UI.-${notify({ pages: false })}`;+${notify({ pages: false, environment })}`;const HEALTH = '@telemetry_event:whatsapp_session_health';-for (const environment of ['dev', 'prod']) {+for (const environment of ['dev', 'prod'] as const) {const env = `env:${environment}`;const label = `[${environment.toUpperCase()}] WhatsApp sessions`;const common = {@@ -44,7 +46,7 @@ for (const environment of ['dev', 'prod']) {name: `${label}: accounts without an owner`,type: 'log alert',query: `logs("${HEALTH} ${env}").index("*").rollup("min", "@accounts_unowned").last("10m") > 0`,- message: `{{value}} connected accounts have had no replica holding their lease for ten minutes, so their groups receive and send nothing. Check replica count, MAX_ACCOUNTS headroom, and the lock connection (DATABASE_URL must be the direct MacroDB instance endpoint).\n${CONTEXT}`,+ message: `{{value}} connected accounts have had no replica holding their lease for ten minutes, so their groups receive and send nothing. Check replica count, MAX_ACCOUNTS headroom, and the lock connection (DATABASE_URL must be the direct MacroDB instance endpoint).\n${context(environment)}`,monitorThresholds: { critical: '0' },// Health events stop when the service is not deployed; that is not zero.onMissingData: 'resolve',@@ -59,7 +61,7 @@ for (const environment of ['dev', 'prod']) {name: `${label}: {{account_id.name}} reconnect churn`,type: 'log alert',query: `logs("@telemetry_event:whatsapp_session_reconnect ${env}").index("*").rollup("count").by("@account_id").last("15m") > 10`,- message: `Account {{account_id.name}} reconnected {{value}} times in 15 minutes. A flapping session can get the number flagged by WhatsApp; check the disconnect reasons on its spans, and whether two replicas are fighting over the lease (lease_epoch bumps).\n${CONTEXT}`,+ message: `Account {{account_id.name}} reconnected {{value}} times in 15 minutes. A flapping session can get the number flagged by WhatsApp; check the disconnect reasons on its spans, and whether two replicas are fighting over the lease (lease_epoch bumps).\n${context(environment)}`,monitorThresholds: { critical: '10', warning: '5' },onMissingData: 'resolve',},@@ -73,7 +75,7 @@ for (const environment of ['dev', 'prod']) {name: `${label}: sends with unknown outcome`,type: 'log alert',query: `logs("@telemetry_event:whatsapp_send_outcome @outcome:unknown ${env}").index("*").rollup("count").last("15m") > 5`,- message: `{{value}} sends timed out after submission in 15 minutes. They are never resent automatically; the sender sees them as unknown. Check WhatsApp connectivity and drain/takeover timing around deploys.\n${CONTEXT}`,+ message: `{{value}} sends timed out after submission in 15 minutes. They are never resent automatically; the sender sees them as unknown. Check WhatsApp connectivity and drain/takeover timing around deploys.\n${context(environment)}`,monitorThresholds: { critical: '5', warning: '0' },onMissingData: 'resolve',},@@ -87,7 +89,7 @@ for (const environment of ['dev', 'prod']) {name: `${label}: oldest unprocessed inbound event`,type: 'log alert',query: `logs("${HEALTH} ${env}").index("*").rollup("max", "@oldest_unprocessed_inbound_seconds").last("10m") > 300`,- message: `An inbound WhatsApp event has waited {{value}}s for ingestion. Messages are reaching the replica but not the channel; check the ingestion spans and MacroDB latency.\n${CONTEXT}`,+ message: `An inbound WhatsApp event has waited {{value}}s for ingestion. Messages are reaching the replica but not the channel; check the ingestion spans and MacroDB latency.\n${context(environment)}`,monitorThresholds: { critical: '300', warning: '60' },onMissingData: 'resolve',},@@ -101,7 +103,7 @@ for (const environment of ['dev', 'prod']) {name: `${label}: drain close to the stop timeout`,type: 'log alert',query: `logs("@telemetry_event:whatsapp_session_drain ${env}").index("*").rollup("max", "@drain_duration_ms").last("1h") > 100000`,- message: `A replica took {{value}}ms to drain on SIGTERM. ECS kills the task at 120s, cutting off in-flight sends (they become unknown). Raise DRAIN_BATCH or lower MAX_ACCOUNTS (services/whatsapp_session_service/src/main.rs).\n${CONTEXT}`,+ message: `A replica took {{value}}ms to drain on SIGTERM. ECS kills the task at 120s, cutting off in-flight sends (they become unknown). Raise DRAIN_BATCH or lower MAX_ACCOUNTS (services/whatsapp_session_service/src/main.rs).\n${context(environment)}`,monitorThresholds: { critical: '100000', warning: '80000' },onMissingData: 'resolve',},diff --git a/infra/stacks/monitoring/notify.ts b/infra/stacks/monitoring/notify.tsindex 670048d99f..cfc335ef36 100644--- a/infra/stacks/monitoring/notify.ts+++ b/infra/stacks/monitoring/notify.ts@@ -30,7 +30,15 @@ export const RENOTIFY_MINUTES = 60;* have warned before it alerted, or recovered from no data; closing an alert* GoAlert does not have open does nothing.*/-export function notify({ pages }: { pages: boolean }): string {+export function notify({+ pages,+ environment = 'prod',+}: {+ pages: boolean;+ /** Dev and staging monitors stay visible in Datadog but notify no one: on-call is for prod. */+ environment?: 'prod' | 'dev' | 'staging';+}): string {+ if (environment !== 'prod') return '';const critical = pages? `{{#is_alert}}Paging on-call. ${GOALERT}{{/is_alert}}`: `{{#is_alert}}${GOALERT_NOTIFY}{{/is_alert}}`;diff --git a/infra/stacks/monitoring/routing.test.ts b/infra/stacks/monitoring/routing.test.tsindex 2a234126bc..06bac7fdc6 100644--- a/infra/stacks/monitoring/routing.test.ts+++ b/infra/stacks/monitoring/routing.test.ts@@ -53,9 +53,24 @@ const handles = (message: string): string[] =>/@(?:webhook|slack|pagerduty|opsgenie|teams)-[\w-]+|[\w.+-]+@[\w-]+\.[\w.]+/g) ?? [];-test('every monitor and synthetic test notifies only GoAlert', () => {+// Dev and staging monitors stay in Datadog but notify no one.+const nonProd = (r: pulumi.runtime.MockResourceArgs) =>+ /\[(DEV|STAGING)\]/.test(r.inputs.name);++test('dev and staging monitors notify no one', () => {+ const quiet = monitors().filter(nonProd);+ expect(quiet.length).toBeGreaterThan(5);+ for (const resource of quiet) {+ expect({+ name: resource.name,+ handles: handles(resource.inputs.message),+ }).toEqual({ name: resource.name, handles: [] });+ }+});++test('every prod monitor and synthetic test notifies only GoAlert', () => {const routed = [...monitors(), ...synthetics()].filter(- (r) => r.name !== 'goalert-health'+ (r) => r.name !== 'goalert-health' && !nonProd(r));expect(routed.length).toBeGreaterThan(80);for (const resource of routed) {diff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.mdindex 844ccd544b..d4240c984d 100644--- a/infra/stacks/observability/README.md+++ b/infra/stacks/observability/README.md@@ -559,9 +559,11 @@ point's alerts with 401 and Grafana shows it as failing.### Escalation-Both policies have one step, webhook `http://127.0.0.1:8090/goalert`, and-repeat it 5 times every 60 minutes, so the channel sees each open alert hourly-until it is acknowledged, plus every acknowledgement and close.+Both policies have one step, webhook `http://127.0.0.1:8090/goalert`. The+channel sees every new alert, acknowledgement and close. `Macro prod` repeats+its step 5 times every 60 minutes, so an unacknowledged page is posted again+hourly; `Macro notify` posts once, so warnings stay quiet after their first+post.- Escalation policy `Macro prod` pages: services `Grafana critical` (Grafana'spaging criticals) and `Datadog critical`.