Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

fix(monitoring): stop alerting on normal load and keep dev monitors out of on-call (#9122)

Wolf Mermelsteinfa28afecb167
diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.md
index 763cdd2b18..3042d786ff 100644
--- a/infra/stacks/monitoring/README.md
+++ b/infra/stacks/monitoring/README.md
@@ -23,11 +23,13 @@ and has two services, each with its pair of webhooks declared in `goalert.ts`:
warnings and no-data go to Notify.
- **Every recovery** closes on each service the monitor could have opened.
Closing an alert GoAlert does not have open does nothing.
-- **The hourly re-post comes from GoAlert**: its escalation posts to the channel
- and repeats every hour until someone acknowledges. Monitors still renotify
+- **Re-posts come from GoAlert**: a page posts to the channel again every hour
+ until someone acknowledges it; a Notify alert posts once. Monitors still renotify
hourly (`RENOTIFY_MINUTES`); the repeat dedups into the open GoAlert alert
(`$ALERT_CYCLE_KEY`) and reopens it if someone closed it while the monitor
still fires.
+- **Dev and staging monitors notify no one** (`notify({ pages, environment })`).
+ They stay visible in Datadog; on-call is for prod.
- **Datadog never posts to the channel itself**, so each alert appears once.
The only exception is the GoAlert health check (`goalert-health`), which
posts straight to `@webhook-macro-alert-hook` (a webhook integration
diff --git a/infra/stacks/monitoring/monitors/compute.ts b/infra/stacks/monitoring/monitors/compute.ts
index 6e47a98dc0..5f3fa7bc3e 100644
--- a/infra/stacks/monitoring/monitors/compute.ts
+++ b/infra/stacks/monitoring/monitors/compute.ts
@@ -6,17 +6,20 @@ import { adopted } from '../adopted';
import { notify } from '../notify';
// https://us5.datadoghq.com/monitors/1062688
+// Average per task, not the sum over a family's tasks: the sum crossed the old
+// 400/500 thresholds whenever a service scaled out. Per-task averages peaked
+// around 50% over the 3 days to 2026-10-29.
adopted('ecs-high-cpu', {
name: '[PROD] High cpu usage across ECS Clusters',
type: 'query alert',
query:
- 'avg(last_5m):sum:ecs.fargate.cpu.percent{(ecs_cluster:macro-cluster-prod OR ecs_cluster:cloud-storage-cluster-prod OR ecs_cluster:macroai-cluster-prod-4cbe721)} by {task_name} > 500',
+ 'avg(last_5m):avg:ecs.fargate.cpu.percent{(ecs_cluster:macro-cluster-prod OR ecs_cluster:cloud-storage-cluster-prod OR ecs_cluster:macroai-cluster-prod-4cbe721)} by {task_name} > 90',
message: `{{task_name.name}} is experiencing high cpu usage in ECS
${notify({ pages: false })}`,
monitorThresholds: {
- critical: '500',
- warning: '400',
+ critical: '90',
+ warning: '80',
},
onMissingData: 'show_no_data',
notifyAudit: false,
@@ -72,7 +75,7 @@ adopted('document-text-extractor-dev', {
'sum(last_4h):(sum:aws.lambda.errors{resource:document-text-extractor-dev}.as_count() / sum:aws.lambda.invocations{resource:document-text-extractor-dev}.as_count()) * 100 > 80',
message: `Document text extractor failing: {{value}}% of invocations errored over 4 hours.
-${notify({ pages: false })}`,
+${notify({ pages: false, environment: 'dev' })}`,
monitorThresholds: {
critical: '80',
warning: '50',
diff --git a/infra/stacks/monitoring/monitors/dynamodb.ts b/infra/stacks/monitoring/monitors/dynamodb.ts
index 6c7c5ed744..cdc1583d9a 100644
--- a/infra/stacks/monitoring/monitors/dynamodb.ts
+++ b/infra/stacks/monitoring/monitors/dynamodb.ts
@@ -26,17 +26,19 @@ ${notify({ pages: false })}`,
});
// https://us5.datadoghq.com/monitors/16917574
+// connection-gateway-table-prod reads normally peak around 27 units an hour
+// (week to 2026-10-29) without throttling, so 10/20 fired on ordinary traffic.
adopted('dynamodb-read-capacity', {
name: 'Dynamodb ConsumedReadCapacityUnits High for {{tablename.name}}',
type: 'query alert',
query:
- 'avg(last_1h):avg:aws.dynamodb.consumed_read_capacity_units{*} by {tablename} > 20',
+ 'avg(last_1h):avg:aws.dynamodb.consumed_read_capacity_units{*} by {tablename} > 60',
message: `DynamoDB table {{tablename.name}} is consuming {{value}} read capacity units.
${notify({ pages: false })}`,
monitorThresholds: {
- critical: '20',
- warning: '10',
+ critical: '60',
+ warning: '40',
},
onMissingData: 'default',
notifyAudit: false,
diff --git a/infra/stacks/monitoring/monitors/graphql-soup.ts b/infra/stacks/monitoring/monitors/graphql-soup.ts
index fc4368593c..819ee1a090 100644
--- a/infra/stacks/monitoring/monitors/graphql-soup.ts
+++ b/infra/stacks/monitoring/monitors/graphql-soup.ts
@@ -282,11 +282,11 @@ rolloutMonitor('graphql-soup-db-cpu', {
rolloutMonitor('graphql-soup-alb-bytes', {
name: '[PROD] GraphQL soup: prod-gateway-alb processed bytes surge',
type: 'query alert',
- query: `sum(last_15m):sum:aws.applicationelb.processed_bytes{${GATEWAY_ALB}}.as_count() > 3000000000`,
+ query: `sum(last_15m):sum:aws.applicationelb.processed_bytes{${GATEWAY_ALB}}.as_count() > 8000000000`,
description:
- 'prod-gateway-alb processed {{value}} bytes in 15 minutes (pre-rollout max was 1.9 GB). GraphQL request bodies are unbounded.',
- critical: '3000000000',
- warning: '1500000000',
+ 'prod-gateway-alb processed {{value}} bytes in 15 minutes (the 7 days to 2026-10-29 averaged 0.7 GB and peaked at 4.1 GB). GraphQL request bodies are unbounded.',
+ critical: '8000000000',
+ warning: '6000000000',
service: 'prod-gateway-alb',
});
diff --git a/infra/stacks/monitoring/monitors/logs.ts b/infra/stacks/monitoring/monitors/logs.ts
index 1d8d4d419e..ea850e0c61 100644
--- a/infra/stacks/monitoring/monitors/logs.ts
+++ b/infra/stacks/monitoring/monitors/logs.ts
@@ -62,7 +62,7 @@ adopted('dss-errors-dev', {
query: `logs("service:cloud-storage-service env:dev status:error ${DSS_EXPIRED_JWT_EXCLUSION}").index("*").rollup("count").last("5m") > 200`,
message: `Dev cloud-storage-service logged {{value}} errors in 5 minutes.
-${notify({ pages: false })}`,
+${notify({ pages: false, environment: 'dev' })}`,
monitorThresholds: {
critical: '200',
warning: '100',
@@ -208,7 +208,7 @@ adopted('channel-invite-spam-dev', {
'logs("service:notification-service-dev message:\\"processing message\\" @notification_event_type:channel_invite").index("*").rollup("count").by("@sender_id").last("1h") > 10',
message: `A user is generating a high number of channel invites. This could mean they are spamming users with invite emails.
-${notify({ pages: false })}`,
+${notify({ pages: false, environment: 'dev' })}`,
monitorThresholds: {
critical: '10',
warning: '5',
diff --git a/infra/stacks/monitoring/monitors/notification-retention.ts b/infra/stacks/monitoring/monitors/notification-retention.ts
index 3d8027c8a4..1ca6a2fde8 100644
--- a/infra/stacks/monitoring/monitors/notification-retention.ts
+++ b/infra/stacks/monitoring/monitors/notification-retention.ts
@@ -5,13 +5,15 @@ import { goalertWebhooks } from '../goalert';
import { notify, RENOTIFY_MINUTES } from '../notify';
const HEALTH = '@telemetry_event:notification_consumer_health';
-const CONTEXT = `Notification-retention rollout: pause scheduled cleanup before investigating.
+const context = (
+ environment: 'dev' | 'prod'
+) => `Notification-retention rollout: pause scheduled cleanup before investigating.
Check consumer health/backlog, notification.delete.publish spans, database latency,
and replica lag. A zero application backlog is not an end-to-end delivery ACK:
Postgres NOTIFY publication and WebSocket delivery can still fail.
-${notify({ pages: false })}`;
+${notify({ pages: false, environment })}`;
-for (const environment of ['dev', 'prod']) {
+for (const environment of ['dev', 'prod'] as const) {
const query = `${HEALTH} env:${environment}`;
const common = {
tags: [
@@ -31,7 +33,7 @@ for (const environment of ['dev', 'prod']) {
name: `[${environment.toUpperCase()}] Notification consumer: application backlog high`,
type: 'log alert',
query: `logs("${query} @consumer_health:healthy").index("*").rollup("avg", "@max_offset_backlog").by("@consumer_instance").last("5m") > 1000`,
- message: `One consumer's maximum partition offset backlog averages over {{threshold}} for five minutes. This is not committed-offset lag. The initial threshold is conservative; tune after observing dry-run and canary baselines.\n${CONTEXT}`,
+ message: `One consumer's maximum partition offset backlog averages over {{threshold}} for five minutes. This is not committed-offset lag. The initial threshold is conservative; tune after observing dry-run and canary baselines.\n${context(environment)}`,
monitorThresholds: { critical: '1000', warning: '100' },
// A retired process must not leave its old lag alert firing forever.
// Stalls in live processes produce explicit health events below.
@@ -46,7 +48,7 @@ for (const environment of ['dev', 'prod']) {
name: `[${environment.toUpperCase()}] Notification consumer: telemetry stale or unavailable`,
type: 'log alert',
query: `logs("${query} @consumer_health:(stale OR unavailable OR disconnected)").index("*").rollup("count").by("@consumer_instance").last("5m") > 0`,
- message: `A live consumer has stale statistics, unknown offsets, or no connected brokers after its startup grace period. Do not interpret missing/partial telemetry as zero lag.\n${CONTEXT}`,
+ message: `A live consumer has stale statistics, unknown offsets, or no connected brokers after its startup grace period. Do not interpret missing/partial telemetry as zero lag.\n${context(environment)}`,
monitorThresholds: { critical: '0' },
onMissingData: 'resolve',
},
@@ -59,7 +61,7 @@ for (const environment of ['dev', 'prod']) {
name: `[${environment.toUpperCase()}] Notification consumer: no telemetry`,
type: 'log alert',
query: `logs("${query}").index("*").rollup("count").last("5m") < 1`,
- message: `No notification-consumer health events arrived for five minutes. Verify DSS consumers are running and log ingestion is healthy. Deploy the consumer instrumentation before enabling this monitor.\n${CONTEXT}`,
+ message: `No notification-consumer health events arrived for five minutes. Verify DSS consumers are running and log ingestion is healthy. Deploy the consumer instrumentation before enabling this monitor.\n${context(environment)}`,
monitorThresholds: { critical: '1' },
onMissingData: 'default',
},
@@ -87,7 +89,7 @@ for (const environment of ['dev', 'prod']) {
name: `[${environment.toUpperCase()}] Notification deletion: realtime publication failed`,
type: 'log alert',
query: `logs("env:${environment} \\"failed to publish notification delete realtime update\\"").index("*").rollup("count").last("5m") > 0`,
- message: `A database deletion could not be forwarded to realtime. Deletion has already committed; PostgreSQL NOTIFY is best-effort, not a durable retry queue.\n${CONTEXT}`,
+ message: `A database deletion could not be forwarded to realtime. Deletion has already committed; PostgreSQL NOTIFY is best-effort, not a durable retry queue.\n${context(environment)}`,
monitorThresholds: { critical: '0' },
onMissingData: 'default',
},
@@ -104,7 +106,7 @@ new datadog.Monitor(
type: 'log alert',
query:
'logs("\\"notification retention telemetry flush\\"").index("*").rollup("count").last("15m") > 0',
- message: `Cleanup may have succeeded while its traces were lost. Check the Datadog extension, its scoped secret access, and OTLP receiver. Do not retry deletion merely to retry telemetry.\n${CONTEXT}`,
+ message: `Cleanup may have succeeded while its traces were lost. Check the Datadog extension, its scoped secret access, and OTLP receiver. Do not retry deletion merely to retry telemetry.\n${context('prod')}`,
tags: ['service:notification-retention', 'rollout:notification-retention'],
priority: '3',
monitorThresholds: { critical: '0' },
diff --git a/infra/stacks/monitoring/monitors/regressions.ts b/infra/stacks/monitoring/monitors/regressions.ts
index 75d8579a7b..a8e408345b 100644
--- a/infra/stacks/monitoring/monitors/regressions.ts
+++ b/infra/stacks/monitoring/monitors/regressions.ts
@@ -33,7 +33,7 @@ const dashboard = new datadog.Dashboard('regression-alerts', {
thresholdGraph({
title: 'Internal span errors abnormally high',
query:
- "anomalies(sum:trace.Internal.errors{env:prod} by {service}.as_count(), 'agile', 3)",
+ "anomalies(sum:trace.Internal.errors{env:prod AND NOT service:web-app} by {service}.as_count(), 'agile', 3)",
}),
thresholdGraph({
title: 'Queue consumer errors abnormally high',
@@ -77,16 +77,18 @@ prodMonitor('apm-server-error-anomaly', {
runbook: 'docs/runbooks/regressions.md#errors',
});
+// Excludes web-app: browser spans fail on users' networks and devices, so its
+// band swings with traffic. Browser errors have their own RUM monitors.
prodMonitor('apm-internal-error-anomaly', {
title: 'Internal span errors abnormally high',
service: 'apm',
query:
- "avg(last_4h):anomalies(sum:trace.Internal.errors{env:prod} by {service}.as_count(), 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",
+ "avg(last_4h):anomalies(sum:trace.Internal.errors{env:prod AND NOT service:web-app} by {service}.as_count(), 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",
monitorThresholds: ANOMALY_THRESHOLDS,
monitorThresholdWindows: ANOMALY_WINDOWS,
pages: false,
description:
- '{{service.name}} is recording more failed internal spans (background jobs, workers, browser operations) than its usual band. Check the latest deploy of that service first.',
+ '{{service.name}} is recording more failed internal spans (background jobs, workers) than its usual band. Check the latest deploy of that service first.',
dashboard: dashboard.url,
runbook: 'docs/runbooks/regressions.md#errors',
});
diff --git a/infra/stacks/monitoring/monitors/rum.ts b/infra/stacks/monitoring/monitors/rum.ts
index f636ac39d5..bd8686c967 100644
--- a/infra/stacks/monitoring/monitors/rum.ts
+++ b/infra/stacks/monitoring/monitors/rum.ts
@@ -13,7 +13,7 @@ adopted('web-app-errors-staging', {
'rum("@type:error @application.name:\\"Web App\\" env:production @view.url_host:staging.macro.com").rollup("count").last("5m") > 90',
message: `The staging web app recorded {{value}} errors in 5 minutes.
-${notify({ pages: false })}`,
+${notify({ pages: false, environment: 'staging' })}`,
monitorThresholds: {
critical: '90',
warning: '50',
diff --git a/infra/stacks/monitoring/monitors/ses.ts b/infra/stacks/monitoring/monitors/ses.ts
index 593e046c9b..9b77d69a06 100644
--- a/infra/stacks/monitoring/monitors/ses.ts
+++ b/infra/stacks/monitoring/monitors/ses.ts
@@ -26,17 +26,18 @@ ${notify({ pages: true })}`,
});
// https://us5.datadoghq.com/monitors/4178105
+// No warning: SES has reported a flat 0.47% complaint rate since at least
+// 2026-10-15 while recording no complaints, so a 0.1% warning never cleared.
adopted('ses-complaint-rate', {
name: '[SES] Complaint Rate Increasing',
type: 'query alert',
query:
'avg(last_1h):avg:aws.ses.reputation_complaint_rate{aws_account:569036502058} > 0.005',
- message: `SES Bounce Rate Increasing. Action needs to be taken to prevent account suspension.
+ message: `SES complaint rate is {{value}}. SES may pause sending at 0.5%.
${notify({ pages: true })}`,
monitorThresholds: {
critical: '0.005',
- warning: '0.001',
},
onMissingData: 'default',
notifyAudit: false,
diff --git a/infra/stacks/monitoring/monitors/whatsapp-session.ts b/infra/stacks/monitoring/monitors/whatsapp-session.ts
index 40290e2924..2d65995053 100644
--- a/infra/stacks/monitoring/monitors/whatsapp-session.ts
+++ b/infra/stacks/monitoring/monitors/whatsapp-session.ts
@@ -18,15 +18,17 @@ import * as datadog from '@pulumi/datadog';
import { goalertWebhooks } from '../goalert';
import { notify, RENOTIFY_MINUTES } from '../notify';
-const CONTEXT = `Runbook: services/whatsapp_session_service/AGENTS.md. To stop all
+const context = (
+ environment: 'dev' | 'prod'
+) => `Runbook: services/whatsapp_session_service/AGENTS.md. To stop all
WhatsApp traffic, scale whatsapp-session-service to 0 tasks; sends recorded
meanwhile expire unsent after ten minutes. Turning off the enable-whatsapp-linked
flag only hides the app UI.
-${notify({ pages: false })}`;
+${notify({ pages: false, environment })}`;
const HEALTH = '@telemetry_event:whatsapp_session_health';
-for (const environment of ['dev', 'prod']) {
+for (const environment of ['dev', 'prod'] as const) {
const env = `env:${environment}`;
const label = `[${environment.toUpperCase()}] WhatsApp sessions`;
const common = {
@@ -44,7 +46,7 @@ for (const environment of ['dev', 'prod']) {
name: `${label}: accounts without an owner`,
type: 'log alert',
query: `logs("${HEALTH} ${env}").index("*").rollup("min", "@accounts_unowned").last("10m") > 0`,
- message: `{{value}} connected accounts have had no replica holding their lease for ten minutes, so their groups receive and send nothing. Check replica count, MAX_ACCOUNTS headroom, and the lock connection (DATABASE_URL must be the direct MacroDB instance endpoint).\n${CONTEXT}`,
+ message: `{{value}} connected accounts have had no replica holding their lease for ten minutes, so their groups receive and send nothing. Check replica count, MAX_ACCOUNTS headroom, and the lock connection (DATABASE_URL must be the direct MacroDB instance endpoint).\n${context(environment)}`,
monitorThresholds: { critical: '0' },
// Health events stop when the service is not deployed; that is not zero.
onMissingData: 'resolve',
@@ -59,7 +61,7 @@ for (const environment of ['dev', 'prod']) {
name: `${label}: {{account_id.name}} reconnect churn`,
type: 'log alert',
query: `logs("@telemetry_event:whatsapp_session_reconnect ${env}").index("*").rollup("count").by("@account_id").last("15m") > 10`,
- message: `Account {{account_id.name}} reconnected {{value}} times in 15 minutes. A flapping session can get the number flagged by WhatsApp; check the disconnect reasons on its spans, and whether two replicas are fighting over the lease (lease_epoch bumps).\n${CONTEXT}`,
+ message: `Account {{account_id.name}} reconnected {{value}} times in 15 minutes. A flapping session can get the number flagged by WhatsApp; check the disconnect reasons on its spans, and whether two replicas are fighting over the lease (lease_epoch bumps).\n${context(environment)}`,
monitorThresholds: { critical: '10', warning: '5' },
onMissingData: 'resolve',
},
@@ -73,7 +75,7 @@ for (const environment of ['dev', 'prod']) {
name: `${label}: sends with unknown outcome`,
type: 'log alert',
query: `logs("@telemetry_event:whatsapp_send_outcome @outcome:unknown ${env}").index("*").rollup("count").last("15m") > 5`,
- message: `{{value}} sends timed out after submission in 15 minutes. They are never resent automatically; the sender sees them as unknown. Check WhatsApp connectivity and drain/takeover timing around deploys.\n${CONTEXT}`,
+ message: `{{value}} sends timed out after submission in 15 minutes. They are never resent automatically; the sender sees them as unknown. Check WhatsApp connectivity and drain/takeover timing around deploys.\n${context(environment)}`,
monitorThresholds: { critical: '5', warning: '0' },
onMissingData: 'resolve',
},
@@ -87,7 +89,7 @@ for (const environment of ['dev', 'prod']) {
name: `${label}: oldest unprocessed inbound event`,
type: 'log alert',
query: `logs("${HEALTH} ${env}").index("*").rollup("max", "@oldest_unprocessed_inbound_seconds").last("10m") > 300`,
- message: `An inbound WhatsApp event has waited {{value}}s for ingestion. Messages are reaching the replica but not the channel; check the ingestion spans and MacroDB latency.\n${CONTEXT}`,
+ message: `An inbound WhatsApp event has waited {{value}}s for ingestion. Messages are reaching the replica but not the channel; check the ingestion spans and MacroDB latency.\n${context(environment)}`,
monitorThresholds: { critical: '300', warning: '60' },
onMissingData: 'resolve',
},
@@ -101,7 +103,7 @@ for (const environment of ['dev', 'prod']) {
name: `${label}: drain close to the stop timeout`,
type: 'log alert',
query: `logs("@telemetry_event:whatsapp_session_drain ${env}").index("*").rollup("max", "@drain_duration_ms").last("1h") > 100000`,
- message: `A replica took {{value}}ms to drain on SIGTERM. ECS kills the task at 120s, cutting off in-flight sends (they become unknown). Raise DRAIN_BATCH or lower MAX_ACCOUNTS (services/whatsapp_session_service/src/main.rs).\n${CONTEXT}`,
+ message: `A replica took {{value}}ms to drain on SIGTERM. ECS kills the task at 120s, cutting off in-flight sends (they become unknown). Raise DRAIN_BATCH or lower MAX_ACCOUNTS (services/whatsapp_session_service/src/main.rs).\n${context(environment)}`,
monitorThresholds: { critical: '100000', warning: '80000' },
onMissingData: 'resolve',
},
diff --git a/infra/stacks/monitoring/notify.ts b/infra/stacks/monitoring/notify.ts
index 670048d99f..cfc335ef36 100644
--- a/infra/stacks/monitoring/notify.ts
+++ b/infra/stacks/monitoring/notify.ts
@@ -30,7 +30,15 @@ export const RENOTIFY_MINUTES = 60;
* have warned before it alerted, or recovered from no data; closing an alert
* GoAlert does not have open does nothing.
*/
-export function notify({ pages }: { pages: boolean }): string {
+export function notify({
+ pages,
+ environment = 'prod',
+}: {
+ pages: boolean;
+ /** Dev and staging monitors stay visible in Datadog but notify no one: on-call is for prod. */
+ environment?: 'prod' | 'dev' | 'staging';
+}): string {
+ if (environment !== 'prod') return '';
const critical = pages
? `{{#is_alert}}Paging on-call. ${GOALERT}{{/is_alert}}`
: `{{#is_alert}}${GOALERT_NOTIFY}{{/is_alert}}`;
diff --git a/infra/stacks/monitoring/routing.test.ts b/infra/stacks/monitoring/routing.test.ts
index 2a234126bc..06bac7fdc6 100644
--- a/infra/stacks/monitoring/routing.test.ts
+++ b/infra/stacks/monitoring/routing.test.ts
@@ -53,9 +53,24 @@ const handles = (message: string): string[] =>
/@(?:webhook|slack|pagerduty|opsgenie|teams)-[\w-]+|[\w.+-]+@[\w-]+\.[\w.]+/g
) ?? [];
-test('every monitor and synthetic test notifies only GoAlert', () => {
+// Dev and staging monitors stay in Datadog but notify no one.
+const nonProd = (r: pulumi.runtime.MockResourceArgs) =>
+ /\[(DEV|STAGING)\]/.test(r.inputs.name);
+
+test('dev and staging monitors notify no one', () => {
+ const quiet = monitors().filter(nonProd);
+ expect(quiet.length).toBeGreaterThan(5);
+ for (const resource of quiet) {
+ expect({
+ name: resource.name,
+ handles: handles(resource.inputs.message),
+ }).toEqual({ name: resource.name, handles: [] });
+ }
+});
+
+test('every prod monitor and synthetic test notifies only GoAlert', () => {
const routed = [...monitors(), ...synthetics()].filter(
- (r) => r.name !== 'goalert-health'
+ (r) => r.name !== 'goalert-health' && !nonProd(r)
);
expect(routed.length).toBeGreaterThan(80);
for (const resource of routed) {
diff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.md
index 844ccd544b..d4240c984d 100644
--- a/infra/stacks/observability/README.md
+++ b/infra/stacks/observability/README.md
@@ -559,9 +559,11 @@ point's alerts with 401 and Grafana shows it as failing.
### Escalation
-Both policies have one step, webhook `http://127.0.0.1:8090/goalert`, and
-repeat it 5 times every 60 minutes, so the channel sees each open alert hourly
-until it is acknowledged, plus every acknowledgement and close.
+Both policies have one step, webhook `http://127.0.0.1:8090/goalert`. The
+channel sees every new alert, acknowledgement and close. `Macro prod` repeats
+its step 5 times every 60 minutes, so an unacknowledged page is posted again
+hourly; `Macro notify` posts once, so warnings stay quiet after their first
+post.
- Escalation policy `Macro prod` pages: services `Grafana critical` (Grafana's
paging criticals) and `Datadog critical`.