Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(observability): route every alert through GoAlert (#9103)

Wolf Mermelstein1b25423c874c
diff --git a/.cursor/mcp.json b/.cursor/mcp.json
index b7466c4525..e3d89496d8 100644
--- a/.cursor/mcp.json
+++ b/.cursor/mcp.json
@@ -9,6 +9,9 @@
"http://127.0.0.1:9222"
]
},
+ "goalert": {
+ "command": "goalert-mcp-macro"
+ },
"grafana": {
"command": "docker",
"args": [
diff --git a/.mcp.json b/.mcp.json
index 777d1669da..581b54bed7 100644
--- a/.mcp.json
+++ b/.mcp.json
@@ -9,6 +9,9 @@
"http://127.0.0.1:9222"
]
},
+ "goalert": {
+ "command": "goalert-mcp-macro"
+ },
"grafana": {
"command": "docker",
"args": [
diff --git a/CLAUDE.md b/CLAUDE.md
index af0cc0e159..4eb65897bd 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -61,6 +61,7 @@ Real files live in `.agents/`; tool-specific directories only symlink to it.
| Giving the user a local preview link | `local-preview` skill |
| Working inside Cursor Cloud | [Cursor Cloud](docs/development/cursor-cloud.md) |
| Driving the app through a browser | [App agent guide](docs/agent-guide/README.md) |
+| Agent MCP servers, including GoAlert | [MCP servers](docs/development/mcp-servers.md) |
| Adding or changing SDK methods | [SDK rule](.agents/rules/sdk.md), `add-sdk-endpoint` skill |
| SDK version bumps and npm publishing | [SDK release guide](packages/sdk/RELEASING.md), [release-sdk skill](.agents/skills/release-sdk/SKILL.md) |
| Adding a service, worker, or Pulumi stack | [New service guide](docs/guides/new-service.md) |
diff --git a/docs/development/mcp-servers.md b/docs/development/mcp-servers.md
new file mode 100644
index 0000000000..b662aa0ff8
--- /dev/null
+++ b/docs/development/mcp-servers.md
@@ -0,0 +1,31 @@
+# Agent MCP servers
+
+The repository registers its MCP servers once per harness: `.mcp.json` (Claude
+Code), `.cursor/mcp.json` (Cursor) and `opencode.json` (opencode). Add or change
+a server in all three. Local commands such as `goalert-mcp-macro` come from the
+Nix dev shell, so start the harness inside `nix develop`.
+
+| Server | What it reaches |
+| --- | --- |
+| `chrome-devtools` | The local agent browser on `127.0.0.1:9222`; see [running locally](running-locally.md#tracing-logs-and-the-debug-browser). |
+| `goalert` | Production GoAlert at `https://oncall.macro-internal.com`: alerts, services, on-call, schedules. |
+| `grafana` | The local LGTM Grafana on `localhost:3001`. |
+| `macro`, `posthog`, `datadog` | Hosted servers that sign in with OAuth. `macro` is Claude Code only, `datadog` opencode only. |
+
+## GoAlert
+
+`goalert-mcp-macro` runs the community
+[goalert-mcp](https://github.com/amitray007/goalert-mcp) server, built from a
+pinned commit in `nix/cloud-storage.nix`. At launch it reads
+`GOALERT_ADMIN_PASSWORD` from Doppler `observability/prd` and signs in as
+GoAlert's `admin`, so it needs a Doppler login with access to that project.
+Nothing is stored in the repository.
+
+It starts in read-only mode: only list/get tools and a read-only GraphQL query
+are exposed. To acknowledge or close alerts in one session, start the harness
+with `GOALERT_READ_ONLY=false`, for example
+`GOALERT_READ_ONLY=false claude`. Every write is made as `admin` against the
+production pager.
+
+To upgrade, review the new source first, then bump `rev`, `hash` and
+`npmDepsHash` together.
diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.md
index f89dd81d55..d9d5f1cca4 100644
--- a/infra/stacks/monitoring/AGENTS.md
+++ b/infra/stacks/monitoring/AGENTS.md
@@ -28,14 +28,20 @@ From `infra/`:
in both. Grafana pages for those, so their Datadog copies use `pages: false`.
- Read `README.md` before deployment: imported resources must be adopted first
to avoid duplicate alerts. Never deploy without explicit approval.
-- Deploying needs the `goalertIntegrationKey` secret in stack config, and
- GoAlert (infra/stacks/observability) reachable at oncall.macro-internal.com.
+- Deploying needs the `goalertIntegrationKey` and `goalertNotifyIntegrationKey`
+ secrets in stack config, and GoAlert (infra/stacks/observability) reachable
+ at oncall.macro-internal.com.
- A prod monitor's graph on its file's dashboard repeats its metric and
thresholds by hand; change both. `routing.test.ts` matches them by title.
-- Never mention a Slack handle or email address in a monitor message; end it
- with `notify({ pages })`. `routing.test.ts` enforces it.
-- Notification retention adds nine monitors and one dashboard. Deploy DSS health
- instrumentation first; missing telemetry is deliberately an alert, not zero lag.
+- End every monitor message with `notify({ pages })`, which routes only to
+ GoAlert, and pass `dependsOn: goalertWebhooks`. Never mention a channel
+ webhook, Slack handle or email address: GoAlert posts to the channel. Only
+ `goalert-health` posts to `@webhook-macro-alert-hook` directly.
+ `routing.test.ts` enforces it.
+- Notification retention adds nine monitors, one dashboard and a downtime muting
+ the prod missing-telemetry alarm; delete the downtime once DSS health
+ instrumentation is deployed. Missing telemetry is deliberately an alert, not
+ zero lag.
- Do not use native committed-offset lag for the independent notification
consumers. Their per-instance log measurements preserve broadcast semantics.
- `notificationRetentionDashboardUrl` is a stack output, not an app route.
diff --git a/infra/stacks/monitoring/Pulumi.prod.yaml b/infra/stacks/monitoring/Pulumi.prod.yaml
index c565ca2d7f..66ddabe53f 100644
--- a/infra/stacks/monitoring/Pulumi.prod.yaml
+++ b/infra/stacks/monitoring/Pulumi.prod.yaml
@@ -4,3 +4,5 @@
config:
monitoring:goalertIntegrationKey:
secure: AAABAKEDvnVx7IjOr5rgXFTFIDwyqUy8gmBPo/bVOSKuV6DsgerGZ6IddCTt5xiG4VtYCz39X2aI4B5bUVXb+WSz0cU=
+ monitoring:goalertNotifyIntegrationKey:
+ secure: AAABAPQIMCniNfR6WaoRVEvQcYjpT0HH3qR97z1+juZNLzR4VrBXzglAM76tHOr/8ImIMGrTmzyl8ZaURy1VMJvRj6g=
diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.md
index 7ed4374cbe..763cdd2b18 100644
--- a/infra/stacks/monitoring/README.md
+++ b/infra/stacks/monitoring/README.md
@@ -6,27 +6,36 @@ Datadog monitors and synthetic tests as code.
Datadog. This stack **adopts** them — it does not recreate them. `import.json`
lists every one by id. Everything else is declared here first and is created
on deploy: the prod threshold and anomaly monitors (`monitors/macrodb.ts`,
-`redis.ts`, `queues.ts`, `regressions.ts`) and their dashboards, the rollout monitors, the GoAlert webhooks and health check, and one log-based metric.
+`redis.ts`, `queues.ts`, `regressions.ts`) and their dashboards, the rollout monitors, the GoAlert webhooks and health check, one log-based metric, and one downtime.
## Routing
-Every monitor and synthetic test notifies through `notify.ts`, and nowhere
-else:
-
-- **All of them** post to the Macro Alerts channel (`@webhook-macro-alert-hook`,
- a webhook integration configured in the Datadog UI).
-- **Paging monitors** also open a GoAlert alert on `{{#is_alert}}` and close it
- on `{{#is_recovery}}` (`@webhook-goalert`, `@webhook-goalert-close`, declared
- in `goalert.ts`). GoAlert runs on the observability host
- (`infra/stacks/observability`) and phones the on-call engineer. Warnings never
- page.
-- **Every monitor renotifies hourly** (`RENOTIFY_MINUTES`) while it stays in
- warning or alert.
+Every monitor and synthetic test notifies GoAlert through `notify.ts`, and
+nothing else. GoAlert runs on the observability host (`infra/stacks/observability`)
+and has two services, each with its pair of webhooks declared in `goalert.ts`:
+
+- **Notify** (`@webhook-goalert-notify`, `@webhook-goalert-notify-close`):
+ posts the alert to the Macro Alerts channel. Every warning, alert and no-data
+ notification opens or updates an alert here, except a paging monitor's
+ critical alert.
+- **Paging** (`@webhook-goalert`, `@webhook-goalert-close`): also phones the
+ on-call engineer. Only a paging monitor's `{{#is_alert}}` opens one; its
+ warnings and no-data go to Notify.
+- **Every recovery** closes on each service the monitor could have opened.
+ Closing an alert GoAlert does not have open does nothing.
+- **The hourly re-post comes from GoAlert**: its escalation posts to the channel
+ and repeats every hour until someone acknowledges. Monitors still renotify
+ hourly (`RENOTIFY_MINUTES`); the repeat dedups into the open GoAlert alert
+ (`$ALERT_CYCLE_KEY`) and reopens it if someone closed it while the monitor
+ still fires.
+- **Datadog never posts to the channel itself**, so each alert appears once.
+ The only exception is the GoAlert health check (`goalert-health`), which
+ posts straight to `@webhook-macro-alert-hook` (a webhook integration
+ configured in the Datadog UI): when GoAlert is down, nothing routed through
+ it arrives.
- `routing.test.ts` fails if any monitor mentions another handle or an email
- address, or lacks the renotify interval.
-
-The GoAlert health check (`goalert-health`) notifies the channel only: when the
-pager is down, a page through it would never arrive.
+ address, if anything but the health check mentions the channel webhook, or if
+ a monitor lacks the renotify interval.
Paging monitors: the app-down synthetic, ALB non-200 responses and low healthy
hosts, Lambda error count, web-app source errors, macro-cache memory, RDS high
@@ -41,8 +50,8 @@ its thresholds many times a day.
the prod threshold, forecast and anomaly monitors with `prodMonitor()`. The ones
on CloudWatch metrics have hand-written copies in Grafana
(`infra/stacks/observability/nixos/prod-alerts.nix`), and Grafana pages on-call for
-the critical ones through GoAlert. Their Datadog copies only notify, so one
-incident pages once. Keep thresholds and windows in sync by hand. The error-logs
+the critical ones through GoAlert. Their Datadog copies only notify (GoAlert's
+Notify service), so one incident pages once. Keep thresholds and windows in sync by hand. The error-logs
anomaly monitor reads `macro.logs.errors`, a log-based metric declared in
`regressions.ts`, because log monitors have no anomaly function.
@@ -55,11 +64,12 @@ graph with matching markers.
## Read this before deploying
-**Set the GoAlert integration key and deploy GoAlert first.** The webhooks
-need the key, and paging monitors reference them:
+**Set the GoAlert integration keys and deploy GoAlert first.** The webhooks
+need the keys, and every monitor references them:
```bash
-pulumi config set --secret goalertIntegrationKey '<Generic API key of the Datadog service in GoAlert>'
+pulumi config set --secret goalertIntegrationKey '<Generic API key of the paging service in GoAlert>'
+pulumi config set --secret goalertNotifyIntegrationKey '<Generic API key of the Notify service in GoAlert>'
```
**Run the import before the first `pulumi up`.** The program declares 34
@@ -93,12 +103,22 @@ and records options omitted during import.
The gate is that preview:
- **34 monitors + 1 synthetic test updated, in `message` and
- `renotifyInterval` only** — the routing move. The two AI-editing monitors,
- already deployed from this stack, update the same way.
+ `renotifyInterval` only** — the routing move to GoAlert. The two AI-editing
+ monitors, already deployed from this stack, update their `message` the same
+ way. `dss-errors-prod` also changes its query and critical threshold
+ (150 → 800).
+- **Already-deployed monitors update their `message`** (GoAlert handles instead
+ of the channel), and the GraphQL soup volume, gateway connections and task
+ count monitors their `query` and thresholds.
- **73 resources to create**: 64 monitors (25 prod threshold and anomaly, 19 GraphQL soup, 9
notification retention, 10 WhatsApp session, 1 native call push), 2 webhooks,
1 log-based metric, the GoAlert synthetic test, the four prod alert
- dashboards, and the notification retention dashboard.
+ dashboards, and the notification retention dashboard. If the paging
+ webhooks and GoAlert synthetic are already deployed, expect only the 2
+ Notify webhooks among the webhooks.
+- **1 downtime to create**, muting `[PROD] Notification consumer: no
+ telemetry`, which fires by design until the DSS consumer health
+ instrumentation ships. Delete it then.
- **Any other update, a replace, or a delete** — the hand-written code in
`monitors/*.ts` drifted from the live monitor. Reconcile against
`imported.generated.ts` before deploying.
@@ -213,7 +233,9 @@ failed delete publication, plus a cross-environment retention trace-flush alarm.
The dashboard URL is exported as `notificationRetentionDashboardUrl`.
Deploy the notification consumer instrumentation in DSS before these monitors;
-otherwise the missing-telemetry alerts intentionally fire. This monitoring stack
+otherwise the missing-telemetry alerts intentionally fire. Until it ships, a
+downtime in `notification-retention.ts` mutes the prod one; delete the downtime
+once the instrumentation is deployed. This monitoring stack
still requires a separate, explicitly approved manual deployment. Adopted
monitors may change only in `message` and `renotifyInterval` (see the gate above).
diff --git a/infra/stacks/monitoring/adopted.ts b/infra/stacks/monitoring/adopted.ts
index 2aea5e94fb..a2cc20841e 100644
--- a/infra/stacks/monitoring/adopted.ts
+++ b/infra/stacks/monitoring/adopted.ts
@@ -27,7 +27,7 @@ export function adopted(
renotifyInterval: RENOTIFY_MINUTES,
...args,
},
- // Messages may mention the GoAlert webhooks, which must exist first.
+ // Messages mention the GoAlert webhooks, which must exist first.
{ protect: true, dependsOn: goalertWebhooks }
);
}
diff --git a/infra/stacks/monitoring/goalert.ts b/infra/stacks/monitoring/goalert.ts
index 964d50d643..d6f7d6293c 100644
--- a/infra/stacks/monitoring/goalert.ts
+++ b/infra/stacks/monitoring/goalert.ts
@@ -1,45 +1,60 @@
// Datadog → GoAlert, the on-call pager on the Ohio observability host
// (infra/stacks/observability).
//
-// Both webhooks post to a GoAlert Generic API integration key. GoAlert keeps
+// Each webhook posts to a GoAlert Generic API integration key: `goalert` and
+// `goalert-close` to the paging service, `goalert-notify` and
+// `goalert-notify-close` to the Notify service, which posts to the Macro Alerts
+// channel without phoning anyone. notify.ts picks between them. GoAlert keeps
// one open alert per `dedup` value: repeat notifications (renotify) update it,
// and `action: close` resolves it. `$ALERT_CYCLE_KEY` stays the same from a
// monitor group's trigger to its recovery.
import * as datadog from '@pulumi/datadog';
import * as pulumi from '@pulumi/pulumi';
-import { notify, RENOTIFY_MINUTES } from './notify';
+import { RENOTIFY_MINUTES } from './notify';
const GOALERT_HOST = 'oncall.macro-internal.com';
-// The integration key authenticates the request; it lives in this stack's
-// encrypted config: `pulumi config set --secret goalertIntegrationKey <key>`.
-const integrationKey = new pulumi.Config().requireSecret(
- 'goalertIntegrationKey'
-);
-const url = pulumi.interpolate`https://${GOALERT_HOST}/api/v2/generic/incoming?token=${integrationKey}`;
const alert = {
summary: '$EVENT_TITLE',
details: '$LINK\n\n$TEXT_ONLY_MSG',
dedup: '$ALERT_CYCLE_KEY',
};
+/** An opening and a closing webhook for one GoAlert service's integration key. */
+function goalertService(name: string, integrationKey: pulumi.Output<string>) {
+ const url = pulumi.interpolate`https://${GOALERT_HOST}/api/v2/generic/incoming?token=${integrationKey}`;
+ return [
+ new datadog.Webhook(name, {
+ name,
+ url,
+ encodeAs: 'json',
+ payload: JSON.stringify(alert),
+ }),
+ new datadog.Webhook(`${name}-close`, {
+ name: `${name}-close`,
+ url,
+ encodeAs: 'json',
+ payload: JSON.stringify({ ...alert, action: 'close' }),
+ }),
+ ];
+}
+
+// The integration keys authenticate the requests; they live in this stack's
+// encrypted config: `pulumi config set --secret <name> <key>`.
+const config = new pulumi.Config();
+
+/** Every webhook notify() mentions. Monitors depend on all of them. */
export const goalertWebhooks = [
- new datadog.Webhook('goalert', {
- name: 'goalert',
- url,
- encodeAs: 'json',
- payload: JSON.stringify(alert),
- }),
- new datadog.Webhook('goalert-close', {
- name: 'goalert-close',
- url,
- encodeAs: 'json',
- payload: JSON.stringify({ ...alert, action: 'close' }),
- }),
+ ...goalertService('goalert', config.requireSecret('goalertIntegrationKey')),
+ ...goalertService(
+ 'goalert-notify',
+ config.requireSecret('goalertNotifyIntegrationKey')
+ ),
];
-// Watches the pager from outside. It notifies the channel only: if GoAlert is
-// down, a page through GoAlert would never arrive.
+// Watches the pager from outside. It is the only monitor that posts to the
+// Macro Alerts channel directly, through the webhook integration configured in
+// the Datadog UI: if GoAlert is down, an alert routed through it never arrives.
new datadog.SyntheticsTest('goalert-health', {
name: 'GoAlert health check',
type: 'api',
@@ -58,9 +73,9 @@ new datadog.SyntheticsTest('goalert-health', {
monitorName: '[PROD] GoAlert unreachable',
monitorOptions: { renotifyInterval: RENOTIFY_MINUTES },
},
- message: `GoAlert at https://${GOALERT_HOST} is not answering its health check, so Datadog and Grafana alerts cannot page anyone. Paging falls back to the observability host's CloudWatch alarms.
+ message: `GoAlert at https://${GOALERT_HOST} is not answering its health check, so Datadog and Grafana alerts reach neither the on-call engineer nor the channel. Paging falls back to the observability host's CloudWatch alarms.
Runbook: https://github.com/macro-inc/macro/blob/main/infra/stacks/observability/README.md
-${notify({ pages: false })}`,
+@webhook-macro-alert-hook`,
});
diff --git a/infra/stacks/monitoring/monitors/ai-editing.ts b/infra/stacks/monitoring/monitors/ai-editing.ts
index 673e35917c..383cd7d4cd 100644
--- a/infra/stacks/monitoring/monitors/ai-editing.ts
+++ b/infra/stacks/monitoring/monitors/ai-editing.ts
@@ -14,6 +14,7 @@
// `documentToken` (macro_sync_service_jwt), so a probe needs somewhere to mint
// one.
import * as datadog from '@pulumi/datadog';
+import { goalertWebhooks } from '../goalert';
import { notify, RENOTIFY_MINUTES } from '../notify';
// The worker's entry span is `http.server.request` with resource_name `POST`.
@@ -46,7 +47,8 @@ ${notify({ pages: false })}`,
evaluationDelay: 60,
includeTags: true,
notifyAudit: false,
- }
+ },
+ { dependsOn: goalertWebhooks }
);
// ReqwestEditingWorkerClient::edit reports "editing worker returned" for
@@ -85,5 +87,6 @@ ${notify({ pages: false })}`,
notifyAudit: false,
enableLogsSample: true,
groupbySimpleMonitor: false,
- }
+ },
+ { dependsOn: goalertWebhooks }
);
diff --git a/infra/stacks/monitoring/monitors/calls.ts b/infra/stacks/monitoring/monitors/calls.ts
index f17cae7ef3..b5720f9926 100644
--- a/infra/stacks/monitoring/monitors/calls.ts
+++ b/infra/stacks/monitoring/monitors/calls.ts
@@ -15,6 +15,7 @@
// rejects them as disabled or missing endpoints, and `@error` carries the
// `PushEndpointUnavailable` context text, so they are excluded.
import * as datadog from '@pulumi/datadog';
+import { goalertWebhooks } from '../goalert';
import { notify, RENOTIFY_MINUTES } from '../notify';
export const nativeCallPushFailing = new datadog.Monitor(
@@ -42,5 +43,6 @@ ${notify({ pages: false })}`,
notifyAudit: false,
enableLogsSample: true,
groupbySimpleMonitor: false,
- }
+ },
+ { dependsOn: goalertWebhooks }
);
diff --git a/infra/stacks/monitoring/monitors/graphql-soup.ts b/infra/stacks/monitoring/monitors/graphql-soup.ts
index 10f15bdac7..fc4368593c 100644
--- a/infra/stacks/monitoring/monitors/graphql-soup.ts
+++ b/infra/stacks/monitoring/monitors/graphql-soup.ts
@@ -4,8 +4,10 @@
// remove them freely once the rollout settles.
//
// Thresholds sit above the 7 days before the 2026-10-11 ramp: none of these
-// would have fired in that week except where noted. PostHog covers the client
-// side (exceptions, browser cache and draft-queue logs, usage) on
+// would have fired in that week except where noted. Volume, gateway
+// connections and task count were raised on 2026-10-29 above the 7 days
+// before it, after the rollout outgrew the pre-ramp numbers. PostHog covers
+// the client side (exceptions, browser cache and draft-queue logs, usage) on
// https://us.posthog.com/project/299018/dashboard/2198651.
//
// GraphQL resolver errors return HTTP 200 on a single path, so the endpoint
@@ -69,7 +71,7 @@ ${notify({ pages: args.severe ?? false })}`,
includeTags: true,
notifyAudit: false,
},
- { dependsOn: args.severe ? goalertWebhooks : [] }
+ { dependsOn: goalertWebhooks }
);
}
@@ -176,15 +178,16 @@ rolloutMonitor('graphql-soup-unauthenticated-mutations', {
});
// Volume. There is no rate limit on the endpoint, so a client retry loop or
-// reconnect herd shows up here before it shows up as errors.
+// reconnect herd shows up here before it shows up as errors. Over the 7 days
+// to 2026-10-29 the average was 16k requests per 15 minutes, peak 42k.
rolloutMonitor('graphql-soup-request-volume', {
name: '[PROD] GraphQL soup: request volume surge',
type: 'query alert',
- query: `sum(last_15m):sum:trace.http.server.request.hits{${GRAPHQL}}.as_count() > 15000`,
+ query: `sum(last_15m):sum:trace.http.server.request.hits{${GRAPHQL}}.as_count() > 50000`,
description:
- 'POST /dss/items/soup/graphql received {{value}} requests in 15 minutes (pre-rollout max was 3,841). Check for a client retry loop before scaling.',
- critical: '15000',
- warning: '8000',
+ 'POST /dss/items/soup/graphql received {{value}} requests in 15 minutes (normal is about 16k, weekly peak 42k). Check for a client retry loop before scaling.',
+ critical: '50000',
+ warning: '45000',
service: 'cloud-storage-service',
});
@@ -223,15 +226,15 @@ rolloutMonitor('graphql-soup-dss-memory', {
});
// Each task opens its own DB pool, so a scale-out also eats DB connection
-// headroom. It scaled to 7 once in the pre-rollout week.
+// headroom. It scaled to at most 9 in the 7 days to 2026-10-29.
rolloutMonitor('graphql-soup-dss-tasks', {
name: '[PROD] GraphQL soup: cloud-storage-service scaled out',
type: 'query alert',
- query: `max(last_15m):avg:aws.ecs.service.running{${DSS}} >= 6`,
+ query: `max(last_15m):avg:aws.ecs.service.running{${DSS}} >= 10`,
description:
- 'cloud-storage-service is running {{value}} tasks (normal is 3). Check macro-db-prod connections: each task adds a connection pool.',
- critical: '6',
- warning: '5',
+ 'cloud-storage-service is running {{value}} tasks (weekly max was 9). Check macro-db-prod connections: each task adds a connection pool.',
+ critical: '10',
+ warning: '9',
service: 'cloud-storage-service',
});
@@ -274,7 +277,8 @@ rolloutMonitor('graphql-soup-db-cpu', {
});
// Load balancer. The gateway ALB is shared by every prod-gateway service and
-// there is no per-route size metric, so these only catch large shifts.
+// there is no per-route size metric, so these only catch large shifts. Active
+// connections peaked at 10.1k in the 7 days to 2026-10-29.
rolloutMonitor('graphql-soup-alb-bytes', {
name: '[PROD] GraphQL soup: prod-gateway-alb processed bytes surge',
type: 'query alert',
@@ -289,11 +293,11 @@ rolloutMonitor('graphql-soup-alb-bytes', {
rolloutMonitor('graphql-soup-alb-connections', {
name: '[PROD] GraphQL soup: prod-gateway-alb active connections high',
type: 'query alert',
- query: `avg(last_15m):sum:aws.applicationelb.active_connection_count{${GATEWAY_ALB}}.as_count() > 6000`,
+ query: `avg(last_15m):sum:aws.applicationelb.active_connection_count{${GATEWAY_ALB}}.as_count() > 15000`,
description:
- 'prod-gateway-alb active connections are at {{value}} (pre-rollout max was 2,922). Long-lived GraphQL websockets hold connections for up to the 3,600s idle timeout.',
- critical: '6000',
- warning: '4000',
+ 'prod-gateway-alb active connections are at {{value}} (weekly peak 10.1k). Long-lived GraphQL websockets hold connections for up to the 3,600s idle timeout.',
+ critical: '15000',
+ warning: '12000',
service: 'prod-gateway-alb',
});
diff --git a/infra/stacks/monitoring/monitors/logs.ts b/infra/stacks/monitoring/monitors/logs.ts
index 3a2bae165b..1d8d4d419e 100644
--- a/infra/stacks/monitoring/monitors/logs.ts
+++ b/infra/stacks/monitoring/monitors/logs.ts
@@ -76,17 +76,18 @@ ${notify({ pages: false })}`,
});
// https://us5.datadoghq.com/monitors/1062128
-// No warning threshold: prod error lines exceed 50 per 5 minutes in roughly
-// one window in eight, so a warning would notify Slack all day.
+// The steady baseline is 300-450 error lines per 5 minutes since the
+// expired-JWT exclusion, so 800 is roughly double a normal window. No warning
+// threshold: one would notify all day.
adopted('dss-errors-prod', {
name: '[PROD] Increased number of errors for DSS',
type: 'log alert',
- query: `logs("service:cloud-storage-service env:prod status:error ${DSS_EXPIRED_JWT_EXCLUSION}").index("*").rollup("count").last("5m") > 150`,
+ query: `logs("service:cloud-storage-service env:prod status:error ${DSS_EXPIRED_JWT_EXCLUSION}").index("*").rollup("count").last("5m") > 800`,
message: `Prod cloud-storage-service logged {{value}} errors in 5 minutes, not counting expired-JWT rejections.
${notify({ pages: false })}`,
monitorThresholds: {
- critical: '150',
+ critical: '800',
},
onMissingData: 'default',
notifyAudit: true,
diff --git a/infra/stacks/monitoring/monitors/notification-retention.test.ts b/infra/stacks/monitoring/monitors/notification-retention.test.ts
index e213808d66..35a2bcef29 100644
--- a/infra/stacks/monitoring/monitors/notification-retention.test.ts
+++ b/infra/stacks/monitoring/monitors/notification-retention.test.ts
@@ -4,11 +4,17 @@ import * as pulumi from '@pulumi/pulumi';
const resources: pulumi.runtime.MockResourceArgs[] = [];
beforeAll(async () => {
+ // The monitors depend on the GoAlert webhooks, which read these keys.
+ pulumi.runtime.setAllConfig({
+ 'monitoring:goalertIntegrationKey': 'integration-key',
+ 'monitoring:goalertNotifyIntegrationKey': 'notify-integration-key',
+ });
pulumi.runtime.setMocks(
{
newResource(args) {
resources.push(args);
- return { id: args.name, state: args.inputs };
+ // Datadog monitor ids are numbers.
+ return { id: String(resources.length), state: args.inputs };
},
call(args) {
return args.inputs;
@@ -76,3 +82,14 @@ test('dashboard charts only known backlog alongside health and deletion progress
resources.filter((r) => r.type.includes('monitor:Monitor'))
).toHaveLength(9);
});
+
+test('only the prod missing-telemetry alarm is muted until instrumentation ships', () => {
+ const downtimes = resources.filter(
+ (r) => r.type === 'datadog:index/downtimeSchedule:DowntimeSchedule'
+ );
+ const heartbeat =
+ resources.indexOf(monitor('notification-consumer-heartbeat-prod')) + 1;
+ expect(downtimes.map((d) => d.inputs.monitorIdentifier)).toEqual([
+ { monitorId: heartbeat },
+ ]);
+});
diff --git a/infra/stacks/monitoring/monitors/notification-retention.ts b/infra/stacks/monitoring/monitors/notification-retention.ts
index 444dc1a8f7..3d8027c8a4 100644
--- a/infra/stacks/monitoring/monitors/notification-retention.ts
+++ b/infra/stacks/monitoring/monitors/notification-retention.ts
@@ -1,6 +1,7 @@
// Independent notification consumers do not commit Kafka offsets. Native MSK
// consumer-group lag is therefore absent; these logs report application backlog.
import * as datadog from '@pulumi/datadog';
+import { goalertWebhooks } from '../goalert';
import { notify, RENOTIFY_MINUTES } from '../notify';
const HEALTH = '@telemetry_event:notification_consumer_health';
@@ -23,61 +24,96 @@ for (const environment of ['dev', 'prod']) {
requireFullWindow: false,
renotifyInterval: RENOTIFY_MINUTES,
};
- new datadog.Monitor(`notification-consumer-backlog-${environment}`, {
- ...common,
- name: `[${environment.toUpperCase()}] Notification consumer: application backlog high`,
- type: 'log alert',
- query: `logs("${query} @consumer_health:healthy").index("*").rollup("avg", "@max_offset_backlog").by("@consumer_instance").last("5m") > 1000`,
- message: `One consumer's maximum partition offset backlog averages over {{threshold}} for five minutes. This is not committed-offset lag. The initial threshold is conservative; tune after observing dry-run and canary baselines.\n${CONTEXT}`,
- monitorThresholds: { critical: '1000', warning: '100' },
- // A retired process must not leave its old lag alert firing forever.
- // Stalls in live processes produce explicit health events below.
- onMissingData: 'resolve',
- });
- new datadog.Monitor(`notification-consumer-health-${environment}`, {
- ...common,
- name: `[${environment.toUpperCase()}] Notification consumer: telemetry stale or unavailable`,
- type: 'log alert',
- query: `logs("${query} @consumer_health:(stale OR unavailable OR disconnected)").index("*").rollup("count").by("@consumer_instance").last("5m") > 0`,
- message: `A live consumer has stale statistics, unknown offsets, or no connected brokers after its startup grace period. Do not interpret missing/partial telemetry as zero lag.\n${CONTEXT}`,
- monitorThresholds: { critical: '0' },
- onMissingData: 'resolve',
- });
- new datadog.Monitor(`notification-consumer-heartbeat-${environment}`, {
- ...common,
- name: `[${environment.toUpperCase()}] Notification consumer: no telemetry`,
- type: 'log alert',
- query: `logs("${query}").index("*").rollup("count").last("5m") < 1`,
- message: `No notification-consumer health events arrived for five minutes. Verify DSS consumers are running and log ingestion is healthy. Deploy the consumer instrumentation before enabling this monitor.\n${CONTEXT}`,
- monitorThresholds: { critical: '1' },
- onMissingData: 'default',
- });
- new datadog.Monitor(`notification-delete-publication-${environment}`, {
- ...common,
- name: `[${environment.toUpperCase()}] Notification deletion: realtime publication failed`,
- type: 'log alert',
- query: `logs("env:${environment} \\"failed to publish notification delete realtime update\\"").index("*").rollup("count").last("5m") > 0`,
- message: `A database deletion could not be forwarded to realtime. Deletion has already committed; PostgreSQL NOTIFY is best-effort, not a durable retry queue.\n${CONTEXT}`,
- monitorThresholds: { critical: '0' },
- onMissingData: 'default',
- });
+ new datadog.Monitor(
+ `notification-consumer-backlog-${environment}`,
+ {
+ ...common,
+ name: `[${environment.toUpperCase()}] Notification consumer: application backlog high`,
+ type: 'log alert',
+ query: `logs("${query} @consumer_health:healthy").index("*").rollup("avg", "@max_offset_backlog").by("@consumer_instance").last("5m") > 1000`,
+ message: `One consumer's maximum partition offset backlog averages over {{threshold}} for five minutes. This is not committed-offset lag. The initial threshold is conservative; tune after observing dry-run and canary baselines.\n${CONTEXT}`,
+ monitorThresholds: { critical: '1000', warning: '100' },
+ // A retired process must not leave its old lag alert firing forever.
+ // Stalls in live processes produce explicit health events below.
+ onMissingData: 'resolve',
+ },
+ { dependsOn: goalertWebhooks }
+ );
+ new datadog.Monitor(
+ `notification-consumer-health-${environment}`,
+ {
+ ...common,
+ name: `[${environment.toUpperCase()}] Notification consumer: telemetry stale or unavailable`,
+ type: 'log alert',
+ query: `logs("${query} @consumer_health:(stale OR unavailable OR disconnected)").index("*").rollup("count").by("@consumer_instance").last("5m") > 0`,
+ message: `A live consumer has stale statistics, unknown offsets, or no connected brokers after its startup grace period. Do not interpret missing/partial telemetry as zero lag.\n${CONTEXT}`,
+ monitorThresholds: { critical: '0' },
+ onMissingData: 'resolve',
+ },
+ { dependsOn: goalertWebhooks }
+ );
+ const heartbeat = new datadog.Monitor(
+ `notification-consumer-heartbeat-${environment}`,
+ {
+ ...common,
+ name: `[${environment.toUpperCase()}] Notification consumer: no telemetry`,
+ type: 'log alert',
+ query: `logs("${query}").index("*").rollup("count").last("5m") < 1`,
+ message: `No notification-consumer health events arrived for five minutes. Verify DSS consumers are running and log ingestion is healthy. Deploy the consumer instrumentation before enabling this monitor.\n${CONTEXT}`,
+ monitorThresholds: { critical: '1' },
+ onMissingData: 'default',
+ },
+ { dependsOn: goalertWebhooks }
+ );
+ // Fires by design until the DSS notification consumer health
+ // instrumentation is deployed in prod. Delete this downtime once it is.
+ if (environment === 'prod') {
+ new datadog.DowntimeSchedule(
+ 'notification-consumer-heartbeat-prod-downtime',
+ {
+ scope: '*',
+ monitorIdentifier: { monitorId: heartbeat.id.apply(Number) },
+ // No start or end: muted from creation until this resource is deleted.
+ oneTimeSchedule: {},
+ message:
+ 'Muted until the DSS notification consumer health instrumentation is deployed.',
+ }
+ );
+ }
+ new datadog.Monitor(
+ `notification-delete-publication-${environment}`,
+ {
+ ...common,
+ name: `[${environment.toUpperCase()}] Notification deletion: realtime publication failed`,
+ type: 'log alert',
+ query: `logs("env:${environment} \\"failed to publish notification delete realtime update\\"").index("*").rollup("count").last("5m") > 0`,
+ message: `A database deletion could not be forwarded to realtime. Deletion has already committed; PostgreSQL NOTIFY is best-effort, not a durable retry queue.\n${CONTEXT}`,
+ monitorThresholds: { critical: '0' },
+ onMissingData: 'default',
+ },
+ { dependsOn: goalertWebhooks }
+ );
}
// CloudWatch's Lambda Errors stays zero when cleanup succeeds but tracing fails.
// Match the application diagnostic rather than assuming Firehose sets env tags.
-new datadog.Monitor('notification-retention-telemetry-export', {
- name: 'Notification retention: trace flush failed',
- type: 'log alert',
- query:
- 'logs("\\"notification retention telemetry flush\\"").index("*").rollup("count").last("15m") > 0',
- message: `Cleanup may have succeeded while its traces were lost. Check the Datadog extension, its scoped secret access, and OTLP receiver. Do not retry deletion merely to retry telemetry.\n${CONTEXT}`,
- tags: ['service:notification-retention', 'rollout:notification-retention'],
- priority: '3',
- monitorThresholds: { critical: '0' },
- requireFullWindow: false,
- renotifyInterval: RENOTIFY_MINUTES,
- onMissingData: 'default',
-});
+new datadog.Monitor(
+ 'notification-retention-telemetry-export',
+ {
+ name: 'Notification retention: trace flush failed',
+ type: 'log alert',
+ query:
+ 'logs("\\"notification retention telemetry flush\\"").index("*").rollup("count").last("15m") > 0',
+ message: `Cleanup may have succeeded while its traces were lost. Check the Datadog extension, its scoped secret access, and OTLP receiver. Do not retry deletion merely to retry telemetry.\n${CONTEXT}`,
+ tags: ['service:notification-retention', 'rollout:notification-retention'],
+ priority: '3',
+ monitorThresholds: { critical: '0' },
+ requireFullWindow: false,
+ renotifyInterval: RENOTIFY_MINUTES,
+ onMissingData: 'default',
+ },
+ { dependsOn: goalertWebhooks }
+);
export const notificationRetentionDashboard = new datadog.Dashboard(
'notification-retention',
diff --git a/infra/stacks/monitoring/monitors/whatsapp-session.ts b/infra/stacks/monitoring/monitors/whatsapp-session.ts
index cfb48c45d9..40290e2924 100644
--- a/infra/stacks/monitoring/monitors/whatsapp-session.ts
+++ b/infra/stacks/monitoring/monitors/whatsapp-session.ts
@@ -15,6 +15,7 @@
// Ids are opaque account uuids; never phone numbers, JIDs, or content.
// Thresholds are starting points: there is no baseline before launch.
import * as datadog from '@pulumi/datadog';
+import { goalertWebhooks } from '../goalert';
import { notify, RENOTIFY_MINUTES } from '../notify';
const CONTEXT = `Runbook: services/whatsapp_session_service/AGENTS.md. To stop all
@@ -36,54 +37,74 @@ for (const environment of ['dev', 'prod']) {
renotifyInterval: RENOTIFY_MINUTES,
};
- new datadog.Monitor(`whatsapp-session-unowned-accounts-${environment}`, {
- ...common,
- name: `${label}: accounts without an owner`,
- type: 'log alert',
- query: `logs("${HEALTH} ${env}").index("*").rollup("min", "@accounts_unowned").last("10m") > 0`,
- message: `{{value}} connected accounts have had no replica holding their lease for ten minutes, so their groups receive and send nothing. Check replica count, MAX_ACCOUNTS headroom, and the lock connection (DATABASE_URL must be the direct MacroDB instance endpoint).\n${CONTEXT}`,
- monitorThresholds: { critical: '0' },
- // Health events stop when the service is not deployed; that is not zero.
- onMissingData: 'resolve',
- });
+ new datadog.Monitor(
+ `whatsapp-session-unowned-accounts-${environment}`,
+ {
+ ...common,
+ name: `${label}: accounts without an owner`,
+ type: 'log alert',
+ query: `logs("${HEALTH} ${env}").index("*").rollup("min", "@accounts_unowned").last("10m") > 0`,
+ message: `{{value}} connected accounts have had no replica holding their lease for ten minutes, so their groups receive and send nothing. Check replica count, MAX_ACCOUNTS headroom, and the lock connection (DATABASE_URL must be the direct MacroDB instance endpoint).\n${CONTEXT}`,
+ monitorThresholds: { critical: '0' },
+ // Health events stop when the service is not deployed; that is not zero.
+ onMissingData: 'resolve',
+ },
+ { dependsOn: goalertWebhooks }
+ );
- new datadog.Monitor(`whatsapp-session-reconnect-churn-${environment}`, {
- ...common,
- name: `${label}: {{account_id.name}} reconnect churn`,
- type: 'log alert',
- query: `logs("@telemetry_event:whatsapp_session_reconnect ${env}").index("*").rollup("count").by("@account_id").last("15m") > 10`,
- message: `Account {{account_id.name}} reconnected {{value}} times in 15 minutes. A flapping session can get the number flagged by WhatsApp; check the disconnect reasons on its spans, and whether two replicas are fighting over the lease (lease_epoch bumps).\n${CONTEXT}`,
- monitorThresholds: { critical: '10', warning: '5' },
- onMissingData: 'resolve',
- });
+ new datadog.Monitor(
+ `whatsapp-session-reconnect-churn-${environment}`,
+ {
+ ...common,
+ name: `${label}: {{account_id.name}} reconnect churn`,
+ type: 'log alert',
+ query: `logs("@telemetry_event:whatsapp_session_reconnect ${env}").index("*").rollup("count").by("@account_id").last("15m") > 10`,
+ message: `Account {{account_id.name}} reconnected {{value}} times in 15 minutes. A flapping session can get the number flagged by WhatsApp; check the disconnect reasons on its spans, and whether two replicas are fighting over the lease (lease_epoch bumps).\n${CONTEXT}`,
+ monitorThresholds: { critical: '10', warning: '5' },
+ onMissingData: 'resolve',
+ },
+ { dependsOn: goalertWebhooks }
+ );
- new datadog.Monitor(`whatsapp-session-unknown-sends-${environment}`, {
- ...common,
- name: `${label}: sends with unknown outcome`,
- type: 'log alert',
- query: `logs("@telemetry_event:whatsapp_send_outcome @outcome:unknown ${env}").index("*").rollup("count").last("15m") > 5`,
- message: `{{value}} sends timed out after submission in 15 minutes. They are never resent automatically; the sender sees them as unknown. Check WhatsApp connectivity and drain/takeover timing around deploys.\n${CONTEXT}`,
- monitorThresholds: { critical: '5', warning: '0' },
- onMissingData: 'resolve',
- });
+ new datadog.Monitor(
+ `whatsapp-session-unknown-sends-${environment}`,
+ {
+ ...common,
+ name: `${label}: sends with unknown outcome`,
+ type: 'log alert',
+ query: `logs("@telemetry_event:whatsapp_send_outcome @outcome:unknown ${env}").index("*").rollup("count").last("15m") > 5`,
+ message: `{{value}} sends timed out after submission in 15 minutes. They are never resent automatically; the sender sees them as unknown. Check WhatsApp connectivity and drain/takeover timing around deploys.\n${CONTEXT}`,
+ monitorThresholds: { critical: '5', warning: '0' },
+ onMissingData: 'resolve',
+ },
+ { dependsOn: goalertWebhooks }
+ );
- new datadog.Monitor(`whatsapp-session-inbound-lag-${environment}`, {
- ...common,
- name: `${label}: oldest unprocessed inbound event`,
- type: 'log alert',
- query: `logs("${HEALTH} ${env}").index("*").rollup("max", "@oldest_unprocessed_inbound_seconds").last("10m") > 300`,
- message: `An inbound WhatsApp event has waited {{value}}s for ingestion. Messages are reaching the replica but not the channel; check the ingestion spans and MacroDB latency.\n${CONTEXT}`,
- monitorThresholds: { critical: '300', warning: '60' },
- onMissingData: 'resolve',
- });
+ new datadog.Monitor(
+ `whatsapp-session-inbound-lag-${environment}`,
+ {
+ ...common,
+ name: `${label}: oldest unprocessed inbound event`,
+ type: 'log alert',
+ query: `logs("${HEALTH} ${env}").index("*").rollup("max", "@oldest_unprocessed_inbound_seconds").last("10m") > 300`,
+ message: `An inbound WhatsApp event has waited {{value}}s for ingestion. Messages are reaching the replica but not the channel; check the ingestion spans and MacroDB latency.\n${CONTEXT}`,
+ monitorThresholds: { critical: '300', warning: '60' },
+ onMissingData: 'resolve',
+ },
+ { dependsOn: goalertWebhooks }
+ );
- new datadog.Monitor(`whatsapp-session-drain-duration-${environment}`, {
- ...common,
- name: `${label}: drain close to the stop timeout`,
- type: 'log alert',
- query: `logs("@telemetry_event:whatsapp_session_drain ${env}").index("*").rollup("max", "@drain_duration_ms").last("1h") > 100000`,
- message: `A replica took {{value}}ms to drain on SIGTERM. ECS kills the task at 120s, cutting off in-flight sends (they become unknown). Raise DRAIN_BATCH or lower MAX_ACCOUNTS (services/whatsapp_session_service/src/main.rs).\n${CONTEXT}`,
- monitorThresholds: { critical: '100000', warning: '80000' },
- onMissingData: 'resolve',
- });
+ new datadog.Monitor(
+ `whatsapp-session-drain-duration-${environment}`,
+ {
+ ...common,
+ name: `${label}: drain close to the stop timeout`,
+ type: 'log alert',
+ query: `logs("@telemetry_event:whatsapp_session_drain ${env}").index("*").rollup("max", "@drain_duration_ms").last("1h") > 100000`,
+ message: `A replica took {{value}}ms to drain on SIGTERM. ECS kills the task at 120s, cutting off in-flight sends (they become unknown). Raise DRAIN_BATCH or lower MAX_ACCOUNTS (services/whatsapp_session_service/src/main.rs).\n${CONTEXT}`,
+ monitorThresholds: { critical: '100000', warning: '80000' },
+ onMissingData: 'resolve',
+ },
+ { dependsOn: goalertWebhooks }
+ );
}
diff --git a/infra/stacks/monitoring/notify.ts b/infra/stacks/monitoring/notify.ts
index 83234c16a3..670048d99f 100644
--- a/infra/stacks/monitoring/notify.ts
+++ b/infra/stacks/monitoring/notify.ts
@@ -1,29 +1,43 @@
import * as pulumi from '@pulumi/pulumi';
-// Where every monitor in this stack notifies.
+// Where every monitor in this stack notifies: GoAlert, and nothing else.
+//
+// GoAlert (infra/stacks/observability) has two services. Notify posts each
+// alert to the Macro Alerts channel and re-posts it hourly until someone
+// acknowledges it; the paging service also phones the on-call engineer.
+// Datadog never posts to the channel itself, so an alert shows up once, from
+// GoAlert. The one exception is the GoAlert health check in goalert.ts.
//
-// All monitors post to the Macro Alerts channel. Paging monitors also open a
-// GoAlert alert, which phones the on-call engineer, and close it on recovery.
// Datadog sends warning and alert notifications again every RENOTIFY_MINUTES
-// while a monitor stays in that state, so an alert nobody fixed keeps showing up.
+// while a monitor stays in that state. The repeat dedups into the open GoAlert
+// alert, and reopens it if someone closed it while the monitor still fires.
-// Datadog webhook integration that posts into the Macro Alerts channel. It is
-// configured in the Datadog UI, not in this stack.
-export const MACRO_ALERTS = '@webhook-macro-alert-hook';
// Declared in goalert.ts.
export const GOALERT = '@webhook-goalert';
export const GOALERT_CLOSE = '@webhook-goalert-close';
+export const GOALERT_NOTIFY = '@webhook-goalert-notify';
+export const GOALERT_NOTIFY_CLOSE = '@webhook-goalert-notify-close';
export const RENOTIFY_MINUTES = 60;
-/** The notification handles that end every monitor message. */
+/**
+ * The notification handles that end every monitor message. A paging monitor's
+ * critical alert opens a page; every other warning, alert and no-data
+ * notification opens a Notify alert.
+ *
+ * Datadog allows one state block per state, never nested, so each state gets
+ * its own. A recovery closes on both services because a paging monitor may
+ * have warned before it alerted, or recovered from no data; closing an alert
+ * GoAlert does not have open does nothing.
+ */
export function notify({ pages }: { pages: boolean }): string {
- if (!pages) return MACRO_ALERTS;
- // A recovery closes the page whether the monitor recovers from alert or from
- // a warning it dropped to first. Closing an alert GoAlert no longer has
- // open does nothing.
- return `{{#is_alert}}Paging on-call. ${GOALERT}{{/is_alert}}{{#is_recovery}}${GOALERT_CLOSE}{{/is_recovery}}
-${MACRO_ALERTS}`;
+ const critical = pages
+ ? `{{#is_alert}}Paging on-call. ${GOALERT}{{/is_alert}}`
+ : `{{#is_alert}}${GOALERT_NOTIFY}{{/is_alert}}`;
+ const recovery = pages
+ ? `${GOALERT_CLOSE} ${GOALERT_NOTIFY_CLOSE}`
+ : GOALERT_NOTIFY_CLOSE;
+ return `${critical}{{#is_warning}}${GOALERT_NOTIFY}{{/is_warning}}{{#is_no_data}}${GOALERT_NOTIFY}{{/is_no_data}}{{#is_recovery}}${recovery}{{/is_recovery}}`;
}
// Datadog's site: dashboard `url`s are paths under it.
diff --git a/infra/stacks/monitoring/prod-monitor.ts b/infra/stacks/monitoring/prod-monitor.ts
index ec8fd8c415..54e13aaac0 100644
--- a/infra/stacks/monitoring/prod-monitor.ts
+++ b/infra/stacks/monitoring/prod-monitor.ts
@@ -48,7 +48,8 @@ export function prodMonitor(
includeTags: true,
notifyAudit: false,
},
- { dependsOn: args.pages ? goalertWebhooks : [] }
+ // The message mentions the GoAlert webhooks, which must exist first.
+ { dependsOn: goalertWebhooks }
);
}
diff --git a/infra/stacks/monitoring/routing.test.ts b/infra/stacks/monitoring/routing.test.ts
index 3d1810571c..2a234126bc 100644
--- a/infra/stacks/monitoring/routing.test.ts
+++ b/infra/stacks/monitoring/routing.test.ts
@@ -5,8 +5,14 @@ const resources: pulumi.runtime.MockResourceArgs[] = [];
beforeAll(async () => {
pulumi.runtime.setAllConfig(
- { 'monitoring:goalertIntegrationKey': 'integration-key' },
- ['monitoring:goalertIntegrationKey']
+ {
+ 'monitoring:goalertIntegrationKey': 'integration-key',
+ 'monitoring:goalertNotifyIntegrationKey': 'notify-integration-key',
+ },
+ [
+ 'monitoring:goalertIntegrationKey',
+ 'monitoring:goalertNotifyIntegrationKey',
+ ]
);
pulumi.runtime.setMocks(
{
@@ -40,30 +46,48 @@ const synthetics = () =>
(r) => r.type === 'datadog:index/syntheticsTest:SyntheticsTest'
);
-test('every monitor and synthetic test notifies only the Macro Alerts channel and GoAlert', () => {
- const routed = [...monitors(), ...synthetics()];
+// Notification handles and email addresses; attribute names like `@error` in
+// message text are not handles.
+const handles = (message: string): string[] =>
+ message.match(
+ /@(?:webhook|slack|pagerduty|opsgenie|teams)-[\w-]+|[\w.+-]+@[\w-]+\.[\w.]+/g
+ ) ?? [];
+
+test('every monitor and synthetic test notifies only GoAlert', () => {
+ const routed = [...monitors(), ...synthetics()].filter(
+ (r) => r.name !== 'goalert-health'
+ );
expect(routed.length).toBeGreaterThan(80);
for (const resource of routed) {
- // Notification handles and email addresses; attribute names like `@error`
- // in message text are not handles.
- const handles: string[] =
- resource.inputs.message.match(
- /@(?:webhook|slack|pagerduty|opsgenie|teams)-[\w-]+|[\w.+-]+@[\w-]+\.[\w.]+/g
- ) ?? [];
- expect({ name: resource.name, handles }).toEqual({
+ const found = handles(resource.inputs.message);
+ expect({ name: resource.name, handles: found }).toEqual({
name: resource.name,
- handles: expect.arrayContaining(['@webhook-macro-alert-hook']),
+ handles: expect.arrayContaining([
+ '@webhook-goalert-notify',
+ '@webhook-goalert-notify-close',
+ ]),
});
- for (const handle of handles) {
+ for (const handle of found) {
expect([
- '@webhook-macro-alert-hook',
'@webhook-goalert',
'@webhook-goalert-close',
+ '@webhook-goalert-notify',
+ '@webhook-goalert-notify-close',
]).toContain(handle);
}
}
});
+test('only the GoAlert health check posts to the channel directly', () => {
+ const health = synthetics().find((r) => r.name === 'goalert-health');
+ expect(handles(health?.inputs.message)).toEqual([
+ '@webhook-macro-alert-hook',
+ ]);
+ expect(health?.inputs.requestDefinition.url).toBe(
+ 'https://oncall.macro-internal.com/health'
+ );
+});
+
test('every monitor renotifies while it stays in alert', () => {
for (const resource of monitors()) {
expect({
@@ -81,35 +105,25 @@ test('every monitor renotifies while it stays in alert', () => {
}
});
-test('paging prod monitors open and close a GoAlert alert; the rest only notify', () => {
+test('a paging monitor pages on critical and notifies otherwise', () => {
const blocked = monitors().find(
(r) => r.name === 'macrodb-blocked-connections'
);
- expect(blocked?.inputs.message).toContain(
- '{{#is_alert}}Paging on-call. @webhook-goalert{{/is_alert}}'
- );
- expect(blocked?.inputs.message).toContain(
- '{{#is_recovery}}@webhook-goalert-close{{/is_recovery}}'
+ expect(blocked?.inputs.message).toEndWith(
+ '{{#is_alert}}Paging on-call. @webhook-goalert{{/is_alert}}{{#is_warning}}@webhook-goalert-notify{{/is_warning}}{{#is_no_data}}@webhook-goalert-notify{{/is_no_data}}{{#is_recovery}}@webhook-goalert-close @webhook-goalert-notify-close{{/is_recovery}}'
);
expect(blocked?.inputs.priority).toBe('2');
+});
+test('a non-paging monitor notifies on every state', () => {
// Grafana pages for this one, so Datadog only notifies.
const freeable = monitors().find(
(r) => r.name === 'macrodb-primary-freeable-memory'
);
- expect(freeable?.inputs.message).not.toContain('@webhook-goalert');
-
- const latency = monitors().find((r) => r.name === 'macrodb-read-latency');
- expect(latency?.inputs.message).not.toContain('@webhook-goalert');
- expect(latency?.inputs.priority).toBe('3');
-});
-
-test('the GoAlert health check does not page through GoAlert', () => {
- const health = synthetics().find((r) => r.name === 'goalert-health');
- expect(health?.inputs.message).not.toContain('@webhook-goalert');
- expect(health?.inputs.requestDefinition.url).toBe(
- 'https://oncall.macro-internal.com/health'
+ expect(freeable?.inputs.message).toEndWith(
+ '{{#is_alert}}@webhook-goalert-notify{{/is_alert}}{{#is_warning}}@webhook-goalert-notify{{/is_warning}}{{#is_no_data}}@webhook-goalert-notify{{/is_no_data}}{{#is_recovery}}@webhook-goalert-notify-close{{/is_recovery}}'
);
+ expect(freeable?.inputs.priority).toBe('3');
});
test('anomaly rules recover at zero and carry their alert window', () => {
diff --git a/infra/stacks/observability/AGENTS.md b/infra/stacks/observability/AGENTS.md
index a0364f759c..adff157400 100644
--- a/infra/stacks/observability/AGENTS.md
+++ b/infra/stacks/observability/AGENTS.md
@@ -45,6 +45,8 @@ From `infra/`: `bun run check`, then Biome on changed `.ts`.
- GoAlert's application config lives in its database; `goalert/apply-config.py`
overwrites it on every start from `goalert.nix`. `set-config` accepts invalid
values silently, so check the journal for `validate config`.
+- Grafana keeps a provisioned contact point removed from `alerting.nix`; list
+ its uid in `deleteContactPoints` to delete it.
- Grafana provisioning expands `$VAR` in contact point settings, but not in rule
queries or annotations, so `$B` and `$values` work there and secrets must come
in through the Grafana unit's environment.
diff --git a/infra/stacks/observability/Pulumi.prod.yaml b/infra/stacks/observability/Pulumi.prod.yaml
index 5de60a5f49..47aa9b3e11 100644
--- a/infra/stacks/observability/Pulumi.prod.yaml
+++ b/infra/stacks/observability/Pulumi.prod.yaml
@@ -2,5 +2,5 @@ config:
aws:region: us-east-2
observability:ciIngestSecretArn: arn:aws:secretsmanager:us-east-1:569036502058:secret:observability/dev-ingest-7UhJbL
observability:alarmTopicArn: arn:aws:sns:us-east-2:569036502058:observability-prod-alerts
- observability:amiId: ami-0af944c17d63ec7c4
+ observability:amiId: ami-02bbbfa6d82d3cff2
observability:secretArn: arn:aws:secretsmanager:us-east-2:569036502058:secret:observability/prod-TwQpBJ
diff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.md
index ff13d8f15d..844ccd544b 100644
--- a/infra/stacks/observability/README.md
+++ b/infra/stacks/observability/README.md
@@ -445,19 +445,24 @@ the `Prod` group, reading CloudWatch in Virginia directly, and
`nixos/prod-alerts-dashboard.nix` draws it as one panel with both thresholds on
the `Prod alerts` dashboard (`/d/prod-alerts`, rows macrodb, Redis and Queues).
Each rule carries its panel as `__dashboardUid__`/`__panelId__`, so the rule
-view and every notification link to the graph of the metric that fired. Each
+view links to the graph of the metric that fired, and as a `panel_url`
+annotation built from Grafana's `externalURL`, because GoAlert's alert details
+list annotations but not Grafana's panel link. Each
has a Datadog copy in `infra/stacks/monitoring/monitors/`; keep thresholds and
windows in sync by hand. Forecasts, anomalies, Database Monitoring, APM and log
rules are Datadog-only.
-Every alert posts to the `Macro Alerts` contact point through the loopback
-relay; critical ones renotify hourly, others every four hours. Grafana's
-`default.message` lists each alert's annotations (summary, description,
-runbook) and a `Panel:` link to its graph, opened an hour before the alert
-started. Critical alerts of rules with `pages = true` also go to the `GoAlert`
-contact point over loopback. GoAlert owns the alert from there: it opens,
-acknowledges and closes it, and its escalation posts each change to the
-channel. Their Datadog copies only notify, so an incident pages once.
+Every prod alert goes to GoAlert over loopback, and GoAlert alone posts to the
+Macro Alerts channel. Critical alerts of rules with `pages = true` go to the
+`GoAlert` contact point (the paging `Grafana critical` service); every other prod
+alert, warnings and non-paging criticals, goes to `GoAlert notify` (the `Notify`
+service), which is also the default receiver. Grafana repeats critical alerts
+hourly and others every four hours; GoAlert dedups them by fingerprint, opens,
+acknowledges and closes the alert, and its escalation posts each change to the
+channel (see [Escalation](#escalation)). The details GoAlert posts carry each
+alert's labels and annotations (summary, description, runbook, panel) and its
+Grafana source link. Paging rules' Datadog copies only notify, so an incident
+pages once.
The dev rules below keep their own route to `Macro team email`.
@@ -486,12 +491,13 @@ AMI; a schema-5 image cannot boot schema-6 user data.
## On-call paging (GoAlert)
-GoAlert is the source of truth for open alerts: Grafana sends it critical
-alerts, and on-call engineers acknowledge or close them in its UI at
+GoAlert is the source of truth for open alerts and the only sender to the Macro
+Alerts channel: Grafana sends it every prod alert, and on-call engineers acknowledge or close them in its UI at
`https://oncall.macro-internal.com`, signing in with Google. It runs on this host
so paging keeps working while production, its database or Macro itself is down.
Today it pages by posting to the Macro Alerts channel; phone paging (Twilio) and
-a rotation are follow-ups.
+a rotation are follow-ups. Coding agents reach it through the `goalert` MCP
+server; see [MCP servers](../../../docs/development/mcp-servers.md#goalert).
- `nixos/goalert/package.nix` builds GoAlert from source with `buildGoModule`,
embedding its web UI. Bump `version` and the hashes together.
@@ -506,9 +512,12 @@ a rotation are follow-ups.
- A nightly timer dumps the database to `s3://<logs bucket>/goalert/`, one object
per weekday. The dump goes to a file first, so a failed dump never replaces a
good backup.
-- `macro-alerts-relay` on `127.0.0.1:8090` turns GoAlert (`/goalert`) and Grafana
- (`/grafana`) webhooks into Macro channel posts and adds the channel's bot token.
- GoAlert webhooks cannot set headers, and the token stays in one service.
+- `macro-alerts-relay` on `127.0.0.1:8090` turns GoAlert webhooks (`/goalert`)
+ into Macro channel posts and adds the channel's bot token. GoAlert webhooks
+ cannot set headers, and the token stays in one service. A new alert posts as
+ `🚨 PAGE [<service>]` (paging services) or `🔔 [Notify]`, its summary, the first
+ 1500 characters of its details and an `Acknowledge:` link; status changes post
+ one line with the GoAlert link.
- nginx serves `oncall.macro-internal.com` from a runtime server block. Only
`/health`, `/api/v2/generic/incoming` and `/api/v2/heartbeat/` are reachable
without a session (GoAlert checks the integration key); other inbound
@@ -517,6 +526,9 @@ a rotation are follow-ups.
`https://oncall.macro-internal.com/api/v2/identity/providers/oidc/callback`.
GoAlert cannot restrict OIDC to a domain, so that client must keep the
Internal audience. The basic-auth `admin` login is the break-glass account.
+- Every `@macro.com` account is a GoAlert admin from its first sign-in:
+ `goalert/workspace-admins.sql` installs a trigger on GoAlert's `users` table
+ on every start. Other accounts stay plain users.
- The ALB sends the on-call host to its own target group, health-checked at
`/goalert-healthz`. Alarm `observability-goalert` publishes to `alarmTopicArn`
when it fails, so a GoAlert outage is reported without GoAlert.
@@ -528,29 +540,33 @@ us-east-2 Secrets Manager secret the host reads, under the lowercase names:
`goalert_encryption_key` (≥32 random characters), `goalert_admin_password`
(≥32 random characters, the `admin` login), `macro_alerts_webhook_url` (HTTPS)
and `macro_alerts_bot_token` (the Alerts channel's webhook and bot token).
-`goalert_grafana_integration_key` is optional until GoAlert has created it;
-without it Grafana's pages are rejected with 401 and the GoAlert contact point
-shows as failing.
+`goalert_grafana_integration_key` (the `Grafana critical` Grafana key) and
+`goalert_grafana_notify_integration_key` (the `Notify` Grafana key) are optional
+until GoAlert has created them; without one, GoAlert rejects that contact
+point's alerts with 401 and Grafana shows it as failing.
### First deployment
1. Add the secrets above, then build, publish and deploy the AMI as usual. The
certificate gains the on-call name, so Pulumi replaces the ACM certificate and
updates the listener.
-2. Sign in as `admin`. Create the escalation policy and service below, and a
- Grafana integration key on `Grafana critical`. Put the key in Doppler and the
- secret as `goalert_grafana_integration_key` and restart
- `observability-secrets`.
+2. Sign in as `admin`. Create the escalation policies and services below, and a
+ Grafana integration key on `Grafana critical` and on `Notify`. Put the keys in
+ Doppler and the secret as `goalert_grafana_integration_key` and
+ `goalert_grafana_notify_integration_key` and restart `observability-secrets`.
3. Send a test alert through the service and check that it reaches the channel
and can be acknowledged and closed.
### Escalation
-- Escalation policy `Macro prod`, one step: webhook
- `http://127.0.0.1:8090/goalert`. The channel sees each alert, acknowledgement
- and close.
-- Service `Grafana critical` on `Macro prod`. Warnings never page; they only
- reach the channel.
+Both policies have one step, webhook `http://127.0.0.1:8090/goalert`, and
+repeat it 5 times every 60 minutes, so the channel sees each open alert hourly
+until it is acknowledged, plus every acknowledgement and close.
+
+- Escalation policy `Macro prod` pages: services `Grafana critical` (Grafana's
+ paging criticals) and `Datadog critical`.
+- Escalation policy `Macro notify` only posts to the channel: service `Notify`
+ (Grafana warnings and non-paging criticals).
- Acknowledge an alert to stop its escalation, close it when handled; Grafana
closes it on recovery. To silence a whole service for a while, use its
maintenance mode.
diff --git a/infra/stacks/observability/assets/refresh-secrets.py b/infra/stacks/observability/assets/refresh-secrets.py
index 2e4b7e2951..adbe943623 100644
--- a/infra/stacks/observability/assets/refresh-secrets.py
+++ b/infra/stacks/observability/assets/refresh-secrets.py
@@ -30,15 +30,23 @@ for key in ['grafana_secret_key', 'otlp_token', 'goalert_encryption_key', 'goale
raise SystemExit(f'Secret too short: {key}')
if not values['macro_alerts_webhook_url'].startswith('https://'):
raise SystemExit('macro_alerts_webhook_url must use HTTPS')
-# GoAlert creates its Grafana integration key, so it can only be added after the
-# first boot. Until then GoAlert rejects Grafana's pages with 401, and Grafana
-# reports the GoAlert contact point as failing.
-grafana_key = values.get('goalert_grafana_integration_key', '')
-if not re.fullmatch(r'[0-9a-f-]{36}|', grafana_key):
- raise SystemExit('goalert_grafana_integration_key must be a GoAlert key UUID')
-if not grafana_key:
- print('goalert_grafana_integration_key is not set; Grafana cannot page through GoAlert')
-values['grafana-goalert.env'] = f'GOALERT_GRAFANA_KEY={grafana_key}\n'
+# GoAlert creates its Grafana integration keys, so they can only be added after
+# the first boot. Until then GoAlert rejects Grafana's alerts with 401, and
+# Grafana reports the GoAlert contact points as failing. The paging key belongs
+# to `Grafana critical`, the notify key to `Notify`.
+grafana_keys = {
+ 'GOALERT_GRAFANA_KEY': ('goalert_grafana_integration_key', 'Grafana cannot page through GoAlert'),
+ 'GOALERT_GRAFANA_NOTIFY_KEY': ('goalert_grafana_notify_integration_key', 'Grafana alerts cannot reach the channel'),
+}
+grafana_env = ''
+for variable, (key, consequence) in grafana_keys.items():
+ value = values.get(key, '')
+ if not re.fullmatch(r'[0-9a-f-]{36}|', value):
+ raise SystemExit(f'{key} must be a GoAlert key UUID')
+ if not value:
+ print(f'{key} is not set; {consequence}')
+ grafana_env += f'{variable}={value}\n'
+values['grafana-goalert.env'] = grafana_env
# Native remote-write uses HTTP Basic over TLS. Only the password hash reaches nginx.
hashed = subprocess.run(
['openssl', 'passwd', '-6', '-stdin'], input=values['otlp_token'] + '\n',
diff --git a/infra/stacks/observability/nixos/alerting.nix b/infra/stacks/observability/nixos/alerting.nix
index 4babf369ae..757015af27 100644
--- a/infra/stacks/observability/nixos/alerting.nix
+++ b/infra/stacks/observability/nixos/alerting.nix
@@ -92,11 +92,14 @@ let
pages = lib.boolToString ((alert.pages or false) && severity == "critical");
};
# Grafana turns the dashboard and panel annotations into the panel link
- # in notifications and the rule view.
+ # in the rule view. GoAlert's details list annotations but not Grafana's
+ # panel link, so `panel_url` repeats it. `externalURL` is the runtime
+ # root_url; trim its trailing slash.
annotations = {
summary = alert.title;
inherit (alert) description;
runbook_url = "https://github.com/macro-inc/macro/blob/main/docs/runbooks/${alert.runbook}";
+ panel_url = ''{{ reReplaceAll "/$" "" externalURL }}/d/${prodAlerts.dashboardUid}?viewPanel=${toString alert.panelId}'';
__dashboardUid__ = prodAlerts.dashboardUid;
__panelId__ = toString alert.panelId;
};
@@ -197,6 +200,14 @@ in
services.grafana.provision.alerting = {
contactPoints.settings = {
apiVersion = 1;
+ # Grafana keeps removed provisioned contact points until told to delete
+ # them. GoAlert now posts to the channel instead.
+ deleteContactPoints = [
+ {
+ orgId = 1;
+ uid = "macro-alerts";
+ }
+ ];
contactPoints = [
{
orgId = 1;
@@ -215,34 +226,32 @@ in
}
{
orgId = 1;
- name = "Macro Alerts";
+ name = "GoAlert";
receivers = [
{
- uid = "macro-alerts";
+ uid = "goalert";
type = "webhook";
disableResolveMessage = false;
- # The loopback relay adds the channel token; see goalert.nix.
+ # Loopback, so paging does not depend on the ALB. GoAlert closes the
+ # alert when Grafana sends the resolved notification.
settings = {
- url = "http://127.0.0.1:8090/grafana";
+ url = "http://127.0.0.1:3030/api/v2/grafana/incoming?token=$GOALERT_GRAFANA_KEY";
httpMethod = "POST";
- title = ''{{ template "default.title" . }}'';
- message = ''{{ template "default.message" . }}'';
};
}
];
}
{
orgId = 1;
- name = "GoAlert";
+ name = "GoAlert notify";
receivers = [
{
- uid = "goalert";
+ uid = "goalert-notify";
type = "webhook";
disableResolveMessage = false;
- # Loopback, so paging does not depend on the ALB. GoAlert closes the
- # alert when Grafana sends the resolved notification.
+ # GoAlert's `Notify` service, whose policy posts to the channel.
settings = {
- url = "http://127.0.0.1:3030/api/v2/grafana/incoming?token=$GOALERT_GRAFANA_KEY";
+ url = "http://127.0.0.1:3030/api/v2/grafana/incoming?token=$GOALERT_GRAFANA_NOTIFY_KEY";
httpMethod = "POST";
};
}
@@ -250,12 +259,16 @@ in
}
];
};
+ # Every prod alert goes to GoAlert, which posts it to the Macro Alerts
+ # channel and re-posts until acknowledged. Paging criticals go to the
+ # paging service, everything else to `Notify`. GoAlert dedups by alert
+ # fingerprint, so repeats only refresh the open alert.
policies.settings = {
apiVersion = 1;
policies = [
{
orgId = 1;
- receiver = "Macro Alerts";
+ receiver = "GoAlert notify";
group_by = [
"grafana_folder"
"alertname"
@@ -276,8 +289,6 @@ in
]
];
}
- # Paging prod rules; `continue` lets the next routes still post them
- # to Macro Alerts.
{
receiver = "GoAlert";
object_matchers = [
@@ -292,10 +303,10 @@ in
"critical"
]
];
- continue = true;
+ repeat_interval = "1h";
}
{
- receiver = "Macro Alerts";
+ receiver = "GoAlert notify";
object_matchers = [
[
"severity"
@@ -305,7 +316,6 @@ in
];
repeat_interval = "1h";
}
- { receiver = "Macro Alerts"; }
];
}
];
diff --git a/infra/stacks/observability/nixos/goalert.nix b/infra/stacks/observability/nixos/goalert.nix
index fea82f3a14..1c6cea2bcd 100644
--- a/infra/stacks/observability/nixos/goalert.nix
+++ b/infra/stacks/observability/nixos/goalert.nix
@@ -108,7 +108,7 @@ in
"google_client_secret"
];
ExecStartPre = [
- "${pkgs.python3}/bin/python3 ${./goalert/apply-config.py} ${lib.getExe goalert} ${pkgs.writeText "goalert-config.json" (builtins.toJSON goalertConfig)} ${config.services.postgresql.package}/bin/psql"
+ "${pkgs.python3}/bin/python3 ${./goalert/apply-config.py} ${lib.getExe goalert} ${pkgs.writeText "goalert-config.json" (builtins.toJSON goalertConfig)} ${config.services.postgresql.package}/bin/psql ${./goalert/workspace-admins.sql}"
];
ExecStart = start;
MemoryMax = "768M";
@@ -118,7 +118,7 @@ in
};
macro-alerts-relay = needsSecrets // {
- description = "Forward GoAlert and Grafana notifications to Macro Alerts";
+ description = "Forward GoAlert notifications to Macro Alerts";
wantedBy = [ "multi-user.target" ];
serviceConfig = {
DynamicUser = true;
diff --git a/infra/stacks/observability/nixos/goalert/apply-config.py b/infra/stacks/observability/nixos/goalert/apply-config.py
index 2180aa71e5..531f3ca065 100644
--- a/infra/stacks/observability/nixos/goalert/apply-config.py
+++ b/infra/stacks/observability/nixos/goalert/apply-config.py
@@ -1,5 +1,6 @@
-"""Migrate GoAlert's database, replace its config with the Nix-declared one, and
-create the `admin` login if it does not exist.
+"""Migrate GoAlert's database, replace its config with the Nix-declared one,
+make Macro Workspace accounts admins, and create the `admin` login if it does
+not exist.
GoAlert keeps application config only in Postgres. Running this before every
start makes the Nix attrset the source of truth: edits in the admin UI last
@@ -12,7 +13,7 @@ from pathlib import Path
import subprocess
import sys
-goalert, template_path, psql = sys.argv[1], sys.argv[2], sys.argv[3]
+goalert, template_path, psql, workspace_admins_sql = sys.argv[1:5]
credentials = Path(os.environ['CREDENTIALS_DIRECTORY'])
@@ -35,6 +36,10 @@ subprocess.run([goalert, 'migrate'], env=environment, check=True)
subprocess.run(
[goalert, 'set-config'], env=environment, input=json.dumps(config), text=True, check=True,
)
+subprocess.run(
+ [psql, '-v', 'ON_ERROR_STOP=1', '-q', '-f', workspace_admins_sql, environment['GOALERT_DB_URL']],
+ env=environment, check=True,
+)
admin_exists = subprocess.run(
[psql, '-tAc', "select 1 from auth_basic_users where username = 'admin'", environment['GOALERT_DB_URL']],
env=environment, capture_output=True, text=True, check=True,
diff --git a/infra/stacks/observability/nixos/goalert/macro-alerts-relay.py b/infra/stacks/observability/nixos/goalert/macro-alerts-relay.py
index 6e203a9b86..d806225f9d 100644
--- a/infra/stacks/observability/nixos/goalert/macro-alerts-relay.py
+++ b/infra/stacks/observability/nixos/goalert/macro-alerts-relay.py
@@ -1,8 +1,10 @@
-"""Forward GoAlert and Grafana webhooks to the Macro Alerts channel.
+"""Forward GoAlert webhooks to the Macro Alerts channel.
-Macro's channel webhook takes `{"content": ...}` and a bot-token header. GoAlert
-cannot set headers, so both senders post here over loopback and the token stays
-in this one service. Each request is forwarded once; the sender owns retries.
+GoAlert is the only sender: Grafana and Datadog alert GoAlert, whose escalation
+policies post each alert, acknowledgement and close here. Macro's channel webhook
+takes `{"content": ...}` and a bot-token header; GoAlert cannot set headers, so it
+posts here over loopback and the token stays in this one service. Each request is
+forwarded once; GoAlert owns retries.
"""
import json
import logging
@@ -19,17 +21,32 @@ BOT_TOKEN = (CREDENTIALS / 'macro_alerts_bot_token').read_text()
FORWARD_TIMEOUT_SECONDS = 2.5
MAX_BODY_BYTES = 256 * 1024
MAX_CONTENT_CHARS = 4000
+MAX_DETAILS_CHARS = 1500
+
+
+def service_prefix(service: str) -> str:
+ """`Notify` only posts to the channel; every other service pages."""
+ marker = '🔔' if 'Notify' in service else '🚨 PAGE'
+ return f"{marker} [{service}]" if service else marker
def goalert_content(payload: dict) -> str | None:
kind = payload.get('Type')
link = payload.get('GoAlertURL', '')
+ service = payload.get('ServiceName', '')
if kind == 'Alert':
- return f"🚨 Paging on-call: {payload.get('Summary', '')}\n{link}"
+ details = payload.get('Details', '').strip()
+ if len(details) > MAX_DETAILS_CHARS:
+ details = details[:MAX_DETAILS_CHARS].rstrip() + '\n…'
+ lines = [f"{service_prefix(service)} {payload.get('Summary', '')}"]
+ if details:
+ lines.append(details)
+ lines.append(f"Acknowledge: {link}")
+ return '\n\n'.join(lines)
if kind == 'AlertStatus':
return f"{payload.get('LogEntry', '')}: {payload.get('Summary', '')}\n{link}"
if kind == 'AlertBundle':
- return f"🚨 Paging on-call: {payload.get('Count')} alerts for {payload.get('ServiceName', '')}\n{link}"
+ return f"{service_prefix(service)} {payload.get('Count')} alerts\nAcknowledge: {link}"
if kind == 'AlertStatusBundle':
return f"{payload.get('LogEntry', '')} ({payload.get('Count')} alerts)\n{link}"
if kind == 'ScheduleOnCallUsers':
@@ -40,30 +57,21 @@ def goalert_content(payload: dict) -> str | None:
return None
-def grafana_content(payload: dict) -> str | None:
- title = payload.get('title')
- message = payload.get('message')
- if not isinstance(title, str) or not isinstance(message, str):
- return None
- return f"{title}\n{message}"
-
-
-SOURCES = {'/goalert': goalert_content, '/grafana': grafana_content}
-
-
class Relay(BaseHTTPRequestHandler):
def do_POST(self) -> None:
- format_content = SOURCES.get(self.path)
length = int(self.headers.get('Content-Length', '0'))
- if format_content is None or length <= 0 or length > MAX_BODY_BYTES:
- self.send_error(404 if format_content is None else 400)
+ if self.path != '/goalert':
+ self.send_error(404)
+ return
+ if length <= 0 or length > MAX_BODY_BYTES:
+ self.send_error(400)
return
try:
payload = json.loads(self.rfile.read(length))
except json.JSONDecodeError:
self.send_error(400)
return
- content = format_content(payload) if isinstance(payload, dict) else None
+ content = goalert_content(payload) if isinstance(payload, dict) else None
if content is None:
logging.warning('unrecognised %s payload', self.path)
self.send_error(422)
diff --git a/infra/stacks/observability/nixos/goalert/workspace-admins.sql b/infra/stacks/observability/nixos/goalert/workspace-admins.sql
new file mode 100644
index 0000000000..42e38e63ad
--- /dev/null
+++ b/infra/stacks/observability/nixos/goalert/workspace-admins.sql
@@ -0,0 +1,17 @@
+-- Every Macro Workspace account is a GoAlert admin, from its first sign-in.
+-- GoAlert has no default-role setting. Its OIDC login cannot restrict the
+-- email domain, so only @macro.com accounts are promoted.
+CREATE OR REPLACE FUNCTION macro_workspace_admin() RETURNS trigger AS $$
+BEGIN
+ IF NEW.email ILIKE '%@macro.com' THEN
+ NEW.role := 'admin';
+ END IF;
+ RETURN NEW;
+END
+$$ LANGUAGE plpgsql;
+
+CREATE OR REPLACE TRIGGER macro_workspace_admin
+ BEFORE INSERT OR UPDATE OF email, role ON users
+ FOR EACH ROW EXECUTE FUNCTION macro_workspace_admin();
+
+UPDATE users SET role = 'admin' WHERE email ILIKE '%@macro.com' AND role <> 'admin';
diff --git a/infra/stacks/observability/nixos/services.nix b/infra/stacks/observability/nixos/services.nix
index c10cf01397..24008bc5f2 100644
--- a/infra/stacks/observability/nixos/services.nix
+++ b/infra/stacks/observability/nixos/services.nix
@@ -68,7 +68,7 @@ in
serviceConfig = {
EnvironmentFile = [
"/opt/observability/runtime.env"
- # Interpolated into the GoAlert contact point by alerting provisioning.
+ # Interpolated into the GoAlert contact points by alerting provisioning.
"/run/macro-observability/grafana-goalert.env"
];
LoadCredential = map credential [
diff --git a/nix/cloud-storage.nix b/nix/cloud-storage.nix
index e28e1d68fd..1b8e2f3b0b 100644
--- a/nix/cloud-storage.nix
+++ b/nix/cloud-storage.nix
@@ -868,10 +868,46 @@
'';
};
+ # GoAlert MCP server (github.com/amitray007/goalert-mcp, MIT), built from
+ # the reviewed v0.1.0 commit rather than the npm tarball. Bump `rev`,
+ # `hash` and `npmDepsHash` together, after re-reviewing the source.
+ goalert-mcp = pkgs.buildNpmPackage {
+ pname = "goalert-mcp";
+ version = "0.1.0";
+ src = pkgs.fetchFromGitHub {
+ owner = "amitray007";
+ repo = "goalert-mcp";
+ rev = "c8c285f35ce31426c576a69c25f5bb1a365a7565";
+ hash = "sha256-TDBI06V6boZ3RihwirGchDcKkZ7nKCyyB1pEebqEMiU=";
+ };
+ npmDepsHash = "sha256-PhanjTUmHoXzXsu9H3wKWVrZ89tfk8T4/QYdFcMGZ+M=";
+ nodejs = pkgs.nodejs_24;
+ # Only dev dependencies (esbuild, fsevents) ship install scripts.
+ npmFlags = [ "--ignore-scripts" ];
+ meta.mainProgram = "goalert-mcp";
+ };
+
+ # Starts goalert-mcp against Macro's GoAlert with the admin login read
+ # from Doppler at launch, read-only unless GOALERT_READ_ONLY=false.
+ goalert-mcp-macro = pkgs.writeShellApplication {
+ name = "goalert-mcp-macro";
+ runtimeInputs = [ pkgs.doppler ];
+ text = ''
+ GOALERT_PASSWORD="$(doppler secrets get GOALERT_ADMIN_PASSWORD --project observability --config prd --plain)"
+ export GOALERT_PASSWORD
+ export GOALERT_BASE_URL=https://oncall.macro-internal.com
+ export GOALERT_USERNAME=admin
+ export GOALERT_READ_ONLY="''${GOALERT_READ_ONLY:-true}"
+ exec ${pkgs.lib.getExe goalert-mcp} "$@"
+ '';
+ };
+
shellTools =
with pkgs;
[
daytona
+ goalert-mcp
+ goalert-mcp-macro
nsc
parallel
docker-compose
@@ -1037,6 +1073,8 @@
openApiBins
dopplerConfigBins
nextestArchive
+ goalert-mcp
+ goalert-mcp-macro
;
default = cargoArtifacts;
}
diff --git a/opencode.json b/opencode.json
index 51c076953b..9339e0a757 100644
--- a/opencode.json
+++ b/opencode.json
@@ -14,6 +14,12 @@
"http://127.0.0.1:9222"
]
},
+ "goalert": {
+ "type": "local",
+ "command": [
+ "goalert-mcp-macro"
+ ]
+ },
"grafana": {
"type": "local",
"command": [