Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
feat(observability): route every alert through GoAlert (#9103)
Wolf Mermelstein1b25423c874c
diff --git a/.cursor/mcp.json b/.cursor/mcp.jsonindex b7466c4525..e3d89496d8 100644--- a/.cursor/mcp.json+++ b/.cursor/mcp.json@@ -9,6 +9,9 @@"http://127.0.0.1:9222"]},+ "goalert": {+ "command": "goalert-mcp-macro"+ },"grafana": {"command": "docker","args": [diff --git a/.mcp.json b/.mcp.jsonindex 777d1669da..581b54bed7 100644--- a/.mcp.json+++ b/.mcp.json@@ -9,6 +9,9 @@"http://127.0.0.1:9222"]},+ "goalert": {+ "command": "goalert-mcp-macro"+ },"grafana": {"command": "docker","args": [diff --git a/CLAUDE.md b/CLAUDE.mdindex af0cc0e159..4eb65897bd 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -61,6 +61,7 @@ Real files live in `.agents/`; tool-specific directories only symlink to it.| Giving the user a local preview link | `local-preview` skill || Working inside Cursor Cloud | [Cursor Cloud](docs/development/cursor-cloud.md) || Driving the app through a browser | [App agent guide](docs/agent-guide/README.md) |+| Agent MCP servers, including GoAlert | [MCP servers](docs/development/mcp-servers.md) || Adding or changing SDK methods | [SDK rule](.agents/rules/sdk.md), `add-sdk-endpoint` skill || SDK version bumps and npm publishing | [SDK release guide](packages/sdk/RELEASING.md), [release-sdk skill](.agents/skills/release-sdk/SKILL.md) || Adding a service, worker, or Pulumi stack | [New service guide](docs/guides/new-service.md) |diff --git a/docs/development/mcp-servers.md b/docs/development/mcp-servers.mdnew file mode 100644index 0000000000..b662aa0ff8--- /dev/null+++ b/docs/development/mcp-servers.md@@ -0,0 +1,31 @@+# Agent MCP servers++The repository registers its MCP servers once per harness: `.mcp.json` (Claude+Code), `.cursor/mcp.json` (Cursor) and `opencode.json` (opencode). Add or change+a server in all three. Local commands such as `goalert-mcp-macro` come from the+Nix dev shell, so start the harness inside `nix develop`.++| Server | What it reaches |+| --- | --- |+| `chrome-devtools` | The local agent browser on `127.0.0.1:9222`; see [running locally](running-locally.md#tracing-logs-and-the-debug-browser). |+| `goalert` | Production GoAlert at `https://oncall.macro-internal.com`: alerts, services, on-call, schedules. |+| `grafana` | The local LGTM Grafana on `localhost:3001`. |+| `macro`, `posthog`, `datadog` | Hosted servers that sign in with OAuth. `macro` is Claude Code only, `datadog` opencode only. |++## GoAlert++`goalert-mcp-macro` runs the community+[goalert-mcp](https://github.com/amitray007/goalert-mcp) server, built from a+pinned commit in `nix/cloud-storage.nix`. At launch it reads+`GOALERT_ADMIN_PASSWORD` from Doppler `observability/prd` and signs in as+GoAlert's `admin`, so it needs a Doppler login with access to that project.+Nothing is stored in the repository.++It starts in read-only mode: only list/get tools and a read-only GraphQL query+are exposed. To acknowledge or close alerts in one session, start the harness+with `GOALERT_READ_ONLY=false`, for example+`GOALERT_READ_ONLY=false claude`. Every write is made as `admin` against the+production pager.++To upgrade, review the new source first, then bump `rev`, `hash` and+`npmDepsHash` together.diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.mdindex f89dd81d55..d9d5f1cca4 100644--- a/infra/stacks/monitoring/AGENTS.md+++ b/infra/stacks/monitoring/AGENTS.md@@ -28,14 +28,20 @@ From `infra/`:in both. Grafana pages for those, so their Datadog copies use `pages: false`.- Read `README.md` before deployment: imported resources must be adopted firstto avoid duplicate alerts. Never deploy without explicit approval.-- Deploying needs the `goalertIntegrationKey` secret in stack config, and- GoAlert (infra/stacks/observability) reachable at oncall.macro-internal.com.+- Deploying needs the `goalertIntegrationKey` and `goalertNotifyIntegrationKey`+ secrets in stack config, and GoAlert (infra/stacks/observability) reachable+ at oncall.macro-internal.com.- A prod monitor's graph on its file's dashboard repeats its metric andthresholds by hand; change both. `routing.test.ts` matches them by title.-- Never mention a Slack handle or email address in a monitor message; end it- with `notify({ pages })`. `routing.test.ts` enforces it.-- Notification retention adds nine monitors and one dashboard. Deploy DSS health- instrumentation first; missing telemetry is deliberately an alert, not zero lag.+- End every monitor message with `notify({ pages })`, which routes only to+ GoAlert, and pass `dependsOn: goalertWebhooks`. Never mention a channel+ webhook, Slack handle or email address: GoAlert posts to the channel. Only+ `goalert-health` posts to `@webhook-macro-alert-hook` directly.+ `routing.test.ts` enforces it.+- Notification retention adds nine monitors, one dashboard and a downtime muting+ the prod missing-telemetry alarm; delete the downtime once DSS health+ instrumentation is deployed. Missing telemetry is deliberately an alert, not+ zero lag.- Do not use native committed-offset lag for the independent notificationconsumers. Their per-instance log measurements preserve broadcast semantics.- `notificationRetentionDashboardUrl` is a stack output, not an app route.diff --git a/infra/stacks/monitoring/Pulumi.prod.yaml b/infra/stacks/monitoring/Pulumi.prod.yamlindex c565ca2d7f..66ddabe53f 100644--- a/infra/stacks/monitoring/Pulumi.prod.yaml+++ b/infra/stacks/monitoring/Pulumi.prod.yaml@@ -4,3 +4,5 @@config:monitoring:goalertIntegrationKey:secure: AAABAKEDvnVx7IjOr5rgXFTFIDwyqUy8gmBPo/bVOSKuV6DsgerGZ6IddCTt5xiG4VtYCz39X2aI4B5bUVXb+WSz0cU=+ monitoring:goalertNotifyIntegrationKey:+ secure: AAABAPQIMCniNfR6WaoRVEvQcYjpT0HH3qR97z1+juZNLzR4VrBXzglAM76tHOr/8ImIMGrTmzyl8ZaURy1VMJvRj6g=diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.mdindex 7ed4374cbe..763cdd2b18 100644--- a/infra/stacks/monitoring/README.md+++ b/infra/stacks/monitoring/README.md@@ -6,27 +6,36 @@ Datadog monitors and synthetic tests as code.Datadog. This stack **adopts** them — it does not recreate them. `import.json`lists every one by id. Everything else is declared here first and is createdon deploy: the prod threshold and anomaly monitors (`monitors/macrodb.ts`,-`redis.ts`, `queues.ts`, `regressions.ts`) and their dashboards, the rollout monitors, the GoAlert webhooks and health check, and one log-based metric.+`redis.ts`, `queues.ts`, `regressions.ts`) and their dashboards, the rollout monitors, the GoAlert webhooks and health check, one log-based metric, and one downtime.## Routing-Every monitor and synthetic test notifies through `notify.ts`, and nowhere-else:--- **All of them** post to the Macro Alerts channel (`@webhook-macro-alert-hook`,- a webhook integration configured in the Datadog UI).-- **Paging monitors** also open a GoAlert alert on `{{#is_alert}}` and close it- on `{{#is_recovery}}` (`@webhook-goalert`, `@webhook-goalert-close`, declared- in `goalert.ts`). GoAlert runs on the observability host- (`infra/stacks/observability`) and phones the on-call engineer. Warnings never- page.-- **Every monitor renotifies hourly** (`RENOTIFY_MINUTES`) while it stays in- warning or alert.+Every monitor and synthetic test notifies GoAlert through `notify.ts`, and+nothing else. GoAlert runs on the observability host (`infra/stacks/observability`)+and has two services, each with its pair of webhooks declared in `goalert.ts`:++- **Notify** (`@webhook-goalert-notify`, `@webhook-goalert-notify-close`):+ posts the alert to the Macro Alerts channel. Every warning, alert and no-data+ notification opens or updates an alert here, except a paging monitor's+ critical alert.+- **Paging** (`@webhook-goalert`, `@webhook-goalert-close`): also phones the+ on-call engineer. Only a paging monitor's `{{#is_alert}}` opens one; its+ warnings and no-data go to Notify.+- **Every recovery** closes on each service the monitor could have opened.+ Closing an alert GoAlert does not have open does nothing.+- **The hourly re-post comes from GoAlert**: its escalation posts to the channel+ and repeats every hour until someone acknowledges. Monitors still renotify+ hourly (`RENOTIFY_MINUTES`); the repeat dedups into the open GoAlert alert+ (`$ALERT_CYCLE_KEY`) and reopens it if someone closed it while the monitor+ still fires.+- **Datadog never posts to the channel itself**, so each alert appears once.+ The only exception is the GoAlert health check (`goalert-health`), which+ posts straight to `@webhook-macro-alert-hook` (a webhook integration+ configured in the Datadog UI): when GoAlert is down, nothing routed through+ it arrives.- `routing.test.ts` fails if any monitor mentions another handle or an email- address, or lacks the renotify interval.--The GoAlert health check (`goalert-health`) notifies the channel only: when the-pager is down, a page through it would never arrive.+ address, if anything but the health check mentions the channel webhook, or if+ a monitor lacks the renotify interval.Paging monitors: the app-down synthetic, ALB non-200 responses and low healthyhosts, Lambda error count, web-app source errors, macro-cache memory, RDS high@@ -41,8 +50,8 @@ its thresholds many times a day.the prod threshold, forecast and anomaly monitors with `prodMonitor()`. The oneson CloudWatch metrics have hand-written copies in Grafana(`infra/stacks/observability/nixos/prod-alerts.nix`), and Grafana pages on-call for-the critical ones through GoAlert. Their Datadog copies only notify, so one-incident pages once. Keep thresholds and windows in sync by hand. The error-logs+the critical ones through GoAlert. Their Datadog copies only notify (GoAlert's+Notify service), so one incident pages once. Keep thresholds and windows in sync by hand. The error-logsanomaly monitor reads `macro.logs.errors`, a log-based metric declared in`regressions.ts`, because log monitors have no anomaly function.@@ -55,11 +64,12 @@ graph with matching markers.## Read this before deploying-**Set the GoAlert integration key and deploy GoAlert first.** The webhooks-need the key, and paging monitors reference them:+**Set the GoAlert integration keys and deploy GoAlert first.** The webhooks+need the keys, and every monitor references them:```bash-pulumi config set --secret goalertIntegrationKey '<Generic API key of the Datadog service in GoAlert>'+pulumi config set --secret goalertIntegrationKey '<Generic API key of the paging service in GoAlert>'+pulumi config set --secret goalertNotifyIntegrationKey '<Generic API key of the Notify service in GoAlert>'```**Run the import before the first `pulumi up`.** The program declares 34@@ -93,12 +103,22 @@ and records options omitted during import.The gate is that preview:- **34 monitors + 1 synthetic test updated, in `message` and- `renotifyInterval` only** — the routing move. The two AI-editing monitors,- already deployed from this stack, update the same way.+ `renotifyInterval` only** — the routing move to GoAlert. The two AI-editing+ monitors, already deployed from this stack, update their `message` the same+ way. `dss-errors-prod` also changes its query and critical threshold+ (150 → 800).+- **Already-deployed monitors update their `message`** (GoAlert handles instead+ of the channel), and the GraphQL soup volume, gateway connections and task+ count monitors their `query` and thresholds.- **73 resources to create**: 64 monitors (25 prod threshold and anomaly, 19 GraphQL soup, 9notification retention, 10 WhatsApp session, 1 native call push), 2 webhooks,1 log-based metric, the GoAlert synthetic test, the four prod alert- dashboards, and the notification retention dashboard.+ dashboards, and the notification retention dashboard. If the paging+ webhooks and GoAlert synthetic are already deployed, expect only the 2+ Notify webhooks among the webhooks.+- **1 downtime to create**, muting `[PROD] Notification consumer: no+ telemetry`, which fires by design until the DSS consumer health+ instrumentation ships. Delete it then.- **Any other update, a replace, or a delete** — the hand-written code in`monitors/*.ts` drifted from the live monitor. Reconcile against`imported.generated.ts` before deploying.@@ -213,7 +233,9 @@ failed delete publication, plus a cross-environment retention trace-flush alarm.The dashboard URL is exported as `notificationRetentionDashboardUrl`.Deploy the notification consumer instrumentation in DSS before these monitors;-otherwise the missing-telemetry alerts intentionally fire. This monitoring stack+otherwise the missing-telemetry alerts intentionally fire. Until it ships, a+downtime in `notification-retention.ts` mutes the prod one; delete the downtime+once the instrumentation is deployed. This monitoring stackstill requires a separate, explicitly approved manual deployment. Adoptedmonitors may change only in `message` and `renotifyInterval` (see the gate above).diff --git a/infra/stacks/monitoring/adopted.ts b/infra/stacks/monitoring/adopted.tsindex 2aea5e94fb..a2cc20841e 100644--- a/infra/stacks/monitoring/adopted.ts+++ b/infra/stacks/monitoring/adopted.ts@@ -27,7 +27,7 @@ export function adopted(renotifyInterval: RENOTIFY_MINUTES,...args,},- // Messages may mention the GoAlert webhooks, which must exist first.+ // Messages mention the GoAlert webhooks, which must exist first.{ protect: true, dependsOn: goalertWebhooks });}diff --git a/infra/stacks/monitoring/goalert.ts b/infra/stacks/monitoring/goalert.tsindex 964d50d643..d6f7d6293c 100644--- a/infra/stacks/monitoring/goalert.ts+++ b/infra/stacks/monitoring/goalert.ts@@ -1,45 +1,60 @@// Datadog → GoAlert, the on-call pager on the Ohio observability host// (infra/stacks/observability).//-// Both webhooks post to a GoAlert Generic API integration key. GoAlert keeps+// Each webhook posts to a GoAlert Generic API integration key: `goalert` and+// `goalert-close` to the paging service, `goalert-notify` and+// `goalert-notify-close` to the Notify service, which posts to the Macro Alerts+// channel without phoning anyone. notify.ts picks between them. GoAlert keeps// one open alert per `dedup` value: repeat notifications (renotify) update it,// and `action: close` resolves it. `$ALERT_CYCLE_KEY` stays the same from a// monitor group's trigger to its recovery.import * as datadog from '@pulumi/datadog';import * as pulumi from '@pulumi/pulumi';-import { notify, RENOTIFY_MINUTES } from './notify';+import { RENOTIFY_MINUTES } from './notify';const GOALERT_HOST = 'oncall.macro-internal.com';-// The integration key authenticates the request; it lives in this stack's-// encrypted config: `pulumi config set --secret goalertIntegrationKey <key>`.-const integrationKey = new pulumi.Config().requireSecret(- 'goalertIntegrationKey'-);-const url = pulumi.interpolate`https://${GOALERT_HOST}/api/v2/generic/incoming?token=${integrationKey}`;const alert = {summary: '$EVENT_TITLE',details: '$LINK\n\n$TEXT_ONLY_MSG',dedup: '$ALERT_CYCLE_KEY',};+/** An opening and a closing webhook for one GoAlert service's integration key. */+function goalertService(name: string, integrationKey: pulumi.Output<string>) {+ const url = pulumi.interpolate`https://${GOALERT_HOST}/api/v2/generic/incoming?token=${integrationKey}`;+ return [+ new datadog.Webhook(name, {+ name,+ url,+ encodeAs: 'json',+ payload: JSON.stringify(alert),+ }),+ new datadog.Webhook(`${name}-close`, {+ name: `${name}-close`,+ url,+ encodeAs: 'json',+ payload: JSON.stringify({ ...alert, action: 'close' }),+ }),+ ];+}++// The integration keys authenticate the requests; they live in this stack's+// encrypted config: `pulumi config set --secret <name> <key>`.+const config = new pulumi.Config();++/** Every webhook notify() mentions. Monitors depend on all of them. */export const goalertWebhooks = [- new datadog.Webhook('goalert', {- name: 'goalert',- url,- encodeAs: 'json',- payload: JSON.stringify(alert),- }),- new datadog.Webhook('goalert-close', {- name: 'goalert-close',- url,- encodeAs: 'json',- payload: JSON.stringify({ ...alert, action: 'close' }),- }),+ ...goalertService('goalert', config.requireSecret('goalertIntegrationKey')),+ ...goalertService(+ 'goalert-notify',+ config.requireSecret('goalertNotifyIntegrationKey')+ ),];-// Watches the pager from outside. It notifies the channel only: if GoAlert is-// down, a page through GoAlert would never arrive.+// Watches the pager from outside. It is the only monitor that posts to the+// Macro Alerts channel directly, through the webhook integration configured in+// the Datadog UI: if GoAlert is down, an alert routed through it never arrives.new datadog.SyntheticsTest('goalert-health', {name: 'GoAlert health check',type: 'api',@@ -58,9 +73,9 @@ new datadog.SyntheticsTest('goalert-health', {monitorName: '[PROD] GoAlert unreachable',monitorOptions: { renotifyInterval: RENOTIFY_MINUTES },},- message: `GoAlert at https://${GOALERT_HOST} is not answering its health check, so Datadog and Grafana alerts cannot page anyone. Paging falls back to the observability host's CloudWatch alarms.+ message: `GoAlert at https://${GOALERT_HOST} is not answering its health check, so Datadog and Grafana alerts reach neither the on-call engineer nor the channel. Paging falls back to the observability host's CloudWatch alarms.Runbook: https://github.com/macro-inc/macro/blob/main/infra/stacks/observability/README.md-${notify({ pages: false })}`,+@webhook-macro-alert-hook`,});diff --git a/infra/stacks/monitoring/monitors/ai-editing.ts b/infra/stacks/monitoring/monitors/ai-editing.tsindex 673e35917c..383cd7d4cd 100644--- a/infra/stacks/monitoring/monitors/ai-editing.ts+++ b/infra/stacks/monitoring/monitors/ai-editing.ts@@ -14,6 +14,7 @@// `documentToken` (macro_sync_service_jwt), so a probe needs somewhere to mint// one.import * as datadog from '@pulumi/datadog';+import { goalertWebhooks } from '../goalert';import { notify, RENOTIFY_MINUTES } from '../notify';// The worker's entry span is `http.server.request` with resource_name `POST`.@@ -46,7 +47,8 @@ ${notify({ pages: false })}`,evaluationDelay: 60,includeTags: true,notifyAudit: false,- }+ },+ { dependsOn: goalertWebhooks });// ReqwestEditingWorkerClient::edit reports "editing worker returned" for@@ -85,5 +87,6 @@ ${notify({ pages: false })}`,notifyAudit: false,enableLogsSample: true,groupbySimpleMonitor: false,- }+ },+ { dependsOn: goalertWebhooks });diff --git a/infra/stacks/monitoring/monitors/calls.ts b/infra/stacks/monitoring/monitors/calls.tsindex f17cae7ef3..b5720f9926 100644--- a/infra/stacks/monitoring/monitors/calls.ts+++ b/infra/stacks/monitoring/monitors/calls.ts@@ -15,6 +15,7 @@// rejects them as disabled or missing endpoints, and `@error` carries the// `PushEndpointUnavailable` context text, so they are excluded.import * as datadog from '@pulumi/datadog';+import { goalertWebhooks } from '../goalert';import { notify, RENOTIFY_MINUTES } from '../notify';export const nativeCallPushFailing = new datadog.Monitor(@@ -42,5 +43,6 @@ ${notify({ pages: false })}`,notifyAudit: false,enableLogsSample: true,groupbySimpleMonitor: false,- }+ },+ { dependsOn: goalertWebhooks });diff --git a/infra/stacks/monitoring/monitors/graphql-soup.ts b/infra/stacks/monitoring/monitors/graphql-soup.tsindex 10f15bdac7..fc4368593c 100644--- a/infra/stacks/monitoring/monitors/graphql-soup.ts+++ b/infra/stacks/monitoring/monitors/graphql-soup.ts@@ -4,8 +4,10 @@// remove them freely once the rollout settles.//// Thresholds sit above the 7 days before the 2026-10-11 ramp: none of these-// would have fired in that week except where noted. PostHog covers the client-// side (exceptions, browser cache and draft-queue logs, usage) on+// would have fired in that week except where noted. Volume, gateway+// connections and task count were raised on 2026-10-29 above the 7 days+// before it, after the rollout outgrew the pre-ramp numbers. PostHog covers+// the client side (exceptions, browser cache and draft-queue logs, usage) on// https://us.posthog.com/project/299018/dashboard/2198651.//// GraphQL resolver errors return HTTP 200 on a single path, so the endpoint@@ -69,7 +71,7 @@ ${notify({ pages: args.severe ?? false })}`,includeTags: true,notifyAudit: false,},- { dependsOn: args.severe ? goalertWebhooks : [] }+ { dependsOn: goalertWebhooks });}@@ -176,15 +178,16 @@ rolloutMonitor('graphql-soup-unauthenticated-mutations', {});// Volume. There is no rate limit on the endpoint, so a client retry loop or-// reconnect herd shows up here before it shows up as errors.+// reconnect herd shows up here before it shows up as errors. Over the 7 days+// to 2026-10-29 the average was 16k requests per 15 minutes, peak 42k.rolloutMonitor('graphql-soup-request-volume', {name: '[PROD] GraphQL soup: request volume surge',type: 'query alert',- query: `sum(last_15m):sum:trace.http.server.request.hits{${GRAPHQL}}.as_count() > 15000`,+ query: `sum(last_15m):sum:trace.http.server.request.hits{${GRAPHQL}}.as_count() > 50000`,description:- 'POST /dss/items/soup/graphql received {{value}} requests in 15 minutes (pre-rollout max was 3,841). Check for a client retry loop before scaling.',- critical: '15000',- warning: '8000',+ 'POST /dss/items/soup/graphql received {{value}} requests in 15 minutes (normal is about 16k, weekly peak 42k). Check for a client retry loop before scaling.',+ critical: '50000',+ warning: '45000',service: 'cloud-storage-service',});@@ -223,15 +226,15 @@ rolloutMonitor('graphql-soup-dss-memory', {});// Each task opens its own DB pool, so a scale-out also eats DB connection-// headroom. It scaled to 7 once in the pre-rollout week.+// headroom. It scaled to at most 9 in the 7 days to 2026-10-29.rolloutMonitor('graphql-soup-dss-tasks', {name: '[PROD] GraphQL soup: cloud-storage-service scaled out',type: 'query alert',- query: `max(last_15m):avg:aws.ecs.service.running{${DSS}} >= 6`,+ query: `max(last_15m):avg:aws.ecs.service.running{${DSS}} >= 10`,description:- 'cloud-storage-service is running {{value}} tasks (normal is 3). Check macro-db-prod connections: each task adds a connection pool.',- critical: '6',- warning: '5',+ 'cloud-storage-service is running {{value}} tasks (weekly max was 9). Check macro-db-prod connections: each task adds a connection pool.',+ critical: '10',+ warning: '9',service: 'cloud-storage-service',});@@ -274,7 +277,8 @@ rolloutMonitor('graphql-soup-db-cpu', {});// Load balancer. The gateway ALB is shared by every prod-gateway service and-// there is no per-route size metric, so these only catch large shifts.+// there is no per-route size metric, so these only catch large shifts. Active+// connections peaked at 10.1k in the 7 days to 2026-10-29.rolloutMonitor('graphql-soup-alb-bytes', {name: '[PROD] GraphQL soup: prod-gateway-alb processed bytes surge',type: 'query alert',@@ -289,11 +293,11 @@ rolloutMonitor('graphql-soup-alb-bytes', {rolloutMonitor('graphql-soup-alb-connections', {name: '[PROD] GraphQL soup: prod-gateway-alb active connections high',type: 'query alert',- query: `avg(last_15m):sum:aws.applicationelb.active_connection_count{${GATEWAY_ALB}}.as_count() > 6000`,+ query: `avg(last_15m):sum:aws.applicationelb.active_connection_count{${GATEWAY_ALB}}.as_count() > 15000`,description:- 'prod-gateway-alb active connections are at {{value}} (pre-rollout max was 2,922). Long-lived GraphQL websockets hold connections for up to the 3,600s idle timeout.',- critical: '6000',- warning: '4000',+ 'prod-gateway-alb active connections are at {{value}} (weekly peak 10.1k). Long-lived GraphQL websockets hold connections for up to the 3,600s idle timeout.',+ critical: '15000',+ warning: '12000',service: 'prod-gateway-alb',});diff --git a/infra/stacks/monitoring/monitors/logs.ts b/infra/stacks/monitoring/monitors/logs.tsindex 3a2bae165b..1d8d4d419e 100644--- a/infra/stacks/monitoring/monitors/logs.ts+++ b/infra/stacks/monitoring/monitors/logs.ts@@ -76,17 +76,18 @@ ${notify({ pages: false })}`,});// https://us5.datadoghq.com/monitors/1062128-// No warning threshold: prod error lines exceed 50 per 5 minutes in roughly-// one window in eight, so a warning would notify Slack all day.+// The steady baseline is 300-450 error lines per 5 minutes since the+// expired-JWT exclusion, so 800 is roughly double a normal window. No warning+// threshold: one would notify all day.adopted('dss-errors-prod', {name: '[PROD] Increased number of errors for DSS',type: 'log alert',- query: `logs("service:cloud-storage-service env:prod status:error ${DSS_EXPIRED_JWT_EXCLUSION}").index("*").rollup("count").last("5m") > 150`,+ query: `logs("service:cloud-storage-service env:prod status:error ${DSS_EXPIRED_JWT_EXCLUSION}").index("*").rollup("count").last("5m") > 800`,message: `Prod cloud-storage-service logged {{value}} errors in 5 minutes, not counting expired-JWT rejections.${notify({ pages: false })}`,monitorThresholds: {- critical: '150',+ critical: '800',},onMissingData: 'default',notifyAudit: true,diff --git a/infra/stacks/monitoring/monitors/notification-retention.test.ts b/infra/stacks/monitoring/monitors/notification-retention.test.tsindex e213808d66..35a2bcef29 100644--- a/infra/stacks/monitoring/monitors/notification-retention.test.ts+++ b/infra/stacks/monitoring/monitors/notification-retention.test.ts@@ -4,11 +4,17 @@ import * as pulumi from '@pulumi/pulumi';const resources: pulumi.runtime.MockResourceArgs[] = [];beforeAll(async () => {+ // The monitors depend on the GoAlert webhooks, which read these keys.+ pulumi.runtime.setAllConfig({+ 'monitoring:goalertIntegrationKey': 'integration-key',+ 'monitoring:goalertNotifyIntegrationKey': 'notify-integration-key',+ });pulumi.runtime.setMocks({newResource(args) {resources.push(args);- return { id: args.name, state: args.inputs };+ // Datadog monitor ids are numbers.+ return { id: String(resources.length), state: args.inputs };},call(args) {return args.inputs;@@ -76,3 +82,14 @@ test('dashboard charts only known backlog alongside health and deletion progressresources.filter((r) => r.type.includes('monitor:Monitor'))).toHaveLength(9);});++test('only the prod missing-telemetry alarm is muted until instrumentation ships', () => {+ const downtimes = resources.filter(+ (r) => r.type === 'datadog:index/downtimeSchedule:DowntimeSchedule'+ );+ const heartbeat =+ resources.indexOf(monitor('notification-consumer-heartbeat-prod')) + 1;+ expect(downtimes.map((d) => d.inputs.monitorIdentifier)).toEqual([+ { monitorId: heartbeat },+ ]);+});diff --git a/infra/stacks/monitoring/monitors/notification-retention.ts b/infra/stacks/monitoring/monitors/notification-retention.tsindex 444dc1a8f7..3d8027c8a4 100644--- a/infra/stacks/monitoring/monitors/notification-retention.ts+++ b/infra/stacks/monitoring/monitors/notification-retention.ts@@ -1,6 +1,7 @@// Independent notification consumers do not commit Kafka offsets. Native MSK// consumer-group lag is therefore absent; these logs report application backlog.import * as datadog from '@pulumi/datadog';+import { goalertWebhooks } from '../goalert';import { notify, RENOTIFY_MINUTES } from '../notify';const HEALTH = '@telemetry_event:notification_consumer_health';@@ -23,61 +24,96 @@ for (const environment of ['dev', 'prod']) {requireFullWindow: false,renotifyInterval: RENOTIFY_MINUTES,};- new datadog.Monitor(`notification-consumer-backlog-${environment}`, {- ...common,- name: `[${environment.toUpperCase()}] Notification consumer: application backlog high`,- type: 'log alert',- query: `logs("${query} @consumer_health:healthy").index("*").rollup("avg", "@max_offset_backlog").by("@consumer_instance").last("5m") > 1000`,- message: `One consumer's maximum partition offset backlog averages over {{threshold}} for five minutes. This is not committed-offset lag. The initial threshold is conservative; tune after observing dry-run and canary baselines.\n${CONTEXT}`,- monitorThresholds: { critical: '1000', warning: '100' },- // A retired process must not leave its old lag alert firing forever.- // Stalls in live processes produce explicit health events below.- onMissingData: 'resolve',- });- new datadog.Monitor(`notification-consumer-health-${environment}`, {- ...common,- name: `[${environment.toUpperCase()}] Notification consumer: telemetry stale or unavailable`,- type: 'log alert',- query: `logs("${query} @consumer_health:(stale OR unavailable OR disconnected)").index("*").rollup("count").by("@consumer_instance").last("5m") > 0`,- message: `A live consumer has stale statistics, unknown offsets, or no connected brokers after its startup grace period. Do not interpret missing/partial telemetry as zero lag.\n${CONTEXT}`,- monitorThresholds: { critical: '0' },- onMissingData: 'resolve',- });- new datadog.Monitor(`notification-consumer-heartbeat-${environment}`, {- ...common,- name: `[${environment.toUpperCase()}] Notification consumer: no telemetry`,- type: 'log alert',- query: `logs("${query}").index("*").rollup("count").last("5m") < 1`,- message: `No notification-consumer health events arrived for five minutes. Verify DSS consumers are running and log ingestion is healthy. Deploy the consumer instrumentation before enabling this monitor.\n${CONTEXT}`,- monitorThresholds: { critical: '1' },- onMissingData: 'default',- });- new datadog.Monitor(`notification-delete-publication-${environment}`, {- ...common,- name: `[${environment.toUpperCase()}] Notification deletion: realtime publication failed`,- type: 'log alert',- query: `logs("env:${environment} \\"failed to publish notification delete realtime update\\"").index("*").rollup("count").last("5m") > 0`,- message: `A database deletion could not be forwarded to realtime. Deletion has already committed; PostgreSQL NOTIFY is best-effort, not a durable retry queue.\n${CONTEXT}`,- monitorThresholds: { critical: '0' },- onMissingData: 'default',- });+ new datadog.Monitor(+ `notification-consumer-backlog-${environment}`,+ {+ ...common,+ name: `[${environment.toUpperCase()}] Notification consumer: application backlog high`,+ type: 'log alert',+ query: `logs("${query} @consumer_health:healthy").index("*").rollup("avg", "@max_offset_backlog").by("@consumer_instance").last("5m") > 1000`,+ message: `One consumer's maximum partition offset backlog averages over {{threshold}} for five minutes. This is not committed-offset lag. The initial threshold is conservative; tune after observing dry-run and canary baselines.\n${CONTEXT}`,+ monitorThresholds: { critical: '1000', warning: '100' },+ // A retired process must not leave its old lag alert firing forever.+ // Stalls in live processes produce explicit health events below.+ onMissingData: 'resolve',+ },+ { dependsOn: goalertWebhooks }+ );+ new datadog.Monitor(+ `notification-consumer-health-${environment}`,+ {+ ...common,+ name: `[${environment.toUpperCase()}] Notification consumer: telemetry stale or unavailable`,+ type: 'log alert',+ query: `logs("${query} @consumer_health:(stale OR unavailable OR disconnected)").index("*").rollup("count").by("@consumer_instance").last("5m") > 0`,+ message: `A live consumer has stale statistics, unknown offsets, or no connected brokers after its startup grace period. Do not interpret missing/partial telemetry as zero lag.\n${CONTEXT}`,+ monitorThresholds: { critical: '0' },+ onMissingData: 'resolve',+ },+ { dependsOn: goalertWebhooks }+ );+ const heartbeat = new datadog.Monitor(+ `notification-consumer-heartbeat-${environment}`,+ {+ ...common,+ name: `[${environment.toUpperCase()}] Notification consumer: no telemetry`,+ type: 'log alert',+ query: `logs("${query}").index("*").rollup("count").last("5m") < 1`,+ message: `No notification-consumer health events arrived for five minutes. Verify DSS consumers are running and log ingestion is healthy. Deploy the consumer instrumentation before enabling this monitor.\n${CONTEXT}`,+ monitorThresholds: { critical: '1' },+ onMissingData: 'default',+ },+ { dependsOn: goalertWebhooks }+ );+ // Fires by design until the DSS notification consumer health+ // instrumentation is deployed in prod. Delete this downtime once it is.+ if (environment === 'prod') {+ new datadog.DowntimeSchedule(+ 'notification-consumer-heartbeat-prod-downtime',+ {+ scope: '*',+ monitorIdentifier: { monitorId: heartbeat.id.apply(Number) },+ // No start or end: muted from creation until this resource is deleted.+ oneTimeSchedule: {},+ message:+ 'Muted until the DSS notification consumer health instrumentation is deployed.',+ }+ );+ }+ new datadog.Monitor(+ `notification-delete-publication-${environment}`,+ {+ ...common,+ name: `[${environment.toUpperCase()}] Notification deletion: realtime publication failed`,+ type: 'log alert',+ query: `logs("env:${environment} \\"failed to publish notification delete realtime update\\"").index("*").rollup("count").last("5m") > 0`,+ message: `A database deletion could not be forwarded to realtime. Deletion has already committed; PostgreSQL NOTIFY is best-effort, not a durable retry queue.\n${CONTEXT}`,+ monitorThresholds: { critical: '0' },+ onMissingData: 'default',+ },+ { dependsOn: goalertWebhooks }+ );}// CloudWatch's Lambda Errors stays zero when cleanup succeeds but tracing fails.// Match the application diagnostic rather than assuming Firehose sets env tags.-new datadog.Monitor('notification-retention-telemetry-export', {- name: 'Notification retention: trace flush failed',- type: 'log alert',- query:- 'logs("\\"notification retention telemetry flush\\"").index("*").rollup("count").last("15m") > 0',- message: `Cleanup may have succeeded while its traces were lost. Check the Datadog extension, its scoped secret access, and OTLP receiver. Do not retry deletion merely to retry telemetry.\n${CONTEXT}`,- tags: ['service:notification-retention', 'rollout:notification-retention'],- priority: '3',- monitorThresholds: { critical: '0' },- requireFullWindow: false,- renotifyInterval: RENOTIFY_MINUTES,- onMissingData: 'default',-});+new datadog.Monitor(+ 'notification-retention-telemetry-export',+ {+ name: 'Notification retention: trace flush failed',+ type: 'log alert',+ query:+ 'logs("\\"notification retention telemetry flush\\"").index("*").rollup("count").last("15m") > 0',+ message: `Cleanup may have succeeded while its traces were lost. Check the Datadog extension, its scoped secret access, and OTLP receiver. Do not retry deletion merely to retry telemetry.\n${CONTEXT}`,+ tags: ['service:notification-retention', 'rollout:notification-retention'],+ priority: '3',+ monitorThresholds: { critical: '0' },+ requireFullWindow: false,+ renotifyInterval: RENOTIFY_MINUTES,+ onMissingData: 'default',+ },+ { dependsOn: goalertWebhooks }+);export const notificationRetentionDashboard = new datadog.Dashboard('notification-retention',diff --git a/infra/stacks/monitoring/monitors/whatsapp-session.ts b/infra/stacks/monitoring/monitors/whatsapp-session.tsindex cfb48c45d9..40290e2924 100644--- a/infra/stacks/monitoring/monitors/whatsapp-session.ts+++ b/infra/stacks/monitoring/monitors/whatsapp-session.ts@@ -15,6 +15,7 @@// Ids are opaque account uuids; never phone numbers, JIDs, or content.// Thresholds are starting points: there is no baseline before launch.import * as datadog from '@pulumi/datadog';+import { goalertWebhooks } from '../goalert';import { notify, RENOTIFY_MINUTES } from '../notify';const CONTEXT = `Runbook: services/whatsapp_session_service/AGENTS.md. To stop all@@ -36,54 +37,74 @@ for (const environment of ['dev', 'prod']) {renotifyInterval: RENOTIFY_MINUTES,};- new datadog.Monitor(`whatsapp-session-unowned-accounts-${environment}`, {- ...common,- name: `${label}: accounts without an owner`,- type: 'log alert',- query: `logs("${HEALTH} ${env}").index("*").rollup("min", "@accounts_unowned").last("10m") > 0`,- message: `{{value}} connected accounts have had no replica holding their lease for ten minutes, so their groups receive and send nothing. Check replica count, MAX_ACCOUNTS headroom, and the lock connection (DATABASE_URL must be the direct MacroDB instance endpoint).\n${CONTEXT}`,- monitorThresholds: { critical: '0' },- // Health events stop when the service is not deployed; that is not zero.- onMissingData: 'resolve',- });+ new datadog.Monitor(+ `whatsapp-session-unowned-accounts-${environment}`,+ {+ ...common,+ name: `${label}: accounts without an owner`,+ type: 'log alert',+ query: `logs("${HEALTH} ${env}").index("*").rollup("min", "@accounts_unowned").last("10m") > 0`,+ message: `{{value}} connected accounts have had no replica holding their lease for ten minutes, so their groups receive and send nothing. Check replica count, MAX_ACCOUNTS headroom, and the lock connection (DATABASE_URL must be the direct MacroDB instance endpoint).\n${CONTEXT}`,+ monitorThresholds: { critical: '0' },+ // Health events stop when the service is not deployed; that is not zero.+ onMissingData: 'resolve',+ },+ { dependsOn: goalertWebhooks }+ );- new datadog.Monitor(`whatsapp-session-reconnect-churn-${environment}`, {- ...common,- name: `${label}: {{account_id.name}} reconnect churn`,- type: 'log alert',- query: `logs("@telemetry_event:whatsapp_session_reconnect ${env}").index("*").rollup("count").by("@account_id").last("15m") > 10`,- message: `Account {{account_id.name}} reconnected {{value}} times in 15 minutes. A flapping session can get the number flagged by WhatsApp; check the disconnect reasons on its spans, and whether two replicas are fighting over the lease (lease_epoch bumps).\n${CONTEXT}`,- monitorThresholds: { critical: '10', warning: '5' },- onMissingData: 'resolve',- });+ new datadog.Monitor(+ `whatsapp-session-reconnect-churn-${environment}`,+ {+ ...common,+ name: `${label}: {{account_id.name}} reconnect churn`,+ type: 'log alert',+ query: `logs("@telemetry_event:whatsapp_session_reconnect ${env}").index("*").rollup("count").by("@account_id").last("15m") > 10`,+ message: `Account {{account_id.name}} reconnected {{value}} times in 15 minutes. A flapping session can get the number flagged by WhatsApp; check the disconnect reasons on its spans, and whether two replicas are fighting over the lease (lease_epoch bumps).\n${CONTEXT}`,+ monitorThresholds: { critical: '10', warning: '5' },+ onMissingData: 'resolve',+ },+ { dependsOn: goalertWebhooks }+ );- new datadog.Monitor(`whatsapp-session-unknown-sends-${environment}`, {- ...common,- name: `${label}: sends with unknown outcome`,- type: 'log alert',- query: `logs("@telemetry_event:whatsapp_send_outcome @outcome:unknown ${env}").index("*").rollup("count").last("15m") > 5`,- message: `{{value}} sends timed out after submission in 15 minutes. They are never resent automatically; the sender sees them as unknown. Check WhatsApp connectivity and drain/takeover timing around deploys.\n${CONTEXT}`,- monitorThresholds: { critical: '5', warning: '0' },- onMissingData: 'resolve',- });+ new datadog.Monitor(+ `whatsapp-session-unknown-sends-${environment}`,+ {+ ...common,+ name: `${label}: sends with unknown outcome`,+ type: 'log alert',+ query: `logs("@telemetry_event:whatsapp_send_outcome @outcome:unknown ${env}").index("*").rollup("count").last("15m") > 5`,+ message: `{{value}} sends timed out after submission in 15 minutes. They are never resent automatically; the sender sees them as unknown. Check WhatsApp connectivity and drain/takeover timing around deploys.\n${CONTEXT}`,+ monitorThresholds: { critical: '5', warning: '0' },+ onMissingData: 'resolve',+ },+ { dependsOn: goalertWebhooks }+ );- new datadog.Monitor(`whatsapp-session-inbound-lag-${environment}`, {- ...common,- name: `${label}: oldest unprocessed inbound event`,- type: 'log alert',- query: `logs("${HEALTH} ${env}").index("*").rollup("max", "@oldest_unprocessed_inbound_seconds").last("10m") > 300`,- message: `An inbound WhatsApp event has waited {{value}}s for ingestion. Messages are reaching the replica but not the channel; check the ingestion spans and MacroDB latency.\n${CONTEXT}`,- monitorThresholds: { critical: '300', warning: '60' },- onMissingData: 'resolve',- });+ new datadog.Monitor(+ `whatsapp-session-inbound-lag-${environment}`,+ {+ ...common,+ name: `${label}: oldest unprocessed inbound event`,+ type: 'log alert',+ query: `logs("${HEALTH} ${env}").index("*").rollup("max", "@oldest_unprocessed_inbound_seconds").last("10m") > 300`,+ message: `An inbound WhatsApp event has waited {{value}}s for ingestion. Messages are reaching the replica but not the channel; check the ingestion spans and MacroDB latency.\n${CONTEXT}`,+ monitorThresholds: { critical: '300', warning: '60' },+ onMissingData: 'resolve',+ },+ { dependsOn: goalertWebhooks }+ );- new datadog.Monitor(`whatsapp-session-drain-duration-${environment}`, {- ...common,- name: `${label}: drain close to the stop timeout`,- type: 'log alert',- query: `logs("@telemetry_event:whatsapp_session_drain ${env}").index("*").rollup("max", "@drain_duration_ms").last("1h") > 100000`,- message: `A replica took {{value}}ms to drain on SIGTERM. ECS kills the task at 120s, cutting off in-flight sends (they become unknown). Raise DRAIN_BATCH or lower MAX_ACCOUNTS (services/whatsapp_session_service/src/main.rs).\n${CONTEXT}`,- monitorThresholds: { critical: '100000', warning: '80000' },- onMissingData: 'resolve',- });+ new datadog.Monitor(+ `whatsapp-session-drain-duration-${environment}`,+ {+ ...common,+ name: `${label}: drain close to the stop timeout`,+ type: 'log alert',+ query: `logs("@telemetry_event:whatsapp_session_drain ${env}").index("*").rollup("max", "@drain_duration_ms").last("1h") > 100000`,+ message: `A replica took {{value}}ms to drain on SIGTERM. ECS kills the task at 120s, cutting off in-flight sends (they become unknown). Raise DRAIN_BATCH or lower MAX_ACCOUNTS (services/whatsapp_session_service/src/main.rs).\n${CONTEXT}`,+ monitorThresholds: { critical: '100000', warning: '80000' },+ onMissingData: 'resolve',+ },+ { dependsOn: goalertWebhooks }+ );}diff --git a/infra/stacks/monitoring/notify.ts b/infra/stacks/monitoring/notify.tsindex 83234c16a3..670048d99f 100644--- a/infra/stacks/monitoring/notify.ts+++ b/infra/stacks/monitoring/notify.ts@@ -1,29 +1,43 @@import * as pulumi from '@pulumi/pulumi';-// Where every monitor in this stack notifies.+// Where every monitor in this stack notifies: GoAlert, and nothing else.+//+// GoAlert (infra/stacks/observability) has two services. Notify posts each+// alert to the Macro Alerts channel and re-posts it hourly until someone+// acknowledges it; the paging service also phones the on-call engineer.+// Datadog never posts to the channel itself, so an alert shows up once, from+// GoAlert. The one exception is the GoAlert health check in goalert.ts.//-// All monitors post to the Macro Alerts channel. Paging monitors also open a-// GoAlert alert, which phones the on-call engineer, and close it on recovery.// Datadog sends warning and alert notifications again every RENOTIFY_MINUTES-// while a monitor stays in that state, so an alert nobody fixed keeps showing up.+// while a monitor stays in that state. The repeat dedups into the open GoAlert+// alert, and reopens it if someone closed it while the monitor still fires.-// Datadog webhook integration that posts into the Macro Alerts channel. It is-// configured in the Datadog UI, not in this stack.-export const MACRO_ALERTS = '@webhook-macro-alert-hook';// Declared in goalert.ts.export const GOALERT = '@webhook-goalert';export const GOALERT_CLOSE = '@webhook-goalert-close';+export const GOALERT_NOTIFY = '@webhook-goalert-notify';+export const GOALERT_NOTIFY_CLOSE = '@webhook-goalert-notify-close';export const RENOTIFY_MINUTES = 60;-/** The notification handles that end every monitor message. */+/**+ * The notification handles that end every monitor message. A paging monitor's+ * critical alert opens a page; every other warning, alert and no-data+ * notification opens a Notify alert.+ *+ * Datadog allows one state block per state, never nested, so each state gets+ * its own. A recovery closes on both services because a paging monitor may+ * have warned before it alerted, or recovered from no data; closing an alert+ * GoAlert does not have open does nothing.+ */export function notify({ pages }: { pages: boolean }): string {- if (!pages) return MACRO_ALERTS;- // A recovery closes the page whether the monitor recovers from alert or from- // a warning it dropped to first. Closing an alert GoAlert no longer has- // open does nothing.- return `{{#is_alert}}Paging on-call. ${GOALERT}{{/is_alert}}{{#is_recovery}}${GOALERT_CLOSE}{{/is_recovery}}-${MACRO_ALERTS}`;+ const critical = pages+ ? `{{#is_alert}}Paging on-call. ${GOALERT}{{/is_alert}}`+ : `{{#is_alert}}${GOALERT_NOTIFY}{{/is_alert}}`;+ const recovery = pages+ ? `${GOALERT_CLOSE} ${GOALERT_NOTIFY_CLOSE}`+ : GOALERT_NOTIFY_CLOSE;+ return `${critical}{{#is_warning}}${GOALERT_NOTIFY}{{/is_warning}}{{#is_no_data}}${GOALERT_NOTIFY}{{/is_no_data}}{{#is_recovery}}${recovery}{{/is_recovery}}`;}// Datadog's site: dashboard `url`s are paths under it.diff --git a/infra/stacks/monitoring/prod-monitor.ts b/infra/stacks/monitoring/prod-monitor.tsindex ec8fd8c415..54e13aaac0 100644--- a/infra/stacks/monitoring/prod-monitor.ts+++ b/infra/stacks/monitoring/prod-monitor.ts@@ -48,7 +48,8 @@ export function prodMonitor(includeTags: true,notifyAudit: false,},- { dependsOn: args.pages ? goalertWebhooks : [] }+ // The message mentions the GoAlert webhooks, which must exist first.+ { dependsOn: goalertWebhooks });}diff --git a/infra/stacks/monitoring/routing.test.ts b/infra/stacks/monitoring/routing.test.tsindex 3d1810571c..2a234126bc 100644--- a/infra/stacks/monitoring/routing.test.ts+++ b/infra/stacks/monitoring/routing.test.ts@@ -5,8 +5,14 @@ const resources: pulumi.runtime.MockResourceArgs[] = [];beforeAll(async () => {pulumi.runtime.setAllConfig(- { 'monitoring:goalertIntegrationKey': 'integration-key' },- ['monitoring:goalertIntegrationKey']+ {+ 'monitoring:goalertIntegrationKey': 'integration-key',+ 'monitoring:goalertNotifyIntegrationKey': 'notify-integration-key',+ },+ [+ 'monitoring:goalertIntegrationKey',+ 'monitoring:goalertNotifyIntegrationKey',+ ]);pulumi.runtime.setMocks({@@ -40,30 +46,48 @@ const synthetics = () =>(r) => r.type === 'datadog:index/syntheticsTest:SyntheticsTest');-test('every monitor and synthetic test notifies only the Macro Alerts channel and GoAlert', () => {- const routed = [...monitors(), ...synthetics()];+// Notification handles and email addresses; attribute names like `@error` in+// message text are not handles.+const handles = (message: string): string[] =>+ message.match(+ /@(?:webhook|slack|pagerduty|opsgenie|teams)-[\w-]+|[\w.+-]+@[\w-]+\.[\w.]+/g+ ) ?? [];++test('every monitor and synthetic test notifies only GoAlert', () => {+ const routed = [...monitors(), ...synthetics()].filter(+ (r) => r.name !== 'goalert-health'+ );expect(routed.length).toBeGreaterThan(80);for (const resource of routed) {- // Notification handles and email addresses; attribute names like `@error`- // in message text are not handles.- const handles: string[] =- resource.inputs.message.match(- /@(?:webhook|slack|pagerduty|opsgenie|teams)-[\w-]+|[\w.+-]+@[\w-]+\.[\w.]+/g- ) ?? [];- expect({ name: resource.name, handles }).toEqual({+ const found = handles(resource.inputs.message);+ expect({ name: resource.name, handles: found }).toEqual({name: resource.name,- handles: expect.arrayContaining(['@webhook-macro-alert-hook']),+ handles: expect.arrayContaining([+ '@webhook-goalert-notify',+ '@webhook-goalert-notify-close',+ ]),});- for (const handle of handles) {+ for (const handle of found) {expect([- '@webhook-macro-alert-hook','@webhook-goalert','@webhook-goalert-close',+ '@webhook-goalert-notify',+ '@webhook-goalert-notify-close',]).toContain(handle);}}});+test('only the GoAlert health check posts to the channel directly', () => {+ const health = synthetics().find((r) => r.name === 'goalert-health');+ expect(handles(health?.inputs.message)).toEqual([+ '@webhook-macro-alert-hook',+ ]);+ expect(health?.inputs.requestDefinition.url).toBe(+ 'https://oncall.macro-internal.com/health'+ );+});+test('every monitor renotifies while it stays in alert', () => {for (const resource of monitors()) {expect({@@ -81,35 +105,25 @@ test('every monitor renotifies while it stays in alert', () => {}});-test('paging prod monitors open and close a GoAlert alert; the rest only notify', () => {+test('a paging monitor pages on critical and notifies otherwise', () => {const blocked = monitors().find((r) => r.name === 'macrodb-blocked-connections');- expect(blocked?.inputs.message).toContain(- '{{#is_alert}}Paging on-call. @webhook-goalert{{/is_alert}}'- );- expect(blocked?.inputs.message).toContain(- '{{#is_recovery}}@webhook-goalert-close{{/is_recovery}}'+ expect(blocked?.inputs.message).toEndWith(+ '{{#is_alert}}Paging on-call. @webhook-goalert{{/is_alert}}{{#is_warning}}@webhook-goalert-notify{{/is_warning}}{{#is_no_data}}@webhook-goalert-notify{{/is_no_data}}{{#is_recovery}}@webhook-goalert-close @webhook-goalert-notify-close{{/is_recovery}}');expect(blocked?.inputs.priority).toBe('2');+});+test('a non-paging monitor notifies on every state', () => {// Grafana pages for this one, so Datadog only notifies.const freeable = monitors().find((r) => r.name === 'macrodb-primary-freeable-memory');- expect(freeable?.inputs.message).not.toContain('@webhook-goalert');-- const latency = monitors().find((r) => r.name === 'macrodb-read-latency');- expect(latency?.inputs.message).not.toContain('@webhook-goalert');- expect(latency?.inputs.priority).toBe('3');-});--test('the GoAlert health check does not page through GoAlert', () => {- const health = synthetics().find((r) => r.name === 'goalert-health');- expect(health?.inputs.message).not.toContain('@webhook-goalert');- expect(health?.inputs.requestDefinition.url).toBe(- 'https://oncall.macro-internal.com/health'+ expect(freeable?.inputs.message).toEndWith(+ '{{#is_alert}}@webhook-goalert-notify{{/is_alert}}{{#is_warning}}@webhook-goalert-notify{{/is_warning}}{{#is_no_data}}@webhook-goalert-notify{{/is_no_data}}{{#is_recovery}}@webhook-goalert-notify-close{{/is_recovery}}');+ expect(freeable?.inputs.priority).toBe('3');});test('anomaly rules recover at zero and carry their alert window', () => {diff --git a/infra/stacks/observability/AGENTS.md b/infra/stacks/observability/AGENTS.mdindex a0364f759c..adff157400 100644--- a/infra/stacks/observability/AGENTS.md+++ b/infra/stacks/observability/AGENTS.md@@ -45,6 +45,8 @@ From `infra/`: `bun run check`, then Biome on changed `.ts`.- GoAlert's application config lives in its database; `goalert/apply-config.py`overwrites it on every start from `goalert.nix`. `set-config` accepts invalidvalues silently, so check the journal for `validate config`.+- Grafana keeps a provisioned contact point removed from `alerting.nix`; list+ its uid in `deleteContactPoints` to delete it.- Grafana provisioning expands `$VAR` in contact point settings, but not in rulequeries or annotations, so `$B` and `$values` work there and secrets must comein through the Grafana unit's environment.diff --git a/infra/stacks/observability/Pulumi.prod.yaml b/infra/stacks/observability/Pulumi.prod.yamlindex 5de60a5f49..47aa9b3e11 100644--- a/infra/stacks/observability/Pulumi.prod.yaml+++ b/infra/stacks/observability/Pulumi.prod.yaml@@ -2,5 +2,5 @@ config:aws:region: us-east-2observability:ciIngestSecretArn: arn:aws:secretsmanager:us-east-1:569036502058:secret:observability/dev-ingest-7UhJbLobservability:alarmTopicArn: arn:aws:sns:us-east-2:569036502058:observability-prod-alerts- observability:amiId: ami-0af944c17d63ec7c4+ observability:amiId: ami-02bbbfa6d82d3cff2observability:secretArn: arn:aws:secretsmanager:us-east-2:569036502058:secret:observability/prod-TwQpBJdiff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.mdindex ff13d8f15d..844ccd544b 100644--- a/infra/stacks/observability/README.md+++ b/infra/stacks/observability/README.md@@ -445,19 +445,24 @@ the `Prod` group, reading CloudWatch in Virginia directly, and`nixos/prod-alerts-dashboard.nix` draws it as one panel with both thresholds onthe `Prod alerts` dashboard (`/d/prod-alerts`, rows macrodb, Redis and Queues).Each rule carries its panel as `__dashboardUid__`/`__panelId__`, so the rule-view and every notification link to the graph of the metric that fired. Each+view links to the graph of the metric that fired, and as a `panel_url`+annotation built from Grafana's `externalURL`, because GoAlert's alert details+list annotations but not Grafana's panel link. Eachhas a Datadog copy in `infra/stacks/monitoring/monitors/`; keep thresholds andwindows in sync by hand. Forecasts, anomalies, Database Monitoring, APM and logrules are Datadog-only.-Every alert posts to the `Macro Alerts` contact point through the loopback-relay; critical ones renotify hourly, others every four hours. Grafana's-`default.message` lists each alert's annotations (summary, description,-runbook) and a `Panel:` link to its graph, opened an hour before the alert-started. Critical alerts of rules with `pages = true` also go to the `GoAlert`-contact point over loopback. GoAlert owns the alert from there: it opens,-acknowledges and closes it, and its escalation posts each change to the-channel. Their Datadog copies only notify, so an incident pages once.+Every prod alert goes to GoAlert over loopback, and GoAlert alone posts to the+Macro Alerts channel. Critical alerts of rules with `pages = true` go to the+`GoAlert` contact point (the paging `Grafana critical` service); every other prod+alert, warnings and non-paging criticals, goes to `GoAlert notify` (the `Notify`+service), which is also the default receiver. Grafana repeats critical alerts+hourly and others every four hours; GoAlert dedups them by fingerprint, opens,+acknowledges and closes the alert, and its escalation posts each change to the+channel (see [Escalation](#escalation)). The details GoAlert posts carry each+alert's labels and annotations (summary, description, runbook, panel) and its+Grafana source link. Paging rules' Datadog copies only notify, so an incident+pages once.The dev rules below keep their own route to `Macro team email`.@@ -486,12 +491,13 @@ AMI; a schema-5 image cannot boot schema-6 user data.## On-call paging (GoAlert)-GoAlert is the source of truth for open alerts: Grafana sends it critical-alerts, and on-call engineers acknowledge or close them in its UI at+GoAlert is the source of truth for open alerts and the only sender to the Macro+Alerts channel: Grafana sends it every prod alert, and on-call engineers acknowledge or close them in its UI at`https://oncall.macro-internal.com`, signing in with Google. It runs on this hostso paging keeps working while production, its database or Macro itself is down.Today it pages by posting to the Macro Alerts channel; phone paging (Twilio) and-a rotation are follow-ups.+a rotation are follow-ups. Coding agents reach it through the `goalert` MCP+server; see [MCP servers](../../../docs/development/mcp-servers.md#goalert).- `nixos/goalert/package.nix` builds GoAlert from source with `buildGoModule`,embedding its web UI. Bump `version` and the hashes together.@@ -506,9 +512,12 @@ a rotation are follow-ups.- A nightly timer dumps the database to `s3://<logs bucket>/goalert/`, one objectper weekday. The dump goes to a file first, so a failed dump never replaces agood backup.-- `macro-alerts-relay` on `127.0.0.1:8090` turns GoAlert (`/goalert`) and Grafana- (`/grafana`) webhooks into Macro channel posts and adds the channel's bot token.- GoAlert webhooks cannot set headers, and the token stays in one service.+- `macro-alerts-relay` on `127.0.0.1:8090` turns GoAlert webhooks (`/goalert`)+ into Macro channel posts and adds the channel's bot token. GoAlert webhooks+ cannot set headers, and the token stays in one service. A new alert posts as+ `🚨 PAGE [<service>]` (paging services) or `🔔 [Notify]`, its summary, the first+ 1500 characters of its details and an `Acknowledge:` link; status changes post+ one line with the GoAlert link.- nginx serves `oncall.macro-internal.com` from a runtime server block. Only`/health`, `/api/v2/generic/incoming` and `/api/v2/heartbeat/` are reachablewithout a session (GoAlert checks the integration key); other inbound@@ -517,6 +526,9 @@ a rotation are follow-ups.`https://oncall.macro-internal.com/api/v2/identity/providers/oidc/callback`.GoAlert cannot restrict OIDC to a domain, so that client must keep theInternal audience. The basic-auth `admin` login is the break-glass account.+- Every `@macro.com` account is a GoAlert admin from its first sign-in:+ `goalert/workspace-admins.sql` installs a trigger on GoAlert's `users` table+ on every start. Other accounts stay plain users.- The ALB sends the on-call host to its own target group, health-checked at`/goalert-healthz`. Alarm `observability-goalert` publishes to `alarmTopicArn`when it fails, so a GoAlert outage is reported without GoAlert.@@ -528,29 +540,33 @@ us-east-2 Secrets Manager secret the host reads, under the lowercase names:`goalert_encryption_key` (≥32 random characters), `goalert_admin_password`(≥32 random characters, the `admin` login), `macro_alerts_webhook_url` (HTTPS)and `macro_alerts_bot_token` (the Alerts channel's webhook and bot token).-`goalert_grafana_integration_key` is optional until GoAlert has created it;-without it Grafana's pages are rejected with 401 and the GoAlert contact point-shows as failing.+`goalert_grafana_integration_key` (the `Grafana critical` Grafana key) and+`goalert_grafana_notify_integration_key` (the `Notify` Grafana key) are optional+until GoAlert has created them; without one, GoAlert rejects that contact+point's alerts with 401 and Grafana shows it as failing.### First deployment1. Add the secrets above, then build, publish and deploy the AMI as usual. Thecertificate gains the on-call name, so Pulumi replaces the ACM certificate andupdates the listener.-2. Sign in as `admin`. Create the escalation policy and service below, and a- Grafana integration key on `Grafana critical`. Put the key in Doppler and the- secret as `goalert_grafana_integration_key` and restart- `observability-secrets`.+2. Sign in as `admin`. Create the escalation policies and services below, and a+ Grafana integration key on `Grafana critical` and on `Notify`. Put the keys in+ Doppler and the secret as `goalert_grafana_integration_key` and+ `goalert_grafana_notify_integration_key` and restart `observability-secrets`.3. Send a test alert through the service and check that it reaches the channeland can be acknowledged and closed.### Escalation-- Escalation policy `Macro prod`, one step: webhook- `http://127.0.0.1:8090/goalert`. The channel sees each alert, acknowledgement- and close.-- Service `Grafana critical` on `Macro prod`. Warnings never page; they only- reach the channel.+Both policies have one step, webhook `http://127.0.0.1:8090/goalert`, and+repeat it 5 times every 60 minutes, so the channel sees each open alert hourly+until it is acknowledged, plus every acknowledgement and close.++- Escalation policy `Macro prod` pages: services `Grafana critical` (Grafana's+ paging criticals) and `Datadog critical`.+- Escalation policy `Macro notify` only posts to the channel: service `Notify`+ (Grafana warnings and non-paging criticals).- Acknowledge an alert to stop its escalation, close it when handled; Grafanacloses it on recovery. To silence a whole service for a while, use itsmaintenance mode.diff --git a/infra/stacks/observability/assets/refresh-secrets.py b/infra/stacks/observability/assets/refresh-secrets.pyindex 2e4b7e2951..adbe943623 100644--- a/infra/stacks/observability/assets/refresh-secrets.py+++ b/infra/stacks/observability/assets/refresh-secrets.py@@ -30,15 +30,23 @@ for key in ['grafana_secret_key', 'otlp_token', 'goalert_encryption_key', 'goaleraise SystemExit(f'Secret too short: {key}')if not values['macro_alerts_webhook_url'].startswith('https://'):raise SystemExit('macro_alerts_webhook_url must use HTTPS')-# GoAlert creates its Grafana integration key, so it can only be added after the-# first boot. Until then GoAlert rejects Grafana's pages with 401, and Grafana-# reports the GoAlert contact point as failing.-grafana_key = values.get('goalert_grafana_integration_key', '')-if not re.fullmatch(r'[0-9a-f-]{36}|', grafana_key):- raise SystemExit('goalert_grafana_integration_key must be a GoAlert key UUID')-if not grafana_key:- print('goalert_grafana_integration_key is not set; Grafana cannot page through GoAlert')-values['grafana-goalert.env'] = f'GOALERT_GRAFANA_KEY={grafana_key}\n'+# GoAlert creates its Grafana integration keys, so they can only be added after+# the first boot. Until then GoAlert rejects Grafana's alerts with 401, and+# Grafana reports the GoAlert contact points as failing. The paging key belongs+# to `Grafana critical`, the notify key to `Notify`.+grafana_keys = {+ 'GOALERT_GRAFANA_KEY': ('goalert_grafana_integration_key', 'Grafana cannot page through GoAlert'),+ 'GOALERT_GRAFANA_NOTIFY_KEY': ('goalert_grafana_notify_integration_key', 'Grafana alerts cannot reach the channel'),+}+grafana_env = ''+for variable, (key, consequence) in grafana_keys.items():+ value = values.get(key, '')+ if not re.fullmatch(r'[0-9a-f-]{36}|', value):+ raise SystemExit(f'{key} must be a GoAlert key UUID')+ if not value:+ print(f'{key} is not set; {consequence}')+ grafana_env += f'{variable}={value}\n'+values['grafana-goalert.env'] = grafana_env# Native remote-write uses HTTP Basic over TLS. Only the password hash reaches nginx.hashed = subprocess.run(['openssl', 'passwd', '-6', '-stdin'], input=values['otlp_token'] + '\n',diff --git a/infra/stacks/observability/nixos/alerting.nix b/infra/stacks/observability/nixos/alerting.nixindex 4babf369ae..757015af27 100644--- a/infra/stacks/observability/nixos/alerting.nix+++ b/infra/stacks/observability/nixos/alerting.nix@@ -92,11 +92,14 @@ letpages = lib.boolToString ((alert.pages or false) && severity == "critical");};# Grafana turns the dashboard and panel annotations into the panel link- # in notifications and the rule view.+ # in the rule view. GoAlert's details list annotations but not Grafana's+ # panel link, so `panel_url` repeats it. `externalURL` is the runtime+ # root_url; trim its trailing slash.annotations = {summary = alert.title;inherit (alert) description;runbook_url = "https://github.com/macro-inc/macro/blob/main/docs/runbooks/${alert.runbook}";+ panel_url = ''{{ reReplaceAll "/$" "" externalURL }}/d/${prodAlerts.dashboardUid}?viewPanel=${toString alert.panelId}'';__dashboardUid__ = prodAlerts.dashboardUid;__panelId__ = toString alert.panelId;};@@ -197,6 +200,14 @@ inservices.grafana.provision.alerting = {contactPoints.settings = {apiVersion = 1;+ # Grafana keeps removed provisioned contact points until told to delete+ # them. GoAlert now posts to the channel instead.+ deleteContactPoints = [+ {+ orgId = 1;+ uid = "macro-alerts";+ }+ ];contactPoints = [{orgId = 1;@@ -215,34 +226,32 @@ in}{orgId = 1;- name = "Macro Alerts";+ name = "GoAlert";receivers = [{- uid = "macro-alerts";+ uid = "goalert";type = "webhook";disableResolveMessage = false;- # The loopback relay adds the channel token; see goalert.nix.+ # Loopback, so paging does not depend on the ALB. GoAlert closes the+ # alert when Grafana sends the resolved notification.settings = {- url = "http://127.0.0.1:8090/grafana";+ url = "http://127.0.0.1:3030/api/v2/grafana/incoming?token=$GOALERT_GRAFANA_KEY";httpMethod = "POST";- title = ''{{ template "default.title" . }}'';- message = ''{{ template "default.message" . }}'';};}];}{orgId = 1;- name = "GoAlert";+ name = "GoAlert notify";receivers = [{- uid = "goalert";+ uid = "goalert-notify";type = "webhook";disableResolveMessage = false;- # Loopback, so paging does not depend on the ALB. GoAlert closes the- # alert when Grafana sends the resolved notification.+ # GoAlert's `Notify` service, whose policy posts to the channel.settings = {- url = "http://127.0.0.1:3030/api/v2/grafana/incoming?token=$GOALERT_GRAFANA_KEY";+ url = "http://127.0.0.1:3030/api/v2/grafana/incoming?token=$GOALERT_GRAFANA_NOTIFY_KEY";httpMethod = "POST";};}@@ -250,12 +259,16 @@ in}];};+ # Every prod alert goes to GoAlert, which posts it to the Macro Alerts+ # channel and re-posts until acknowledged. Paging criticals go to the+ # paging service, everything else to `Notify`. GoAlert dedups by alert+ # fingerprint, so repeats only refresh the open alert.policies.settings = {apiVersion = 1;policies = [{orgId = 1;- receiver = "Macro Alerts";+ receiver = "GoAlert notify";group_by = ["grafana_folder""alertname"@@ -276,8 +289,6 @@ in]];}- # Paging prod rules; `continue` lets the next routes still post them- # to Macro Alerts.{receiver = "GoAlert";object_matchers = [@@ -292,10 +303,10 @@ in"critical"]];- continue = true;+ repeat_interval = "1h";}{- receiver = "Macro Alerts";+ receiver = "GoAlert notify";object_matchers = [["severity"@@ -305,7 +316,6 @@ in];repeat_interval = "1h";}- { receiver = "Macro Alerts"; }];}];diff --git a/infra/stacks/observability/nixos/goalert.nix b/infra/stacks/observability/nixos/goalert.nixindex fea82f3a14..1c6cea2bcd 100644--- a/infra/stacks/observability/nixos/goalert.nix+++ b/infra/stacks/observability/nixos/goalert.nix@@ -108,7 +108,7 @@ in"google_client_secret"];ExecStartPre = [- "${pkgs.python3}/bin/python3 ${./goalert/apply-config.py} ${lib.getExe goalert} ${pkgs.writeText "goalert-config.json" (builtins.toJSON goalertConfig)} ${config.services.postgresql.package}/bin/psql"+ "${pkgs.python3}/bin/python3 ${./goalert/apply-config.py} ${lib.getExe goalert} ${pkgs.writeText "goalert-config.json" (builtins.toJSON goalertConfig)} ${config.services.postgresql.package}/bin/psql ${./goalert/workspace-admins.sql}"];ExecStart = start;MemoryMax = "768M";@@ -118,7 +118,7 @@ in};macro-alerts-relay = needsSecrets // {- description = "Forward GoAlert and Grafana notifications to Macro Alerts";+ description = "Forward GoAlert notifications to Macro Alerts";wantedBy = [ "multi-user.target" ];serviceConfig = {DynamicUser = true;diff --git a/infra/stacks/observability/nixos/goalert/apply-config.py b/infra/stacks/observability/nixos/goalert/apply-config.pyindex 2180aa71e5..531f3ca065 100644--- a/infra/stacks/observability/nixos/goalert/apply-config.py+++ b/infra/stacks/observability/nixos/goalert/apply-config.py@@ -1,5 +1,6 @@-"""Migrate GoAlert's database, replace its config with the Nix-declared one, and-create the `admin` login if it does not exist.+"""Migrate GoAlert's database, replace its config with the Nix-declared one,+make Macro Workspace accounts admins, and create the `admin` login if it does+not exist.GoAlert keeps application config only in Postgres. Running this before everystart makes the Nix attrset the source of truth: edits in the admin UI last@@ -12,7 +13,7 @@ from pathlib import Pathimport subprocessimport sys-goalert, template_path, psql = sys.argv[1], sys.argv[2], sys.argv[3]+goalert, template_path, psql, workspace_admins_sql = sys.argv[1:5]credentials = Path(os.environ['CREDENTIALS_DIRECTORY'])@@ -35,6 +36,10 @@ subprocess.run([goalert, 'migrate'], env=environment, check=True)subprocess.run([goalert, 'set-config'], env=environment, input=json.dumps(config), text=True, check=True,)+subprocess.run(+ [psql, '-v', 'ON_ERROR_STOP=1', '-q', '-f', workspace_admins_sql, environment['GOALERT_DB_URL']],+ env=environment, check=True,+)admin_exists = subprocess.run([psql, '-tAc', "select 1 from auth_basic_users where username = 'admin'", environment['GOALERT_DB_URL']],env=environment, capture_output=True, text=True, check=True,diff --git a/infra/stacks/observability/nixos/goalert/macro-alerts-relay.py b/infra/stacks/observability/nixos/goalert/macro-alerts-relay.pyindex 6e203a9b86..d806225f9d 100644--- a/infra/stacks/observability/nixos/goalert/macro-alerts-relay.py+++ b/infra/stacks/observability/nixos/goalert/macro-alerts-relay.py@@ -1,8 +1,10 @@-"""Forward GoAlert and Grafana webhooks to the Macro Alerts channel.+"""Forward GoAlert webhooks to the Macro Alerts channel.-Macro's channel webhook takes `{"content": ...}` and a bot-token header. GoAlert-cannot set headers, so both senders post here over loopback and the token stays-in this one service. Each request is forwarded once; the sender owns retries.+GoAlert is the only sender: Grafana and Datadog alert GoAlert, whose escalation+policies post each alert, acknowledgement and close here. Macro's channel webhook+takes `{"content": ...}` and a bot-token header; GoAlert cannot set headers, so it+posts here over loopback and the token stays in this one service. Each request is+forwarded once; GoAlert owns retries."""import jsonimport logging@@ -19,17 +21,32 @@ BOT_TOKEN = (CREDENTIALS / 'macro_alerts_bot_token').read_text()FORWARD_TIMEOUT_SECONDS = 2.5MAX_BODY_BYTES = 256 * 1024MAX_CONTENT_CHARS = 4000+MAX_DETAILS_CHARS = 1500+++def service_prefix(service: str) -> str:+ """`Notify` only posts to the channel; every other service pages."""+ marker = '🔔' if 'Notify' in service else '🚨 PAGE'+ return f"{marker} [{service}]" if service else markerdef goalert_content(payload: dict) -> str | None:kind = payload.get('Type')link = payload.get('GoAlertURL', '')+ service = payload.get('ServiceName', '')if kind == 'Alert':- return f"🚨 Paging on-call: {payload.get('Summary', '')}\n{link}"+ details = payload.get('Details', '').strip()+ if len(details) > MAX_DETAILS_CHARS:+ details = details[:MAX_DETAILS_CHARS].rstrip() + '\n…'+ lines = [f"{service_prefix(service)} {payload.get('Summary', '')}"]+ if details:+ lines.append(details)+ lines.append(f"Acknowledge: {link}")+ return '\n\n'.join(lines)if kind == 'AlertStatus':return f"{payload.get('LogEntry', '')}: {payload.get('Summary', '')}\n{link}"if kind == 'AlertBundle':- return f"🚨 Paging on-call: {payload.get('Count')} alerts for {payload.get('ServiceName', '')}\n{link}"+ return f"{service_prefix(service)} {payload.get('Count')} alerts\nAcknowledge: {link}"if kind == 'AlertStatusBundle':return f"{payload.get('LogEntry', '')} ({payload.get('Count')} alerts)\n{link}"if kind == 'ScheduleOnCallUsers':@@ -40,30 +57,21 @@ def goalert_content(payload: dict) -> str | None:return None-def grafana_content(payload: dict) -> str | None:- title = payload.get('title')- message = payload.get('message')- if not isinstance(title, str) or not isinstance(message, str):- return None- return f"{title}\n{message}"---SOURCES = {'/goalert': goalert_content, '/grafana': grafana_content}--class Relay(BaseHTTPRequestHandler):def do_POST(self) -> None:- format_content = SOURCES.get(self.path)length = int(self.headers.get('Content-Length', '0'))- if format_content is None or length <= 0 or length > MAX_BODY_BYTES:- self.send_error(404 if format_content is None else 400)+ if self.path != '/goalert':+ self.send_error(404)+ return+ if length <= 0 or length > MAX_BODY_BYTES:+ self.send_error(400)returntry:payload = json.loads(self.rfile.read(length))except json.JSONDecodeError:self.send_error(400)return- content = format_content(payload) if isinstance(payload, dict) else None+ content = goalert_content(payload) if isinstance(payload, dict) else Noneif content is None:logging.warning('unrecognised %s payload', self.path)self.send_error(422)diff --git a/infra/stacks/observability/nixos/goalert/workspace-admins.sql b/infra/stacks/observability/nixos/goalert/workspace-admins.sqlnew file mode 100644index 0000000000..42e38e63ad--- /dev/null+++ b/infra/stacks/observability/nixos/goalert/workspace-admins.sql@@ -0,0 +1,17 @@+-- Every Macro Workspace account is a GoAlert admin, from its first sign-in.+-- GoAlert has no default-role setting. Its OIDC login cannot restrict the+-- email domain, so only @macro.com accounts are promoted.+CREATE OR REPLACE FUNCTION macro_workspace_admin() RETURNS trigger AS $$+BEGIN+ IF NEW.email ILIKE '%@macro.com' THEN+ NEW.role := 'admin';+ END IF;+ RETURN NEW;+END+$$ LANGUAGE plpgsql;++CREATE OR REPLACE TRIGGER macro_workspace_admin+ BEFORE INSERT OR UPDATE OF email, role ON users+ FOR EACH ROW EXECUTE FUNCTION macro_workspace_admin();++UPDATE users SET role = 'admin' WHERE email ILIKE '%@macro.com' AND role <> 'admin';diff --git a/infra/stacks/observability/nixos/services.nix b/infra/stacks/observability/nixos/services.nixindex c10cf01397..24008bc5f2 100644--- a/infra/stacks/observability/nixos/services.nix+++ b/infra/stacks/observability/nixos/services.nix@@ -68,7 +68,7 @@ inserviceConfig = {EnvironmentFile = ["/opt/observability/runtime.env"- # Interpolated into the GoAlert contact point by alerting provisioning.+ # Interpolated into the GoAlert contact points by alerting provisioning."/run/macro-observability/grafana-goalert.env"];LoadCredential = map credential [diff --git a/nix/cloud-storage.nix b/nix/cloud-storage.nixindex e28e1d68fd..1b8e2f3b0b 100644--- a/nix/cloud-storage.nix+++ b/nix/cloud-storage.nix@@ -868,10 +868,46 @@'';};+ # GoAlert MCP server (github.com/amitray007/goalert-mcp, MIT), built from+ # the reviewed v0.1.0 commit rather than the npm tarball. Bump `rev`,+ # `hash` and `npmDepsHash` together, after re-reviewing the source.+ goalert-mcp = pkgs.buildNpmPackage {+ pname = "goalert-mcp";+ version = "0.1.0";+ src = pkgs.fetchFromGitHub {+ owner = "amitray007";+ repo = "goalert-mcp";+ rev = "c8c285f35ce31426c576a69c25f5bb1a365a7565";+ hash = "sha256-TDBI06V6boZ3RihwirGchDcKkZ7nKCyyB1pEebqEMiU=";+ };+ npmDepsHash = "sha256-PhanjTUmHoXzXsu9H3wKWVrZ89tfk8T4/QYdFcMGZ+M=";+ nodejs = pkgs.nodejs_24;+ # Only dev dependencies (esbuild, fsevents) ship install scripts.+ npmFlags = [ "--ignore-scripts" ];+ meta.mainProgram = "goalert-mcp";+ };++ # Starts goalert-mcp against Macro's GoAlert with the admin login read+ # from Doppler at launch, read-only unless GOALERT_READ_ONLY=false.+ goalert-mcp-macro = pkgs.writeShellApplication {+ name = "goalert-mcp-macro";+ runtimeInputs = [ pkgs.doppler ];+ text = ''+ GOALERT_PASSWORD="$(doppler secrets get GOALERT_ADMIN_PASSWORD --project observability --config prd --plain)"+ export GOALERT_PASSWORD+ export GOALERT_BASE_URL=https://oncall.macro-internal.com+ export GOALERT_USERNAME=admin+ export GOALERT_READ_ONLY="''${GOALERT_READ_ONLY:-true}"+ exec ${pkgs.lib.getExe goalert-mcp} "$@"+ '';+ };+shellTools =with pkgs;[daytona+ goalert-mcp+ goalert-mcp-macronscparalleldocker-compose@@ -1037,6 +1073,8 @@openApiBinsdopplerConfigBinsnextestArchive+ goalert-mcp+ goalert-mcp-macro;default = cargoArtifacts;}diff --git a/opencode.json b/opencode.jsonindex 51c076953b..9339e0a757 100644--- a/opencode.json+++ b/opencode.json@@ -14,6 +14,12 @@"http://127.0.0.1:9222"]},+ "goalert": {+ "type": "local",+ "command": [+ "goalert-mcp-macro"+ ]+ },"grafana": {"type": "local","command": [