Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
feat(observability): page on-call through GoAlert from Grafana and add prod database alerts (#9067)
Wolf Mermelstein0a19515960a6
diff --git a/docs/README.md b/docs/README.mdindex 309905a332..1b3a6e5151 100644--- a/docs/README.md+++ b/docs/README.md@@ -6,7 +6,7 @@| [`guides/`](guides/) | End-to-end walkthroughs: adding a service, entity, block, and more. || [`architecture/`](architecture/) | How shipped systems work today. || [`design/`](design/) | Proposals, plans, and audits; check each doc's status. |-| [`runbooks/`](runbooks/) | Rollouts, migrations, imports, and demos with operational steps. |+| [`runbooks/`](runbooks/) | Rollouts, migrations, imports, demos, and the response steps that prod alerts link to. || [`agent-guide/`](agent-guide/) | How browser agents operate the app, not how to develop it. |Docs that belong to one app, service, or crate live next to its code.diff --git a/docs/runbooks/macrodb.md b/docs/runbooks/macrodb.mdnew file mode 100644index 0000000000..1ed4428e90--- /dev/null+++ b/docs/runbooks/macrodb.md@@ -0,0 +1,94 @@+# MacroDB alerts++Runbook for the `macro-db-prod` and `macro-db-prod-read-replica` monitors in+[`monitors/macrodb.ts`](../../infra/stacks/monitoring/monitors/macrodb.ts). Both instances are+`db.m9g.2xlarge`: 8 vCPU, about 30 GB of memory, `max_connections` 3,439.++Start in three places:++1. **Performance Insights** for the instance (RDS console → the instance →+ Monitoring → Database load). Group by waits, then by SQL, to see what the+ active sessions are doing.+2. **Datadog Database Monitoring** for `macro-db-prod` (Query Metrics, Query+ Samples, and the blocking queries view).+3. **Recent deploys and queue backlogs.** Most overloads have started with a+ deploy, a backfill, or a queue draining all at once. Check the ECS service+ events and the SQS monitors in the same window.++## Memory++Freeable memory and swap alerts. Every Postgres backend keeps several MB of+private memory, about 14 MB per connection under prod load, so memory falls as+connections rise. The OS page cache shrinks first; once it is gone the+instance swaps and every query slows.++- Check `aws.rds.database_connections` against the same window. If connections+ climbed, find which services added them (see [Connections](#connections)).+- Check swap (`aws.rds.swap_usage`). Sustained swap above a few GB means the+ instance is out of memory, not just low on cache.+- Levers: lower the pool size of the service that grew; pause backfills; as a+ last resort fail over or resize the instance (a resize restarts it).++## Load++Database load (average active sessions) relative to vCPUs, CPU, and disk read+latency. Load above the vCPU count means sessions are queueing for CPU, locks+or IO.++- In Performance Insights, the top wait event says which: `CPU`, `Lock:*` /+ `LWLock:*`, or `IO:*`.+- If lock waits dominate, go to [Locks](#locks).+- If one query dominates CPU, find its caller from the query sample's+ application name and the APM span with the same statement.+- Levers: pause the backfill or worker driving the load; roll back the deploy+ that introduced the query; add the missing index in a follow-up.++## Connections++Connection count on the primary or replica. Each ECS task opens its own pool,+so connections scale with task count times pool size.++- Group `postgresql.connections` (Database Monitoring) or Performance Insights+ sessions by application or user to see which service holds them.+- Check whether a service scaled out (`aws.ecs.service.running`) or a pool+ setting changed in a recent deploy.+- Levers: lower that service's pool maximum or task count. RDS Proxy is not a+ quick fix: moving a service onto it has caused pool timeouts before.++## Replica lag++The read replica is behind the primary. GraphQL Soup reads go to the replica,+so users see stale lists and items they just created go missing.++- Check primary write volume and WAL (`aws.rds.transaction_logs_generation`)+ and replica CPU and IO. Lag usually follows a write burst on the primary or a+ long query on the replica holding back replay.+- Look for long-running queries on the replica in Database Monitoring.+- Levers: stop the write burst (often a backfill); cancel the long replica+ query.++## Storage++Free storage space, now or forecast to run low within a week.++- Compare `aws.rds.free_storage_space` with+ `aws.rds.transaction_logs_disk_usage` and+ `aws.rds.replication_slot_disk_usage`: WAL held by a replication slot+ grows without bound.+- Otherwise find the tables that grew (`pg_total_relation_size` on the largest+ relations) and what started writing to them.+- Levers: drop an abandoned replication slot; stop the writer; increase+ allocated storage (online, but only once every six hours).++## Locks++Connections blocked on locks, or a transaction open for a long time.++- In Database Monitoring's blocking queries view, find the blocking session and+ its query. Advisory locks (`pg_advisory_xact_lock`) and tuple locks from+ concurrent updates of the same rows have caused past overloads.+- A long open transaction holds its locks and stops vacuum. Find it in+ `pg_stat_activity` (`xact_start`, `state = 'idle in transaction'`).+- Levers: cancel (`pg_cancel_backend`) or terminate (`pg_terminate_backend`)+ the blocking session once you know what it is; stop the job that keeps+ starting new ones.diff --git a/docs/runbooks/queues.md b/docs/runbooks/queues.mdnew file mode 100644index 0000000000..d0a4511a44--- /dev/null+++ b/docs/runbooks/queues.md@@ -0,0 +1,34 @@+# Queue alerts++Runbook for the SQS monitors in+[`monitors/queues.ts`](../../infra/stacks/monitoring/monitors/queues.ts).++## Dead letters++Messages arrived in a prod dead-letter queue. Most DLQs hold old messages+permanently, so the alert counts new arrivals, not depth.++- Read a few messages from the DLQ (`aws sqs receive-message` with a short+ visibility timeout, so they return to the queue) and find the consumer's+ error logs for the same ids around the time they moved.+- Check the consumer's latest deploy: a new message shape or a bug failing+ every message moves them all to the DLQ after the retry limit.+- Levers: fix and deploy the consumer, then redrive the DLQ to the source queue+ (SQS console → Start DLQ redrive). Do not redrive before the fix; the+ messages come straight back.++## Backlog++The oldest message in a work queue is old, or the email backfill backlog grew+abnormally.++- Check that the consumer is running (`aws.ecs.service.running`) and not+ crashing (task stop reasons, error logs).+- If it is running, check what it waits on. The email workers write to MacroDB+ and call Gmail per mailbox; a slow database or Gmail rate limits stall them.+ See the [MacroDB runbook](macrodb.md).+- A sudden rise in enqueues (`aws.sqs.number_of_messages_sent`) on the backfill+ queue means many mailboxes started backfilling at once, which also loads+ MacroDB.+- Levers: scale the consumer if the database has headroom; otherwise throttle+ the producer or pause the backfill until the database recovers.diff --git a/docs/runbooks/redis.md b/docs/runbooks/redis.mdnew file mode 100644index 0000000000..191f24e719--- /dev/null+++ b/docs/runbooks/redis.md@@ -0,0 +1,33 @@+# Redis alerts++Runbook for the ElastiCache monitors in+[`monitors/redis.ts`](../../infra/stacks/monitoring/monitors/redis.ts): memory used,+memory forecast, and evictions, per node of every prod cache.++## Memory++The prod caches run `volatile-lru`, which evicts only keys with a TTL. When+most keys have no TTL, a full cache rejects writes (`OOM command not allowed`)+instead of evicting, and every feature that writes to it fails.++Start in three places:++1. **Key mix.** Sample keys by prefix on a replica (`redis-cli --scan+ --pattern '<prefix>*' | head`, `MEMORY USAGE <key>`) and count keys without+ a TTL (`TTL <key>` returns `-1`). A single prefix growing without a TTL is+ the usual cause.+2. **Growth over time.** Graph `aws.elasticache.database_memory_usage_percentage`+ and `aws.elasticache.curr_items` over the last month. Steady linear growth+ points at a key family that is written and never deleted.+3. **Evictions.** `aws.elasticache.evictions` above zero means the cache is+ already full and dropping keys that do have a TTL.++Levers:++- Delete or expire the runaway key family (in batches, with `SCAN` and+ `UNLINK`, never `KEYS`).+- Resize the node type. A resize can move the primary to another node, and+ services that connect to a node endpoint rather than the primary endpoint+ then get `READONLY` errors until it fails back.+- Fix the writer so the keys get a TTL or are deleted when no longer+ referenced.diff --git a/docs/runbooks/regressions.md b/docs/runbooks/regressions.mdnew file mode 100644index 0000000000..838ba3ac87--- /dev/null+++ b/docs/runbooks/regressions.md@@ -0,0 +1,31 @@+# Error and latency regression alerts++Runbook for the per-service anomaly monitors in+[`monitors/regressions.ts`](../../infra/stacks/monitoring/monitors/regressions.ts). Each compares a+service's errors or latency with its own usual band for that time of day, so it+fires on a change, not on a fixed count.++## Errors++A service's HTTP server errors, failed internal or consumer spans, or error+logs are above their usual band.++- Find what changed. Compare the start of the rise with the service's deploys+ (ECS service events, the web app's release) and with flag changes.+- Group the errors by `resource_name` (APM) or by message pattern (Logs →+ Patterns) for the alert window. A new error appearing from zero is usually+ the regression; a familiar one growing usually follows load or a dependency.+- Check Error Tracking for issues first seen in the window.+- Levers: roll back the deploy or turn off the flag; if a dependency is+ failing, follow that dependency's runbook.++## Latency++A service's p95 HTTP latency is above its usual band.++- In APM, sort the service's resources by p95 change to find the slow+ endpoint, then open a slow trace and see which span holds the time.+- Database time points at the [MacroDB runbook](macrodb.md); time waiting for a+ pool connection points at pool exhaustion in that service.+- Levers: roll back the deploy that introduced the slow path; reduce load from+ the caller driving it.diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.mdindex e83f993df0..39999c4aa3 100644--- a/infra/stacks/monitoring/AGENTS.md+++ b/infra/stacks/monitoring/AGENTS.md@@ -4,19 +4,33 @@Org-global Datadog monitors, synthetics, and rollout dashboards. This stack ismanual and has only a prod stack, even for monitors covering both environments.+Prod threshold and anomaly monitors are in `monitors/{macrodb,redis,queues,regressions}.ts`+(`prodMonitor()` in `prod-monitor.ts`); every monitor notifies through `notify.ts`.## ChecksFrom `infra/`:-- `bun test stacks/monitoring/monitors/notification-retention.test.ts`+- `bun test --isolate stacks/monitoring` (routing and notification retention+ tests). `--isolate` matters: the Pulumi mock tests+ each import modules that register resources, and a shared module cache+ hides them from the second file.- `bun run check`- Biome on changed files.-- An authenticated `pulumi preview` must leave adopted monitors unchanged.+- An authenticated `pulumi preview` may change adopted monitors only in+ `message` and `renotifyInterval`; README.md lists the expected counts.## Gotchas+- The CloudWatch-based monitors in `macrodb.ts`, `redis.ts` and `queues.ts`+ have hand-written Grafana copies in+ `infra/stacks/observability/nixos/alerting.nix`. Change thresholds and windows+ in both. Grafana pages for those, so their Datadog copies use `pages: false`.- Read `README.md` before deployment: imported resources must be adopted firstto avoid duplicate alerts. Never deploy without explicit approval.+- Deploying needs the `goalertIntegrationKey` secret in stack config, and+ GoAlert (infra/stacks/observability) reachable at oncall.macro-internal.com.+- Never mention a Slack handle or email address in a monitor message; end it+ with `notify({ pages })`. `routing.test.ts` enforces it.- Notification retention adds nine monitors and one dashboard. Deploy DSS healthinstrumentation first; missing telemetry is deliberately an alert, not zero lag.- Do not use native committed-offset lag for the independent notificationdiff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.mdindex 14677af3c4..2a233b8558 100644--- a/infra/stacks/monitoring/README.md+++ b/infra/stacks/monitoring/README.md@@ -4,15 +4,61 @@ Datadog monitors and synthetic tests as code.36 resources here (35 monitors and one synthetic test) already exist inDatadog. This stack **adopts** them — it does not recreate them. `import.json`-lists every one by id. The two AI-editing monitors in `monitors/ai-editing.ts`-are new and are meant to be created.+lists every one by id. Everything else is declared here first and is created+on deploy: the prod threshold and anomaly monitors (`monitors/macrodb.ts`,+`redis.ts`, `queues.ts`, `regressions.ts`), the rollout monitors, the GoAlert webhooks and health check, and one log-based metric.++## Routing++Every monitor and synthetic test notifies through `notify.ts`, and nowhere+else:++- **All of them** post to the Macro Alerts channel (`@webhook-macro-alert-hook`,+ a webhook integration configured in the Datadog UI).+- **Paging monitors** also open a GoAlert alert on `{{#is_alert}}` and close it+ on `{{#is_recovery}}` (`@webhook-goalert`, `@webhook-goalert-close`, declared+ in `goalert.ts`). GoAlert runs on the observability host+ (`infra/stacks/observability`) and phones the on-call engineer. Warnings never+ page.+- **Every monitor renotifies hourly** (`RENOTIFY_MINUTES`) while it stays in+ warning or alert.+- `routing.test.ts` fails if any monitor mentions another handle or an email+ address, or lacks the renotify interval.++The GoAlert health check (`goalert-health`) notifies the channel only: when the+pager is down, a page through it would never arrive.++Paging monitors: the app-down synthetic, ALB non-200 responses and low healthy+hosts, Lambda error count, web-app source errors, macro-cache memory, RDS high+CPU, RDS storage, RDS proxy connections, the three SES reputation monitors, the+GraphQL soup DSS 5xx and DB connections monitors, and macrodb blocked+connections. `error-logs-overall` notifies without paging because it crosses+its thresholds many times a day.++## Prod threshold monitors++`monitors/macrodb.ts`, `redis.ts`, `queues.ts` and `regressions.ts` declare+the prod threshold, forecast and anomaly monitors with `prodMonitor()`. The ones+on CloudWatch metrics have hand-written copies in Grafana+(`infra/stacks/observability/nixos/alerting.nix`), and Grafana pages on-call for+the critical ones through GoAlert. Their Datadog copies only notify, so one+incident pages once. Keep thresholds and windows in sync by hand. The error-logs+anomaly monitor reads `macro.logs.errors`, a log-based metric declared in+`regressions.ts`, because log monitors have no anomaly function.## Read this before deploying+**Set the GoAlert integration key and deploy GoAlert first.** The webhooks+need the key, and paging monitors reference them:++```bash+pulumi config set --secret goalertIntegrationKey '<Generic API key of the Datadog service in GoAlert>'+```+**Run the import before the first `pulumi up`.** The program declares 35monitors that already exist in Datadog. Until `pulumi import` has told Pulumi-they exist, a deploy would create 35 duplicates — duplicate pages to-`#holy-shit-alarms` included.+they exist, a deploy would create 35 duplicates — duplicate pages+included.Adoption is a one-time local run. `pulumi import` reads Datadog and writesPulumi state — it does not create, modify, or pause anything in Datadog. The@@ -39,11 +85,16 @@ and records options omitted during import.The gate is that preview:-- **35 monitors + 1 synthetic test unchanged** — adoption is exact.-- **2 monitors to create** (`ai-editing-*`) — expected, that's the new alerting.-- **Any update, replace, or delete** — the hand-written code in `monitors/*.ts`- drifted from the live monitor. Reconcile against `imported.generated.ts`- before deploying. An `update` here silently changes who gets paged.+- **35 monitors + 1 synthetic test updated, in `message` and+ `renotifyInterval` only** — the routing move. The two AI-editing monitors,+ already deployed from this stack, update the same way.+- **69 resources to create**: 64 monitors (25 prod threshold and anomaly, 19 GraphQL soup, 9+ notification retention, 10 WhatsApp session, 1 native call push), 2 webhooks,+ 1 log-based metric, the GoAlert synthetic test, and the notification+ retention dashboard.+- **Any other update, a replace, or a delete** — the hand-written code in+ `monitors/*.ts` drifted from the live monitor. Reconcile against+ `imported.generated.ts` before deploying.Only then deploy: `pulumi up` locally, or **Actions → "deploy pulumi service"**with `pulumi-service-name: monitoring`, `environment: prod` (that action already@@ -106,10 +157,16 @@ is applied. Reach for `restrictedRoles` if convention stops being enough.## Adding a monitor-Declare it in the matching `monitors/*.ts` with `new datadog.Monitor(...)`. Use-the `adopted()` helper only for monitors that already exist in Datadog — it sets-`protect`, which is there to stop a program that no longer declares a-pre-existing monitor from deleting it.+Add a prod threshold or anomaly monitor with `prodMonitor()` in the matching+`monitors/*.ts`, and its Grafana copy in `alerting.nix` when the metric is in+CloudWatch. Declare anything else with `new datadog.Monitor(...)`, ending its+message with `notify({ pages })` and setting `renotifyInterval: RENOTIFY_MINUTES`. Use the `adopted()` helper only+for monitors that already exist in Datadog — it sets `protect`, which is there+to stop a program that no longer declares a pre-existing monitor from deleting+it, and the hourly renotify default.++The MemoryDB (`cloud-storage-cache-*`) and burst-balance monitors report no+data; retire them in a separate commit that drops `protect` first.## AI-editing coverage@@ -125,23 +182,21 @@ follow-up.## Native call push coverage`monitors/calls.ts` adds one log monitor that alerts on any rejected iOS or-Android call push, excluding stale device registrations. A `pulumi preview`-after it is added should show that one create and nothing else.+Android call push, excluding stale device registrations.It sees only publishes SNS rejects. A push that SNS accepts and the device thendrops leaves no backend trace.## GraphQL soup rollout coverage-`monitors/graphql-soup.ts` declares 15 new monitors for the `enable-graphql-soup`-rollout: GraphQL latency, errors and volume, websocket upgrades,-cloud-storage-service capacity and 5xx, macro-db-prod connections and CPU, the-gateway ALB, and web-app cache logs. A `pulumi preview` after they are added-should show exactly those 15 creates and nothing else.+`monitors/graphql-soup.ts` declares 19 new monitors for the `enable-graphql-soup`+rollout: GraphQL latency, errors and volume, per-operation errors and latency,+unauthenticated mutations, websocket upgrades, cloud-storage-service capacity+and 5xx, macro-db-prod connections and CPU, the gateway ALB, and web-app cache+logs.-Resolver errors still return HTTP 200, so the error monitors are proxies. Add-per-operation error and latency monitors once the server emits operation names-and error counts.+Resolver errors still return HTTP 200, so the endpoint error monitors are+proxies; the per-operation monitors read the `graphql.execute` span instead.## Notification retention coverage@@ -152,8 +207,8 @@ The dashboard URL is exported as `notificationRetentionDashboardUrl`.Deploy the notification consumer instrumentation in DSS before these monitors;otherwise the missing-telemetry alerts intentionally fire. This monitoring stack-still requires a separate, explicitly approved manual deployment. The preview-must not change adopted monitors.+still requires a separate, explicitly approved manual deployment. Adopted+monitors may change only in `message` and `renotifyInterval` (see the gate above).Consumers are intentionally ungrouped and do not commit offsets, so native MSKconsumer-group lag is not meaningful. The new logs expose broker high watermarkdiff --git a/infra/stacks/monitoring/adopted.ts b/infra/stacks/monitoring/adopted.tsindex 58eaaf9902..2aea5e94fb 100644--- a/infra/stacks/monitoring/adopted.ts+++ b/infra/stacks/monitoring/adopted.ts@@ -1,4 +1,6 @@import * as datadog from '@pulumi/datadog';+import { goalertWebhooks } from './goalert';+import { RENOTIFY_MINUTES } from './notify';/*** A monitor that already exists in Datadog and was adopted into this stack by@@ -19,7 +21,13 @@ export function adopted(// default explicit after normalizing imported state; group delay wins.return new datadog.Monitor(name,- { requireFullWindow: false, newHostDelay: 300, ...args },- { protect: true }+ {+ requireFullWindow: false,+ newHostDelay: 300,+ renotifyInterval: RENOTIFY_MINUTES,+ ...args,+ },+ // Messages may mention the GoAlert webhooks, which must exist first.+ { protect: true, dependsOn: goalertWebhooks });}diff --git a/infra/stacks/monitoring/goalert.ts b/infra/stacks/monitoring/goalert.tsnew file mode 100644index 0000000000..964d50d643--- /dev/null+++ b/infra/stacks/monitoring/goalert.ts@@ -0,0 +1,66 @@+// Datadog → GoAlert, the on-call pager on the Ohio observability host+// (infra/stacks/observability).+//+// Both webhooks post to a GoAlert Generic API integration key. GoAlert keeps+// one open alert per `dedup` value: repeat notifications (renotify) update it,+// and `action: close` resolves it. `$ALERT_CYCLE_KEY` stays the same from a+// monitor group's trigger to its recovery.+import * as datadog from '@pulumi/datadog';+import * as pulumi from '@pulumi/pulumi';+import { notify, RENOTIFY_MINUTES } from './notify';++const GOALERT_HOST = 'oncall.macro-internal.com';++// The integration key authenticates the request; it lives in this stack's+// encrypted config: `pulumi config set --secret goalertIntegrationKey <key>`.+const integrationKey = new pulumi.Config().requireSecret(+ 'goalertIntegrationKey'+);+const url = pulumi.interpolate`https://${GOALERT_HOST}/api/v2/generic/incoming?token=${integrationKey}`;+const alert = {+ summary: '$EVENT_TITLE',+ details: '$LINK\n\n$TEXT_ONLY_MSG',+ dedup: '$ALERT_CYCLE_KEY',+};++export const goalertWebhooks = [+ new datadog.Webhook('goalert', {+ name: 'goalert',+ url,+ encodeAs: 'json',+ payload: JSON.stringify(alert),+ }),+ new datadog.Webhook('goalert-close', {+ name: 'goalert-close',+ url,+ encodeAs: 'json',+ payload: JSON.stringify({ ...alert, action: 'close' }),+ }),+];++// Watches the pager from outside. It notifies the channel only: if GoAlert is+// down, a page through GoAlert would never arrive.+new datadog.SyntheticsTest('goalert-health', {+ name: 'GoAlert health check',+ type: 'api',+ subtype: 'http',+ status: 'live',+ tags: ['env:prod', 'service:goalert'],+ locations: ['aws:us-east-1', 'aws:us-west-2'],+ requestDefinition: { method: 'GET', url: `https://${GOALERT_HOST}/health` },+ assertions: [{ type: 'statusCode', operator: 'is', target: '200' }],+ optionsList: {+ tickEvery: 300,+ minFailureDuration: 300,+ // Both locations must fail, so one probe's network trouble is not a page outage.+ minLocationFailed: 2,+ retry: { count: 1, interval: 30000 },+ monitorName: '[PROD] GoAlert unreachable',+ monitorOptions: { renotifyInterval: RENOTIFY_MINUTES },+ },+ message: `GoAlert at https://${GOALERT_HOST} is not answering its health check, so Datadog and Grafana alerts cannot page anyone. Paging falls back to the observability host's CloudWatch alarms.++Runbook: https://github.com/macro-inc/macro/blob/main/infra/stacks/observability/README.md++${notify({ pages: false })}`,+});diff --git a/infra/stacks/monitoring/index.ts b/infra/stacks/monitoring/index.tsindex 9be3ec78f4..440efbe8bc 100644--- a/infra/stacks/monitoring/index.ts+++ b/infra/stacks/monitoring/index.ts@@ -1,8 +1,8 @@// Datadog monitors and synthetic tests.//-// Every monitor in the org is declared here. See README.md before deploying —-// the 35 monitors under `monitors/` are adopted, not created, and a program that-// stops declaring one deletes it.+// Every monitor in the org is declared here, and every one notifies through+// notify.ts. See README.md before deploying — the 35 monitors under `monitors/`+// are adopted, not created, and a program that stops declaring one deletes it.// Adopted from the Datadog UI.import './monitors/apm';@@ -18,9 +18,14 @@ import './monitors/ses';import './synthetics';// Declared here first.+import './goalert';import './monitors/ai-editing';import './monitors/calls';import './monitors/graphql-soup';+import './monitors/macrodb';+import './monitors/queues';+import './monitors/redis';+import './monitors/regressions';import './monitors/whatsapp-session';import { notificationRetentionDashboard } from './monitors/notification-retention';diff --git a/infra/stacks/monitoring/monitors/ai-editing.ts b/infra/stacks/monitoring/monitors/ai-editing.tsindex c32f166e5e..673e35917c 100644--- a/infra/stacks/monitoring/monitors/ai-editing.ts+++ b/infra/stacks/monitoring/monitors/ai-editing.ts@@ -14,6 +14,7 @@// `documentToken` (macro_sync_service_jwt), so a probe needs somewhere to mint// one.import * as datadog from '@pulumi/datadog';+import { notify, RENOTIFY_MINUTES } from '../notify';// The worker's entry span is `http.server.request` with resource_name `POST`.// Custom spans like `edit.session` do NOT produce trace metrics — Datadog only@@ -35,9 +36,10 @@ cancels; only the former counts as an error here.- Check provider outages first: the supervisor chain falls back Anthropic ->OpenAI, so a single provider failing shows up as latency, not errors.-@slack-monitoring`,+${notify({ pages: false })}`,tags: ['env:prod', 'service:ai-editing-worker', 'product:apm'],monitorThresholds: { critical: '50', warning: '20' },+ renotifyInterval: RENOTIFY_MINUTES,// Ratio of two count metrics: no traffic means no data, which must not// alert. Quiet-hours coverage is the canary's job, not this monitor's.onMissingData: 'default',@@ -74,9 +76,10 @@ reads the callers' logs rather than the worker's telemetry. The common causehistorically is \`initial sync failed: timeout\` — the worker could not open thedocument over the sync service.-@slack-monitoring`,+${notify({ pages: false })}`,tags: ['env:prod', 'service:ai-editing-worker'],monitorThresholds: { critical: '25', warning: '12' },+ renotifyInterval: RENOTIFY_MINUTES,onMissingData: 'default',includeTags: false,notifyAudit: false,diff --git a/infra/stacks/monitoring/monitors/apm.ts b/infra/stacks/monitoring/monitors/apm.tsindex 6bc600024f..3ae9f75cf7 100644--- a/infra/stacks/monitoring/monitors/apm.ts+++ b/infra/stacks/monitoring/monitors/apm.ts@@ -3,6 +3,7 @@// Adopted from monitors created by hand in the Datadog UI. Do not edit a// monitor here and in the UI at the same time — this file wins on deploy.import { adopted } from '../adopted';+import { notify } from '../notify';// https://us5.datadoghq.com/monitors/19271629adopted('apm-get-channel-messages-p95', {@@ -28,7 +29,9 @@ p95 response time has exceeded **100ms** (threshold). During the Apr 17 incident### Useful Links- [Slow spans explorer](https://us5.datadoghq.com/apm/traces?query=service%3Acloud-storage-service+resource_name%3Aget_channel_messages+env%3Aprod+%40duration%3A%3E100000000)-- [Investigation notebook](https://us5.datadoghq.com/notebook/277323)`,+- [Investigation notebook](https://us5.datadoghq.com/notebook/277323)++${notify({ pages: false })}`,priority: '5',tags: ['service:cloud-storage-service',diff --git a/infra/stacks/monitoring/monitors/cache.ts b/infra/stacks/monitoring/monitors/cache.tsindex c40df50b75..72a64fcc02 100644--- a/infra/stacks/monitoring/monitors/cache.ts+++ b/infra/stacks/monitoring/monitors/cache.ts@@ -3,6 +3,7 @@// Adopted from monitors created by hand in the Datadog UI. Do not edit a// monitor here and in the UI at the same time — this file wins on deploy.import { adopted } from '../adopted';+import { notify } from '../notify';// https://us5.datadoghq.com/monitors/1062487adopted('macro-cache-available-memory-low', {@@ -10,19 +11,9 @@ adopted('macro-cache-available-memory-low', {type: 'query alert',query:'avg(last_1h):avg:aws.elasticache.database_memory_usage_percentage{name:macro-cache-prod*} > 90',- message: `{{#is_alert}}-@slack-holy-shit-alarms-{{/is_alert}}-{{#is_alert_recovery}}-@slack-holy-shit-alarms-{{/is_alert_recovery}}+ message: `macro-cache-prod is using {{value}}% of its memory. Keys without a TTL are never evicted, so at 100% writes fail.-{{#is_warning_recovery}}-@slack-monitoring-{{/is_warning_recovery}}-{{#is_warning}}-@slack-monitoring-{{/is_warning}}`,+${notify({ pages: true })}`,monitorThresholds: {critical: '90',warning: '80',@@ -40,8 +31,10 @@ adopted('cloud-storage-cache-high-cpu', {type: 'query alert',query:'avg(last_1h):avg:aws.memorydb.cpuutilization{clustername:cloud-storage-cache-memorydb-prod} > 90',- message:- '{{#is_alert}} @slack-holy-shit-alarms {{/is_alert}} {{#is_alert_recovery}} @slack-holy-shit-alarms {{/is_alert_recovery}}',+ // Reports no data: the MemoryDB cluster appears to be gone. Retire separately.+ message: `cloud-storage-cache MemoryDB CPU is at {{value}}%.++${notify({ pages: false })}`,monitorThresholds: {critical: '90',warning: '80',@@ -59,8 +52,10 @@ adopted('cloud-storage-cache-low-memory', {type: 'query alert',query:'avg(last_1h):avg:aws.memorydb.database_memory_usage_percentage{*} > 90',- message:- '{{#is_alert}} @slack-holy-shit-alarms {{/is_alert}} {{#is_alert_recovery}} @slack-holy-shit-alarms {{/is_alert_recovery}}',+ // Reports no data: the MemoryDB cluster appears to be gone. Retire separately.+ message: `MemoryDB memory usage is at {{value}}%.++${notify({ pages: false })}`,monitorThresholds: {critical: '90',warning: '60',diff --git a/infra/stacks/monitoring/monitors/calls.ts b/infra/stacks/monitoring/monitors/calls.tsindex 91089c3414..f17cae7ef3 100644--- a/infra/stacks/monitoring/monitors/calls.ts+++ b/infra/stacks/monitoring/monitors/calls.ts@@ -15,6 +15,7 @@// rejects them as disabled or missing endpoints, and `@error` carries the// `PushEndpointUnavailable` context text, so they are excluded.import * as datadog from '@pulumi/datadog';+import { notify, RENOTIFY_MINUTES } from '../notify';export const nativeCallPushFailing = new datadog.Monitor('native-call-push-failing',@@ -32,9 +33,10 @@ task role lacks \`sns:Publish\` on that platform application: add it to- [Failure logs](https://us5.datadoghq.com/logs?query=service%3Acloud-storage-service%20env%3Aprod%20%22voip%20push%3A%20SNS%20delivery%20failed%22)-@webhook-macro-alert-hook`,+${notify({ pages: false })}`,tags: ['env:prod', 'service:cloud-storage-service'],monitorThresholds: { critical: '1' },+ renotifyInterval: RENOTIFY_MINUTES,onMissingData: 'default',includeTags: false,notifyAudit: false,diff --git a/infra/stacks/monitoring/monitors/compute.ts b/infra/stacks/monitoring/monitors/compute.tsindex 350e6e6c9b..6e47a98dc0 100644--- a/infra/stacks/monitoring/monitors/compute.ts+++ b/infra/stacks/monitoring/monitors/compute.ts@@ -3,6 +3,7 @@// Adopted from monitors created by hand in the Datadog UI. Do not edit a// monitor here and in the UI at the same time — this file wins on deploy.import { adopted } from '../adopted';+import { notify } from '../notify';// https://us5.datadoghq.com/monitors/1062688adopted('ecs-high-cpu', {@@ -12,7 +13,7 @@ adopted('ecs-high-cpu', {'avg(last_5m):sum:ecs.fargate.cpu.percent{(ecs_cluster:macro-cluster-prod OR ecs_cluster:cloud-storage-cluster-prod OR ecs_cluster:macroai-cluster-prod-4cbe721)} by {task_name} > 500',message: `{{task_name.name}} is experiencing high cpu usage in ECS-@slack-monitoring`,+${notify({ pages: false })}`,monitorThresholds: {critical: '500',warning: '400',@@ -29,10 +30,9 @@ adopted('lambda-error-count', {type: 'query alert',query:'sum(last_1h):sum:aws.lambda.errors{(level:prod OR environment:prod ) AND NOT functionname:ws-api-gateway-authorizer-lambda-prod} by {functionname}.as_rate() > 500',- message: `{{#is_alert}}@slack-holy-shit-alarms {{/is_alert}}-{{#is_alert_recovery}}@slack-holy-shit-alarms{{/is_alert_recovery}}+ message: `Lambda {{functionname.name}} is failing at {{value}} errors per second.-@slack-monitoring`,+${notify({ pages: true })}`,monitorThresholds: {critical: '500',warning: '100',@@ -50,9 +50,9 @@ adopted('document-text-extractor-prod', {type: 'query alert',query:'sum(last_4h):(sum:aws.lambda.errors{resource:document-text-extractor-prod}.as_count() / sum:aws.lambda.invocations{resource:document-text-extractor-prod}.as_count()) * 100 > 80',- message: `@[email protected]-@slack-monitoring-Document text extractor failing`,+ message: `Document text extractor failing: {{value}}% of invocations errored over 4 hours.++${notify({ pages: false })}`,monitorThresholds: {critical: '80',warning: '50',@@ -62,7 +62,6 @@ Document text extractor failing`,includeTags: false,newHostDelay: 300,evaluationDelay: 900,- renotifyInterval: 0,});// https://us5.datadoghq.com/monitors/1291772@@ -71,9 +70,9 @@ adopted('document-text-extractor-dev', {type: 'query alert',query:'sum(last_4h):(sum:aws.lambda.errors{resource:document-text-extractor-dev}.as_count() / sum:aws.lambda.invocations{resource:document-text-extractor-dev}.as_count()) * 100 > 80',- message: `@[email protected]-@slack-monitoring-Document text extractor failing`,+ message: `Document text extractor failing: {{value}}% of invocations errored over 4 hours.++${notify({ pages: false })}`,monitorThresholds: {critical: '80',warning: '50',@@ -83,5 +82,4 @@ Document text extractor failing`,includeTags: false,newHostDelay: 300,evaluationDelay: 900,- renotifyInterval: 0,});diff --git a/infra/stacks/monitoring/monitors/dynamodb.ts b/infra/stacks/monitoring/monitors/dynamodb.tsindex 067f457696..6c7c5ed744 100644--- a/infra/stacks/monitoring/monitors/dynamodb.ts+++ b/infra/stacks/monitoring/monitors/dynamodb.ts@@ -3,6 +3,7 @@// Adopted from monitors created by hand in the Datadog UI. Do not edit a// monitor here and in the UI at the same time — this file wins on deploy.import { adopted } from '../adopted';+import { notify } from '../notify';// https://us5.datadoghq.com/monitors/16917541adopted('dynamodb-write-capacity', {@@ -10,8 +11,9 @@ adopted('dynamodb-write-capacity', {type: 'query alert',query:'avg(last_1h):avg:aws.dynamodb.consumed_write_capacity_units{*} by {tablename} > 500',- message: `@[email protected]+ message: `DynamoDB table {{tablename.name}} is consuming {{value}} write capacity units.++${notify({ pages: false })}`,monitorThresholds: {critical: '500',warning: '300',@@ -29,8 +31,9 @@ adopted('dynamodb-read-capacity', {type: 'query alert',query:'avg(last_1h):avg:aws.dynamodb.consumed_read_capacity_units{*} by {tablename} > 20',- message: `@[email protected]+ message: `DynamoDB table {{tablename.name}} is consuming {{value}} read capacity units.++${notify({ pages: false })}`,monitorThresholds: {critical: '20',warning: '10',diff --git a/infra/stacks/monitoring/monitors/graphql-soup.ts b/infra/stacks/monitoring/monitors/graphql-soup.tsindex 120a25db72..10f15bdac7 100644--- a/infra/stacks/monitoring/monitors/graphql-soup.ts+++ b/infra/stacks/monitoring/monitors/graphql-soup.ts@@ -14,6 +14,8 @@// metric tags) once https://github.com/macro-inc/macro/pull/7911 deploys; until// then they have no data.import * as datadog from '@pulumi/datadog';+import { goalertWebhooks } from '../goalert';+import { notify, RENOTIFY_MINUTES } from '../notify';const GRAPHQL ='service:cloud-storage-service,env:prod,resource_name:post_/dss/items/soup/graphql';@@ -32,16 +34,6 @@ const GATEWAY_ALB = 'loadbalancer:app/prod-gateway-alb/8e99d35bdc70b1c9';const ROLLOUT_CONTEXT = `Rollout: \`enable-graphql-soup\` (https://us.posthog.com/project/299018/feature_flags/738783).Client-side view: https://us.posthog.com/project/299018/dashboard/2198651`;-// Datadog webhook integration that posts into the Macro "Alerts" channel-// (https://macro.com/app/channel/019fb34d-fcfd-7548-9a0f-7084ea16416c). There-// is no separate severe channel, so severity is marked in the message instead.-const MACRO_ALERTS_CHANNEL = '@webhook-macro-alert-hook';--const notify = (severe: boolean) =>- severe- ? `{{#is_alert}}SEVERE: user-facing impact likely, act now.{{/is_alert}}\n${MACRO_ALERTS_CHANNEL}`- : MACRO_ALERTS_CHANNEL;-function rolloutMonitor(name: string,args: {@@ -55,25 +47,30 @@ function rolloutMonitor(severe?: boolean;}): datadog.Monitor {- return new datadog.Monitor(name, {- name: args.name,- type: args.type,- query: args.query,- message: `${args.description}+ return new datadog.Monitor(+ name,+ {+ name: args.name,+ type: args.type,+ query: args.query,+ message: `${args.description}${ROLLOUT_CONTEXT}-${notify(args.severe ?? false)}`,- tags: ['env:prod', `service:${args.service}`, 'rollout:graphql-soup'],- priority: args.severe ? '2' : '3',- monitorThresholds: { critical: args.critical, warning: args.warning },- // APM and log counts are sparse at night; a full window would skip- // evaluations, and an empty count must read as zero rather than alert.- requireFullWindow: false,- onMissingData: 'default',- includeTags: true,- notifyAudit: false,- });+${notify({ pages: args.severe ?? false })}`,+ tags: ['env:prod', `service:${args.service}`, 'rollout:graphql-soup'],+ priority: args.severe ? '2' : '3',+ monitorThresholds: { critical: args.critical, warning: args.warning },+ renotifyInterval: RENOTIFY_MINUTES,+ // APM and log counts are sparse at night; a full window would skip+ // evaluations, and an empty count must read as zero rather than alert.+ requireFullWindow: false,+ onMissingData: 'default',+ includeTags: true,+ notifyAudit: false,+ },+ { dependsOn: args.severe ? goalertWebhooks : [] }+ );}// Latency. The endpoint's p95 is long-tailed even when healthy (30-minute p95diff --git a/infra/stacks/monitoring/monitors/load-balancer.ts b/infra/stacks/monitoring/monitors/load-balancer.tsindex dddff183ed..4468adbb12 100644--- a/infra/stacks/monitoring/monitors/load-balancer.ts+++ b/infra/stacks/monitoring/monitors/load-balancer.ts@@ -3,6 +3,7 @@// Adopted from monitors created by hand in the Datadog UI. Do not edit a// monitor here and in the UI at the same time — this file wins on deploy.import { adopted } from '../adopted';+import { notify } from '../notify';// https://us5.datadoghq.com/monitors/1062605adopted('alb-non-200-responses', {@@ -12,20 +13,7 @@ adopted('alb-non-200-responses', {'sum(last_1h):(sum:aws.applicationelb.httpcode_elb_5xx{level:prod OR environment:prod*} by {loadbalancer}.as_count() / sum:aws.applicationelb.request_count{level:prod OR environment:prod*} by {loadbalancer}.as_count()) * 100 > 90',message: `Detected a high number of non-200 responses for ALB {{loadbalancer.name}}--{{#is_alert}}-@slack-holy-shit-alarms-{{/is_alert}}-{{#is_alert_recovery}}-@slack-holy-shit-alarms-{{/is_alert_recovery}}--@slack-monitoring---@slack-monitoring`,+${notify({ pages: true })}`,monitorThresholds: {critical: '90',warning: '60',@@ -35,7 +23,6 @@ adopted('alb-non-200-responses', {includeTags: false,newGroupDelay: 0,evaluationDelay: 900,- renotifyInterval: 0,requireFullWindow: false,});@@ -47,9 +34,7 @@ adopted('alb-low-healthy-hosts', {'avg(last_1h):abs(avg:aws.applicationelb.healthy_host_count{*} by {loadbalancer}) <= 0.25',message: `The application load balancer for {{loadbalancer.name}} has low healthy host count.-+${notify({ pages: true })}`,monitorThresholds: {critical: '0.25',},@@ -68,7 +53,7 @@ adopted('alb-high-response-time', {'avg(last_1h):avg:aws.applicationelb.target_response_time.average{! name:proxy-service-alb-prod-153de67 , ! name:store-service-alb-prod-4db6301} by {loadbalancer} > 3',message: `Loadbalancer {{loadbalancer.name}} has a high average response time+${notify({ pages: false })}`,monitorThresholds: {critical: '3',},diff --git a/infra/stacks/monitoring/monitors/logs.ts b/infra/stacks/monitoring/monitors/logs.tsindex 3527d51354..3a2bae165b 100644--- a/infra/stacks/monitoring/monitors/logs.ts+++ b/infra/stacks/monitoring/monitors/logs.ts@@ -3,6 +3,7 @@// Adopted from monitors created by hand in the Datadog UI. Do not edit a// monitor here and in the UI at the same time — this file wins on deploy.import { adopted } from '../adopted';+import { notify } from '../notify';// DSS logs every expired-JWT rejection at ERROR. These are routine client token// refreshes and make up about 60% of its prod error lines, so the DSS error@@ -18,8 +19,9 @@ adopted('macroai-error-logs', {type: 'log alert',query:'logs("service:(macroai-query-service-prod OR macroai-store-service-prod OR macroai-proxy-service-prod) status:error").index("*").rollup("count").last("5m") > 2500',- message: `@[email protected]-@slack-monitoring`,+ message: `macroai services logged {{value}} errors in 5 minutes.++${notify({ pages: false })}`,tags: ['macro-ai'],monitorThresholds: {critical: '2500',@@ -29,7 +31,6 @@ adopted('macroai-error-logs', {notifyAudit: false,includeTags: false,newHostDelay: 300,- renotifyInterval: 0,enableLogsSample: true,groupbySimpleMonitor: false,});@@ -40,7 +41,9 @@ adopted('macroapi-error-logs', {type: 'log alert',query:'logs("service:macro-gql-service-prod status:error").index("*").rollup("count").last("5m") > 50',- message: '@slack-monitoring',+ message: `macro-gql-service logged {{value}} errors in 5 minutes.++${notify({ pages: false })}`,monitorThresholds: {critical: '50',},@@ -57,7 +60,9 @@ adopted('dss-errors-dev', {name: '[DEV] Increased number of errors for DSS',type: 'log alert',query: `logs("service:cloud-storage-service env:dev status:error ${DSS_EXPIRED_JWT_EXCLUSION}").index("*").rollup("count").last("5m") > 200`,- message: '@slack-monitoring',+ message: `Dev cloud-storage-service logged {{value}} errors in 5 minutes.++${notify({ pages: false })}`,monitorThresholds: {critical: '200',warning: '100',@@ -77,7 +82,9 @@ adopted('dss-errors-prod', {name: '[PROD] Increased number of errors for DSS',type: 'log alert',query: `logs("service:cloud-storage-service env:prod status:error ${DSS_EXPIRED_JWT_EXCLUSION}").index("*").rollup("count").last("5m") > 150`,- message: '@slack-monitoring',+ message: `Prod cloud-storage-service logged {{value}} errors in 5 minutes, not counting expired-JWT rejections.++${notify({ pages: false })}`,monitorThresholds: {critical: '150',},@@ -94,17 +101,12 @@ adopted('error-logs-overall', {name: '[PROD] Increase in Error Logs',type: 'log alert',query: 'formula("query - query1").last("5m") > 25000',+ // Crosses its thresholds many times a day, so it notifies without paging.+ // The per-service anomaly monitors in regressions.ts cover regressions.message: `Increase in overall count of error logs across production services.Please view datadog to checkout what specific items may be causing the issues.-{{#is_alert}}-@slack-holy-shit-alarms-{{/is_alert}}-{{#is_alert_recovery}}-@slack-holy-shit-alarms-{{/is_alert_recovery}}--@slack-monitoring`,+${notify({ pages: false })}`,priority: '1',tags: ['prod'],monitorThresholds: {@@ -156,7 +158,9 @@ adopted('graphql-redis-connect', {type: 'log alert',query:'logs("service:macro-gql-service-prod status:error \\"Failed to reconnect to Redis cache at URL*\\"").index("*").rollup("count").last("5m") >= 1',- message: '@slack-monitoring',+ message: `macro-gql-service could not reconnect to its Redis cache.++${notify({ pages: false })}`,priority: '3',tags: ['prod'],monitorThresholds: {@@ -176,12 +180,13 @@ adopted('high-channel-invites', {type: 'log alert',query:'logs("service:notification-service-prod @notification_event_type:channel_invite").index("*").rollup("count", "@sender_id").last("1h") > 100',- message: `@slack-monitoring-High number of channel invites+ message: `High number of channel invitesInvestigate whether 1 user is causing this with this log query-\`service:notification-service-prod @notification_event_type:channel_invite\``,+\`service:notification-service-prod @notification_event_type:channel_invite\`++${notify({ pages: false })}`,monitorThresholds: {critical: '100',warning: '50',@@ -202,9 +207,7 @@ adopted('channel-invite-spam-dev', {'logs("service:notification-service-dev message:\\"processing message\\" @notification_event_type:channel_invite").index("*").rollup("count").by("@sender_id").last("1h") > 10',message: `A user is generating a high number of channel invites. This could mean they are spamming users with invite emails.-@slack-monitoring-+${notify({ pages: false })}`,monitorThresholds: {critical: '10',warning: '5',@@ -225,9 +228,7 @@ adopted('channel-invite-spam-prod', {'logs("service:notification-service-prod message:\\"processing message\\" @notification_event_type:channel_invite").index("*").rollup("count").by("@sender_id").last("1h") > 10',message: `A user is generating a high number of channel invites. This could mean they are spamming users with invite emails.-@slack-monitoring-+${notify({ pages: false })}`,monitorThresholds: {critical: '10',warning: '5',diff --git a/infra/stacks/monitoring/monitors/macrodb.ts b/infra/stacks/monitoring/monitors/macrodb.tsnew file mode 100644index 0000000000..b191b943c1--- /dev/null+++ b/infra/stacks/monitoring/monitors/macrodb.ts@@ -0,0 +1,166 @@+// MacroDB (macro-db-prod and its read replica) health.+//+// Thresholds come from two weeks of 5-minute data before the October overload.+// Monitors on CloudWatch metrics also exist in Grafana+// (infra/stacks/observability/nixos/alerting.nix); change both together.+// Grafana pages on-call for those through GoAlert, so they only notify here.+import { prodMonitor } from '../prod-monitor';++prodMonitor('macrodb-primary-freeable-memory', {+ title: 'macro-db-prod freeable memory low',+ service: 'macro-db-prod',+ query:+ 'avg(last_15m):avg:aws.rds.freeable_memory{dbinstanceidentifier:macro-db-prod} < 1000000000',+ monitorThresholds: { critical: '1000000000', warning: '2000000000' },+ pages: false,+ description:+ 'Freeable memory is {{value}} bytes. Normal is 7-18 GB; it fell to about 0.45 GB during the October overload. Each connection holds about 14 MB of private memory, so check connection count and swap first.',+ runbook: 'docs/runbooks/macrodb.md#memory',+});++prodMonitor('macrodb-replica-freeable-memory', {+ title: 'macro-db-prod-read-replica freeable memory low',+ service: 'macro-db-prod',+ query:+ 'avg(last_15m):avg:aws.rds.freeable_memory{dbinstanceidentifier:macro-db-prod-read-replica} < 750000000',+ monitorThresholds: { critical: '750000000', warning: '1500000000' },+ pages: false,+ description:+ 'Replica freeable memory is {{value}} bytes. Normal minimum is about 3 GB. The replica serves GraphQL reads, so memory pressure shows up as slow lists.',+ runbook: 'docs/runbooks/macrodb.md#memory',+});++prodMonitor('macrodb-swap', {+ title: 'macro-db-prod swapping',+ service: 'macro-db-prod',+ query:+ 'avg(last_15m):max:aws.rds.swap_usage{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier} > 5000000000',+ monitorThresholds: { critical: '5000000000', warning: '2000000000' },+ pages: false,+ description:+ '{{dbinstanceidentifier.name}} is using {{value}} bytes of swap. Normal is under 0.6 GB. Swap above 5 GB preceded both October overloads by hours.',+ runbook: 'docs/runbooks/macrodb.md#memory',+});++prodMonitor('macrodb-load', {+ title: 'macro-db-prod database load above vCPU count',+ service: 'macro-db-prod',+ query:+ 'avg(last_15m):avg:aws.rds.dbload_relative_to_num_vcpus{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier} > 4',+ monitorThresholds: { critical: '4', warning: '2' },+ pages: false,+ description:+ '{{dbinstanceidentifier.name}} has {{value}}x as many active sessions as vCPUs. Normal is under 1.3; the October overload reached 8. Look at Performance Insights top waits (locks, IO) and the services holding the most connections.',+ runbook: 'docs/runbooks/macrodb.md#load',+});++prodMonitor('macrodb-cpu', {+ title: 'macro-db-prod CPU high',+ service: 'macro-db-prod',+ query:+ 'avg(last_15m):avg:aws.rds.cpuutilization{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier} > 85',+ monitorThresholds: { critical: '85', warning: '70' },+ pages: false,+ description:+ '{{dbinstanceidentifier.name}} CPU averaged {{value}}% over 15 minutes. Normal peaks are about 30%.',+ runbook: 'docs/runbooks/macrodb.md#load',+});++prodMonitor('macrodb-primary-connections', {+ title: 'macro-db-prod connections high',+ service: 'macro-db-prod',+ query:+ 'avg(last_10m):max:aws.rds.database_connections{dbinstanceidentifier:macro-db-prod} > 2200',+ monitorThresholds: { critical: '2200', warning: '1700' },+ pages: false,+ description:+ 'macro-db-prod has {{value}} connections. Normal 15-minute peaks are about 1,100; max_connections is 3,439. Each connection costs about 14 MB, so this also drains freeable memory.',+ runbook: 'docs/runbooks/macrodb.md#connections',+});++prodMonitor('macrodb-replica-connections', {+ title: 'macro-db-prod-read-replica connections high',+ service: 'macro-db-prod',+ query:+ 'avg(last_10m):max:aws.rds.database_connections{dbinstanceidentifier:macro-db-prod-read-replica} > 700',+ monitorThresholds: { critical: '700', warning: '550' },+ pages: false,+ description:+ 'The read replica has {{value}} connections. Normal peaks are about 460.',+ runbook: 'docs/runbooks/macrodb.md#connections',+});++prodMonitor('macrodb-replica-lag', {+ title: 'macro-db-prod-read-replica lagging',+ service: 'macro-db-prod',+ query:+ 'avg(last_10m):max:aws.rds.replica_lag{dbinstanceidentifier:macro-db-prod-read-replica} > 300',+ monitorThresholds: { critical: '300', warning: '60' },+ pages: false,+ description:+ 'The read replica is {{value}} seconds behind the primary. Normal is under 6 seconds. GraphQL reads use the replica, so users see stale data and missing new items.',+ runbook: 'docs/runbooks/macrodb.md#replica-lag',+});++prodMonitor('macrodb-read-latency', {+ title: 'macro-db-prod disk read latency high',+ service: 'macro-db-prod',+ query:+ 'avg(last_15m):avg:aws.rds.read_latency{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier} > 0.01',+ monitorThresholds: { critical: '0.01', warning: '0.005' },+ pages: false,+ description:+ '{{dbinstanceidentifier.name}} disk reads take {{value}} seconds on average. Normal is about 0.6 ms.',+ runbook: 'docs/runbooks/macrodb.md#load',+});++prodMonitor('macrodb-free-storage', {+ title: 'macro-db-prod free storage low',+ service: 'macro-db-prod',+ query:+ 'avg(last_1h):min:aws.rds.free_storage_space{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier} < 1000000000000',+ monitorThresholds: { critical: '1000000000000', warning: '1500000000000' },+ pages: false,+ description:+ '{{dbinstanceidentifier.name}} has {{value}} bytes of free storage out of 5.37 TB.',+ runbook: 'docs/runbooks/macrodb.md#storage',+});++// Datadog only: Grafana alerting has no forecast function.+prodMonitor('macrodb-free-storage-forecast', {+ title: 'macro-db-prod free storage forecast to run low within a week',+ service: 'macro-db-prod',+ query:+ "min(next_1w):forecast(min:aws.rds.free_storage_space{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}, 'linear', 1, interval='60m', history='1w', model='default') <= 500000000000",+ monitorThresholds: { critical: '500000000000' },+ pages: false,+ description:+ 'At the current write rate {{dbinstanceidentifier.name}} drops below 500 GB free within a week. Find what started writing; recent growth was 100-200 GB a day.',+ runbook: 'docs/runbooks/macrodb.md#storage',+});++// Datadog only: Database Monitoring metrics; Grafana's Postgres collector runs+// against dev only.+prodMonitor('macrodb-blocked-connections', {+ title: 'macro-db-prod connections blocked on locks',+ service: 'macro-db-prod',+ query:+ 'avg(last_10m):max:postgresql.activity.blocked_connections{host:macro-db-prod*} by {host} > 50',+ monitorThresholds: { critical: '50', warning: '20' },+ pages: true,+ description:+ "{{host.name}} has {{value}} connections waiting on locks. Normal is under 7. Advisory and tuple locks dominated the October overload; see Database Monitoring's blocking queries view.",+ runbook: 'docs/runbooks/macrodb.md#locks',+});++prodMonitor('macrodb-long-transactions', {+ title: 'macro-db-prod long-running transaction',+ service: 'macro-db-prod',+ query:+ 'avg(last_10m):max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host} > 1800',+ monitorThresholds: { critical: '1800', warning: '600' },+ pages: false,+ description:+ 'The oldest open transaction on {{host.name}} is {{value}} seconds old. Long transactions hold locks and stop vacuum.',+ runbook: 'docs/runbooks/macrodb.md#locks',+});diff --git a/infra/stacks/monitoring/monitors/notification-retention.ts b/infra/stacks/monitoring/monitors/notification-retention.tsindex 70e343f99f..444dc1a8f7 100644--- a/infra/stacks/monitoring/monitors/notification-retention.ts+++ b/infra/stacks/monitoring/monitors/notification-retention.ts@@ -1,13 +1,14 @@// Independent notification consumers do not commit Kafka offsets. Native MSK// consumer-group lag is therefore absent; these logs report application backlog.import * as datadog from '@pulumi/datadog';+import { notify, RENOTIFY_MINUTES } from '../notify';const HEALTH = '@telemetry_event:notification_consumer_health';const CONTEXT = `Notification-retention rollout: pause scheduled cleanup before investigating.Check consumer health/backlog, notification.delete.publish spans, database latency,and replica lag. A zero application backlog is not an end-to-end delivery ACK:Postgres NOTIFY publication and WebSocket delivery can still fail.-@webhook-macro-alert-hook`;+${notify({ pages: false })}`;for (const environment of ['dev', 'prod']) {const query = `${HEALTH} env:${environment}`;@@ -20,6 +21,7 @@ for (const environment of ['dev', 'prod']) {priority: environment === 'prod' ? '2' : '3',includeTags: true,requireFullWindow: false,+ renotifyInterval: RENOTIFY_MINUTES,};new datadog.Monitor(`notification-consumer-backlog-${environment}`, {...common,@@ -73,6 +75,7 @@ new datadog.Monitor('notification-retention-telemetry-export', {priority: '3',monitorThresholds: { critical: '0' },requireFullWindow: false,+ renotifyInterval: RENOTIFY_MINUTES,onMissingData: 'default',});diff --git a/infra/stacks/monitoring/monitors/opensearch.ts b/infra/stacks/monitoring/monitors/opensearch.tsindex 022cf98aad..27b22d11e7 100644--- a/infra/stacks/monitoring/monitors/opensearch.ts+++ b/infra/stacks/monitoring/monitors/opensearch.ts@@ -3,6 +3,7 @@// Adopted from monitors created by hand in the Datadog UI. Do not edit a// monitor here and in the UI at the same time — this file wins on deploy.import { adopted } from '../adopted';+import { notify } from '../notify';// https://us5.datadoghq.com/monitors/21836655adopted('opensearch-storage-low', {@@ -18,8 +19,6 @@ Storage utilization: {{value}}%Critical threshold: {{threshold}}%Opensearch storage utilization has reached a critical level. Investigate storage growth or increase the allocated storage.--@webhook-macro-alert-hook{{/is_alert}}{{#is_warning}}@@ -30,8 +29,6 @@ Storage utilization: {{value}}%Warning threshold: {{warn_threshold}}%Opensearch storage utilization is approaching the critical threshold.--@webhook-macro-alert-hook{{/is_warning}}{{#is_recovery}}@@ -41,11 +38,9 @@ Environment: {{environment}}Storage utilization: {{value}}%Opensearch storage utilization has returned to an acceptable level.--@webhook-macro-alert-hook{{/is_recovery}}+${notify({ pages: false })}`,monitorThresholds: {critical: '10000',warning: '20000',diff --git a/infra/stacks/monitoring/monitors/queues.ts b/infra/stacks/monitoring/monitors/queues.tsnew file mode 100644index 0000000000..393de6b269--- /dev/null+++ b/infra/stacks/monitoring/monitors/queues.ts@@ -0,0 +1,61 @@+// Prod SQS queues: dead letters, backlog age and backlog growth.+//+// The two queue-age monitors also exist in Grafana+// (infra/stacks/observability/nixos/alerting.nix); change both together.+// Grafana pages on-call for the work queues, so they only notify here.+import { prodMonitor } from '../prod-monitor';++// Datadog only: CloudWatch dimensions can't be filtered by name pattern.+prodMonitor('sqs-dlq-inflow', {+ title: 'Messages landing in a prod dead-letter queue',+ service: 'sqs',+ query:+ 'sum(last_30m):sum:aws.sqs.number_of_messages_sent{queuename:*-dlq-prod} by {queuename}.as_count() > 50',+ monitorThresholds: { critical: '50', warning: '0' },+ pages: false,+ description:+ '{{value}} messages moved to {{queuename.name}} in 30 minutes. Most DLQs hold old messages permanently, so this alerts on new arrivals only.',+ runbook: 'docs/runbooks/queues.md#dead-letters',+});++prodMonitor('sqs-email-backfill-age', {+ title: 'Email backfill queue falling behind',+ service: 'email-service',+ query:+ 'avg(last_15m):max:aws.sqs.approximate_age_of_oldest_message{queuename:email-service-backfill-queue-prod} > 10800',+ monitorThresholds: { critical: '10800', warning: '3600' },+ pages: false,+ description:+ 'The oldest message in email-service-backfill-queue-prod is {{value}} seconds old. Normal peaks are about 50 minutes; new users wait this long for their mailbox to fill.',+ runbook: 'docs/runbooks/queues.md#backlog',+});++prodMonitor('sqs-work-queue-age', {+ title: 'Prod work queue falling behind',+ service: 'sqs',+ query:+ 'avg(last_15m):max:aws.sqs.approximate_age_of_oldest_message{queuename:email-service-refresh-queue-prod OR queuename:convert-service-queue-prod OR queuename:ai-projection-queue-prod OR queuename:email-service-gmail-webhook-queue-prod OR queuename:notification-ingress-queue-prod} by {queuename} > 7200',+ monitorThresholds: { critical: '7200', warning: '1800' },+ pages: false,+ description:+ 'The oldest message in {{queuename.name}} is {{value}} seconds old. These queues normally drain within minutes.',+ runbook: 'docs/runbooks/queues.md#backlog',+});++// Datadog only: Grafana alerting has no anomaly detection.+prodMonitor('sqs-email-backfill-backlog-anomaly', {+ title: 'Email backfill backlog growing abnormally',+ service: 'email-service',+ query:+ "avg(last_4h):anomalies(max:aws.sqs.approximate_number_of_messages_visible{queuename:email-service-backfill-queue-prod}, 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",+ // An anomaly monitor recovers once no point in its window is outside the band.+ monitorThresholds: { critical: '1', criticalRecovery: '0' },+ monitorThresholdWindows: {+ triggerWindow: 'last_30m',+ recoveryWindow: 'last_15m',+ },+ pages: false,+ description:+ 'Visible messages in email-service-backfill-queue-prod are above their usual band. A jump in enqueues loads macrodb through the backfill workers.',+ runbook: 'docs/runbooks/queues.md#backlog',+});diff --git a/infra/stacks/monitoring/monitors/rds.ts b/infra/stacks/monitoring/monitors/rds.tsindex 6e81abc719..9d2b1b6956 100644--- a/infra/stacks/monitoring/monitors/rds.ts+++ b/infra/stacks/monitoring/monitors/rds.ts@@ -3,6 +3,7 @@// Adopted from monitors created by hand in the Datadog UI. Do not edit a// monitor here and in the UI at the same time — this file wins on deploy.import { adopted } from '../adopted';+import { notify } from '../notify';// https://us5.datadoghq.com/monitors/1062630adopted('rds-high-cpu', {@@ -10,12 +11,9 @@ adopted('rds-high-cpu', {type: 'query alert',query:'avg(last_1h):sum:aws.rds.cpuutilization{name:*-prod} by {name}.weighted() >= 95',- message: `@[email protected]-@webhook-macro-alert-hook+ message: `RDS DB {{name.name}} is experiencing high CPU utilization.-RDS DB {{name.name}} is experiencing high CPU utilization.`,+${notify({ pages: true })}`,monitorThresholds: {critical: '95',warning: '80',@@ -25,7 +23,6 @@ RDS DB {{name.name}} is experiencing high CPU utilization.`,includeTags: true,newGroupDelay: 0,evaluationDelay: 900,- renotifyInterval: 0,requireFullWindow: false,});@@ -34,7 +31,9 @@ adopted('macrodb-low-burst-balance', {name: '[PROD] MacroDB Low Burst Balance',type: 'query alert',query: 'avg(last_1h):avg:aws.rds.burst_balance{name:macro-db-prod} <= 10',- message: '@slack-monitoring',+ message: `macro-db-prod burst balance is at {{value}}%.++${notify({ pages: false })}`,monitorThresholds: {critical: '10',warning: '25',@@ -54,7 +53,7 @@ adopted('rds-proxy-connections', {'avg(last_1h):avg:aws.rds.proxy.database_connections{proxyname:macrodb-proxy-prod} > 500',message: `We have excessive concurrent connections in the rds proxy. This can lead to slow database calls and potential outage if other services can not secure a connection-@slack-holy-shit-alarms`,+${notify({ pages: true })}`,monitorThresholds: {critical: '500',},@@ -80,8 +79,6 @@ Storage utilization: {{value}}%Critical threshold: {{threshold}}%RDS storage utilization has reached a critical level. Investigate storage growth or increase the allocated storage.--@webhook-macro-alert-hook{{/is_alert}}{{#is_warning}}@@ -93,8 +90,6 @@ Storage utilization: {{value}}%Warning threshold: {{warn_threshold}}%RDS storage utilization is approaching the critical threshold.--@webhook-macro-alert-hook{{/is_warning}}{{#is_recovery}}@@ -105,11 +100,9 @@ Engine: {{engine.name}}Storage utilization: {{value}}%RDS storage utilization has returned to an acceptable level.--@webhook-macro-alert-hook{{/is_recovery}}+${notify({ pages: true })}`,tags: ['integration:amazon_rds'],monitorThresholds: {critical: '90',@@ -120,7 +113,6 @@ RDS storage utilization has returned to an acceptable level.includeTags: true,newGroupDelay: 60,evaluationDelay: 900,- renotifyInterval: 0,requireFullWindow: false,});@@ -132,7 +124,7 @@ adopted('rds-connection-anomaly', {"avg(last_1d):anomalies(avg:aws.rds.database_connections{name:*-prod} by {dbinstanceidentifier}, 'basic', 2, direction='above', interval=300, alert_window='last_1h', count_default_zero='true') >= 0.99",message: `The number of connections for RDS database {{dbinstanceidentifier.name}} is outside of the typical range.-@slack-monitoring`,+${notify({ pages: false })}`,tags: ['integration:amazon_rds'],monitorThresholds: {critical: '0.99',@@ -147,7 +139,6 @@ adopted('rds-connection-anomaly', {includeTags: true,newGroupDelay: 60,evaluationDelay: 900,- renotifyInterval: 0,requireFullWindow: false,});@@ -157,8 +148,11 @@ adopted('rds-low-freeable-memory', {type: 'query alert',query:'avg(last_1h):avg:aws.rds.freeable_memory{environment:prod OR database_instance:macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com} by {database_instance} < 2000000000',- message: `@[email protected]+ // macrodb.ts and Grafana watch the same metric over a shorter window; this+ // one only notifies.+ message: `{{database_instance.name}} freeable memory is {{value}} bytes.++${notify({ pages: false })}`,monitorThresholds: {critical: '2000000000',warning: '3000000000',@@ -176,10 +170,9 @@ adopted('rds-high-iops', {type: 'query alert',query:'avg(last_1h):avg:aws.rds.write_iops{environment:prod OR database_instance:macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com} by {database_instance}.as_rate() + avg:aws.rds.read_iops{environment:prod OR database_instance:macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com} by {database_instance}.as_rate() > 10000',- message: `@[email protected]+ message: `{{database_instance.name}} read plus write IOPS are at {{value}}.-@webhook-alarm_webhook_test`,+${notify({ pages: false })}`,monitorThresholds: {critical: '10000',warning: '9000',diff --git a/infra/stacks/monitoring/monitors/redis.ts b/infra/stacks/monitoring/monitors/redis.tsnew file mode 100644index 0000000000..5165ac7afe--- /dev/null+++ b/infra/stacks/monitoring/monitors/redis.ts@@ -0,0 +1,44 @@+// ElastiCache memory for every prod cache, per node.+//+// macro-cache-prod climbed from 60% to 100% over a month before writes failed.+// Memory and evictions also exist in Grafana+// (infra/stacks/observability/nixos/alerting.nix); change both together.+// Grafana pages on-call for memory, so it only notifies here.+import { prodMonitor } from '../prod-monitor';++prodMonitor('redis-memory', {+ title: 'Redis memory high',+ service: 'elasticache',+ query:+ 'avg(last_10m):max:aws.elasticache.database_memory_usage_percentage{cacheclusterid:*prod*} by {cacheclusterid} > 90',+ monitorThresholds: { critical: '90', warning: '75' },+ pages: false,+ description:+ '{{cacheclusterid.name}} is using {{value}}% of maxmemory. Keys without a TTL are never evicted under volatile-lru, so at 100% writes fail.',+ runbook: 'docs/runbooks/redis.md#memory',+});++// Datadog only: Grafana alerting has no forecast function.+prodMonitor('redis-memory-forecast', {+ title: 'Redis memory forecast to fill within three days',+ service: 'elasticache',+ query:+ "max(next_3d):forecast(max:aws.elasticache.database_memory_usage_percentage{cacheclusterid:*prod*} by {cacheclusterid}, 'linear', 1, interval='60m', history='1w', model='default') >= 95",+ monitorThresholds: { critical: '95' },+ pages: false,+ description:+ '{{cacheclusterid.name}} memory is growing fast enough to pass 95% within three days. macro-cache-prod climbed from 60% to 100% over a month before writes failed.',+ runbook: 'docs/runbooks/redis.md#memory',+});++prodMonitor('redis-evictions', {+ title: 'Redis evicting keys',+ service: 'elasticache',+ query:+ 'sum(last_10m):sum:aws.elasticache.evictions{cacheclusterid:*prod*} by {cacheclusterid}.as_count() > 1000',+ monitorThresholds: { critical: '1000', warning: '0' },+ pages: false,+ description:+ '{{cacheclusterid.name}} evicted {{value}} keys in 10 minutes. Prod caches normally evict nothing.',+ runbook: 'docs/runbooks/redis.md#memory',+});diff --git a/infra/stacks/monitoring/monitors/regressions.ts b/infra/stacks/monitoring/monitors/regressions.tsnew file mode 100644index 0000000000..86ff6c09e8--- /dev/null+++ b/infra/stacks/monitoring/monitors/regressions.ts@@ -0,0 +1,88 @@+// Per-service error and latency anomalies: each compares a service with its+// own usual band, so it fires on a change rather than a fixed count.+//+// Datadog only: prod traces and logs don't reach Grafana, and Grafana alerting+// has no anomaly detection.+import * as datadog from '@pulumi/datadog';+import { prodMonitor } from '../prod-monitor';++// Datadog has no anomaly function for log monitors, so the error-logs anomaly+// monitor reads this log-based metric instead. Counts every prod error log+// line, tagged by service.+new datadog.LogsMetric('macro-logs-errors', {+ name: 'macro.logs.errors',+ compute: { aggregationType: 'count' },+ filter: { query: 'env:prod status:error' },+ groupBies: [{ path: 'service', tagName: 'service' }],+});++// An anomaly monitor recovers once no point in its window is outside the band.+const ANOMALY_THRESHOLDS = { critical: '1', criticalRecovery: '0' };+const ANOMALY_WINDOWS = {+ triggerWindow: 'last_30m',+ recoveryWindow: 'last_15m',+};++prodMonitor('apm-server-error-anomaly', {+ title: 'HTTP server errors abnormally high',+ service: 'apm',+ query:+ "avg(last_4h):anomalies(sum:trace.http.server.request.errors{env:prod} by {service}.as_count(), 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",+ monitorThresholds: ANOMALY_THRESHOLDS,+ monitorThresholdWindows: ANOMALY_WINDOWS,+ pages: false,+ description:+ '{{service.name}} is returning more HTTP server errors than its usual band. Check the latest deploy of that service first.',+ runbook: 'docs/runbooks/regressions.md#errors',+});++prodMonitor('apm-internal-error-anomaly', {+ title: 'Internal span errors abnormally high',+ service: 'apm',+ query:+ "avg(last_4h):anomalies(sum:trace.Internal.errors{env:prod} by {service}.as_count(), 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",+ monitorThresholds: ANOMALY_THRESHOLDS,+ monitorThresholdWindows: ANOMALY_WINDOWS,+ pages: false,+ description:+ '{{service.name}} is recording more failed internal spans (background jobs, workers, browser operations) than its usual band. Check the latest deploy of that service first.',+ runbook: 'docs/runbooks/regressions.md#errors',+});++prodMonitor('apm-consumer-error-anomaly', {+ title: 'Queue consumer errors abnormally high',+ service: 'apm',+ query:+ "avg(last_4h):anomalies(sum:trace.Consumer.errors{env:prod} by {service}.as_count(), 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",+ monitorThresholds: ANOMALY_THRESHOLDS,+ monitorThresholdWindows: ANOMALY_WINDOWS,+ pages: false,+ description:+ '{{service.name}} is failing more queue and Kafka messages than its usual band.',+ runbook: 'docs/runbooks/regressions.md#errors',+});++prodMonitor('apm-latency-anomaly', {+ title: 'HTTP latency abnormally high',+ service: 'apm',+ query:+ "avg(last_4h):anomalies(p95:trace.http.server.request{env:prod AND (service:cloud-storage-service OR service:authentication-service)} by {service}, 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",+ monitorThresholds: ANOMALY_THRESHOLDS,+ monitorThresholdWindows: ANOMALY_WINDOWS,+ pages: false,+ description: '{{service.name}} p95 HTTP latency is above its usual band.',+ runbook: 'docs/runbooks/regressions.md#latency',+});++prodMonitor('error-logs-anomaly', {+ title: 'Error logs abnormally high',+ service: 'logs',+ query:+ "avg(last_4h):anomalies(sum:macro.logs.errors{*} by {service}.as_count(), 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",+ monitorThresholds: ANOMALY_THRESHOLDS,+ monitorThresholdWindows: ANOMALY_WINDOWS,+ pages: false,+ description:+ '{{service.name}} is logging more errors than its usual band. Check the latest deploy of that service first.',+ runbook: 'docs/runbooks/regressions.md#errors',+});diff --git a/infra/stacks/monitoring/monitors/rum.ts b/infra/stacks/monitoring/monitors/rum.tsindex 4ed2f44b76..f636ac39d5 100644--- a/infra/stacks/monitoring/monitors/rum.ts+++ b/infra/stacks/monitoring/monitors/rum.ts@@ -3,6 +3,7 @@// Adopted from monitors created by hand in the Datadog UI. Do not edit a// monitor here and in the UI at the same time — this file wins on deploy.import { adopted } from '../adopted';+import { notify } from '../notify';// https://us5.datadoghq.com/monitors/1177082adopted('web-app-errors-staging', {@@ -10,7 +11,9 @@ adopted('web-app-errors-staging', {type: 'rum alert',query:'rum("@type:error @application.name:\\"Web App\\" env:production @view.url_host:staging.macro.com").rollup("count").last("5m") > 90',- message: '@slack-monitoring',+ message: `The staging web app recorded {{value}} errors in 5 minutes.++${notify({ pages: false })}`,monitorThresholds: {critical: '90',warning: '50',@@ -28,8 +31,7 @@ adopted('web-app-source-errors-prod', {query: 'formula("(1 - (query - query1) / query) * 100").last("1h") > 90',message: `{{value}} % of all views in Web App are throwing errors from source.-@slack-monitoring-{{#is_alert}}@slack-holy-shit-alarms {{/is_alert}}`,+${notify({ pages: true })}`,priority: '2',tags: ['prod'],monitorThresholds: {diff --git a/infra/stacks/monitoring/monitors/ses.ts b/infra/stacks/monitoring/monitors/ses.tsindex ce427ff29b..593e046c9b 100644--- a/infra/stacks/monitoring/monitors/ses.ts+++ b/infra/stacks/monitoring/monitors/ses.ts@@ -3,6 +3,7 @@// Adopted from monitors created by hand in the Datadog UI. Do not edit a// monitor here and in the UI at the same time — this file wins on deploy.import { adopted } from '../adopted';+import { notify } from '../notify';// https://us5.datadoghq.com/monitors/1333601adopted('ses-bounce-rate', {@@ -12,9 +13,7 @@ adopted('ses-bounce-rate', {'avg(last_1h):avg:aws.ses.reputation_bounce_rate{aws_account:569036502058} > 0.05',message: `SES Bounce Rate Increasing.-{{#is_alert}} @slack-holy-shit-alarms {{/is_alert}} {{#is_alert_recovery}} @slack-holy-shit-alarms {{/is_alert_recovery}}--@slack-monitoring`,+${notify({ pages: true })}`,monitorThresholds: {critical: '0.05',warning: '0.03',@@ -34,13 +33,7 @@ adopted('ses-complaint-rate', {'avg(last_1h):avg:aws.ses.reputation_complaint_rate{aws_account:569036502058} > 0.005',message: `SES Bounce Rate Increasing. Action needs to be taken to prevent account suspension.-{{#is_warning}}@slack-monitoring{{/is_warning}}-{{#is_warning_recovery}}@slack-monitoring {{/is_warning_recovery}}--{{#is_alert}}@slack-holy-shit-alarms {{/is_alert}}-{{#is_alert_recovery}}@slack-holy-shit-alarms @slack-monitoring {{/is_alert_recovery}}-+${notify({ pages: true })}`,monitorThresholds: {critical: '0.005',warning: '0.001',@@ -59,13 +52,7 @@ adopted('ses-high-email-sends', {query: 'sum(last_1h):avg:aws.ses.send.sum{*} by {aws_account} > 250',message: `SES is experiencing high email sends. This could be related to a potential attack and may require users to be blocked.-{{#is_warning}}@slack-monitoring{{/is_warning}}-{{#is_warning_recovery}}@slack-monitoring {{/is_warning_recovery}}--{{#is_alert}}@slack-holy-shit-alarms {{/is_alert}}-{{#is_alert_recovery}}@slack-holy-shit-alarms @slack-monitoring {{/is_alert_recovery}}-+${notify({ pages: true })}`,monitorThresholds: {critical: '250',warning: '200',diff --git a/infra/stacks/monitoring/monitors/whatsapp-session.ts b/infra/stacks/monitoring/monitors/whatsapp-session.tsindex 98136ff4a6..cfb48c45d9 100644--- a/infra/stacks/monitoring/monitors/whatsapp-session.ts+++ b/infra/stacks/monitoring/monitors/whatsapp-session.ts@@ -15,12 +15,13 @@// Ids are opaque account uuids; never phone numbers, JIDs, or content.// Thresholds are starting points: there is no baseline before launch.import * as datadog from '@pulumi/datadog';+import { notify, RENOTIFY_MINUTES } from '../notify';const CONTEXT = `Runbook: services/whatsapp_session_service/AGENTS.md. To stop allWhatsApp traffic, scale whatsapp-session-service to 0 tasks; sends recordedmeanwhile expire unsent after ten minutes. Turning off the enable-whatsapp-linkedflag only hides the app UI.-@webhook-macro-alert-hook`;+${notify({ pages: false })}`;const HEALTH = '@telemetry_event:whatsapp_session_health';@@ -32,6 +33,7 @@ for (const environment of ['dev', 'prod']) {priority: environment === 'prod' ? '2' : '3',includeTags: true,requireFullWindow: false,+ renotifyInterval: RENOTIFY_MINUTES,};new datadog.Monitor(`whatsapp-session-unowned-accounts-${environment}`, {diff --git a/infra/stacks/monitoring/notify.ts b/infra/stacks/monitoring/notify.tsnew file mode 100644index 0000000000..4a4380d61e--- /dev/null+++ b/infra/stacks/monitoring/notify.ts@@ -0,0 +1,47 @@+// Where every monitor in this stack notifies.+//+// All monitors post to the Macro Alerts channel. Paging monitors also open a+// GoAlert alert, which phones the on-call engineer, and close it on recovery.+// Datadog sends warning and alert notifications again every RENOTIFY_MINUTES+// while a monitor stays in that state, so an alert nobody fixed keeps showing up.++// Datadog webhook integration that posts into the Macro Alerts channel. It is+// configured in the Datadog UI, not in this stack.+export const MACRO_ALERTS = '@webhook-macro-alert-hook';+// Declared in goalert.ts.+export const GOALERT = '@webhook-goalert';+export const GOALERT_CLOSE = '@webhook-goalert-close';++export const RENOTIFY_MINUTES = 60;++/** The notification handles that end every monitor message. */+export function notify({ pages }: { pages: boolean }): string {+ if (!pages) return MACRO_ALERTS;+ // A recovery closes the page whether the monitor recovers from alert or from+ // a warning it dropped to first. Closing an alert GoAlert no longer has+ // open does nothing.+ return `{{#is_alert}}Paging on-call. ${GOALERT}{{/is_alert}}{{#is_recovery}}${GOALERT_CLOSE}{{/is_recovery}}+${MACRO_ALERTS}`;+}++/**+ * A monitor message: the state and value against its threshold, what it+ * means, where to start, and the notification handles.+ */+export function alertMessage({+ description,+ runbook,+ pages,+}: {+ description: string;+ runbook: string;+ pages: boolean;+}): string {+ return `{{#is_alert}}🔴 CRITICAL: {{value}} (threshold {{threshold}}){{/is_alert}}{{#is_warning}}🟠 WARNING: {{value}} (warning threshold {{warn_threshold}}){{/is_warning}}{{#is_recovery}}✅ RECOVERED: {{value}}{{/is_recovery}}++${description}++Runbook: https://github.com/macro-inc/macro/blob/main/${runbook}++${notify({ pages })}`;+}diff --git a/infra/stacks/monitoring/prod-monitor.ts b/infra/stacks/monitoring/prod-monitor.tsnew file mode 100644index 0000000000..eca574ef86--- /dev/null+++ b/infra/stacks/monitoring/prod-monitor.ts@@ -0,0 +1,49 @@+import * as datadog from '@pulumi/datadog';+import { goalertWebhooks } from './goalert';+import { alertMessage, RENOTIFY_MINUTES } from './notify';++/**+ * A prod monitor declared in this stack. The rules with a CloudWatch metric+ * also exist in Grafana (infra/stacks/observability/nixos/alerting.nix); keep+ * their thresholds and windows in sync there by hand.+ */+export function prodMonitor(+ name: string,+ args: {+ title: string;+ service: string;+ query: string;+ monitorThresholds: datadog.types.input.MonitorMonitorThresholds;+ /** Anomaly monitors only: the query's `alert_window`, and a recovery window. */+ monitorThresholdWindows?: datadog.types.input.MonitorMonitorThresholdWindows;+ /** A critical alert pages on-call through GoAlert. Warnings never page. */+ pages: boolean;+ description: string;+ /** Repository path with a heading anchor, e.g. `docs/runbooks/macrodb.md#memory`. */+ runbook: string;+ }+): datadog.Monitor {+ return new datadog.Monitor(+ name,+ {+ name: `[PROD] ${args.title}`,+ type: 'query alert',+ query: args.query,+ message: alertMessage({+ description: args.description,+ runbook: args.runbook,+ pages: args.pages,+ }),+ tags: ['env:prod', `service:${args.service}`],+ priority: args.pages ? '2' : '3',+ monitorThresholds: args.monitorThresholds,+ monitorThresholdWindows: args.monitorThresholdWindows,+ renotifyInterval: RENOTIFY_MINUTES,+ requireFullWindow: false,+ onMissingData: 'default',+ includeTags: true,+ notifyAudit: false,+ },+ { dependsOn: args.pages ? goalertWebhooks : [] }+ );+}diff --git a/infra/stacks/monitoring/routing.test.ts b/infra/stacks/monitoring/routing.test.tsnew file mode 100644index 0000000000..b140d1d5e0--- /dev/null+++ b/infra/stacks/monitoring/routing.test.ts@@ -0,0 +1,118 @@+import { beforeAll, expect, test } from 'bun:test';+import * as pulumi from '@pulumi/pulumi';++const resources: pulumi.runtime.MockResourceArgs[] = [];++beforeAll(async () => {+ pulumi.runtime.setAllConfig(+ { 'monitoring:goalertIntegrationKey': 'integration-key' },+ ['monitoring:goalertIntegrationKey']+ );+ pulumi.runtime.setMocks(+ {+ newResource(args) {+ resources.push(args);+ return { id: args.name, state: args.inputs };+ },+ call(args) {+ return args.inputs;+ },+ },+ 'monitoring',+ 'prod',+ false+ );+ await import('./index');+ await pulumi.runtime.waitForRPCs();+});++const monitors = () =>+ resources.filter((r) => r.type === 'datadog:index/monitor:Monitor');+const synthetics = () =>+ resources.filter(+ (r) => r.type === 'datadog:index/syntheticsTest:SyntheticsTest'+ );++test('every monitor and synthetic test notifies only the Macro Alerts channel and GoAlert', () => {+ const routed = [...monitors(), ...synthetics()];+ expect(routed.length).toBeGreaterThan(80);+ for (const resource of routed) {+ // Notification handles and email addresses; attribute names like `@error`+ // in message text are not handles.+ const handles: string[] =+ resource.inputs.message.match(+ /@(?:webhook|slack|pagerduty|opsgenie|teams)-[\w-]+|[\w.+-]+@[\w-]+\.[\w.]+/g+ ) ?? [];+ expect({ name: resource.name, handles }).toEqual({+ name: resource.name,+ handles: expect.arrayContaining(['@webhook-macro-alert-hook']),+ });+ for (const handle of handles) {+ expect([+ '@webhook-macro-alert-hook',+ '@webhook-goalert',+ '@webhook-goalert-close',+ ]).toContain(handle);+ }+ }+});++test('every monitor renotifies while it stays in alert', () => {+ for (const resource of monitors()) {+ expect({+ name: resource.name,+ renotify: resource.inputs.renotifyInterval,+ }).toEqual({+ name: resource.name,+ renotify: 60,+ });+ }+ for (const resource of synthetics()) {+ expect(resource.inputs.optionsList.monitorOptions.renotifyInterval).toBe(+ 60+ );+ }+});++test('paging prod monitors open and close a GoAlert alert; the rest only notify', () => {+ const blocked = monitors().find(+ (r) => r.name === 'macrodb-blocked-connections'+ );+ expect(blocked?.inputs.message).toContain(+ '{{#is_alert}}Paging on-call. @webhook-goalert{{/is_alert}}'+ );+ expect(blocked?.inputs.message).toContain(+ '{{#is_recovery}}@webhook-goalert-close{{/is_recovery}}'+ );+ expect(blocked?.inputs.priority).toBe('2');++ // Grafana pages for this one, so Datadog only notifies.+ const freeable = monitors().find(+ (r) => r.name === 'macrodb-primary-freeable-memory'+ );+ expect(freeable?.inputs.message).not.toContain('@webhook-goalert');++ const latency = monitors().find((r) => r.name === 'macrodb-read-latency');+ expect(latency?.inputs.message).not.toContain('@webhook-goalert');+ expect(latency?.inputs.priority).toBe('3');+});++test('the GoAlert health check does not page through GoAlert', () => {+ const health = synthetics().find((r) => r.name === 'goalert-health');+ expect(health?.inputs.message).not.toContain('@webhook-goalert');+ expect(health?.inputs.requestDefinition.url).toBe(+ 'https://oncall.macro-internal.com/health'+ );+});++test('anomaly rules recover at zero and carry their alert window', () => {+ const anomaly = monitors().find((r) => r.name === 'apm-server-error-anomaly');+ expect(anomaly?.inputs.monitorThresholds).toEqual({+ critical: '1',+ criticalRecovery: '0',+ });+ expect(anomaly?.inputs.monitorThresholdWindows).toEqual({+ triggerWindow: 'last_30m',+ recoveryWindow: 'last_15m',+ });+});diff --git a/infra/stacks/monitoring/synthetics.ts b/infra/stacks/monitoring/synthetics.tsindex 9738e72f4f..cdd721a588 100644--- a/infra/stacks/monitoring/synthetics.ts+++ b/infra/stacks/monitoring/synthetics.ts@@ -1,4 +1,6 @@import * as datadog from '@pulumi/datadog';+import { goalertWebhooks } from './goalert';+import { notify, RENOTIFY_MINUTES } from './notify';// The only external probe of the product. It owns its own alert monitor// (19726068, "Macro App is Down") through `optionsList.monitorName` — that@@ -11,8 +13,8 @@ import * as datadog from '@pulumi/datadog';// here.//// `message` below is Datadog's default synthetics notification template,-// reproduced verbatim from the live test. Backticks are escaped for the-// template literal; the rendered text is unchanged.+// reproduced from the live test with its notification handles replaced by+// `notify`. Backticks are escaped for the template literal.export const appWelcomeHealthCheck = new datadog.SyntheticsTest('app-welcome-health-check',{@@ -43,12 +45,13 @@ export const appWelcomeHealthCheck = new datadog.SyntheticsTest(httpVersion: 'any',retry: { count: 0, interval: 300 },monitorName: 'Macro App is Down',- monitorOptions: { notificationPresetName: 'show_all' },+ monitorOptions: {+ notificationPresetName: 'show_all',+ renotifyInterval: RENOTIFY_MINUTES,+ },},message: `{{! Test result details }}-@slack-holy-shit-alarms @[email protected] @[email protected]-Your test {{#is_alert}}failed{{else}}recovered{{/is_alert}} after running for {{eval "synthetics.attributes.result.duration/1000" }}s on the {{#if synthetics.attributes.location.privateLocation}}Private{{else}}Managed{{/if}} Location {{synthetics.attributes.location.id}}.{{! If alert, provide details about the failure }}@@ -98,7 +101,9 @@ The test successfully extracted the following variables:{{#each synthetics.attributes.result.variables.extracted}}* **Name:** \`{{name}}\` (Global Variable ID: \`{{id}}\`)Value: {{#if secure}}*Obfuscated (value hidden)*{{else}}\`{{{val}}}\`{{/if}}{{/each}}-{{/if}}`,+{{/if}}++${notify({ pages: true })}`,},- { protect: true }+ { protect: true, dependsOn: goalertWebhooks });diff --git a/infra/stacks/observability/AGENTS.md b/infra/stacks/observability/AGENTS.mdnew file mode 100644index 0000000000..d995967ba9--- /dev/null+++ b/infra/stacks/observability/AGENTS.md@@ -0,0 +1,48 @@+# observability++## What it is++The Ohio (us-east-2) Grafana host: Grafana, Loki, Tempo, Prometheus, Alloy and+the GoAlert on-call pager on one NixOS EC2 instance, separate from production in+us-east-1. `README.md` is the full operations guide.++## Start here++- `nixos/host.nix`: boot order, retained volume, stable uids, service hardening.+- `nixos/services.nix`, `nixos/goalert.nix`: the systemd services and credentials.+- `index.ts`: VPC, ALB, certificate, DNS, alarms and EC2 user data.++## Recipes++- Change a Grafana prod alert rule: edit `prodRules` in `nixos/alerting.nix`, and+ its Datadog copy in `infra/stacks/monitoring/monitors/` by hand.+- Add a runtime setting: `settings.ts` → `render.ts` (bump the schema version) →+ `nixos/read-user-data.py` (keys, pattern, version) → the consuming `.nix` file.+- Add a secret: `assets/refresh-secrets.py` (validation) → `LoadCredential` in+ the unit → README secret list. Add the unit to `secretConsumers` in `host.nix`.+- Upgrade GoAlert: bump `version` and the hashes in `nixos/goalert/package.nix`.++## Checks++From this directory:+- `nix build --dry-run .#nixosConfigurations.observability.config.system.build.toplevel`+- `nix fmt -- <changed .nix files>`; `python3 -m py_compile` on changed Python.++From `infra/`: `bun run check`, then Biome on changed `.ts`.++## Gotchas++- Changes ship only as a new AMI plus an approved deploy; never `pulumi up` or+ publish an image without explicit permission.+- The host reads secrets from the stack's own Secrets Manager secret in+ us-east-2, so it never depends on production to boot. Doppler+ `observability/prd` holds the values; copy changes into the secret.+- A service listed in `host.nix` `services` gets `ProtectSystem=strict`, metadata+ blocking and the bootstrap dependency; one that needs the instance role must be+ in the `IPAddressDeny` exception list or run as its own unit.+- GoAlert's application config lives in its database; `goalert/apply-config.py`+ overwrites it on every start from `goalert.nix`. `set-config` accepts invalid+ values silently, so check the journal for `validate config`.+- Grafana provisioning expands `$VAR` in contact point settings, but not in rule+ queries or annotations, so `$B` and `$values` work there and secrets must come+ in through the Grafana unit's environment.diff --git a/infra/stacks/observability/CLAUDE.md b/infra/stacks/observability/CLAUDE.mdnew file mode 120000index 0000000000..47dc3e3d86--- /dev/null+++ b/infra/stacks/observability/CLAUDE.md@@ -0,0 +1 @@+AGENTS.md\ No newline at end of filediff --git a/infra/stacks/observability/Pulumi.prod.yaml b/infra/stacks/observability/Pulumi.prod.yamlindex aff03f765a..e233257080 100644--- a/infra/stacks/observability/Pulumi.prod.yaml+++ b/infra/stacks/observability/Pulumi.prod.yaml@@ -2,5 +2,5 @@ config:aws:region: us-east-2observability:ciIngestSecretArn: arn:aws:secretsmanager:us-east-1:569036502058:secret:observability/dev-ingest-7UhJbLobservability:alarmTopicArn: arn:aws:sns:us-east-2:569036502058:observability-prod-alerts- observability:amiId: ami-0e88a39db857c8c52+ observability:amiId: ami-0e775059ad164977cobservability:secretArn: arn:aws:secretsmanager:us-east-2:569036502058:secret:observability/prod-TwQpBJdiff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.mdindex d51de3801d..43fd8ccde9 100644--- a/infra/stacks/observability/README.md+++ b/infra/stacks/observability/README.md@@ -62,7 +62,8 @@ historical backfill, or CI alert notifications. Validate a successful and failedworkflow, a rerun, and a multi-job trace after rollout before relying on the dashboard.This stack starts Grafana alongside Datadog on one private EC2 instance. It-runs Grafana, Loki, Tempo, Prometheus, Alloy and nginx as native systemd services+runs Grafana, Loki, Tempo, Prometheus, Alloy, nginx and the GoAlert on-call+pager (with its own Postgres) as native systemd serviceson a prebuilt NixOS image. There is no Docker or Compose dependency. Nix ownspackages, configuration, service users, resource limits and startup dependencies:@@ -73,6 +74,9 @@ packages, configuration, service users, resource limits and startup dependencies- `nixos/host-telemetry.nix` and `nixos/host.alloy`: host metrics and journal logs.- `nixos/cloudwatch.nix`: independent disk and memory health metrics.- `nixos/proxy.nix`: native nginx virtual hosts and query-only backend gateway.+- `nixos/goalert.nix` and `nixos/goalert/`: GoAlert, its Postgres, backups and+ the Macro Alerts relay (see [On-call paging](#on-call-paging-goalert)).+- `nixos/alerting.nix`: Grafana alert rules, contact points and routing.- `nixos/host.nix`: boot dependencies, retained storage and stable service identities.NixOS modules generate application configuration from Nix attributes. Alloy's@@ -90,6 +94,7 @@ OTEL exporter -- HTTPS + bearer token --> ALB --> nginx --> Alloy--> Loki --> S3 logs--> Tempo --> S3 traces--> Prometheus --> EBS+Datadog, on-call browsers -- HTTPS --> ALB --> nginx --> GoAlert --> Postgres --> EBS```- One `m7i.xlarge` (4 vCPU / 16 GiB) in a new VPC's private `us-east-2a`@@ -111,7 +116,8 @@ OTEL exporter -- HTTPS + bearer token --> ALB --> nginx --> Alloyan application-consistent or cross-region backup. Snapshot recovery can loseup to a day of local state. S3 does not protect telemetry still buffered locally.- No RDS, shared filesystem, Kafka or cluster. Grafana's SQLite stores users,- dashboards and settings, not log/trace/metric history.+ dashboards and settings, not log/trace/metric history. GoAlert's schedules,+ users and alerts live in a local Postgres 17 on the same volume.- All service and OS versions are pinned by this stack's `flake.lock` on NixOS26.05. Grafana plugin auto-install/update is disabled. Reviewed lock updatesand replacement AMIs deliver upgrades; there is no installation or build at boot.@@ -230,10 +236,13 @@ DNS-validated regional ACM certificate. Region validation rejects `us-east-1`.and secrets per environment. Confirm Workspace MFA enforcement.2. Through the approved secret-management process, create a Secrets Manager JSONsecret in **us-east-2** containing `google_client_id`, `google_client_secret`,- `grafana_secret_key`, and `otlp_token`. Generate independent cryptographically- random values of at least 32 characters for the last two. Preserve- `grafana_secret_key` through recovery; changing it can make stored credentials- unreadable. Never paste secret values into shell commands, this file or PRs.+ `grafana_secret_key`, and `otlp_token`, plus the GoAlert fields listed under+ [On-call paging](#on-call-paging-goalert). Generate independent cryptographically+ random values of at least 32 characters for `grafana_secret_key`, `otlp_token`+ and `goalert_encryption_key`. Preserve `grafana_secret_key` and+ `goalert_encryption_key` through recovery; changing them makes stored+ credentials unreadable. Never paste secret values into shell commands, this+ file or PRs.3. Select an existing **us-east-2** SNS topicwith a confirmed, monitored subscription for infrastructure alarms. Itsdelivery destination must remain accessible during a production outage.@@ -426,8 +435,24 @@ models. Grafana uses the EC2 role for regional metric reads and publishing to thconfigured alarm SNS topic. Other unprivileged HTTP/ingest services still cannotaccess instance metadata.-`nixos/alerting.nix` starts with the three dev monitors declared in the monitoring-stack: DSS errors (1013923), channel-invite spam (4876245), and document text+`prodRules` in `nixos/alerting.nix` holds the prod CloudWatch rules: macrodb+memory, swap, load, CPU, connections, replica lag, read latency and storage;+Redis memory and evictions; SQS queue age. Each is a warning and a critical rule+in the `Prod` group, reading CloudWatch in Virginia directly. Each has a Datadog+copy in `infra/stacks/monitoring/monitors/`; keep thresholds and windows in sync+by hand. Forecasts, anomalies, Database Monitoring, APM and log rules are+Datadog-only.++Every alert posts to the `Macro Alerts` contact point through the loopback+relay; critical ones renotify hourly, others every four hours. Critical alerts+of rules with `pages = true` also go to the `GoAlert` contact point over+loopback. GoAlert owns the alert from there: it opens, acknowledges and closes+it, and its escalation posts each change to the channel. Their Datadog copies+only notify, so an incident pages once.++The dev rules below keep their own route to `Macro team email`.++The three dev monitors declared in the monitoring stack: DSS errors (1013923), channel-invite spam (4876245), and document textextractor failure ratio (1291772). Warning and critical rules are separate andmutually exclusive. Their windows and thresholds come from those definitions;the Lambda ratio sums four hours ending fifteen minutes ago. No-data is OK for@@ -447,8 +472,82 @@ without SMTP credentials. The configured topic already has [email protected] and[email protected] subscriptions; each recipient must confirm their SNS subscription.Test the contact point and verify receipt before relying on notifications.-This release adds `alarmTopicArn` to user-data schema 5. Deploy it together with-the new reviewed AMI; a schema-4 image cannot boot schema-5 user data.+User-data schema 6 adds `oncallHost`. Deploy it together with the new reviewed+AMI; a schema-5 image cannot boot schema-6 user data.++## On-call paging (GoAlert)++GoAlert is the source of truth for open alerts: Grafana sends it critical+alerts, and on-call engineers acknowledge or close them in its UI at+`https://oncall.macro-internal.com`. It runs on this host so paging keeps working+while production, its database or Macro itself is down. Today it pages by+posting to the Macro Alerts channel; phone paging (Twilio), Google login and a+rotation are follow-ups.++- `nixos/goalert/package.nix` builds GoAlert from source with `buildGoModule`,+ embedding its web UI. Bump `version` and the hashes together.+- `nixos/goalert.nix` runs Postgres 17 on a Unix socket only (data in+ `/srv/observability/postgres`), and GoAlert on `127.0.0.1:3030`. Its+ application config is a Nix attrset. GoAlert stores that config only in its+ database, so `apply-config.py` migrates the schema, runs `goalert set-config`+ and creates the `admin` login before every start. Edits in GoAlert's admin+ config page are overwritten on the next restart. `set-config` stores invalid+ values without failing; GoAlert logs `validate config` errors when it loads+ them, so check the journal after changing config.+- A nightly timer dumps the database to `s3://<logs bucket>/goalert/`, one object+ per weekday. The dump goes to a file first, so a failed dump never replaces a+ good backup.+- `macro-alerts-relay` on `127.0.0.1:8090` turns GoAlert (`/goalert`) and Grafana+ (`/grafana`) webhooks into Macro channel posts and adds the channel's bot token.+ GoAlert webhooks cannot set headers, and the token stays in one service.+- nginx serves `oncall.macro-internal.com` from a runtime server block. Only+ `/health`, `/api/v2/generic/incoming` and `/api/v2/heartbeat/` are reachable+ without a session (GoAlert checks the integration key); other inbound+ integrations return 404. The rest is GoAlert's UI behind its login.+- The ALB sends the on-call host to its own target group, health-checked at+ `/goalert-healthz`. Alarm `observability-goalert` publishes to `alarmTopicArn`+ when it fails, so a GoAlert outage is reported without GoAlert.++### Secrets++Doppler `observability/prd` is the source of truth; copy the values into the+us-east-2 Secrets Manager secret the host reads, under the lowercase names:+`goalert_encryption_key` (≥32 random characters), `goalert_admin_password`+(≥32 random characters, the `admin` login), `macro_alerts_webhook_url` (HTTPS)+and `macro_alerts_bot_token` (the Alerts channel's webhook and bot token).+`goalert_grafana_integration_key` is optional until GoAlert has created it;+without it Grafana's pages are rejected with 401 and the GoAlert contact point+shows as failing.++### First deployment++1. Add the secrets above, then build, publish and deploy the AMI as usual. The+ certificate gains the on-call name, so Pulumi replaces the ACM certificate and+ updates the listener.+2. Sign in as `admin`. Create the escalation policy and service below, and a+ Grafana integration key on `Grafana critical`. Put the key in Doppler and the+ secret as `goalert_grafana_integration_key` and restart+ `observability-secrets`.+3. Send a test alert through the service and check that it reaches the channel+ and can be acknowledged and closed.++### Escalation++- Escalation policy `Macro prod`, one step: webhook+ `http://127.0.0.1:8090/goalert`. The channel sees each alert, acknowledgement+ and close.+- Service `Grafana critical` on `Macro prod`. Warnings never page; they only+ reach the channel.+- Acknowledge an alert to stop its escalation, close it when handled; Grafana+ closes it on recovery. To silence a whole service for a while, use its+ maintenance mode.++### Recovery++GoAlert's state is on the retained volume, so a replacement instance resumes it.+For a lost volume, restore the latest dump with+`pg_restore --clean -d goalert` before starting `goalert`, using the same+`goalert_encryption_key`.## Dev dual exportdiff --git a/infra/stacks/observability/assets/refresh-secrets.py b/infra/stacks/observability/assets/refresh-secrets.pyindex dde22d99f0..2e4b7e2951 100644--- a/infra/stacks/observability/assets/refresh-secrets.py+++ b/infra/stacks/observability/assets/refresh-secrets.py@@ -3,6 +3,7 @@import jsonimport osfrom pathlib import Path+import reimport subprocessconfig = json.loads(Path('/opt/observability/bootstrap.json').read_text())@@ -12,7 +13,11 @@ result = subprocess.run(check=True, capture_output=True, text=True,)values = json.loads(json.loads(result.stdout)['SecretString'])-keys = ['google_client_id', 'google_client_secret', 'grafana_secret_key', 'otlp_token']+keys = [+ 'google_client_id', 'google_client_secret', 'grafana_secret_key', 'otlp_token',+ 'goalert_encryption_key', 'goalert_admin_password',+ 'macro_alerts_webhook_url', 'macro_alerts_bot_token',+]# Validate the entire bundle before writing any file. Token is at least 32 bytes.for key in keys:value = values.get(key)@@ -20,16 +25,27 @@ for key in keys:raise SystemExit(f'Missing or invalid secret field: {key}')if '\n' in value or '\r' in value:raise SystemExit(f'Secret must be a single line: {key}')-for key in ['grafana_secret_key', 'otlp_token']:+for key in ['grafana_secret_key', 'otlp_token', 'goalert_encryption_key', 'goalert_admin_password']:if len(values[key]) < 32:raise SystemExit(f'Secret too short: {key}')+if not values['macro_alerts_webhook_url'].startswith('https://'):+ raise SystemExit('macro_alerts_webhook_url must use HTTPS')+# GoAlert creates its Grafana integration key, so it can only be added after the+# first boot. Until then GoAlert rejects Grafana's pages with 401, and Grafana+# reports the GoAlert contact point as failing.+grafana_key = values.get('goalert_grafana_integration_key', '')+if not re.fullmatch(r'[0-9a-f-]{36}|', grafana_key):+ raise SystemExit('goalert_grafana_integration_key must be a GoAlert key UUID')+if not grafana_key:+ print('goalert_grafana_integration_key is not set; Grafana cannot page through GoAlert')+values['grafana-goalert.env'] = f'GOALERT_GRAFANA_KEY={grafana_key}\n'# Native remote-write uses HTTP Basic over TLS. Only the password hash reaches nginx.hashed = subprocess.run(['openssl', 'passwd', '-6', '-stdin'], input=values['otlp_token'] + '\n',check=True, capture_output=True, text=True,).stdout.strip()values['metrics.htpasswd'] = 'metrics:' + hashed + '\n'-keys.append('metrics.htpasswd')+keys += ['metrics.htpasswd', 'grafana-goalert.env']directory = Path('/run/macro-observability')directory.mkdir(mode=0o700, exist_ok=True)os.chown(directory, 0, 0)diff --git a/infra/stacks/observability/index.ts b/infra/stacks/observability/index.tsindex b9e110520e..def1a85f02 100644--- a/infra/stacks/observability/index.ts+++ b/infra/stacks/observability/index.ts@@ -20,6 +20,7 @@ const baseDomain = 'macro-internal.com';const suffix = stack === 'prod' ? '' : '-dev';const grafanaHost = `grafana${suffix}.${baseDomain}`;const otlpHost = `otlp${suffix}.${baseDomain}`;+const oncallHost = `oncall${suffix}.${baseDomain}`;const tags = { project: 'observability', environment: stack };const durable = { protect: true, retainOnDelete: true };const secretArn = config.require('secretArn');@@ -34,6 +35,7 @@ validateSettings({region,grafanaHost,otlpHost,+ oncallHost,secretArn,alarmTopicArn,volumeId: 'vol-0',@@ -59,24 +61,26 @@ const zone = aws.route53.getZoneOutput({});const certificate = new aws.acm.Certificate('observability', {domainName: grafanaHost,- subjectAlternativeNames: [otlpHost],+ subjectAlternativeNames: [otlpHost, oncallHost],validationMethod: 'DNS',tags,});-const validationRecords = [grafanaHost, otlpHost].map((hostname) => {- const option = certificate.domainValidationOptions.apply((options) => {- const match = options.find((entry) => entry.domainName === hostname);- if (!match) throw new Error(`Missing ACM validation for ${hostname}`);- return match;- });- return new aws.route53.Record(`observability-certificate-${hostname}`, {- zoneId: zone.zoneId,- name: option.resourceRecordName,- type: option.resourceRecordType,- records: [option.resourceRecordValue],- ttl: 300,- });-});+const validationRecords = [grafanaHost, otlpHost, oncallHost].map(+ (hostname) => {+ const option = certificate.domainValidationOptions.apply((options) => {+ const match = options.find((entry) => entry.domainName === hostname);+ if (!match) throw new Error(`Missing ACM validation for ${hostname}`);+ return match;+ });+ return new aws.route53.Record(`observability-certificate-${hostname}`, {+ zoneId: zone.zoneId,+ name: option.resourceRecordName,+ type: option.resourceRecordType,+ records: [option.resourceRecordValue],+ ttl: 300,+ });+ }+);const validatedCertificate = new aws.acm.CertificateValidation('observability',{@@ -281,6 +285,7 @@ const instance = new aws.ec2.Instance(region,grafanaHost,otlpHost,+ oncallHost,secretArn,alarmTopicArn,volumeId,@@ -346,7 +351,29 @@ new aws.lb.ListenerRule('observability-hosts', {conditions: [{ hostHeader: { values: [grafanaHost, otlpHost] } }],actions: [{ type: 'forward', targetGroupArn: target.arn }],});-for (const hostname of [grafanaHost, otlpHost]) {+// Same nginx port, but its own health check so a GoAlert outage alarms+// separately from Grafana.+const oncallTarget = new aws.lb.TargetGroup('observability-oncall', {+ port: 8080,+ protocol: 'HTTP',+ targetType: 'instance',+ vpcId: vpc.vpcId,+ healthCheck: { path: '/goalert-healthz', matcher: '200', interval: 30 },+ deregistrationDelay: 30,+ tags,+});+new aws.lb.TargetGroupAttachment('observability-oncall', {+ targetGroupArn: oncallTarget.arn,+ targetId: instance.id,+ port: 8080,+});+new aws.lb.ListenerRule('observability-oncall', {+ listenerArn: listener.arn,+ priority: 2,+ conditions: [{ hostHeader: { values: [oncallHost] } }],+ actions: [{ type: 'forward', targetGroupArn: oncallTarget.arn }],+});+for (const hostname of [grafanaHost, otlpHost, oncallHost]) {new aws.route53.Record(hostname, {zoneId: zone.zoneId,name: hostname,@@ -418,6 +445,20 @@ new aws.cloudwatch.MetricAlarm('observability-ui', {dimensions: { LoadBalancer: alb.arnSuffix, TargetGroup: target.arnSuffix },treatMissingData: 'breaching',});+// Paging must not depend on GoAlert to report that GoAlert is down. On-call+// phones subscribe to this topic by SMS directly (see README).+new aws.cloudwatch.MetricAlarm('observability-goalert', {+ ...alarm,+ alarmDescription: `GoAlert at ${oncallHost} is failing health checks; pages are not being sent.`,+ namespace: 'AWS/ApplicationELB',+ metricName: 'UnHealthyHostCount',+ threshold: 0,+ dimensions: {+ LoadBalancer: alb.arnSuffix,+ TargetGroup: oncallTarget.arnSuffix,+ },+ treatMissingData: 'breaching',+});for (const [name, path] of [['observability-disk', '/srv/observability'],['observability-root-disk', '/'],@@ -444,6 +485,7 @@ new aws.cloudwatch.MetricAlarm('observability-memory', {export const grafanaUrl = `https://${grafanaHost}`;export const otlpHttpEndpoint = `https://${otlpHost}`;+export const goalertUrl = `https://${oncallHost}`;export const instanceId = instance.id;export const dataVolumeId = data.id;export const logsBucket = logs.bucket;diff --git a/infra/stacks/observability/nixos/alerting.nix b/infra/stacks/observability/nixos/alerting.nixindex dacf000b2e..e761fa4a35 100644--- a/infra/stacks/observability/nixos/alerting.nix+++ b/infra/stacks/observability/nixos/alerting.nix@@ -1,4 +1,4 @@-{ ... }:+{ lib, ... }:letlokiQuery = expr: {refId = "A";@@ -33,36 +33,527 @@ letdatadog_url = "https://us5.datadoghq.com/monitors/${monitorId}";description = "Migrated threshold and window; Datadog remains active during validation.";};- data = query ++ [- {+ data = query ++ expressions "last" condition;+ };+ # Reduce query A to one value per series (B), then compare it (C).+ expressions = reducer: condition: [+ {+ refId = "B";+ datasourceUid = "__expr__";+ relativeTimeRange = {+ from = 0;+ to = 0;+ };+ model = {refId = "B";- datasourceUid = "__expr__";- relativeTimeRange = {- from = 0;- to = 0;- };- model = {- refId = "B";- type = "reduce";- expression = "A";- reducer = "last";- };- }- {+ type = "reduce";+ expression = "A";+ inherit reducer;+ };+ }+ {+ refId = "C";+ datasourceUid = "__expr__";+ relativeTimeRange = {+ from = 0;+ to = 0;+ };+ model = {refId = "C";- datasourceUid = "__expr__";+ type = "math";+ expression = condition;+ };+ }+ ];++ # Prod rules. Each has a matching Datadog monitor in+ # infra/stacks/monitoring/monitors/{macrodb,redis,queues}.ts; keep thresholds+ # and windows in sync by hand. Grafana pages for the rules marked `pages`, so+ # their Datadog copies only notify.+ cloudwatch =+ {+ namespace,+ metricName,+ dimensions,+ statistic,+ }:+ window: [+ {+ refId = "A";+ datasourceUid = "cloudwatch";relativeTimeRange = {- from = 0;+ from = window;to = 0;};model = {- refId = "C";- type = "math";- expression = condition;+ refId = "A";+ id = "a";+ region = "us-east-1";+ inherit+ namespace+ metricName+ dimensions+ statistic+ ;+ period = "60";+ matchExact = true;+ metricQueryType = 0;+ metricEditorMode = 0;+ queryMode = "Metrics";};}];+ # `window` is in seconds; `condition` compares the reduced value $B.+ prodRule =+ {+ uid,+ title,+ severity,+ service,+ pages ? false,+ metric,+ window,+ reducer,+ condition,+ description,+ runbook,+ }:+ {+ inherit uid;+ title = "[PROD] ${title} (${severity})";+ condition = "C";+ for = "0s";+ # Sparse CloudWatch counters have no datapoints when idle.+ noDataState = "OK";+ execErrState = "Error";+ isPaused = false;+ labels = {+ environment = "prod";+ inherit severity service;+ # Only critical alerts page, through GoAlert.+ pages = lib.boolToString (pages && severity == "critical");+ };+ annotations = {+ summary = title;+ inherit description;+ runbook_url = "https://github.com/macro-inc/macro/blob/main/docs/runbooks/${runbook}";+ };+ data = cloudwatch metric window ++ expressions reducer condition;+ };++ rds = metricName: instances: statistic: {+ namespace = "AWS/RDS";+ inherit metricName statistic;+ dimensions.DBInstanceIdentifier = instances;+ };+ bothInstances = [+ "macro-db-prod"+ "macro-db-prod-read-replica"+ ];+ redisNodes = [+ "macro-cache-prod-001"+ "macro-cache-prod-002"+ "email-service-redis-prod-c862c10"+ "calendar-service-redis-prod-aee80d2"+ "connection-gateway-redis-prod-f9d8cd0"+ ];+ redis = metricName: statistic: {+ namespace = "AWS/ElastiCache";+ inherit metricName statistic;+ dimensions.CacheClusterId = redisNodes;+ };+ queueAge = queues: {+ namespace = "AWS/SQS";+ metricName = "ApproximateAgeOfOldestMessage";+ statistic = "Maximum";+ dimensions.QueueName = queues;+ };+ workQueues = [+ "email-service-refresh-queue-prod"+ "convert-service-queue-prod"+ "ai-projection-queue-prod"+ "email-service-gmail-webhook-queue-prod"+ "notification-ingress-queue-prod"+ ];++ primaryMemory = {+ service = "macro-db-prod";+ pages = true;+ metric = rds "FreeableMemory" [ "macro-db-prod" ] "Average";+ window = 900;+ reducer = "mean";+ description = "Freeable memory is {{ $values.B }} bytes. Normal is 7-18 GB; it fell to about 0.45 GB during the October overload. Each connection holds about 14 MB of private memory, so check connection count and swap first.";+ runbook = "macrodb.md#memory";+ };+ replicaMemory = {+ service = "macro-db-prod";+ pages = true;+ metric = rds "FreeableMemory" [ "macro-db-prod-read-replica" ] "Average";+ window = 900;+ reducer = "mean";+ description = "Replica freeable memory is {{ $values.B }} bytes. Normal minimum is about 3 GB. The replica serves GraphQL reads, so memory pressure shows up as slow lists.";+ runbook = "macrodb.md#memory";+ };+ swap = {+ service = "macro-db-prod";+ pages = true;+ metric = rds "SwapUsage" bothInstances "Maximum";+ window = 900;+ reducer = "mean";+ description = "{{ $labels.DBInstanceIdentifier }} is using {{ $values.B }} bytes of swap. Normal is under 0.6 GB. Swap above 5 GB preceded both October overloads by hours.";+ runbook = "macrodb.md#memory";+ };+ load = {+ service = "macro-db-prod";+ pages = true;+ metric = rds "DBLoadRelativeToNumVCPUs" bothInstances "Average";+ window = 900;+ reducer = "mean";+ description = "{{ $labels.DBInstanceIdentifier }} has {{ $values.B }}x as many active sessions as vCPUs. Normal is under 1.3; the October overload reached 8. Look at Performance Insights top waits (locks, IO) and the services holding the most connections.";+ runbook = "macrodb.md#load";+ };+ cpu = {+ service = "macro-db-prod";+ pages = true;+ metric = rds "CPUUtilization" bothInstances "Average";+ window = 900;+ reducer = "mean";+ description = "{{ $labels.DBInstanceIdentifier }} CPU averaged {{ $values.B }}% over 15 minutes. Normal peaks are about 30%.";+ runbook = "macrodb.md#load";+ };+ primaryConnections = {+ service = "macro-db-prod";+ pages = true;+ metric = rds "DatabaseConnections" [ "macro-db-prod" ] "Maximum";+ window = 600;+ reducer = "mean";+ description = "macro-db-prod has {{ $values.B }} connections. Normal 15-minute peaks are about 1,100; max_connections is 3,439. Each connection costs about 14 MB, so this also drains freeable memory.";+ runbook = "macrodb.md#connections";+ };+ replicaConnections = {+ service = "macro-db-prod";+ metric = rds "DatabaseConnections" [ "macro-db-prod-read-replica" ] "Maximum";+ window = 600;+ reducer = "mean";+ description = "The read replica has {{ $values.B }} connections. Normal peaks are about 460.";+ runbook = "macrodb.md#connections";};+ replicaLag = {+ service = "macro-db-prod";+ pages = true;+ metric = rds "ReplicaLag" [ "macro-db-prod-read-replica" ] "Maximum";+ window = 600;+ reducer = "mean";+ description = "The read replica is {{ $values.B }} seconds behind the primary. Normal is under 6 seconds. GraphQL reads use the replica, so users see stale data and missing new items.";+ runbook = "macrodb.md#replica-lag";+ };+ readLatency = {+ service = "macro-db-prod";+ metric = rds "ReadLatency" bothInstances "Average";+ window = 900;+ reducer = "mean";+ description = "{{ $labels.DBInstanceIdentifier }} disk reads take {{ $values.B }} seconds on average. Normal is about 0.6 ms.";+ runbook = "macrodb.md#load";+ };+ freeStorage = {+ service = "macro-db-prod";+ pages = true;+ metric = rds "FreeStorageSpace" bothInstances "Minimum";+ window = 3600;+ reducer = "min";+ description = "{{ $labels.DBInstanceIdentifier }} has {{ $values.B }} bytes of free storage out of 5.37 TB.";+ runbook = "macrodb.md#storage";+ };+ redisMemory = {+ service = "elasticache";+ pages = true;+ metric = redis "DatabaseMemoryUsagePercentage" "Maximum";+ window = 600;+ reducer = "mean";+ description = "{{ $labels.CacheClusterId }} is using {{ $values.B }}% of maxmemory. Keys without a TTL are never evicted under volatile-lru, so at 100% writes fail.";+ runbook = "redis.md#memory";+ };+ redisEvictions = {+ service = "elasticache";+ metric = redis "Evictions" "Sum";+ window = 600;+ reducer = "sum";+ description = "{{ $labels.CacheClusterId }} evicted {{ $values.B }} keys in 10 minutes. Prod caches normally evict nothing.";+ runbook = "redis.md#memory";+ };+ backfillAge = {+ service = "email-service";+ metric = queueAge [ "email-service-backfill-queue-prod" ];+ window = 900;+ reducer = "mean";+ description = "The oldest message in email-service-backfill-queue-prod is {{ $values.B }} seconds old. Normal peaks are about 50 minutes; new users wait this long for their mailbox to fill.";+ runbook = "queues.md#backlog";+ };+ workQueueAge = {+ service = "sqs";+ pages = true;+ metric = queueAge workQueues;+ window = 900;+ reducer = "mean";+ description = "The oldest message in {{ $labels.QueueName }} is {{ $values.B }} seconds old. These queues normally drain within minutes.";+ runbook = "queues.md#backlog";+ };++ # Warning and critical are mutually exclusive bands, as in the dev rules.+ prodRules = map prodRule [+ (+ primaryMemory+ // {+ uid = "macrodb-primary-freeable-memory-crit";+ title = "macro-db-prod freeable memory low";+ severity = "critical";+ condition = "$B < 1000000000";+ }+ )+ (+ primaryMemory+ // {+ uid = "macrodb-primary-freeable-memory-warn";+ title = "macro-db-prod freeable memory low";+ severity = "warning";+ condition = "$B < 2000000000 && $B >= 1000000000";+ }+ )+ (+ replicaMemory+ // {+ uid = "macrodb-replica-freeable-memory-crit";+ title = "macro-db-prod-read-replica freeable memory low";+ severity = "critical";+ condition = "$B < 750000000";+ }+ )+ (+ replicaMemory+ // {+ uid = "macrodb-replica-freeable-memory-warn";+ title = "macro-db-prod-read-replica freeable memory low";+ severity = "warning";+ condition = "$B < 1500000000 && $B >= 750000000";+ }+ )+ (+ swap+ // {+ uid = "macrodb-swap-crit";+ title = "macro-db-prod swapping";+ severity = "critical";+ condition = "$B > 5000000000";+ }+ )+ (+ swap+ // {+ uid = "macrodb-swap-warn";+ title = "macro-db-prod swapping";+ severity = "warning";+ condition = "$B > 2000000000 && $B <= 5000000000";+ }+ )+ (+ load+ // {+ uid = "macrodb-load-crit";+ title = "macro-db-prod database load above vCPU count";+ severity = "critical";+ condition = "$B > 4";+ }+ )+ (+ load+ // {+ uid = "macrodb-load-warn";+ title = "macro-db-prod database load above vCPU count";+ severity = "warning";+ condition = "$B > 2 && $B <= 4";+ }+ )+ (+ cpu+ // {+ uid = "macrodb-cpu-crit";+ title = "macro-db-prod CPU high";+ severity = "critical";+ condition = "$B > 85";+ }+ )+ (+ cpu+ // {+ uid = "macrodb-cpu-warn";+ title = "macro-db-prod CPU high";+ severity = "warning";+ condition = "$B > 70 && $B <= 85";+ }+ )+ (+ primaryConnections+ // {+ uid = "macrodb-primary-connections-crit";+ title = "macro-db-prod connections high";+ severity = "critical";+ condition = "$B > 2200";+ }+ )+ (+ primaryConnections+ // {+ uid = "macrodb-primary-connections-warn";+ title = "macro-db-prod connections high";+ severity = "warning";+ condition = "$B > 1700 && $B <= 2200";+ }+ )+ (+ replicaConnections+ // {+ uid = "macrodb-replica-connections-crit";+ title = "macro-db-prod-read-replica connections high";+ severity = "critical";+ condition = "$B > 700";+ }+ )+ (+ replicaConnections+ // {+ uid = "macrodb-replica-connections-warn";+ title = "macro-db-prod-read-replica connections high";+ severity = "warning";+ condition = "$B > 550 && $B <= 700";+ }+ )+ (+ replicaLag+ // {+ uid = "macrodb-replica-lag-crit";+ title = "macro-db-prod-read-replica lagging";+ severity = "critical";+ condition = "$B > 300";+ }+ )+ (+ replicaLag+ // {+ uid = "macrodb-replica-lag-warn";+ title = "macro-db-prod-read-replica lagging";+ severity = "warning";+ condition = "$B > 60 && $B <= 300";+ }+ )+ (+ readLatency+ // {+ uid = "macrodb-read-latency-crit";+ title = "macro-db-prod disk read latency high";+ severity = "critical";+ condition = "$B > 0.01";+ }+ )+ (+ readLatency+ // {+ uid = "macrodb-read-latency-warn";+ title = "macro-db-prod disk read latency high";+ severity = "warning";+ condition = "$B > 0.005 && $B <= 0.01";+ }+ )+ (+ freeStorage+ // {+ uid = "macrodb-free-storage-crit";+ title = "macro-db-prod free storage low";+ severity = "critical";+ condition = "$B < 1000000000000";+ }+ )+ (+ freeStorage+ // {+ uid = "macrodb-free-storage-warn";+ title = "macro-db-prod free storage low";+ severity = "warning";+ condition = "$B < 1500000000000 && $B >= 1000000000000";+ }+ )+ (+ redisMemory+ // {+ uid = "redis-memory-crit";+ title = "Redis memory high";+ severity = "critical";+ condition = "$B > 90";+ }+ )+ (+ redisMemory+ // {+ uid = "redis-memory-warn";+ title = "Redis memory high";+ severity = "warning";+ condition = "$B > 75 && $B <= 90";+ }+ )+ (+ redisEvictions+ // {+ uid = "redis-evictions-crit";+ title = "Redis evicting keys";+ severity = "critical";+ condition = "$B > 1000";+ }+ )+ (+ redisEvictions+ // {+ uid = "redis-evictions-warn";+ title = "Redis evicting keys";+ severity = "warning";+ condition = "$B > 0 && $B <= 1000";+ }+ )+ (+ backfillAge+ // {+ uid = "sqs-email-backfill-age-crit";+ title = "Email backfill queue falling behind";+ severity = "critical";+ condition = "$B > 10800";+ }+ )+ (+ backfillAge+ // {+ uid = "sqs-email-backfill-age-warn";+ title = "Email backfill queue falling behind";+ severity = "warning";+ condition = "$B > 3600 && $B <= 10800";+ }+ )+ (+ workQueueAge+ // {+ uid = "sqs-work-queue-age-crit";+ title = "Prod work queue falling behind";+ severity = "critical";+ condition = "$B > 7200";+ }+ )+ (+ workQueueAge+ // {+ uid = "sqs-work-queue-age-warn";+ title = "Prod work queue falling behind";+ severity = "warning";+ condition = "$B > 1800 && $B <= 7200";+ }+ )+ ];dss = [(lokiQuery ''sum(count_over_time({service_name="cloud-storage-service",deployment_environment="dev"} |~ "(?i)\\berror\\w*" [5m]))'')];@@ -133,6 +624,41 @@ in}];}+ {+ orgId = 1;+ name = "Macro Alerts";+ receivers = [+ {+ uid = "macro-alerts";+ type = "webhook";+ disableResolveMessage = false;+ # The loopback relay adds the channel token; see goalert.nix.+ settings = {+ url = "http://127.0.0.1:8090/grafana";+ httpMethod = "POST";+ title = ''{{ template "default.title" . }}'';+ message = ''{{ template "default.message" . }}'';+ };+ }+ ];+ }+ {+ orgId = 1;+ name = "GoAlert";+ receivers = [+ {+ uid = "goalert";+ type = "webhook";+ disableResolveMessage = false;+ # Loopback, so paging does not depend on the ALB. GoAlert closes the+ # alert when Grafana sends the resolved notification.+ settings = {+ url = "http://127.0.0.1:3030/api/v2/grafana/incoming?token=$GOALERT_GRAFANA_KEY";+ httpMethod = "POST";+ };+ }+ ];+ }];};policies.settings = {@@ -140,21 +666,71 @@ inpolicies = [{orgId = 1;- receiver = "Macro team email";+ receiver = "Macro Alerts";group_by = ["grafana_folder""alertname""severity"+ "pages"];group_wait = "30s";group_interval = "5m";repeat_interval = "4h";+ routes = [+ {+ receiver = "Macro team email";+ object_matchers = [+ [+ "environment"+ "="+ "dev"+ ]+ ];+ }+ # Paging prod rules; `continue` lets the next routes still post them+ # to Macro Alerts.+ {+ receiver = "GoAlert";+ object_matchers = [+ [+ "pages"+ "="+ "true"+ ]+ [+ "severity"+ "="+ "critical"+ ]+ ];+ continue = true;+ }+ {+ receiver = "Macro Alerts";+ object_matchers = [+ [+ "severity"+ "="+ "critical"+ ]+ ];+ repeat_interval = "1h";+ }+ { receiver = "Macro Alerts"; }+ ];}];};rules.settings = {apiVersion = 1;groups = [+ {+ orgId = 1;+ name = "Prod";+ folder = "Macro";+ interval = "1m";+ rules = prodRules;+ }{orgId = 1;name = "Datadog dev migration";diff --git a/infra/stacks/observability/nixos/goalert.nix b/infra/stacks/observability/nixos/goalert.nixnew file mode 100644index 0000000000..480fb341af--- /dev/null+++ b/infra/stacks/observability/nixos/goalert.nix@@ -0,0 +1,150 @@+# GoAlert on-call paging, its Postgres, and the Macro Alerts relay.+#+# Everything here runs in Ohio and talks only to loopback, so paging keeps+# working while production or its database is down. Pages reach the Macro+# Alerts channel through the relay; phone paging (Twilio) and Google login are+# not set up yet.+{+ config,+ lib,+ pkgs,+ ...+}:+let+ goalert = pkgs.callPackage ./goalert/package.nix { };+ credential = name: "${name}:/run/macro-observability/${name}";+ needsSecrets = {+ requires = [ "observability-secrets.service" ];+ after = [ "observability-secrets.service" ];+ partOf = [ "observability-secrets.service" ];+ };+ relayUrl = "http://127.0.0.1:8090/goalert";+ # GoAlert's config.Config, applied with `goalert set-config` before every+ # start. `@name@` strings are replaced with the credential of that name.+ goalertConfig = {+ General.ApplicationName = "Macro On-Call";+ # One `admin` login, created by apply-config.py from goalert_admin_password.+ Auth.DisableBasic = false;+ Webhook = {+ Enable = true;+ AllowedURLs = [ relayUrl ];+ };+ Maintenance = {+ AlertCleanupDays = 180;+ AlertAutoCloseDays = 0;+ };+ };+ dbUrl = "postgres://goalert@/goalert?host=/run/postgresql&sslmode=disable";+ goalertEnvironment = {+ GOALERT_DB_URL = dbUrl;+ GOALERT_LISTEN = "127.0.0.1:3030";+ GOMEMLIMIT = "600MiB";+ };+ # The encryption key only ever lives in this process's environment.+ start = pkgs.writeShellScript "goalert-start" ''+ GOALERT_DATA_ENCRYPTION_KEY="$(< "$CREDENTIALS_DIRECTORY/goalert_encryption_key")"+ export GOALERT_DATA_ENCRYPTION_KEY+ exec ${lib.getExe goalert}+ '';+ backup = pkgs.writeShellScript "goalert-backup" ''+ set -euo pipefail+ # Dump to a file first, so a failed dump never replaces a good backup.+ dump="$(mktemp)"+ ${config.services.postgresql.package}/bin/pg_dump --format=custom --file="$dump" goalert+ # One object per weekday keeps a rolling week without a lifecycle rule on+ # a bucket whose retention Loki owns.+ ${lib.getExe pkgs.awscli2} s3 cp --only-show-errors "$dump" \+ "s3://$LOGS_BUCKET/goalert/goalert-$(date +%a).dump"+ '';+in+{+ services.postgresql = {+ enable = true;+ package = pkgs.postgresql_17;+ dataDir = "/srv/observability/postgres";+ enableTCPIP = false;+ settings.listen_addresses = lib.mkForce "";+ ensureDatabases = [ "goalert" ];+ ensureUsers = [+ {+ name = "goalert";+ ensureDBOwnership = true;+ }+ ];+ };++ systemd.services = {+ postgresql = {+ serviceConfig.MemoryMax = "512M";+ # Like the other services, retry forever instead of the module's limit.+ unitConfig.StartLimitIntervalSec = lib.mkForce 0;+ };++ goalert = needsSecrets // {+ description = "GoAlert on-call paging";+ wantedBy = [ "multi-user.target" ];+ requires = needsSecrets.requires ++ [ "postgresql.target" ];+ after = needsSecrets.after ++ [ "postgresql.target" ];+ environment = goalertEnvironment;+ serviceConfig = {+ User = "goalert";+ Group = "goalert";+ EnvironmentFile = "/opt/observability/runtime.env";+ LoadCredential = map credential [+ "goalert_encryption_key"+ "goalert_admin_password"+ ];+ ExecStartPre = [+ "${pkgs.python3}/bin/python3 ${./goalert/apply-config.py} ${lib.getExe goalert} ${pkgs.writeText "goalert-config.json" (builtins.toJSON goalertConfig)} ${config.services.postgresql.package}/bin/psql"+ ];+ ExecStart = start;+ MemoryMax = "768M";+ CapabilityBoundingSet = [ "" ];+ PrivateDevices = true;+ };+ };++ macro-alerts-relay = needsSecrets // {+ description = "Forward GoAlert and Grafana notifications to Macro Alerts";+ wantedBy = [ "multi-user.target" ];+ serviceConfig = {+ DynamicUser = true;+ LoadCredential = map credential [+ "macro_alerts_webhook_url"+ "macro_alerts_bot_token"+ ];+ ExecStart = "${pkgs.python3}/bin/python3 ${./goalert/macro-alerts-relay.py}";+ MemoryMax = "128M";+ CapabilityBoundingSet = [ "" ];+ PrivateDevices = true;+ };+ };++ goalert-backup = {+ description = "Dump the GoAlert database to S3";+ requires = [ "postgresql.target" ];+ after = [ "postgresql.target" ];+ serviceConfig = {+ Type = "oneshot";+ User = "goalert";+ Group = "goalert";+ EnvironmentFile = "/opt/observability/runtime.env";+ ExecStart = backup;+ PrivateTmp = true;+ ProtectSystem = "strict";+ ProtectHome = true;+ NoNewPrivileges = true;+ # The AWS CLI writes its cache under HOME.+ Environment = "HOME=/tmp";+ };+ };+ };++ systemd.timers.goalert-backup = {+ wantedBy = [ "timers.target" ];+ timerConfig = {+ OnCalendar = "*-*-* 05:00:00";+ Persistent = true;+ };+ };+}diff --git a/infra/stacks/observability/nixos/goalert/apply-config.py b/infra/stacks/observability/nixos/goalert/apply-config.pynew file mode 100644index 0000000000..2180aa71e5--- /dev/null+++ b/infra/stacks/observability/nixos/goalert/apply-config.py@@ -0,0 +1,48 @@+"""Migrate GoAlert's database, replace its config with the Nix-declared one, and+create the `admin` login if it does not exist.++GoAlert keeps application config only in Postgres. Running this before every+start makes the Nix attrset the source of truth: edits in the admin UI last+only until the next restart. Secret placeholders such as `@goalert_admin_password@`+are replaced with credential files; values never pass through a shell.+"""+import json+import os+from pathlib import Path+import subprocess+import sys++goalert, template_path, psql = sys.argv[1], sys.argv[2], sys.argv[3]+credentials = Path(os.environ['CREDENTIALS_DIRECTORY'])+++def substitute(value):+ if isinstance(value, dict):+ return {key: substitute(item) for key, item in value.items()}+ if isinstance(value, list):+ return [substitute(item) for item in value]+ if isinstance(value, str) and value.startswith('@') and value.endswith('@'):+ return (credentials / value[1:-1]).read_text()+ return value+++config = substitute(json.loads(Path(template_path).read_text()))+environment = {+ **os.environ,+ 'GOALERT_DATA_ENCRYPTION_KEY': (credentials / 'goalert_encryption_key').read_text(),+}+subprocess.run([goalert, 'migrate'], env=environment, check=True)+subprocess.run(+ [goalert, 'set-config'], env=environment, input=json.dumps(config), text=True, check=True,+)+admin_exists = subprocess.run(+ [psql, '-tAc', "select 1 from auth_basic_users where username = 'admin'", environment['GOALERT_DB_URL']],+ env=environment, capture_output=True, text=True, check=True,+).stdout.strip()+if not admin_exists:+ # `add-user` takes the password only as an argument; this runs once, on first boot.+ subprocess.run(+ [goalert, 'add-user', '--admin', '--user', 'admin', '--email', '[email protected]',+ '--pass', (credentials / 'goalert_admin_password').read_text()],+ env=environment, check=True,+ )diff --git a/infra/stacks/observability/nixos/goalert/macro-alerts-relay.py b/infra/stacks/observability/nixos/goalert/macro-alerts-relay.pynew file mode 100644index 0000000000..6e203a9b86--- /dev/null+++ b/infra/stacks/observability/nixos/goalert/macro-alerts-relay.py@@ -0,0 +1,102 @@+"""Forward GoAlert and Grafana webhooks to the Macro Alerts channel.++Macro's channel webhook takes `{"content": ...}` and a bot-token header. GoAlert+cannot set headers, so both senders post here over loopback and the token stays+in this one service. Each request is forwarded once; the sender owns retries.+"""+import json+import logging+import os+import time+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from pathlib import Path+import urllib.request++CREDENTIALS = Path(os.environ['CREDENTIALS_DIRECTORY'])+WEBHOOK_URL = (CREDENTIALS / 'macro_alerts_webhook_url').read_text()+BOT_TOKEN = (CREDENTIALS / 'macro_alerts_bot_token').read_text()+# GoAlert abandons a webhook after three seconds.+FORWARD_TIMEOUT_SECONDS = 2.5+MAX_BODY_BYTES = 256 * 1024+MAX_CONTENT_CHARS = 4000+++def goalert_content(payload: dict) -> str | None:+ kind = payload.get('Type')+ link = payload.get('GoAlertURL', '')+ if kind == 'Alert':+ return f"🚨 Paging on-call: {payload.get('Summary', '')}\n{link}"+ if kind == 'AlertStatus':+ return f"{payload.get('LogEntry', '')}: {payload.get('Summary', '')}\n{link}"+ if kind == 'AlertBundle':+ return f"🚨 Paging on-call: {payload.get('Count')} alerts for {payload.get('ServiceName', '')}\n{link}"+ if kind == 'AlertStatusBundle':+ return f"{payload.get('LogEntry', '')} ({payload.get('Count')} alerts)\n{link}"+ if kind == 'ScheduleOnCallUsers':+ names = ', '.join(user.get('Name', '') for user in payload.get('Users', []))+ return f"On call for {payload.get('ScheduleName', '')}: {names or 'nobody'}\n{payload.get('ScheduleURL', '')}"+ if kind in ('Test', 'Verification'):+ return f"GoAlert {kind.lower()} message for the Macro Alerts webhook."+ return None+++def grafana_content(payload: dict) -> str | None:+ title = payload.get('title')+ message = payload.get('message')+ if not isinstance(title, str) or not isinstance(message, str):+ return None+ return f"{title}\n{message}"+++SOURCES = {'/goalert': goalert_content, '/grafana': grafana_content}+++class Relay(BaseHTTPRequestHandler):+ def do_POST(self) -> None:+ format_content = SOURCES.get(self.path)+ length = int(self.headers.get('Content-Length', '0'))+ if format_content is None or length <= 0 or length > MAX_BODY_BYTES:+ self.send_error(404 if format_content is None else 400)+ return+ try:+ payload = json.loads(self.rfile.read(length))+ except json.JSONDecodeError:+ self.send_error(400)+ return+ content = format_content(payload) if isinstance(payload, dict) else None+ if content is None:+ logging.warning('unrecognised %s payload', self.path)+ self.send_error(422)+ return+ request = urllib.request.Request(+ WEBHOOK_URL,+ data=json.dumps({'content': content[:MAX_CONTENT_CHARS]}).encode(),+ headers={'Content-Type': 'application/json', 'x-macro-channel-bot-token': BOT_TOKEN},+ method='POST',+ )+ started = time.monotonic()+ try:+ with urllib.request.urlopen(request, timeout=FORWARD_TIMEOUT_SECONDS) as response:+ status = response.status+ except OSError as error:+ logging.warning(+ 'forward source=%s outcome=failed elapsed_ms=%d error=%s',+ self.path, (time.monotonic() - started) * 1000, error,+ )+ self.send_error(502)+ return+ logging.info(+ 'forward source=%s status=%d elapsed_ms=%d',+ self.path, status, (time.monotonic() - started) * 1000,+ )+ self.send_response(204 if 200 <= status < 300 else 502)+ self.end_headers()++ def log_message(self, format: str, *args: object) -> None:+ # Request lines carry no secrets, but errors are logged explicitly above.+ pass+++if __name__ == '__main__':+ logging.basicConfig(level=logging.INFO, format='%(levelname)s %(message)s')+ ThreadingHTTPServer(('127.0.0.1', 8090), Relay).serve_forever()diff --git a/infra/stacks/observability/nixos/goalert/package.nix b/infra/stacks/observability/nixos/goalert/package.nixnew file mode 100644index 0000000000..f22931e8ca--- /dev/null+++ b/infra/stacks/observability/nixos/goalert/package.nix@@ -0,0 +1,95 @@+# GoAlert is not in nixpkgs, so build it here from the tagged source. The Go+# binary embeds the web UI (web/src/build), which is bundled with bun first.+{+ lib,+ stdenvNoCC,+ buildGoModule,+ fetchFromGitHub,+ bun,+}:+let+ version = "0.35.0";+ # `git rev-parse v0.35.0` and its commit date, for `goalert version`.+ commit = "db9829187a1c1f96a7c57112b74dba54442d7c6b";+ commitDate = "2026-09-15T21:24:20Z";++ src = fetchFromGitHub {+ owner = "target";+ repo = "goalert";+ tag = "v${version}";+ hash = "sha256-huzessc9Iy/j689miscZJ6/MZIClA+qG3EX1tOokk54=";+ };++ nodeModules = stdenvNoCC.mkDerivation {+ pname = "goalert-node-modules";+ inherit version src;+ nativeBuildInputs = [ bun ];+ dontConfigure = true;+ dontFixup = true;+ buildPhase = ''+ runHook preBuild+ export HOME=$TMPDIR+ bun install --frozen-lockfile --ignore-scripts --no-progress+ runHook postBuild+ '';+ installPhase = ''+ runHook preInstall+ cp -R node_modules $out+ runHook postInstall+ '';+ outputHash = "sha256-ECi5MztQ+pavprPftLFHu4q9wxSg9SZywFecRD4WFJI=";+ outputHashMode = "recursive";+ };++ # Mirrors the Makefile's web/src/build/static/app.js target.+ ui = stdenvNoCC.mkDerivation {+ pname = "goalert-ui";+ inherit version src;+ nativeBuildInputs = [ bun ];+ dontConfigure = true;+ buildPhase = ''+ runHook preBuild+ cp -R ${nodeModules} node_modules+ chmod -R u+w node_modules+ patchShebangs node_modules+ mkdir -p web/src/build/static+ cp web/src/app/public/icons/favicon-* web/src/app/public/logos/lightmode_* \+ web/src/app/public/logos/darkmode_* web/src/app/public/logos/goalert-alt-logo.png \+ web/src/build/static/+ GOALERT_VERSION=v${version} bun web/src/esbuild.config.js --prod+ runHook postBuild+ '';+ installPhase = ''+ runHook preInstall+ cp -R web/src/build $out+ runHook postInstall+ '';+ };+in+buildGoModule {+ pname = "goalert";+ inherit version src;+ vendorHash = "sha256-htUZy6mHcuINTa6ih/cD2hF5pHjZWPqX1vuiJAu1zKQ=";+ subPackages = [ "cmd/goalert" ];+ preBuild = ''+ rm -rf web/src/build+ cp -R ${ui} web/src/build+ '';+ ldflags = [+ "-s"+ "-w"+ "-X github.com/target/goalert/version.gitVersion=v${version}"+ "-X github.com/target/goalert/version.gitCommit=${commit}"+ "-X github.com/target/goalert/version.gitTreeState=clean"+ "-X github.com/target/goalert/version.buildDate=${commitDate}"+ ];+ # The test suite needs a Postgres database.+ doCheck = false;+ meta = {+ description = "On-call scheduling, automated escalations and notifications";+ homepage = "https://github.com/target/goalert";+ license = lib.licenses.asl20;+ mainProgram = "goalert";+ platforms = [ "x86_64-linux" ];+ };+}diff --git a/infra/stacks/observability/nixos/host.nix b/infra/stacks/observability/nixos/host.nixindex 5be83e4920..ba1838783b 100644--- a/infra/stacks/observability/nixos/host.nix+++ b/infra/stacks/observability/nixos/host.nix@@ -16,11 +16,16 @@ let"alloy""alloy-ingest""nginx"+ "postgresql"+ "goalert"+ "macro-alerts-relay"];secretConsumers = ["grafana.service""alloy-ingest.service""nginx.service"+ "goalert.service"+ "macro-alerts-relay.service"];dataUsers = {grafana = 20001;@@ -29,6 +34,8 @@ letprometheus = 20004;alloy = 20005;alloy-ingest = 20006;+ postgres = 20007;+ goalert = 20008;};in{diff --git a/infra/stacks/observability/nixos/proxy.nix b/infra/stacks/observability/nixos/proxy.nixindex 72f8e34126..bb0eea6a94 100644--- a/infra/stacks/observability/nixos/proxy.nix+++ b/infra/stacks/observability/nixos/proxy.nix@@ -3,6 +3,20 @@ letdeny = {extraConfig = "return 404;";};+ # TLS ends at the ALB; GoAlert redirects plain-HTTP requests without this.+ goalertProxy = ''+ proxy_pass http://127.0.0.1:3030;+ proxy_set_header Host $host;+ proxy_set_header X-Forwarded-Proto https;+ proxy_set_header X-Forwarded-For $remote_addr;+ '';+ # Unauthenticated paging endpoints. GoAlert checks the integration key on each.+ pagingIngress = ''+ limit_except POST { deny all; }+ limit_req zone=paging burst=50 nodelay;+ limit_req_status 429;+ ${goalertProxy}+ '';websocketHeaders = ''proxy_http_version 1.1;proxy_set_header Upgrade $http_upgrade;@@ -28,7 +42,9 @@ inlog_format safe '$remote_addr $host $request_method $uri $status';access_log syslog:server=unix:/dev/log,facility=local7,tag=nginx_access,severity=info safe;limit_req_zone $server_name zone=ingest:1m rate=100r/s;+ limit_req_zone $server_name zone=paging:1m rate=20r/s;map $http_upgrade $connection_upgrade { default upgrade; ''' close; }+ include /opt/observability/oncall-server.conf;'';virtualHosts.public = {listen = [@@ -45,6 +61,11 @@ inproxyPass = "http://127.0.0.1:3000/api/health";extraConfig = "access_log off; proxy_set_header Host $grafana_host;";};+ # The on-call target group's health check arrives without the on-call host.+ "= /goalert-healthz" = {+ proxyPass = "http://127.0.0.1:3030/health";+ extraConfig = "access_log off; proxy_set_header X-Forwarded-Proto https;";+ };"/" = {proxyPass = "http://127.0.0.1:3000";extraConfig = ''@@ -127,5 +148,24 @@ in};};};+ # Included by the runtime on-call server block, which carries the hostname.+ environment.etc."nginx/oncall-locations.conf".text = ''+ add_header Strict-Transport-Security 'max-age=31536000' always;+ location = /health {+ access_log off;+ limit_except GET { deny all; }+ ${goalertProxy}+ }+ location ^~ /api/v2/generic/incoming { ${pagingIngress} }+ location ^~ /api/v2/heartbeat/ { ${pagingIngress} }+ # Grafana pages GoAlert over loopback; other inbound integrations are unused.+ location ~ ^/(api/v2/(grafana|prometheusalertmanager|site24x7|mailgun|uik|twilio)|v1/webhooks|health/) {+ return 404;+ }+ location / {+ ${goalertProxy}+ ${websocketHeaders}+ }+ '';systemd.services.nginx.serviceConfig.MemoryMax = "256M";}diff --git a/infra/stacks/observability/nixos/read-user-data.py b/infra/stacks/observability/nixos/read-user-data.pyindex a6cd081a08..d0d381b40f 100644--- a/infra/stacks/observability/nixos/read-user-data.py+++ b/infra/stacks/observability/nixos/read-user-data.py@@ -7,13 +7,13 @@ import urllib.requestCONFIG_ROOT = Path('/opt/observability')SETTING_KEYS = {- 'region', 'grafanaHost', 'otlpHost',+ 'region', 'grafanaHost', 'otlpHost', 'oncallHost','secretArn', 'alarmTopicArn', 'volumeId', 'logsBucket', 'tracesBucket',}def validate_payload(payload):- if not isinstance(payload, dict) or set(payload) != {'version', 'settings'} or payload['version'] != 5:+ if not isinstance(payload, dict) or set(payload) != {'version', 'settings'} or payload['version'] != 6:raise ValueError('Unsupported observability user-data schema')settings = payload['settings']if not isinstance(settings, dict) or set(settings) != SETTING_KEYS:@@ -22,6 +22,7 @@ def validate_payload(payload):'region': r'us-east-2','grafanaHost': r'[a-z0-9-]+\.macro-internal\.com','otlpHost': r'[a-z0-9-]+\.macro-internal\.com',+ 'oncallHost': r'[a-z0-9-]+\.macro-internal\.com','alarmTopicArn': r'arn:aws:sns:us-east-2:\d{12}:[a-zA-Z0-9_-]+','secretArn': r'arn:aws:secretsmanager:us-east-2:\d{12}:secret:[a-zA-Z0-9/_+=.@-]+','volumeId': r'vol-[a-f0-9]+',@@ -31,8 +32,9 @@ def validate_payload(payload):for key, pattern in patterns.items():if not isinstance(settings[key], str) or not re.fullmatch(pattern, settings[key]):raise ValueError(f'Invalid runtime setting: {key}')- if settings['grafanaHost'] == settings['otlpHost']:- raise ValueError('UI and ingestion require separate hostnames')+ hosts = [settings['grafanaHost'], settings['otlpHost'], settings['oncallHost']]+ if len(set(hosts)) != len(hosts):+ raise ValueError('Grafana, ingestion and on-call require separate hostnames')return settings@@ -43,6 +45,7 @@ def write_runtime(payload, output_root):'GRAFANA_ALERT_TOPIC_ARN': settings['alarmTopicArn'],'GRAFANA_HOST': settings['grafanaHost'],'GRAFANA_ROOT_URL': 'https://' + settings['grafanaHost'] + '/',+ 'GOALERT_PUBLIC_URL': 'https://' + settings['oncallHost'],'LOGS_BUCKET': settings['logsBucket'],'TRACES_BUCKET': settings['tracesBucket'],}@@ -56,6 +59,12 @@ def write_runtime(payload, output_root):'}\nmap "" $grafana_host {\n'f" default {settings['grafanaHost']};\n" + '}\n'),+ # GoAlert gets its own server so its paths never overlap Grafana's.+ 'oncall-server.conf': (+ 'server {\n listen 0.0.0.0:8080;\n'+ f" server_name {settings['oncallHost']};\n"+ ' include /etc/nginx/oncall-locations.conf;\n}\n'+ ),'bootstrap.json': json.dumps({'region': settings['region'], 'secretArn': settings['secretArn']}),'volume-id': settings['volumeId'],}diff --git a/infra/stacks/observability/nixos/services.nix b/infra/stacks/observability/nixos/services.nixindex c10a6932b8..e7b15641d9 100644--- a/infra/stacks/observability/nixos/services.nix+++ b/infra/stacks/observability/nixos/services.nix@@ -20,6 +20,7 @@ in./database-dashboard.nix./database-queries-dashboard.nix./alerting.nix+ ./goalert.nix];services.grafana = {enable = true;@@ -64,7 +65,11 @@ in};grafana = needsSecrets // {serviceConfig = {- EnvironmentFile = "/opt/observability/runtime.env";+ EnvironmentFile = [+ "/opt/observability/runtime.env"+ # Interpolated into the GoAlert contact point by alerting provisioning.+ "/run/macro-observability/grafana-goalert.env"+ ];LoadCredential = map credential ["google_client_id""google_client_secret"diff --git a/infra/stacks/observability/render.ts b/infra/stacks/observability/render.tsindex 0d1a10d3ef..3eb8485611 100644--- a/infra/stacks/observability/render.ts+++ b/infra/stacks/observability/render.ts@@ -4,7 +4,7 @@ import { type Settings, validateSettings } from './settings';// Nix owns service configuration. User data carries only validated runtime values.export function renderUserData(settings: Settings): string {validateSettings(settings);- const compressed = gzipSync(JSON.stringify({ version: 5, settings }));+ const compressed = gzipSync(JSON.stringify({ version: 6, settings }));if (compressed.length > 16 * 1024) {throw new Error('Configuration exceeds the EC2 user-data limit');}diff --git a/infra/stacks/observability/settings.ts b/infra/stacks/observability/settings.tsindex 9f65d5ba3e..7af8bcf53a 100644--- a/infra/stacks/observability/settings.ts+++ b/infra/stacks/observability/settings.ts@@ -2,6 +2,7 @@ export interface Settings {region: string;grafanaHost: string;otlpHost: string;+ oncallHost: string;secretArn: string;alarmTopicArn: string;volumeId: string;@@ -38,15 +39,18 @@ export function validateRegionalArn(// Validate runtime values before they reach the NixOS configuration renderer.export function validateSettings(settings: Settings): void {- for (const host of [settings.grafanaHost, settings.otlpHost]) {+ const hosts = [settings.grafanaHost, settings.otlpHost, settings.oncallHost];+ for (const host of hosts) {if (!/^[a-z0-9-]+\.macro-internal\.com$/.test(host)) {throw new Error('Observability hostnames must be subdomains of macro-internal.com');}}- if (settings.grafanaHost === settings.otlpHost) {- throw new Error('UI and ingestion require separate hostnames');+ if (new Set(hosts).size !== hosts.length) {+ throw new Error(+ 'Grafana, ingestion and on-call require separate hostnames'+ );}if (!/^arn:aws:secretsmanager:[a-z0-9-]+:\d{12}:secret:[a-zA-Z0-9/_+=.@-]+$/.test(