Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(observability): page on-call through GoAlert from Grafana and add prod database alerts (#9067)

Wolf Mermelstein0a19515960a6
diff --git a/docs/README.md b/docs/README.md
index 309905a332..1b3a6e5151 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -6,7 +6,7 @@
| [`guides/`](guides/) | End-to-end walkthroughs: adding a service, entity, block, and more. |
| [`architecture/`](architecture/) | How shipped systems work today. |
| [`design/`](design/) | Proposals, plans, and audits; check each doc's status. |
-| [`runbooks/`](runbooks/) | Rollouts, migrations, imports, and demos with operational steps. |
+| [`runbooks/`](runbooks/) | Rollouts, migrations, imports, demos, and the response steps that prod alerts link to. |
| [`agent-guide/`](agent-guide/) | How browser agents operate the app, not how to develop it. |
Docs that belong to one app, service, or crate live next to its code.
diff --git a/docs/runbooks/macrodb.md b/docs/runbooks/macrodb.md
new file mode 100644
index 0000000000..1ed4428e90
--- /dev/null
+++ b/docs/runbooks/macrodb.md
@@ -0,0 +1,94 @@
+# MacroDB alerts
+
+Runbook for the `macro-db-prod` and `macro-db-prod-read-replica` monitors in
+[`monitors/macrodb.ts`](../../infra/stacks/monitoring/monitors/macrodb.ts). Both instances are
+`db.m9g.2xlarge`: 8 vCPU, about 30 GB of memory, `max_connections` 3,439.
+
+Start in three places:
+
+1. **Performance Insights** for the instance (RDS console → the instance →
+ Monitoring → Database load). Group by waits, then by SQL, to see what the
+ active sessions are doing.
+2. **Datadog Database Monitoring** for `macro-db-prod` (Query Metrics, Query
+ Samples, and the blocking queries view).
+3. **Recent deploys and queue backlogs.** Most overloads have started with a
+ deploy, a backfill, or a queue draining all at once. Check the ECS service
+ events and the SQS monitors in the same window.
+
+## Memory
+
+Freeable memory and swap alerts. Every Postgres backend keeps several MB of
+private memory, about 14 MB per connection under prod load, so memory falls as
+connections rise. The OS page cache shrinks first; once it is gone the
+instance swaps and every query slows.
+
+- Check `aws.rds.database_connections` against the same window. If connections
+ climbed, find which services added them (see [Connections](#connections)).
+- Check swap (`aws.rds.swap_usage`). Sustained swap above a few GB means the
+ instance is out of memory, not just low on cache.
+- Levers: lower the pool size of the service that grew; pause backfills; as a
+ last resort fail over or resize the instance (a resize restarts it).
+
+## Load
+
+Database load (average active sessions) relative to vCPUs, CPU, and disk read
+latency. Load above the vCPU count means sessions are queueing for CPU, locks
+or IO.
+
+- In Performance Insights, the top wait event says which: `CPU`, `Lock:*` /
+ `LWLock:*`, or `IO:*`.
+- If lock waits dominate, go to [Locks](#locks).
+- If one query dominates CPU, find its caller from the query sample's
+ application name and the APM span with the same statement.
+- Levers: pause the backfill or worker driving the load; roll back the deploy
+ that introduced the query; add the missing index in a follow-up.
+
+## Connections
+
+Connection count on the primary or replica. Each ECS task opens its own pool,
+so connections scale with task count times pool size.
+
+- Group `postgresql.connections` (Database Monitoring) or Performance Insights
+ sessions by application or user to see which service holds them.
+- Check whether a service scaled out (`aws.ecs.service.running`) or a pool
+ setting changed in a recent deploy.
+- Levers: lower that service's pool maximum or task count. RDS Proxy is not a
+ quick fix: moving a service onto it has caused pool timeouts before.
+
+## Replica lag
+
+The read replica is behind the primary. GraphQL Soup reads go to the replica,
+so users see stale lists and items they just created go missing.
+
+- Check primary write volume and WAL (`aws.rds.transaction_logs_generation`)
+ and replica CPU and IO. Lag usually follows a write burst on the primary or a
+ long query on the replica holding back replay.
+- Look for long-running queries on the replica in Database Monitoring.
+- Levers: stop the write burst (often a backfill); cancel the long replica
+ query.
+
+## Storage
+
+Free storage space, now or forecast to run low within a week.
+
+- Compare `aws.rds.free_storage_space` with
+ `aws.rds.transaction_logs_disk_usage` and
+ `aws.rds.replication_slot_disk_usage`: WAL held by a replication slot
+ grows without bound.
+- Otherwise find the tables that grew (`pg_total_relation_size` on the largest
+ relations) and what started writing to them.
+- Levers: drop an abandoned replication slot; stop the writer; increase
+ allocated storage (online, but only once every six hours).
+
+## Locks
+
+Connections blocked on locks, or a transaction open for a long time.
+
+- In Database Monitoring's blocking queries view, find the blocking session and
+ its query. Advisory locks (`pg_advisory_xact_lock`) and tuple locks from
+ concurrent updates of the same rows have caused past overloads.
+- A long open transaction holds its locks and stops vacuum. Find it in
+ `pg_stat_activity` (`xact_start`, `state = 'idle in transaction'`).
+- Levers: cancel (`pg_cancel_backend`) or terminate (`pg_terminate_backend`)
+ the blocking session once you know what it is; stop the job that keeps
+ starting new ones.
diff --git a/docs/runbooks/queues.md b/docs/runbooks/queues.md
new file mode 100644
index 0000000000..d0a4511a44
--- /dev/null
+++ b/docs/runbooks/queues.md
@@ -0,0 +1,34 @@
+# Queue alerts
+
+Runbook for the SQS monitors in
+[`monitors/queues.ts`](../../infra/stacks/monitoring/monitors/queues.ts).
+
+## Dead letters
+
+Messages arrived in a prod dead-letter queue. Most DLQs hold old messages
+permanently, so the alert counts new arrivals, not depth.
+
+- Read a few messages from the DLQ (`aws sqs receive-message` with a short
+ visibility timeout, so they return to the queue) and find the consumer's
+ error logs for the same ids around the time they moved.
+- Check the consumer's latest deploy: a new message shape or a bug failing
+ every message moves them all to the DLQ after the retry limit.
+- Levers: fix and deploy the consumer, then redrive the DLQ to the source queue
+ (SQS console → Start DLQ redrive). Do not redrive before the fix; the
+ messages come straight back.
+
+## Backlog
+
+The oldest message in a work queue is old, or the email backfill backlog grew
+abnormally.
+
+- Check that the consumer is running (`aws.ecs.service.running`) and not
+ crashing (task stop reasons, error logs).
+- If it is running, check what it waits on. The email workers write to MacroDB
+ and call Gmail per mailbox; a slow database or Gmail rate limits stall them.
+ See the [MacroDB runbook](macrodb.md).
+- A sudden rise in enqueues (`aws.sqs.number_of_messages_sent`) on the backfill
+ queue means many mailboxes started backfilling at once, which also loads
+ MacroDB.
+- Levers: scale the consumer if the database has headroom; otherwise throttle
+ the producer or pause the backfill until the database recovers.
diff --git a/docs/runbooks/redis.md b/docs/runbooks/redis.md
new file mode 100644
index 0000000000..191f24e719
--- /dev/null
+++ b/docs/runbooks/redis.md
@@ -0,0 +1,33 @@
+# Redis alerts
+
+Runbook for the ElastiCache monitors in
+[`monitors/redis.ts`](../../infra/stacks/monitoring/monitors/redis.ts): memory used,
+memory forecast, and evictions, per node of every prod cache.
+
+## Memory
+
+The prod caches run `volatile-lru`, which evicts only keys with a TTL. When
+most keys have no TTL, a full cache rejects writes (`OOM command not allowed`)
+instead of evicting, and every feature that writes to it fails.
+
+Start in three places:
+
+1. **Key mix.** Sample keys by prefix on a replica (`redis-cli --scan
+ --pattern '<prefix>*' | head`, `MEMORY USAGE <key>`) and count keys without
+ a TTL (`TTL <key>` returns `-1`). A single prefix growing without a TTL is
+ the usual cause.
+2. **Growth over time.** Graph `aws.elasticache.database_memory_usage_percentage`
+ and `aws.elasticache.curr_items` over the last month. Steady linear growth
+ points at a key family that is written and never deleted.
+3. **Evictions.** `aws.elasticache.evictions` above zero means the cache is
+ already full and dropping keys that do have a TTL.
+
+Levers:
+
+- Delete or expire the runaway key family (in batches, with `SCAN` and
+ `UNLINK`, never `KEYS`).
+- Resize the node type. A resize can move the primary to another node, and
+ services that connect to a node endpoint rather than the primary endpoint
+ then get `READONLY` errors until it fails back.
+- Fix the writer so the keys get a TTL or are deleted when no longer
+ referenced.
diff --git a/docs/runbooks/regressions.md b/docs/runbooks/regressions.md
new file mode 100644
index 0000000000..838ba3ac87
--- /dev/null
+++ b/docs/runbooks/regressions.md
@@ -0,0 +1,31 @@
+# Error and latency regression alerts
+
+Runbook for the per-service anomaly monitors in
+[`monitors/regressions.ts`](../../infra/stacks/monitoring/monitors/regressions.ts). Each compares a
+service's errors or latency with its own usual band for that time of day, so it
+fires on a change, not on a fixed count.
+
+## Errors
+
+A service's HTTP server errors, failed internal or consumer spans, or error
+logs are above their usual band.
+
+- Find what changed. Compare the start of the rise with the service's deploys
+ (ECS service events, the web app's release) and with flag changes.
+- Group the errors by `resource_name` (APM) or by message pattern (Logs →
+ Patterns) for the alert window. A new error appearing from zero is usually
+ the regression; a familiar one growing usually follows load or a dependency.
+- Check Error Tracking for issues first seen in the window.
+- Levers: roll back the deploy or turn off the flag; if a dependency is
+ failing, follow that dependency's runbook.
+
+## Latency
+
+A service's p95 HTTP latency is above its usual band.
+
+- In APM, sort the service's resources by p95 change to find the slow
+ endpoint, then open a slow trace and see which span holds the time.
+- Database time points at the [MacroDB runbook](macrodb.md); time waiting for a
+ pool connection points at pool exhaustion in that service.
+- Levers: roll back the deploy that introduced the slow path; reduce load from
+ the caller driving it.
diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.md
index e83f993df0..39999c4aa3 100644
--- a/infra/stacks/monitoring/AGENTS.md
+++ b/infra/stacks/monitoring/AGENTS.md
@@ -4,19 +4,33 @@
Org-global Datadog monitors, synthetics, and rollout dashboards. This stack is
manual and has only a prod stack, even for monitors covering both environments.
+Prod threshold and anomaly monitors are in `monitors/{macrodb,redis,queues,regressions}.ts`
+(`prodMonitor()` in `prod-monitor.ts`); every monitor notifies through `notify.ts`.
## Checks
From `infra/`:
-- `bun test stacks/monitoring/monitors/notification-retention.test.ts`
+- `bun test --isolate stacks/monitoring` (routing and notification retention
+ tests). `--isolate` matters: the Pulumi mock tests
+ each import modules that register resources, and a shared module cache
+ hides them from the second file.
- `bun run check`
- Biome on changed files.
-- An authenticated `pulumi preview` must leave adopted monitors unchanged.
+- An authenticated `pulumi preview` may change adopted monitors only in
+ `message` and `renotifyInterval`; README.md lists the expected counts.
## Gotchas
+- The CloudWatch-based monitors in `macrodb.ts`, `redis.ts` and `queues.ts`
+ have hand-written Grafana copies in
+ `infra/stacks/observability/nixos/alerting.nix`. Change thresholds and windows
+ in both. Grafana pages for those, so their Datadog copies use `pages: false`.
- Read `README.md` before deployment: imported resources must be adopted first
to avoid duplicate alerts. Never deploy without explicit approval.
+- Deploying needs the `goalertIntegrationKey` secret in stack config, and
+ GoAlert (infra/stacks/observability) reachable at oncall.macro-internal.com.
+- Never mention a Slack handle or email address in a monitor message; end it
+ with `notify({ pages })`. `routing.test.ts` enforces it.
- Notification retention adds nine monitors and one dashboard. Deploy DSS health
instrumentation first; missing telemetry is deliberately an alert, not zero lag.
- Do not use native committed-offset lag for the independent notification
diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.md
index 14677af3c4..2a233b8558 100644
--- a/infra/stacks/monitoring/README.md
+++ b/infra/stacks/monitoring/README.md
@@ -4,15 +4,61 @@ Datadog monitors and synthetic tests as code.
36 resources here (35 monitors and one synthetic test) already exist in
Datadog. This stack **adopts** them — it does not recreate them. `import.json`
-lists every one by id. The two AI-editing monitors in `monitors/ai-editing.ts`
-are new and are meant to be created.
+lists every one by id. Everything else is declared here first and is created
+on deploy: the prod threshold and anomaly monitors (`monitors/macrodb.ts`,
+`redis.ts`, `queues.ts`, `regressions.ts`), the rollout monitors, the GoAlert webhooks and health check, and one log-based metric.
+
+## Routing
+
+Every monitor and synthetic test notifies through `notify.ts`, and nowhere
+else:
+
+- **All of them** post to the Macro Alerts channel (`@webhook-macro-alert-hook`,
+ a webhook integration configured in the Datadog UI).
+- **Paging monitors** also open a GoAlert alert on `{{#is_alert}}` and close it
+ on `{{#is_recovery}}` (`@webhook-goalert`, `@webhook-goalert-close`, declared
+ in `goalert.ts`). GoAlert runs on the observability host
+ (`infra/stacks/observability`) and phones the on-call engineer. Warnings never
+ page.
+- **Every monitor renotifies hourly** (`RENOTIFY_MINUTES`) while it stays in
+ warning or alert.
+- `routing.test.ts` fails if any monitor mentions another handle or an email
+ address, or lacks the renotify interval.
+
+The GoAlert health check (`goalert-health`) notifies the channel only: when the
+pager is down, a page through it would never arrive.
+
+Paging monitors: the app-down synthetic, ALB non-200 responses and low healthy
+hosts, Lambda error count, web-app source errors, macro-cache memory, RDS high
+CPU, RDS storage, RDS proxy connections, the three SES reputation monitors, the
+GraphQL soup DSS 5xx and DB connections monitors, and macrodb blocked
+connections. `error-logs-overall` notifies without paging because it crosses
+its thresholds many times a day.
+
+## Prod threshold monitors
+
+`monitors/macrodb.ts`, `redis.ts`, `queues.ts` and `regressions.ts` declare
+the prod threshold, forecast and anomaly monitors with `prodMonitor()`. The ones
+on CloudWatch metrics have hand-written copies in Grafana
+(`infra/stacks/observability/nixos/alerting.nix`), and Grafana pages on-call for
+the critical ones through GoAlert. Their Datadog copies only notify, so one
+incident pages once. Keep thresholds and windows in sync by hand. The error-logs
+anomaly monitor reads `macro.logs.errors`, a log-based metric declared in
+`regressions.ts`, because log monitors have no anomaly function.
## Read this before deploying
+**Set the GoAlert integration key and deploy GoAlert first.** The webhooks
+need the key, and paging monitors reference them:
+
+```bash
+pulumi config set --secret goalertIntegrationKey '<Generic API key of the Datadog service in GoAlert>'
+```
+
**Run the import before the first `pulumi up`.** The program declares 35
monitors that already exist in Datadog. Until `pulumi import` has told Pulumi
-they exist, a deploy would create 35 duplicates — duplicate pages to
-`#holy-shit-alarms` included.
+they exist, a deploy would create 35 duplicates — duplicate pages
+included.
Adoption is a one-time local run. `pulumi import` reads Datadog and writes
Pulumi state — it does not create, modify, or pause anything in Datadog. The
@@ -39,11 +85,16 @@ and records options omitted during import.
The gate is that preview:
-- **35 monitors + 1 synthetic test unchanged** — adoption is exact.
-- **2 monitors to create** (`ai-editing-*`) — expected, that's the new alerting.
-- **Any update, replace, or delete** — the hand-written code in `monitors/*.ts`
- drifted from the live monitor. Reconcile against `imported.generated.ts`
- before deploying. An `update` here silently changes who gets paged.
+- **35 monitors + 1 synthetic test updated, in `message` and
+ `renotifyInterval` only** — the routing move. The two AI-editing monitors,
+ already deployed from this stack, update the same way.
+- **69 resources to create**: 64 monitors (25 prod threshold and anomaly, 19 GraphQL soup, 9
+ notification retention, 10 WhatsApp session, 1 native call push), 2 webhooks,
+ 1 log-based metric, the GoAlert synthetic test, and the notification
+ retention dashboard.
+- **Any other update, a replace, or a delete** — the hand-written code in
+ `monitors/*.ts` drifted from the live monitor. Reconcile against
+ `imported.generated.ts` before deploying.
Only then deploy: `pulumi up` locally, or **Actions → "deploy pulumi service"**
with `pulumi-service-name: monitoring`, `environment: prod` (that action already
@@ -106,10 +157,16 @@ is applied. Reach for `restrictedRoles` if convention stops being enough.
## Adding a monitor
-Declare it in the matching `monitors/*.ts` with `new datadog.Monitor(...)`. Use
-the `adopted()` helper only for monitors that already exist in Datadog — it sets
-`protect`, which is there to stop a program that no longer declares a
-pre-existing monitor from deleting it.
+Add a prod threshold or anomaly monitor with `prodMonitor()` in the matching
+`monitors/*.ts`, and its Grafana copy in `alerting.nix` when the metric is in
+CloudWatch. Declare anything else with `new datadog.Monitor(...)`, ending its
+message with `notify({ pages })` and setting `renotifyInterval: RENOTIFY_MINUTES`. Use the `adopted()` helper only
+for monitors that already exist in Datadog — it sets `protect`, which is there
+to stop a program that no longer declares a pre-existing monitor from deleting
+it, and the hourly renotify default.
+
+The MemoryDB (`cloud-storage-cache-*`) and burst-balance monitors report no
+data; retire them in a separate commit that drops `protect` first.
## AI-editing coverage
@@ -125,23 +182,21 @@ follow-up.
## Native call push coverage
`monitors/calls.ts` adds one log monitor that alerts on any rejected iOS or
-Android call push, excluding stale device registrations. A `pulumi preview`
-after it is added should show that one create and nothing else.
+Android call push, excluding stale device registrations.
It sees only publishes SNS rejects. A push that SNS accepts and the device then
drops leaves no backend trace.
## GraphQL soup rollout coverage
-`monitors/graphql-soup.ts` declares 15 new monitors for the `enable-graphql-soup`
-rollout: GraphQL latency, errors and volume, websocket upgrades,
-cloud-storage-service capacity and 5xx, macro-db-prod connections and CPU, the
-gateway ALB, and web-app cache logs. A `pulumi preview` after they are added
-should show exactly those 15 creates and nothing else.
+`monitors/graphql-soup.ts` declares 19 new monitors for the `enable-graphql-soup`
+rollout: GraphQL latency, errors and volume, per-operation errors and latency,
+unauthenticated mutations, websocket upgrades, cloud-storage-service capacity
+and 5xx, macro-db-prod connections and CPU, the gateway ALB, and web-app cache
+logs.
-Resolver errors still return HTTP 200, so the error monitors are proxies. Add
-per-operation error and latency monitors once the server emits operation names
-and error counts.
+Resolver errors still return HTTP 200, so the endpoint error monitors are
+proxies; the per-operation monitors read the `graphql.execute` span instead.
## Notification retention coverage
@@ -152,8 +207,8 @@ The dashboard URL is exported as `notificationRetentionDashboardUrl`.
Deploy the notification consumer instrumentation in DSS before these monitors;
otherwise the missing-telemetry alerts intentionally fire. This monitoring stack
-still requires a separate, explicitly approved manual deployment. The preview
-must not change adopted monitors.
+still requires a separate, explicitly approved manual deployment. Adopted
+monitors may change only in `message` and `renotifyInterval` (see the gate above).
Consumers are intentionally ungrouped and do not commit offsets, so native MSK
consumer-group lag is not meaningful. The new logs expose broker high watermark
diff --git a/infra/stacks/monitoring/adopted.ts b/infra/stacks/monitoring/adopted.ts
index 58eaaf9902..2aea5e94fb 100644
--- a/infra/stacks/monitoring/adopted.ts
+++ b/infra/stacks/monitoring/adopted.ts
@@ -1,4 +1,6 @@
import * as datadog from '@pulumi/datadog';
+import { goalertWebhooks } from './goalert';
+import { RENOTIFY_MINUTES } from './notify';
/**
* A monitor that already exists in Datadog and was adopted into this stack by
@@ -19,7 +21,13 @@ export function adopted(
// default explicit after normalizing imported state; group delay wins.
return new datadog.Monitor(
name,
- { requireFullWindow: false, newHostDelay: 300, ...args },
- { protect: true }
+ {
+ requireFullWindow: false,
+ newHostDelay: 300,
+ renotifyInterval: RENOTIFY_MINUTES,
+ ...args,
+ },
+ // Messages may mention the GoAlert webhooks, which must exist first.
+ { protect: true, dependsOn: goalertWebhooks }
);
}
diff --git a/infra/stacks/monitoring/goalert.ts b/infra/stacks/monitoring/goalert.ts
new file mode 100644
index 0000000000..964d50d643
--- /dev/null
+++ b/infra/stacks/monitoring/goalert.ts
@@ -0,0 +1,66 @@
+// Datadog → GoAlert, the on-call pager on the Ohio observability host
+// (infra/stacks/observability).
+//
+// Both webhooks post to a GoAlert Generic API integration key. GoAlert keeps
+// one open alert per `dedup` value: repeat notifications (renotify) update it,
+// and `action: close` resolves it. `$ALERT_CYCLE_KEY` stays the same from a
+// monitor group's trigger to its recovery.
+import * as datadog from '@pulumi/datadog';
+import * as pulumi from '@pulumi/pulumi';
+import { notify, RENOTIFY_MINUTES } from './notify';
+
+const GOALERT_HOST = 'oncall.macro-internal.com';
+
+// The integration key authenticates the request; it lives in this stack's
+// encrypted config: `pulumi config set --secret goalertIntegrationKey <key>`.
+const integrationKey = new pulumi.Config().requireSecret(
+ 'goalertIntegrationKey'
+);
+const url = pulumi.interpolate`https://${GOALERT_HOST}/api/v2/generic/incoming?token=${integrationKey}`;
+const alert = {
+ summary: '$EVENT_TITLE',
+ details: '$LINK\n\n$TEXT_ONLY_MSG',
+ dedup: '$ALERT_CYCLE_KEY',
+};
+
+export const goalertWebhooks = [
+ new datadog.Webhook('goalert', {
+ name: 'goalert',
+ url,
+ encodeAs: 'json',
+ payload: JSON.stringify(alert),
+ }),
+ new datadog.Webhook('goalert-close', {
+ name: 'goalert-close',
+ url,
+ encodeAs: 'json',
+ payload: JSON.stringify({ ...alert, action: 'close' }),
+ }),
+];
+
+// Watches the pager from outside. It notifies the channel only: if GoAlert is
+// down, a page through GoAlert would never arrive.
+new datadog.SyntheticsTest('goalert-health', {
+ name: 'GoAlert health check',
+ type: 'api',
+ subtype: 'http',
+ status: 'live',
+ tags: ['env:prod', 'service:goalert'],
+ locations: ['aws:us-east-1', 'aws:us-west-2'],
+ requestDefinition: { method: 'GET', url: `https://${GOALERT_HOST}/health` },
+ assertions: [{ type: 'statusCode', operator: 'is', target: '200' }],
+ optionsList: {
+ tickEvery: 300,
+ minFailureDuration: 300,
+ // Both locations must fail, so one probe's network trouble is not a page outage.
+ minLocationFailed: 2,
+ retry: { count: 1, interval: 30000 },
+ monitorName: '[PROD] GoAlert unreachable',
+ monitorOptions: { renotifyInterval: RENOTIFY_MINUTES },
+ },
+ message: `GoAlert at https://${GOALERT_HOST} is not answering its health check, so Datadog and Grafana alerts cannot page anyone. Paging falls back to the observability host's CloudWatch alarms.
+
+Runbook: https://github.com/macro-inc/macro/blob/main/infra/stacks/observability/README.md
+
+${notify({ pages: false })}`,
+});
diff --git a/infra/stacks/monitoring/index.ts b/infra/stacks/monitoring/index.ts
index 9be3ec78f4..440efbe8bc 100644
--- a/infra/stacks/monitoring/index.ts
+++ b/infra/stacks/monitoring/index.ts
@@ -1,8 +1,8 @@
// Datadog monitors and synthetic tests.
//
-// Every monitor in the org is declared here. See README.md before deploying —
-// the 35 monitors under `monitors/` are adopted, not created, and a program that
-// stops declaring one deletes it.
+// Every monitor in the org is declared here, and every one notifies through
+// notify.ts. See README.md before deploying — the 35 monitors under `monitors/`
+// are adopted, not created, and a program that stops declaring one deletes it.
// Adopted from the Datadog UI.
import './monitors/apm';
@@ -18,9 +18,14 @@ import './monitors/ses';
import './synthetics';
// Declared here first.
+import './goalert';
import './monitors/ai-editing';
import './monitors/calls';
import './monitors/graphql-soup';
+import './monitors/macrodb';
+import './monitors/queues';
+import './monitors/redis';
+import './monitors/regressions';
import './monitors/whatsapp-session';
import { notificationRetentionDashboard } from './monitors/notification-retention';
diff --git a/infra/stacks/monitoring/monitors/ai-editing.ts b/infra/stacks/monitoring/monitors/ai-editing.ts
index c32f166e5e..673e35917c 100644
--- a/infra/stacks/monitoring/monitors/ai-editing.ts
+++ b/infra/stacks/monitoring/monitors/ai-editing.ts
@@ -14,6 +14,7 @@
// `documentToken` (macro_sync_service_jwt), so a probe needs somewhere to mint
// one.
import * as datadog from '@pulumi/datadog';
+import { notify, RENOTIFY_MINUTES } from '../notify';
// The worker's entry span is `http.server.request` with resource_name `POST`.
// Custom spans like `edit.session` do NOT produce trace metrics — Datadog only
@@ -35,9 +36,10 @@ cancels; only the former counts as an error here.
- Check provider outages first: the supervisor chain falls back Anthropic ->
OpenAI, so a single provider failing shows up as latency, not errors.
-@slack-monitoring`,
+${notify({ pages: false })}`,
tags: ['env:prod', 'service:ai-editing-worker', 'product:apm'],
monitorThresholds: { critical: '50', warning: '20' },
+ renotifyInterval: RENOTIFY_MINUTES,
// Ratio of two count metrics: no traffic means no data, which must not
// alert. Quiet-hours coverage is the canary's job, not this monitor's.
onMissingData: 'default',
@@ -74,9 +76,10 @@ reads the callers' logs rather than the worker's telemetry. The common cause
historically is \`initial sync failed: timeout\` — the worker could not open the
document over the sync service.
-@slack-monitoring`,
+${notify({ pages: false })}`,
tags: ['env:prod', 'service:ai-editing-worker'],
monitorThresholds: { critical: '25', warning: '12' },
+ renotifyInterval: RENOTIFY_MINUTES,
onMissingData: 'default',
includeTags: false,
notifyAudit: false,
diff --git a/infra/stacks/monitoring/monitors/apm.ts b/infra/stacks/monitoring/monitors/apm.ts
index 6bc600024f..3ae9f75cf7 100644
--- a/infra/stacks/monitoring/monitors/apm.ts
+++ b/infra/stacks/monitoring/monitors/apm.ts
@@ -3,6 +3,7 @@
// Adopted from monitors created by hand in the Datadog UI. Do not edit a
// monitor here and in the UI at the same time — this file wins on deploy.
import { adopted } from '../adopted';
+import { notify } from '../notify';
// https://us5.datadoghq.com/monitors/19271629
adopted('apm-get-channel-messages-p95', {
@@ -28,7 +29,9 @@ p95 response time has exceeded **100ms** (threshold). During the Apr 17 incident
### Useful Links
- [Slow spans explorer](https://us5.datadoghq.com/apm/traces?query=service%3Acloud-storage-service+resource_name%3Aget_channel_messages+env%3Aprod+%40duration%3A%3E100000000)
-- [Investigation notebook](https://us5.datadoghq.com/notebook/277323)`,
+- [Investigation notebook](https://us5.datadoghq.com/notebook/277323)
+
+${notify({ pages: false })}`,
priority: '5',
tags: [
'service:cloud-storage-service',
diff --git a/infra/stacks/monitoring/monitors/cache.ts b/infra/stacks/monitoring/monitors/cache.ts
index c40df50b75..72a64fcc02 100644
--- a/infra/stacks/monitoring/monitors/cache.ts
+++ b/infra/stacks/monitoring/monitors/cache.ts
@@ -3,6 +3,7 @@
// Adopted from monitors created by hand in the Datadog UI. Do not edit a
// monitor here and in the UI at the same time — this file wins on deploy.
import { adopted } from '../adopted';
+import { notify } from '../notify';
// https://us5.datadoghq.com/monitors/1062487
adopted('macro-cache-available-memory-low', {
@@ -10,19 +11,9 @@ adopted('macro-cache-available-memory-low', {
type: 'query alert',
query:
'avg(last_1h):avg:aws.elasticache.database_memory_usage_percentage{name:macro-cache-prod*} > 90',
- message: `{{#is_alert}}
-@slack-holy-shit-alarms
-{{/is_alert}}
-{{#is_alert_recovery}}
-@slack-holy-shit-alarms
-{{/is_alert_recovery}}
+ message: `macro-cache-prod is using {{value}}% of its memory. Keys without a TTL are never evicted, so at 100% writes fail.
-{{#is_warning_recovery}}
-@slack-monitoring
-{{/is_warning_recovery}}
-{{#is_warning}}
-@slack-monitoring
-{{/is_warning}}`,
+${notify({ pages: true })}`,
monitorThresholds: {
critical: '90',
warning: '80',
@@ -40,8 +31,10 @@ adopted('cloud-storage-cache-high-cpu', {
type: 'query alert',
query:
'avg(last_1h):avg:aws.memorydb.cpuutilization{clustername:cloud-storage-cache-memorydb-prod} > 90',
- message:
- '{{#is_alert}} @slack-holy-shit-alarms {{/is_alert}} {{#is_alert_recovery}} @slack-holy-shit-alarms {{/is_alert_recovery}}',
+ // Reports no data: the MemoryDB cluster appears to be gone. Retire separately.
+ message: `cloud-storage-cache MemoryDB CPU is at {{value}}%.
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '90',
warning: '80',
@@ -59,8 +52,10 @@ adopted('cloud-storage-cache-low-memory', {
type: 'query alert',
query:
'avg(last_1h):avg:aws.memorydb.database_memory_usage_percentage{*} > 90',
- message:
- '{{#is_alert}} @slack-holy-shit-alarms {{/is_alert}} {{#is_alert_recovery}} @slack-holy-shit-alarms {{/is_alert_recovery}}',
+ // Reports no data: the MemoryDB cluster appears to be gone. Retire separately.
+ message: `MemoryDB memory usage is at {{value}}%.
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '90',
warning: '60',
diff --git a/infra/stacks/monitoring/monitors/calls.ts b/infra/stacks/monitoring/monitors/calls.ts
index 91089c3414..f17cae7ef3 100644
--- a/infra/stacks/monitoring/monitors/calls.ts
+++ b/infra/stacks/monitoring/monitors/calls.ts
@@ -15,6 +15,7 @@
// rejects them as disabled or missing endpoints, and `@error` carries the
// `PushEndpointUnavailable` context text, so they are excluded.
import * as datadog from '@pulumi/datadog';
+import { notify, RENOTIFY_MINUTES } from '../notify';
export const nativeCallPushFailing = new datadog.Monitor(
'native-call-push-failing',
@@ -32,9 +33,10 @@ task role lacks \`sns:Publish\` on that platform application: add it to
- [Failure logs](https://us5.datadoghq.com/logs?query=service%3Acloud-storage-service%20env%3Aprod%20%22voip%20push%3A%20SNS%20delivery%20failed%22)
-@webhook-macro-alert-hook`,
+${notify({ pages: false })}`,
tags: ['env:prod', 'service:cloud-storage-service'],
monitorThresholds: { critical: '1' },
+ renotifyInterval: RENOTIFY_MINUTES,
onMissingData: 'default',
includeTags: false,
notifyAudit: false,
diff --git a/infra/stacks/monitoring/monitors/compute.ts b/infra/stacks/monitoring/monitors/compute.ts
index 350e6e6c9b..6e47a98dc0 100644
--- a/infra/stacks/monitoring/monitors/compute.ts
+++ b/infra/stacks/monitoring/monitors/compute.ts
@@ -3,6 +3,7 @@
// Adopted from monitors created by hand in the Datadog UI. Do not edit a
// monitor here and in the UI at the same time — this file wins on deploy.
import { adopted } from '../adopted';
+import { notify } from '../notify';
// https://us5.datadoghq.com/monitors/1062688
adopted('ecs-high-cpu', {
@@ -12,7 +13,7 @@ adopted('ecs-high-cpu', {
'avg(last_5m):sum:ecs.fargate.cpu.percent{(ecs_cluster:macro-cluster-prod OR ecs_cluster:cloud-storage-cluster-prod OR ecs_cluster:macroai-cluster-prod-4cbe721)} by {task_name} > 500',
message: `{{task_name.name}} is experiencing high cpu usage in ECS
-@slack-monitoring`,
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '500',
warning: '400',
@@ -29,10 +30,9 @@ adopted('lambda-error-count', {
type: 'query alert',
query:
'sum(last_1h):sum:aws.lambda.errors{(level:prod OR environment:prod ) AND NOT functionname:ws-api-gateway-authorizer-lambda-prod} by {functionname}.as_rate() > 500',
- message: `{{#is_alert}}@slack-holy-shit-alarms {{/is_alert}}
-{{#is_alert_recovery}}@slack-holy-shit-alarms{{/is_alert_recovery}}
+ message: `Lambda {{functionname.name}} is failing at {{value}} errors per second.
-@slack-monitoring`,
+${notify({ pages: true })}`,
monitorThresholds: {
critical: '500',
warning: '100',
@@ -50,9 +50,9 @@ adopted('document-text-extractor-prod', {
type: 'query alert',
query:
'sum(last_4h):(sum:aws.lambda.errors{resource:document-text-extractor-prod}.as_count() / sum:aws.lambda.invocations{resource:document-text-extractor-prod}.as_count()) * 100 > 80',
- message: `@[email protected]
-@slack-monitoring
-Document text extractor failing`,
+ message: `Document text extractor failing: {{value}}% of invocations errored over 4 hours.
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '80',
warning: '50',
@@ -62,7 +62,6 @@ Document text extractor failing`,
includeTags: false,
newHostDelay: 300,
evaluationDelay: 900,
- renotifyInterval: 0,
});
// https://us5.datadoghq.com/monitors/1291772
@@ -71,9 +70,9 @@ adopted('document-text-extractor-dev', {
type: 'query alert',
query:
'sum(last_4h):(sum:aws.lambda.errors{resource:document-text-extractor-dev}.as_count() / sum:aws.lambda.invocations{resource:document-text-extractor-dev}.as_count()) * 100 > 80',
- message: `@[email protected]
-@slack-monitoring
-Document text extractor failing`,
+ message: `Document text extractor failing: {{value}}% of invocations errored over 4 hours.
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '80',
warning: '50',
@@ -83,5 +82,4 @@ Document text extractor failing`,
includeTags: false,
newHostDelay: 300,
evaluationDelay: 900,
- renotifyInterval: 0,
});
diff --git a/infra/stacks/monitoring/monitors/dynamodb.ts b/infra/stacks/monitoring/monitors/dynamodb.ts
index 067f457696..6c7c5ed744 100644
--- a/infra/stacks/monitoring/monitors/dynamodb.ts
+++ b/infra/stacks/monitoring/monitors/dynamodb.ts
@@ -3,6 +3,7 @@
// Adopted from monitors created by hand in the Datadog UI. Do not edit a
// monitor here and in the UI at the same time — this file wins on deploy.
import { adopted } from '../adopted';
+import { notify } from '../notify';
// https://us5.datadoghq.com/monitors/16917541
adopted('dynamodb-write-capacity', {
@@ -10,8 +11,9 @@ adopted('dynamodb-write-capacity', {
type: 'query alert',
query:
'avg(last_1h):avg:aws.dynamodb.consumed_write_capacity_units{*} by {tablename} > 500',
- message: `@[email protected]
+ message: `DynamoDB table {{tablename.name}} is consuming {{value}} write capacity units.
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '500',
warning: '300',
@@ -29,8 +31,9 @@ adopted('dynamodb-read-capacity', {
type: 'query alert',
query:
'avg(last_1h):avg:aws.dynamodb.consumed_read_capacity_units{*} by {tablename} > 20',
- message: `@[email protected]
+ message: `DynamoDB table {{tablename.name}} is consuming {{value}} read capacity units.
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '20',
warning: '10',
diff --git a/infra/stacks/monitoring/monitors/graphql-soup.ts b/infra/stacks/monitoring/monitors/graphql-soup.ts
index 120a25db72..10f15bdac7 100644
--- a/infra/stacks/monitoring/monitors/graphql-soup.ts
+++ b/infra/stacks/monitoring/monitors/graphql-soup.ts
@@ -14,6 +14,8 @@
// metric tags) once https://github.com/macro-inc/macro/pull/7911 deploys; until
// then they have no data.
import * as datadog from '@pulumi/datadog';
+import { goalertWebhooks } from '../goalert';
+import { notify, RENOTIFY_MINUTES } from '../notify';
const GRAPHQL =
'service:cloud-storage-service,env:prod,resource_name:post_/dss/items/soup/graphql';
@@ -32,16 +34,6 @@ const GATEWAY_ALB = 'loadbalancer:app/prod-gateway-alb/8e99d35bdc70b1c9';
const ROLLOUT_CONTEXT = `Rollout: \`enable-graphql-soup\` (https://us.posthog.com/project/299018/feature_flags/738783).
Client-side view: https://us.posthog.com/project/299018/dashboard/2198651`;
-// Datadog webhook integration that posts into the Macro "Alerts" channel
-// (https://macro.com/app/channel/019fb34d-fcfd-7548-9a0f-7084ea16416c). There
-// is no separate severe channel, so severity is marked in the message instead.
-const MACRO_ALERTS_CHANNEL = '@webhook-macro-alert-hook';
-
-const notify = (severe: boolean) =>
- severe
- ? `{{#is_alert}}SEVERE: user-facing impact likely, act now.{{/is_alert}}\n${MACRO_ALERTS_CHANNEL}`
- : MACRO_ALERTS_CHANNEL;
-
function rolloutMonitor(
name: string,
args: {
@@ -55,25 +47,30 @@ function rolloutMonitor(
severe?: boolean;
}
): datadog.Monitor {
- return new datadog.Monitor(name, {
- name: args.name,
- type: args.type,
- query: args.query,
- message: `${args.description}
+ return new datadog.Monitor(
+ name,
+ {
+ name: args.name,
+ type: args.type,
+ query: args.query,
+ message: `${args.description}
${ROLLOUT_CONTEXT}
-${notify(args.severe ?? false)}`,
- tags: ['env:prod', `service:${args.service}`, 'rollout:graphql-soup'],
- priority: args.severe ? '2' : '3',
- monitorThresholds: { critical: args.critical, warning: args.warning },
- // APM and log counts are sparse at night; a full window would skip
- // evaluations, and an empty count must read as zero rather than alert.
- requireFullWindow: false,
- onMissingData: 'default',
- includeTags: true,
- notifyAudit: false,
- });
+${notify({ pages: args.severe ?? false })}`,
+ tags: ['env:prod', `service:${args.service}`, 'rollout:graphql-soup'],
+ priority: args.severe ? '2' : '3',
+ monitorThresholds: { critical: args.critical, warning: args.warning },
+ renotifyInterval: RENOTIFY_MINUTES,
+ // APM and log counts are sparse at night; a full window would skip
+ // evaluations, and an empty count must read as zero rather than alert.
+ requireFullWindow: false,
+ onMissingData: 'default',
+ includeTags: true,
+ notifyAudit: false,
+ },
+ { dependsOn: args.severe ? goalertWebhooks : [] }
+ );
}
// Latency. The endpoint's p95 is long-tailed even when healthy (30-minute p95
diff --git a/infra/stacks/monitoring/monitors/load-balancer.ts b/infra/stacks/monitoring/monitors/load-balancer.ts
index dddff183ed..4468adbb12 100644
--- a/infra/stacks/monitoring/monitors/load-balancer.ts
+++ b/infra/stacks/monitoring/monitors/load-balancer.ts
@@ -3,6 +3,7 @@
// Adopted from monitors created by hand in the Datadog UI. Do not edit a
// monitor here and in the UI at the same time — this file wins on deploy.
import { adopted } from '../adopted';
+import { notify } from '../notify';
// https://us5.datadoghq.com/monitors/1062605
adopted('alb-non-200-responses', {
@@ -12,20 +13,7 @@ adopted('alb-non-200-responses', {
'sum(last_1h):(sum:aws.applicationelb.httpcode_elb_5xx{level:prod OR environment:prod*} by {loadbalancer}.as_count() / sum:aws.applicationelb.request_count{level:prod OR environment:prod*} by {loadbalancer}.as_count()) * 100 > 90',
message: `Detected a high number of non-200 responses for ALB {{loadbalancer.name}}
-
-{{#is_alert}}
-@slack-holy-shit-alarms
-{{/is_alert}}
-{{#is_alert_recovery}}
-@slack-holy-shit-alarms
-{{/is_alert_recovery}}
-
-@slack-monitoring
-
-
-@slack-monitoring`,
+${notify({ pages: true })}`,
monitorThresholds: {
critical: '90',
warning: '60',
@@ -35,7 +23,6 @@ adopted('alb-non-200-responses', {
includeTags: false,
newGroupDelay: 0,
evaluationDelay: 900,
- renotifyInterval: 0,
requireFullWindow: false,
});
@@ -47,9 +34,7 @@ adopted('alb-low-healthy-hosts', {
'avg(last_1h):abs(avg:aws.applicationelb.healthy_host_count{*} by {loadbalancer}) <= 0.25',
message: `The application load balancer for {{loadbalancer.name}} has low healthy host count.
-
+${notify({ pages: true })}`,
monitorThresholds: {
critical: '0.25',
},
@@ -68,7 +53,7 @@ adopted('alb-high-response-time', {
'avg(last_1h):avg:aws.applicationelb.target_response_time.average{! name:proxy-service-alb-prod-153de67 , ! name:store-service-alb-prod-4db6301} by {loadbalancer} > 3',
message: `Loadbalancer {{loadbalancer.name}} has a high average response time
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '3',
},
diff --git a/infra/stacks/monitoring/monitors/logs.ts b/infra/stacks/monitoring/monitors/logs.ts
index 3527d51354..3a2bae165b 100644
--- a/infra/stacks/monitoring/monitors/logs.ts
+++ b/infra/stacks/monitoring/monitors/logs.ts
@@ -3,6 +3,7 @@
// Adopted from monitors created by hand in the Datadog UI. Do not edit a
// monitor here and in the UI at the same time — this file wins on deploy.
import { adopted } from '../adopted';
+import { notify } from '../notify';
// DSS logs every expired-JWT rejection at ERROR. These are routine client token
// refreshes and make up about 60% of its prod error lines, so the DSS error
@@ -18,8 +19,9 @@ adopted('macroai-error-logs', {
type: 'log alert',
query:
'logs("service:(macroai-query-service-prod OR macroai-store-service-prod OR macroai-proxy-service-prod) status:error").index("*").rollup("count").last("5m") > 2500',
- message: `@[email protected]
-@slack-monitoring`,
+ message: `macroai services logged {{value}} errors in 5 minutes.
+
+${notify({ pages: false })}`,
tags: ['macro-ai'],
monitorThresholds: {
critical: '2500',
@@ -29,7 +31,6 @@ adopted('macroai-error-logs', {
notifyAudit: false,
includeTags: false,
newHostDelay: 300,
- renotifyInterval: 0,
enableLogsSample: true,
groupbySimpleMonitor: false,
});
@@ -40,7 +41,9 @@ adopted('macroapi-error-logs', {
type: 'log alert',
query:
'logs("service:macro-gql-service-prod status:error").index("*").rollup("count").last("5m") > 50',
- message: '@slack-monitoring',
+ message: `macro-gql-service logged {{value}} errors in 5 minutes.
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '50',
},
@@ -57,7 +60,9 @@ adopted('dss-errors-dev', {
name: '[DEV] Increased number of errors for DSS',
type: 'log alert',
query: `logs("service:cloud-storage-service env:dev status:error ${DSS_EXPIRED_JWT_EXCLUSION}").index("*").rollup("count").last("5m") > 200`,
- message: '@slack-monitoring',
+ message: `Dev cloud-storage-service logged {{value}} errors in 5 minutes.
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '200',
warning: '100',
@@ -77,7 +82,9 @@ adopted('dss-errors-prod', {
name: '[PROD] Increased number of errors for DSS',
type: 'log alert',
query: `logs("service:cloud-storage-service env:prod status:error ${DSS_EXPIRED_JWT_EXCLUSION}").index("*").rollup("count").last("5m") > 150`,
- message: '@slack-monitoring',
+ message: `Prod cloud-storage-service logged {{value}} errors in 5 minutes, not counting expired-JWT rejections.
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '150',
},
@@ -94,17 +101,12 @@ adopted('error-logs-overall', {
name: '[PROD] Increase in Error Logs',
type: 'log alert',
query: 'formula("query - query1").last("5m") > 25000',
+ // Crosses its thresholds many times a day, so it notifies without paging.
+ // The per-service anomaly monitors in regressions.ts cover regressions.
message: `Increase in overall count of error logs across production services.
Please view datadog to checkout what specific items may be causing the issues.
-{{#is_alert}}
-@slack-holy-shit-alarms
-{{/is_alert}}
-{{#is_alert_recovery}}
-@slack-holy-shit-alarms
-{{/is_alert_recovery}}
-
-@slack-monitoring`,
+${notify({ pages: false })}`,
priority: '1',
tags: ['prod'],
monitorThresholds: {
@@ -156,7 +158,9 @@ adopted('graphql-redis-connect', {
type: 'log alert',
query:
'logs("service:macro-gql-service-prod status:error \\"Failed to reconnect to Redis cache at URL*\\"").index("*").rollup("count").last("5m") >= 1',
- message: '@slack-monitoring',
+ message: `macro-gql-service could not reconnect to its Redis cache.
+
+${notify({ pages: false })}`,
priority: '3',
tags: ['prod'],
monitorThresholds: {
@@ -176,12 +180,13 @@ adopted('high-channel-invites', {
type: 'log alert',
query:
'logs("service:notification-service-prod @notification_event_type:channel_invite").index("*").rollup("count", "@sender_id").last("1h") > 100',
- message: `@slack-monitoring
-High number of channel invites
+ message: `High number of channel invites
Investigate whether 1 user is causing this with this log query
-\`service:notification-service-prod @notification_event_type:channel_invite\``,
+\`service:notification-service-prod @notification_event_type:channel_invite\`
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '100',
warning: '50',
@@ -202,9 +207,7 @@ adopted('channel-invite-spam-dev', {
'logs("service:notification-service-dev message:\\"processing message\\" @notification_event_type:channel_invite").index("*").rollup("count").by("@sender_id").last("1h") > 10',
message: `A user is generating a high number of channel invites. This could mean they are spamming users with invite emails.
-@slack-monitoring
-
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '10',
warning: '5',
@@ -225,9 +228,7 @@ adopted('channel-invite-spam-prod', {
'logs("service:notification-service-prod message:\\"processing message\\" @notification_event_type:channel_invite").index("*").rollup("count").by("@sender_id").last("1h") > 10',
message: `A user is generating a high number of channel invites. This could mean they are spamming users with invite emails.
-@slack-monitoring
-
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '10',
warning: '5',
diff --git a/infra/stacks/monitoring/monitors/macrodb.ts b/infra/stacks/monitoring/monitors/macrodb.ts
new file mode 100644
index 0000000000..b191b943c1
--- /dev/null
+++ b/infra/stacks/monitoring/monitors/macrodb.ts
@@ -0,0 +1,166 @@
+// MacroDB (macro-db-prod and its read replica) health.
+//
+// Thresholds come from two weeks of 5-minute data before the October overload.
+// Monitors on CloudWatch metrics also exist in Grafana
+// (infra/stacks/observability/nixos/alerting.nix); change both together.
+// Grafana pages on-call for those through GoAlert, so they only notify here.
+import { prodMonitor } from '../prod-monitor';
+
+prodMonitor('macrodb-primary-freeable-memory', {
+ title: 'macro-db-prod freeable memory low',
+ service: 'macro-db-prod',
+ query:
+ 'avg(last_15m):avg:aws.rds.freeable_memory{dbinstanceidentifier:macro-db-prod} < 1000000000',
+ monitorThresholds: { critical: '1000000000', warning: '2000000000' },
+ pages: false,
+ description:
+ 'Freeable memory is {{value}} bytes. Normal is 7-18 GB; it fell to about 0.45 GB during the October overload. Each connection holds about 14 MB of private memory, so check connection count and swap first.',
+ runbook: 'docs/runbooks/macrodb.md#memory',
+});
+
+prodMonitor('macrodb-replica-freeable-memory', {
+ title: 'macro-db-prod-read-replica freeable memory low',
+ service: 'macro-db-prod',
+ query:
+ 'avg(last_15m):avg:aws.rds.freeable_memory{dbinstanceidentifier:macro-db-prod-read-replica} < 750000000',
+ monitorThresholds: { critical: '750000000', warning: '1500000000' },
+ pages: false,
+ description:
+ 'Replica freeable memory is {{value}} bytes. Normal minimum is about 3 GB. The replica serves GraphQL reads, so memory pressure shows up as slow lists.',
+ runbook: 'docs/runbooks/macrodb.md#memory',
+});
+
+prodMonitor('macrodb-swap', {
+ title: 'macro-db-prod swapping',
+ service: 'macro-db-prod',
+ query:
+ 'avg(last_15m):max:aws.rds.swap_usage{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier} > 5000000000',
+ monitorThresholds: { critical: '5000000000', warning: '2000000000' },
+ pages: false,
+ description:
+ '{{dbinstanceidentifier.name}} is using {{value}} bytes of swap. Normal is under 0.6 GB. Swap above 5 GB preceded both October overloads by hours.',
+ runbook: 'docs/runbooks/macrodb.md#memory',
+});
+
+prodMonitor('macrodb-load', {
+ title: 'macro-db-prod database load above vCPU count',
+ service: 'macro-db-prod',
+ query:
+ 'avg(last_15m):avg:aws.rds.dbload_relative_to_num_vcpus{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier} > 4',
+ monitorThresholds: { critical: '4', warning: '2' },
+ pages: false,
+ description:
+ '{{dbinstanceidentifier.name}} has {{value}}x as many active sessions as vCPUs. Normal is under 1.3; the October overload reached 8. Look at Performance Insights top waits (locks, IO) and the services holding the most connections.',
+ runbook: 'docs/runbooks/macrodb.md#load',
+});
+
+prodMonitor('macrodb-cpu', {
+ title: 'macro-db-prod CPU high',
+ service: 'macro-db-prod',
+ query:
+ 'avg(last_15m):avg:aws.rds.cpuutilization{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier} > 85',
+ monitorThresholds: { critical: '85', warning: '70' },
+ pages: false,
+ description:
+ '{{dbinstanceidentifier.name}} CPU averaged {{value}}% over 15 minutes. Normal peaks are about 30%.',
+ runbook: 'docs/runbooks/macrodb.md#load',
+});
+
+prodMonitor('macrodb-primary-connections', {
+ title: 'macro-db-prod connections high',
+ service: 'macro-db-prod',
+ query:
+ 'avg(last_10m):max:aws.rds.database_connections{dbinstanceidentifier:macro-db-prod} > 2200',
+ monitorThresholds: { critical: '2200', warning: '1700' },
+ pages: false,
+ description:
+ 'macro-db-prod has {{value}} connections. Normal 15-minute peaks are about 1,100; max_connections is 3,439. Each connection costs about 14 MB, so this also drains freeable memory.',
+ runbook: 'docs/runbooks/macrodb.md#connections',
+});
+
+prodMonitor('macrodb-replica-connections', {
+ title: 'macro-db-prod-read-replica connections high',
+ service: 'macro-db-prod',
+ query:
+ 'avg(last_10m):max:aws.rds.database_connections{dbinstanceidentifier:macro-db-prod-read-replica} > 700',
+ monitorThresholds: { critical: '700', warning: '550' },
+ pages: false,
+ description:
+ 'The read replica has {{value}} connections. Normal peaks are about 460.',
+ runbook: 'docs/runbooks/macrodb.md#connections',
+});
+
+prodMonitor('macrodb-replica-lag', {
+ title: 'macro-db-prod-read-replica lagging',
+ service: 'macro-db-prod',
+ query:
+ 'avg(last_10m):max:aws.rds.replica_lag{dbinstanceidentifier:macro-db-prod-read-replica} > 300',
+ monitorThresholds: { critical: '300', warning: '60' },
+ pages: false,
+ description:
+ 'The read replica is {{value}} seconds behind the primary. Normal is under 6 seconds. GraphQL reads use the replica, so users see stale data and missing new items.',
+ runbook: 'docs/runbooks/macrodb.md#replica-lag',
+});
+
+prodMonitor('macrodb-read-latency', {
+ title: 'macro-db-prod disk read latency high',
+ service: 'macro-db-prod',
+ query:
+ 'avg(last_15m):avg:aws.rds.read_latency{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier} > 0.01',
+ monitorThresholds: { critical: '0.01', warning: '0.005' },
+ pages: false,
+ description:
+ '{{dbinstanceidentifier.name}} disk reads take {{value}} seconds on average. Normal is about 0.6 ms.',
+ runbook: 'docs/runbooks/macrodb.md#load',
+});
+
+prodMonitor('macrodb-free-storage', {
+ title: 'macro-db-prod free storage low',
+ service: 'macro-db-prod',
+ query:
+ 'avg(last_1h):min:aws.rds.free_storage_space{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier} < 1000000000000',
+ monitorThresholds: { critical: '1000000000000', warning: '1500000000000' },
+ pages: false,
+ description:
+ '{{dbinstanceidentifier.name}} has {{value}} bytes of free storage out of 5.37 TB.',
+ runbook: 'docs/runbooks/macrodb.md#storage',
+});
+
+// Datadog only: Grafana alerting has no forecast function.
+prodMonitor('macrodb-free-storage-forecast', {
+ title: 'macro-db-prod free storage forecast to run low within a week',
+ service: 'macro-db-prod',
+ query:
+ "min(next_1w):forecast(min:aws.rds.free_storage_space{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}, 'linear', 1, interval='60m', history='1w', model='default') <= 500000000000",
+ monitorThresholds: { critical: '500000000000' },
+ pages: false,
+ description:
+ 'At the current write rate {{dbinstanceidentifier.name}} drops below 500 GB free within a week. Find what started writing; recent growth was 100-200 GB a day.',
+ runbook: 'docs/runbooks/macrodb.md#storage',
+});
+
+// Datadog only: Database Monitoring metrics; Grafana's Postgres collector runs
+// against dev only.
+prodMonitor('macrodb-blocked-connections', {
+ title: 'macro-db-prod connections blocked on locks',
+ service: 'macro-db-prod',
+ query:
+ 'avg(last_10m):max:postgresql.activity.blocked_connections{host:macro-db-prod*} by {host} > 50',
+ monitorThresholds: { critical: '50', warning: '20' },
+ pages: true,
+ description:
+ "{{host.name}} has {{value}} connections waiting on locks. Normal is under 7. Advisory and tuple locks dominated the October overload; see Database Monitoring's blocking queries view.",
+ runbook: 'docs/runbooks/macrodb.md#locks',
+});
+
+prodMonitor('macrodb-long-transactions', {
+ title: 'macro-db-prod long-running transaction',
+ service: 'macro-db-prod',
+ query:
+ 'avg(last_10m):max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host} > 1800',
+ monitorThresholds: { critical: '1800', warning: '600' },
+ pages: false,
+ description:
+ 'The oldest open transaction on {{host.name}} is {{value}} seconds old. Long transactions hold locks and stop vacuum.',
+ runbook: 'docs/runbooks/macrodb.md#locks',
+});
diff --git a/infra/stacks/monitoring/monitors/notification-retention.ts b/infra/stacks/monitoring/monitors/notification-retention.ts
index 70e343f99f..444dc1a8f7 100644
--- a/infra/stacks/monitoring/monitors/notification-retention.ts
+++ b/infra/stacks/monitoring/monitors/notification-retention.ts
@@ -1,13 +1,14 @@
// Independent notification consumers do not commit Kafka offsets. Native MSK
// consumer-group lag is therefore absent; these logs report application backlog.
import * as datadog from '@pulumi/datadog';
+import { notify, RENOTIFY_MINUTES } from '../notify';
const HEALTH = '@telemetry_event:notification_consumer_health';
const CONTEXT = `Notification-retention rollout: pause scheduled cleanup before investigating.
Check consumer health/backlog, notification.delete.publish spans, database latency,
and replica lag. A zero application backlog is not an end-to-end delivery ACK:
Postgres NOTIFY publication and WebSocket delivery can still fail.
-@webhook-macro-alert-hook`;
+${notify({ pages: false })}`;
for (const environment of ['dev', 'prod']) {
const query = `${HEALTH} env:${environment}`;
@@ -20,6 +21,7 @@ for (const environment of ['dev', 'prod']) {
priority: environment === 'prod' ? '2' : '3',
includeTags: true,
requireFullWindow: false,
+ renotifyInterval: RENOTIFY_MINUTES,
};
new datadog.Monitor(`notification-consumer-backlog-${environment}`, {
...common,
@@ -73,6 +75,7 @@ new datadog.Monitor('notification-retention-telemetry-export', {
priority: '3',
monitorThresholds: { critical: '0' },
requireFullWindow: false,
+ renotifyInterval: RENOTIFY_MINUTES,
onMissingData: 'default',
});
diff --git a/infra/stacks/monitoring/monitors/opensearch.ts b/infra/stacks/monitoring/monitors/opensearch.ts
index 022cf98aad..27b22d11e7 100644
--- a/infra/stacks/monitoring/monitors/opensearch.ts
+++ b/infra/stacks/monitoring/monitors/opensearch.ts
@@ -3,6 +3,7 @@
// Adopted from monitors created by hand in the Datadog UI. Do not edit a
// monitor here and in the UI at the same time — this file wins on deploy.
import { adopted } from '../adopted';
+import { notify } from '../notify';
// https://us5.datadoghq.com/monitors/21836655
adopted('opensearch-storage-low', {
@@ -18,8 +19,6 @@ Storage utilization: {{value}}%
Critical threshold: {{threshold}}%
Opensearch storage utilization has reached a critical level. Investigate storage growth or increase the allocated storage.
-
-@webhook-macro-alert-hook
{{/is_alert}}
{{#is_warning}}
@@ -30,8 +29,6 @@ Storage utilization: {{value}}%
Warning threshold: {{warn_threshold}}%
Opensearch storage utilization is approaching the critical threshold.
-
-@webhook-macro-alert-hook
{{/is_warning}}
{{#is_recovery}}
@@ -41,11 +38,9 @@ Environment: {{environment}}
Storage utilization: {{value}}%
Opensearch storage utilization has returned to an acceptable level.
-
-@webhook-macro-alert-hook
{{/is_recovery}}
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '10000',
warning: '20000',
diff --git a/infra/stacks/monitoring/monitors/queues.ts b/infra/stacks/monitoring/monitors/queues.ts
new file mode 100644
index 0000000000..393de6b269
--- /dev/null
+++ b/infra/stacks/monitoring/monitors/queues.ts
@@ -0,0 +1,61 @@
+// Prod SQS queues: dead letters, backlog age and backlog growth.
+//
+// The two queue-age monitors also exist in Grafana
+// (infra/stacks/observability/nixos/alerting.nix); change both together.
+// Grafana pages on-call for the work queues, so they only notify here.
+import { prodMonitor } from '../prod-monitor';
+
+// Datadog only: CloudWatch dimensions can't be filtered by name pattern.
+prodMonitor('sqs-dlq-inflow', {
+ title: 'Messages landing in a prod dead-letter queue',
+ service: 'sqs',
+ query:
+ 'sum(last_30m):sum:aws.sqs.number_of_messages_sent{queuename:*-dlq-prod} by {queuename}.as_count() > 50',
+ monitorThresholds: { critical: '50', warning: '0' },
+ pages: false,
+ description:
+ '{{value}} messages moved to {{queuename.name}} in 30 minutes. Most DLQs hold old messages permanently, so this alerts on new arrivals only.',
+ runbook: 'docs/runbooks/queues.md#dead-letters',
+});
+
+prodMonitor('sqs-email-backfill-age', {
+ title: 'Email backfill queue falling behind',
+ service: 'email-service',
+ query:
+ 'avg(last_15m):max:aws.sqs.approximate_age_of_oldest_message{queuename:email-service-backfill-queue-prod} > 10800',
+ monitorThresholds: { critical: '10800', warning: '3600' },
+ pages: false,
+ description:
+ 'The oldest message in email-service-backfill-queue-prod is {{value}} seconds old. Normal peaks are about 50 minutes; new users wait this long for their mailbox to fill.',
+ runbook: 'docs/runbooks/queues.md#backlog',
+});
+
+prodMonitor('sqs-work-queue-age', {
+ title: 'Prod work queue falling behind',
+ service: 'sqs',
+ query:
+ 'avg(last_15m):max:aws.sqs.approximate_age_of_oldest_message{queuename:email-service-refresh-queue-prod OR queuename:convert-service-queue-prod OR queuename:ai-projection-queue-prod OR queuename:email-service-gmail-webhook-queue-prod OR queuename:notification-ingress-queue-prod} by {queuename} > 7200',
+ monitorThresholds: { critical: '7200', warning: '1800' },
+ pages: false,
+ description:
+ 'The oldest message in {{queuename.name}} is {{value}} seconds old. These queues normally drain within minutes.',
+ runbook: 'docs/runbooks/queues.md#backlog',
+});
+
+// Datadog only: Grafana alerting has no anomaly detection.
+prodMonitor('sqs-email-backfill-backlog-anomaly', {
+ title: 'Email backfill backlog growing abnormally',
+ service: 'email-service',
+ query:
+ "avg(last_4h):anomalies(max:aws.sqs.approximate_number_of_messages_visible{queuename:email-service-backfill-queue-prod}, 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",
+ // An anomaly monitor recovers once no point in its window is outside the band.
+ monitorThresholds: { critical: '1', criticalRecovery: '0' },
+ monitorThresholdWindows: {
+ triggerWindow: 'last_30m',
+ recoveryWindow: 'last_15m',
+ },
+ pages: false,
+ description:
+ 'Visible messages in email-service-backfill-queue-prod are above their usual band. A jump in enqueues loads macrodb through the backfill workers.',
+ runbook: 'docs/runbooks/queues.md#backlog',
+});
diff --git a/infra/stacks/monitoring/monitors/rds.ts b/infra/stacks/monitoring/monitors/rds.ts
index 6e81abc719..9d2b1b6956 100644
--- a/infra/stacks/monitoring/monitors/rds.ts
+++ b/infra/stacks/monitoring/monitors/rds.ts
@@ -3,6 +3,7 @@
// Adopted from monitors created by hand in the Datadog UI. Do not edit a
// monitor here and in the UI at the same time — this file wins on deploy.
import { adopted } from '../adopted';
+import { notify } from '../notify';
// https://us5.datadoghq.com/monitors/1062630
adopted('rds-high-cpu', {
@@ -10,12 +11,9 @@ adopted('rds-high-cpu', {
type: 'query alert',
query:
'avg(last_1h):sum:aws.rds.cpuutilization{name:*-prod} by {name}.weighted() >= 95',
- message: `@[email protected]
-@webhook-macro-alert-hook
+ message: `RDS DB {{name.name}} is experiencing high CPU utilization.
-RDS DB {{name.name}} is experiencing high CPU utilization.`,
+${notify({ pages: true })}`,
monitorThresholds: {
critical: '95',
warning: '80',
@@ -25,7 +23,6 @@ RDS DB {{name.name}} is experiencing high CPU utilization.`,
includeTags: true,
newGroupDelay: 0,
evaluationDelay: 900,
- renotifyInterval: 0,
requireFullWindow: false,
});
@@ -34,7 +31,9 @@ adopted('macrodb-low-burst-balance', {
name: '[PROD] MacroDB Low Burst Balance',
type: 'query alert',
query: 'avg(last_1h):avg:aws.rds.burst_balance{name:macro-db-prod} <= 10',
- message: '@slack-monitoring',
+ message: `macro-db-prod burst balance is at {{value}}%.
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '10',
warning: '25',
@@ -54,7 +53,7 @@ adopted('rds-proxy-connections', {
'avg(last_1h):avg:aws.rds.proxy.database_connections{proxyname:macrodb-proxy-prod} > 500',
message: `We have excessive concurrent connections in the rds proxy. This can lead to slow database calls and potential outage if other services can not secure a connection
-@slack-holy-shit-alarms`,
+${notify({ pages: true })}`,
monitorThresholds: {
critical: '500',
},
@@ -80,8 +79,6 @@ Storage utilization: {{value}}%
Critical threshold: {{threshold}}%
RDS storage utilization has reached a critical level. Investigate storage growth or increase the allocated storage.
-
-@webhook-macro-alert-hook
{{/is_alert}}
{{#is_warning}}
@@ -93,8 +90,6 @@ Storage utilization: {{value}}%
Warning threshold: {{warn_threshold}}%
RDS storage utilization is approaching the critical threshold.
-
-@webhook-macro-alert-hook
{{/is_warning}}
{{#is_recovery}}
@@ -105,11 +100,9 @@ Engine: {{engine.name}}
Storage utilization: {{value}}%
RDS storage utilization has returned to an acceptable level.
-
-@webhook-macro-alert-hook
{{/is_recovery}}
+${notify({ pages: true })}`,
tags: ['integration:amazon_rds'],
monitorThresholds: {
critical: '90',
@@ -120,7 +113,6 @@ RDS storage utilization has returned to an acceptable level.
includeTags: true,
newGroupDelay: 60,
evaluationDelay: 900,
- renotifyInterval: 0,
requireFullWindow: false,
});
@@ -132,7 +124,7 @@ adopted('rds-connection-anomaly', {
"avg(last_1d):anomalies(avg:aws.rds.database_connections{name:*-prod} by {dbinstanceidentifier}, 'basic', 2, direction='above', interval=300, alert_window='last_1h', count_default_zero='true') >= 0.99",
message: `The number of connections for RDS database {{dbinstanceidentifier.name}} is outside of the typical range.
-@slack-monitoring`,
+${notify({ pages: false })}`,
tags: ['integration:amazon_rds'],
monitorThresholds: {
critical: '0.99',
@@ -147,7 +139,6 @@ adopted('rds-connection-anomaly', {
includeTags: true,
newGroupDelay: 60,
evaluationDelay: 900,
- renotifyInterval: 0,
requireFullWindow: false,
});
@@ -157,8 +148,11 @@ adopted('rds-low-freeable-memory', {
type: 'query alert',
query:
'avg(last_1h):avg:aws.rds.freeable_memory{environment:prod OR database_instance:macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com} by {database_instance} < 2000000000',
- message: `@[email protected]
+ // macrodb.ts and Grafana watch the same metric over a shorter window; this
+ // one only notifies.
+ message: `{{database_instance.name}} freeable memory is {{value}} bytes.
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '2000000000',
warning: '3000000000',
@@ -176,10 +170,9 @@ adopted('rds-high-iops', {
type: 'query alert',
query:
'avg(last_1h):avg:aws.rds.write_iops{environment:prod OR database_instance:macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com} by {database_instance}.as_rate() + avg:aws.rds.read_iops{environment:prod OR database_instance:macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com} by {database_instance}.as_rate() > 10000',
- message: `@[email protected]
+ message: `{{database_instance.name}} read plus write IOPS are at {{value}}.
-@webhook-alarm_webhook_test`,
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '10000',
warning: '9000',
diff --git a/infra/stacks/monitoring/monitors/redis.ts b/infra/stacks/monitoring/monitors/redis.ts
new file mode 100644
index 0000000000..5165ac7afe
--- /dev/null
+++ b/infra/stacks/monitoring/monitors/redis.ts
@@ -0,0 +1,44 @@
+// ElastiCache memory for every prod cache, per node.
+//
+// macro-cache-prod climbed from 60% to 100% over a month before writes failed.
+// Memory and evictions also exist in Grafana
+// (infra/stacks/observability/nixos/alerting.nix); change both together.
+// Grafana pages on-call for memory, so it only notifies here.
+import { prodMonitor } from '../prod-monitor';
+
+prodMonitor('redis-memory', {
+ title: 'Redis memory high',
+ service: 'elasticache',
+ query:
+ 'avg(last_10m):max:aws.elasticache.database_memory_usage_percentage{cacheclusterid:*prod*} by {cacheclusterid} > 90',
+ monitorThresholds: { critical: '90', warning: '75' },
+ pages: false,
+ description:
+ '{{cacheclusterid.name}} is using {{value}}% of maxmemory. Keys without a TTL are never evicted under volatile-lru, so at 100% writes fail.',
+ runbook: 'docs/runbooks/redis.md#memory',
+});
+
+// Datadog only: Grafana alerting has no forecast function.
+prodMonitor('redis-memory-forecast', {
+ title: 'Redis memory forecast to fill within three days',
+ service: 'elasticache',
+ query:
+ "max(next_3d):forecast(max:aws.elasticache.database_memory_usage_percentage{cacheclusterid:*prod*} by {cacheclusterid}, 'linear', 1, interval='60m', history='1w', model='default') >= 95",
+ monitorThresholds: { critical: '95' },
+ pages: false,
+ description:
+ '{{cacheclusterid.name}} memory is growing fast enough to pass 95% within three days. macro-cache-prod climbed from 60% to 100% over a month before writes failed.',
+ runbook: 'docs/runbooks/redis.md#memory',
+});
+
+prodMonitor('redis-evictions', {
+ title: 'Redis evicting keys',
+ service: 'elasticache',
+ query:
+ 'sum(last_10m):sum:aws.elasticache.evictions{cacheclusterid:*prod*} by {cacheclusterid}.as_count() > 1000',
+ monitorThresholds: { critical: '1000', warning: '0' },
+ pages: false,
+ description:
+ '{{cacheclusterid.name}} evicted {{value}} keys in 10 minutes. Prod caches normally evict nothing.',
+ runbook: 'docs/runbooks/redis.md#memory',
+});
diff --git a/infra/stacks/monitoring/monitors/regressions.ts b/infra/stacks/monitoring/monitors/regressions.ts
new file mode 100644
index 0000000000..86ff6c09e8
--- /dev/null
+++ b/infra/stacks/monitoring/monitors/regressions.ts
@@ -0,0 +1,88 @@
+// Per-service error and latency anomalies: each compares a service with its
+// own usual band, so it fires on a change rather than a fixed count.
+//
+// Datadog only: prod traces and logs don't reach Grafana, and Grafana alerting
+// has no anomaly detection.
+import * as datadog from '@pulumi/datadog';
+import { prodMonitor } from '../prod-monitor';
+
+// Datadog has no anomaly function for log monitors, so the error-logs anomaly
+// monitor reads this log-based metric instead. Counts every prod error log
+// line, tagged by service.
+new datadog.LogsMetric('macro-logs-errors', {
+ name: 'macro.logs.errors',
+ compute: { aggregationType: 'count' },
+ filter: { query: 'env:prod status:error' },
+ groupBies: [{ path: 'service', tagName: 'service' }],
+});
+
+// An anomaly monitor recovers once no point in its window is outside the band.
+const ANOMALY_THRESHOLDS = { critical: '1', criticalRecovery: '0' };
+const ANOMALY_WINDOWS = {
+ triggerWindow: 'last_30m',
+ recoveryWindow: 'last_15m',
+};
+
+prodMonitor('apm-server-error-anomaly', {
+ title: 'HTTP server errors abnormally high',
+ service: 'apm',
+ query:
+ "avg(last_4h):anomalies(sum:trace.http.server.request.errors{env:prod} by {service}.as_count(), 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",
+ monitorThresholds: ANOMALY_THRESHOLDS,
+ monitorThresholdWindows: ANOMALY_WINDOWS,
+ pages: false,
+ description:
+ '{{service.name}} is returning more HTTP server errors than its usual band. Check the latest deploy of that service first.',
+ runbook: 'docs/runbooks/regressions.md#errors',
+});
+
+prodMonitor('apm-internal-error-anomaly', {
+ title: 'Internal span errors abnormally high',
+ service: 'apm',
+ query:
+ "avg(last_4h):anomalies(sum:trace.Internal.errors{env:prod} by {service}.as_count(), 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",
+ monitorThresholds: ANOMALY_THRESHOLDS,
+ monitorThresholdWindows: ANOMALY_WINDOWS,
+ pages: false,
+ description:
+ '{{service.name}} is recording more failed internal spans (background jobs, workers, browser operations) than its usual band. Check the latest deploy of that service first.',
+ runbook: 'docs/runbooks/regressions.md#errors',
+});
+
+prodMonitor('apm-consumer-error-anomaly', {
+ title: 'Queue consumer errors abnormally high',
+ service: 'apm',
+ query:
+ "avg(last_4h):anomalies(sum:trace.Consumer.errors{env:prod} by {service}.as_count(), 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",
+ monitorThresholds: ANOMALY_THRESHOLDS,
+ monitorThresholdWindows: ANOMALY_WINDOWS,
+ pages: false,
+ description:
+ '{{service.name}} is failing more queue and Kafka messages than its usual band.',
+ runbook: 'docs/runbooks/regressions.md#errors',
+});
+
+prodMonitor('apm-latency-anomaly', {
+ title: 'HTTP latency abnormally high',
+ service: 'apm',
+ query:
+ "avg(last_4h):anomalies(p95:trace.http.server.request{env:prod AND (service:cloud-storage-service OR service:authentication-service)} by {service}, 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",
+ monitorThresholds: ANOMALY_THRESHOLDS,
+ monitorThresholdWindows: ANOMALY_WINDOWS,
+ pages: false,
+ description: '{{service.name}} p95 HTTP latency is above its usual band.',
+ runbook: 'docs/runbooks/regressions.md#latency',
+});
+
+prodMonitor('error-logs-anomaly', {
+ title: 'Error logs abnormally high',
+ service: 'logs',
+ query:
+ "avg(last_4h):anomalies(sum:macro.logs.errors{*} by {service}.as_count(), 'agile', 3, direction='above', interval=300, alert_window='last_30m', count_default_zero='true') >= 1",
+ monitorThresholds: ANOMALY_THRESHOLDS,
+ monitorThresholdWindows: ANOMALY_WINDOWS,
+ pages: false,
+ description:
+ '{{service.name}} is logging more errors than its usual band. Check the latest deploy of that service first.',
+ runbook: 'docs/runbooks/regressions.md#errors',
+});
diff --git a/infra/stacks/monitoring/monitors/rum.ts b/infra/stacks/monitoring/monitors/rum.ts
index 4ed2f44b76..f636ac39d5 100644
--- a/infra/stacks/monitoring/monitors/rum.ts
+++ b/infra/stacks/monitoring/monitors/rum.ts
@@ -3,6 +3,7 @@
// Adopted from monitors created by hand in the Datadog UI. Do not edit a
// monitor here and in the UI at the same time — this file wins on deploy.
import { adopted } from '../adopted';
+import { notify } from '../notify';
// https://us5.datadoghq.com/monitors/1177082
adopted('web-app-errors-staging', {
@@ -10,7 +11,9 @@ adopted('web-app-errors-staging', {
type: 'rum alert',
query:
'rum("@type:error @application.name:\\"Web App\\" env:production @view.url_host:staging.macro.com").rollup("count").last("5m") > 90',
- message: '@slack-monitoring',
+ message: `The staging web app recorded {{value}} errors in 5 minutes.
+
+${notify({ pages: false })}`,
monitorThresholds: {
critical: '90',
warning: '50',
@@ -28,8 +31,7 @@ adopted('web-app-source-errors-prod', {
query: 'formula("(1 - (query - query1) / query) * 100").last("1h") > 90',
message: `{{value}} % of all views in Web App are throwing errors from source.
-@slack-monitoring
-{{#is_alert}}@slack-holy-shit-alarms {{/is_alert}}`,
+${notify({ pages: true })}`,
priority: '2',
tags: ['prod'],
monitorThresholds: {
diff --git a/infra/stacks/monitoring/monitors/ses.ts b/infra/stacks/monitoring/monitors/ses.ts
index ce427ff29b..593e046c9b 100644
--- a/infra/stacks/monitoring/monitors/ses.ts
+++ b/infra/stacks/monitoring/monitors/ses.ts
@@ -3,6 +3,7 @@
// Adopted from monitors created by hand in the Datadog UI. Do not edit a
// monitor here and in the UI at the same time — this file wins on deploy.
import { adopted } from '../adopted';
+import { notify } from '../notify';
// https://us5.datadoghq.com/monitors/1333601
adopted('ses-bounce-rate', {
@@ -12,9 +13,7 @@ adopted('ses-bounce-rate', {
'avg(last_1h):avg:aws.ses.reputation_bounce_rate{aws_account:569036502058} > 0.05',
message: `SES Bounce Rate Increasing.
-{{#is_alert}} @slack-holy-shit-alarms {{/is_alert}} {{#is_alert_recovery}} @slack-holy-shit-alarms {{/is_alert_recovery}}
-
-@slack-monitoring`,
+${notify({ pages: true })}`,
monitorThresholds: {
critical: '0.05',
warning: '0.03',
@@ -34,13 +33,7 @@ adopted('ses-complaint-rate', {
'avg(last_1h):avg:aws.ses.reputation_complaint_rate{aws_account:569036502058} > 0.005',
message: `SES Bounce Rate Increasing. Action needs to be taken to prevent account suspension.
-{{#is_warning}}@slack-monitoring{{/is_warning}}
-{{#is_warning_recovery}}@slack-monitoring {{/is_warning_recovery}}
-
-{{#is_alert}}@slack-holy-shit-alarms {{/is_alert}}
-{{#is_alert_recovery}}@slack-holy-shit-alarms @slack-monitoring {{/is_alert_recovery}}
-
+${notify({ pages: true })}`,
monitorThresholds: {
critical: '0.005',
warning: '0.001',
@@ -59,13 +52,7 @@ adopted('ses-high-email-sends', {
query: 'sum(last_1h):avg:aws.ses.send.sum{*} by {aws_account} > 250',
message: `SES is experiencing high email sends. This could be related to a potential attack and may require users to be blocked.
-{{#is_warning}}@slack-monitoring{{/is_warning}}
-{{#is_warning_recovery}}@slack-monitoring {{/is_warning_recovery}}
-
-{{#is_alert}}@slack-holy-shit-alarms {{/is_alert}}
-{{#is_alert_recovery}}@slack-holy-shit-alarms @slack-monitoring {{/is_alert_recovery}}
-
+${notify({ pages: true })}`,
monitorThresholds: {
critical: '250',
warning: '200',
diff --git a/infra/stacks/monitoring/monitors/whatsapp-session.ts b/infra/stacks/monitoring/monitors/whatsapp-session.ts
index 98136ff4a6..cfb48c45d9 100644
--- a/infra/stacks/monitoring/monitors/whatsapp-session.ts
+++ b/infra/stacks/monitoring/monitors/whatsapp-session.ts
@@ -15,12 +15,13 @@
// Ids are opaque account uuids; never phone numbers, JIDs, or content.
// Thresholds are starting points: there is no baseline before launch.
import * as datadog from '@pulumi/datadog';
+import { notify, RENOTIFY_MINUTES } from '../notify';
const CONTEXT = `Runbook: services/whatsapp_session_service/AGENTS.md. To stop all
WhatsApp traffic, scale whatsapp-session-service to 0 tasks; sends recorded
meanwhile expire unsent after ten minutes. Turning off the enable-whatsapp-linked
flag only hides the app UI.
-@webhook-macro-alert-hook`;
+${notify({ pages: false })}`;
const HEALTH = '@telemetry_event:whatsapp_session_health';
@@ -32,6 +33,7 @@ for (const environment of ['dev', 'prod']) {
priority: environment === 'prod' ? '2' : '3',
includeTags: true,
requireFullWindow: false,
+ renotifyInterval: RENOTIFY_MINUTES,
};
new datadog.Monitor(`whatsapp-session-unowned-accounts-${environment}`, {
diff --git a/infra/stacks/monitoring/notify.ts b/infra/stacks/monitoring/notify.ts
new file mode 100644
index 0000000000..4a4380d61e
--- /dev/null
+++ b/infra/stacks/monitoring/notify.ts
@@ -0,0 +1,47 @@
+// Where every monitor in this stack notifies.
+//
+// All monitors post to the Macro Alerts channel. Paging monitors also open a
+// GoAlert alert, which phones the on-call engineer, and close it on recovery.
+// Datadog sends warning and alert notifications again every RENOTIFY_MINUTES
+// while a monitor stays in that state, so an alert nobody fixed keeps showing up.
+
+// Datadog webhook integration that posts into the Macro Alerts channel. It is
+// configured in the Datadog UI, not in this stack.
+export const MACRO_ALERTS = '@webhook-macro-alert-hook';
+// Declared in goalert.ts.
+export const GOALERT = '@webhook-goalert';
+export const GOALERT_CLOSE = '@webhook-goalert-close';
+
+export const RENOTIFY_MINUTES = 60;
+
+/** The notification handles that end every monitor message. */
+export function notify({ pages }: { pages: boolean }): string {
+ if (!pages) return MACRO_ALERTS;
+ // A recovery closes the page whether the monitor recovers from alert or from
+ // a warning it dropped to first. Closing an alert GoAlert no longer has
+ // open does nothing.
+ return `{{#is_alert}}Paging on-call. ${GOALERT}{{/is_alert}}{{#is_recovery}}${GOALERT_CLOSE}{{/is_recovery}}
+${MACRO_ALERTS}`;
+}
+
+/**
+ * A monitor message: the state and value against its threshold, what it
+ * means, where to start, and the notification handles.
+ */
+export function alertMessage({
+ description,
+ runbook,
+ pages,
+}: {
+ description: string;
+ runbook: string;
+ pages: boolean;
+}): string {
+ return `{{#is_alert}}🔴 CRITICAL: {{value}} (threshold {{threshold}}){{/is_alert}}{{#is_warning}}🟠 WARNING: {{value}} (warning threshold {{warn_threshold}}){{/is_warning}}{{#is_recovery}}✅ RECOVERED: {{value}}{{/is_recovery}}
+
+${description}
+
+Runbook: https://github.com/macro-inc/macro/blob/main/${runbook}
+
+${notify({ pages })}`;
+}
diff --git a/infra/stacks/monitoring/prod-monitor.ts b/infra/stacks/monitoring/prod-monitor.ts
new file mode 100644
index 0000000000..eca574ef86
--- /dev/null
+++ b/infra/stacks/monitoring/prod-monitor.ts
@@ -0,0 +1,49 @@
+import * as datadog from '@pulumi/datadog';
+import { goalertWebhooks } from './goalert';
+import { alertMessage, RENOTIFY_MINUTES } from './notify';
+
+/**
+ * A prod monitor declared in this stack. The rules with a CloudWatch metric
+ * also exist in Grafana (infra/stacks/observability/nixos/alerting.nix); keep
+ * their thresholds and windows in sync there by hand.
+ */
+export function prodMonitor(
+ name: string,
+ args: {
+ title: string;
+ service: string;
+ query: string;
+ monitorThresholds: datadog.types.input.MonitorMonitorThresholds;
+ /** Anomaly monitors only: the query's `alert_window`, and a recovery window. */
+ monitorThresholdWindows?: datadog.types.input.MonitorMonitorThresholdWindows;
+ /** A critical alert pages on-call through GoAlert. Warnings never page. */
+ pages: boolean;
+ description: string;
+ /** Repository path with a heading anchor, e.g. `docs/runbooks/macrodb.md#memory`. */
+ runbook: string;
+ }
+): datadog.Monitor {
+ return new datadog.Monitor(
+ name,
+ {
+ name: `[PROD] ${args.title}`,
+ type: 'query alert',
+ query: args.query,
+ message: alertMessage({
+ description: args.description,
+ runbook: args.runbook,
+ pages: args.pages,
+ }),
+ tags: ['env:prod', `service:${args.service}`],
+ priority: args.pages ? '2' : '3',
+ monitorThresholds: args.monitorThresholds,
+ monitorThresholdWindows: args.monitorThresholdWindows,
+ renotifyInterval: RENOTIFY_MINUTES,
+ requireFullWindow: false,
+ onMissingData: 'default',
+ includeTags: true,
+ notifyAudit: false,
+ },
+ { dependsOn: args.pages ? goalertWebhooks : [] }
+ );
+}
diff --git a/infra/stacks/monitoring/routing.test.ts b/infra/stacks/monitoring/routing.test.ts
new file mode 100644
index 0000000000..b140d1d5e0
--- /dev/null
+++ b/infra/stacks/monitoring/routing.test.ts
@@ -0,0 +1,118 @@
+import { beforeAll, expect, test } from 'bun:test';
+import * as pulumi from '@pulumi/pulumi';
+
+const resources: pulumi.runtime.MockResourceArgs[] = [];
+
+beforeAll(async () => {
+ pulumi.runtime.setAllConfig(
+ { 'monitoring:goalertIntegrationKey': 'integration-key' },
+ ['monitoring:goalertIntegrationKey']
+ );
+ pulumi.runtime.setMocks(
+ {
+ newResource(args) {
+ resources.push(args);
+ return { id: args.name, state: args.inputs };
+ },
+ call(args) {
+ return args.inputs;
+ },
+ },
+ 'monitoring',
+ 'prod',
+ false
+ );
+ await import('./index');
+ await pulumi.runtime.waitForRPCs();
+});
+
+const monitors = () =>
+ resources.filter((r) => r.type === 'datadog:index/monitor:Monitor');
+const synthetics = () =>
+ resources.filter(
+ (r) => r.type === 'datadog:index/syntheticsTest:SyntheticsTest'
+ );
+
+test('every monitor and synthetic test notifies only the Macro Alerts channel and GoAlert', () => {
+ const routed = [...monitors(), ...synthetics()];
+ expect(routed.length).toBeGreaterThan(80);
+ for (const resource of routed) {
+ // Notification handles and email addresses; attribute names like `@error`
+ // in message text are not handles.
+ const handles: string[] =
+ resource.inputs.message.match(
+ /@(?:webhook|slack|pagerduty|opsgenie|teams)-[\w-]+|[\w.+-]+@[\w-]+\.[\w.]+/g
+ ) ?? [];
+ expect({ name: resource.name, handles }).toEqual({
+ name: resource.name,
+ handles: expect.arrayContaining(['@webhook-macro-alert-hook']),
+ });
+ for (const handle of handles) {
+ expect([
+ '@webhook-macro-alert-hook',
+ '@webhook-goalert',
+ '@webhook-goalert-close',
+ ]).toContain(handle);
+ }
+ }
+});
+
+test('every monitor renotifies while it stays in alert', () => {
+ for (const resource of monitors()) {
+ expect({
+ name: resource.name,
+ renotify: resource.inputs.renotifyInterval,
+ }).toEqual({
+ name: resource.name,
+ renotify: 60,
+ });
+ }
+ for (const resource of synthetics()) {
+ expect(resource.inputs.optionsList.monitorOptions.renotifyInterval).toBe(
+ 60
+ );
+ }
+});
+
+test('paging prod monitors open and close a GoAlert alert; the rest only notify', () => {
+ const blocked = monitors().find(
+ (r) => r.name === 'macrodb-blocked-connections'
+ );
+ expect(blocked?.inputs.message).toContain(
+ '{{#is_alert}}Paging on-call. @webhook-goalert{{/is_alert}}'
+ );
+ expect(blocked?.inputs.message).toContain(
+ '{{#is_recovery}}@webhook-goalert-close{{/is_recovery}}'
+ );
+ expect(blocked?.inputs.priority).toBe('2');
+
+ // Grafana pages for this one, so Datadog only notifies.
+ const freeable = monitors().find(
+ (r) => r.name === 'macrodb-primary-freeable-memory'
+ );
+ expect(freeable?.inputs.message).not.toContain('@webhook-goalert');
+
+ const latency = monitors().find((r) => r.name === 'macrodb-read-latency');
+ expect(latency?.inputs.message).not.toContain('@webhook-goalert');
+ expect(latency?.inputs.priority).toBe('3');
+});
+
+test('the GoAlert health check does not page through GoAlert', () => {
+ const health = synthetics().find((r) => r.name === 'goalert-health');
+ expect(health?.inputs.message).not.toContain('@webhook-goalert');
+ expect(health?.inputs.requestDefinition.url).toBe(
+ 'https://oncall.macro-internal.com/health'
+ );
+});
+
+test('anomaly rules recover at zero and carry their alert window', () => {
+ const anomaly = monitors().find((r) => r.name === 'apm-server-error-anomaly');
+ expect(anomaly?.inputs.monitorThresholds).toEqual({
+ critical: '1',
+ criticalRecovery: '0',
+ });
+ expect(anomaly?.inputs.monitorThresholdWindows).toEqual({
+ triggerWindow: 'last_30m',
+ recoveryWindow: 'last_15m',
+ });
+});
diff --git a/infra/stacks/monitoring/synthetics.ts b/infra/stacks/monitoring/synthetics.ts
index 9738e72f4f..cdd721a588 100644
--- a/infra/stacks/monitoring/synthetics.ts
+++ b/infra/stacks/monitoring/synthetics.ts
@@ -1,4 +1,6 @@
import * as datadog from '@pulumi/datadog';
+import { goalertWebhooks } from './goalert';
+import { notify, RENOTIFY_MINUTES } from './notify';
// The only external probe of the product. It owns its own alert monitor
// (19726068, "Macro App is Down") through `optionsList.monitorName` — that
@@ -11,8 +13,8 @@ import * as datadog from '@pulumi/datadog';
// here.
//
// `message` below is Datadog's default synthetics notification template,
-// reproduced verbatim from the live test. Backticks are escaped for the
-// template literal; the rendered text is unchanged.
+// reproduced from the live test with its notification handles replaced by
+// `notify`. Backticks are escaped for the template literal.
export const appWelcomeHealthCheck = new datadog.SyntheticsTest(
'app-welcome-health-check',
{
@@ -43,12 +45,13 @@ export const appWelcomeHealthCheck = new datadog.SyntheticsTest(
httpVersion: 'any',
retry: { count: 0, interval: 300 },
monitorName: 'Macro App is Down',
- monitorOptions: { notificationPresetName: 'show_all' },
+ monitorOptions: {
+ notificationPresetName: 'show_all',
+ renotifyInterval: RENOTIFY_MINUTES,
+ },
},
message: `{{! Test result details }}
-@slack-holy-shit-alarms @[email protected] @[email protected]
-
Your test {{#is_alert}}failed{{else}}recovered{{/is_alert}} after running for {{eval "synthetics.attributes.result.duration/1000" }}s on the {{#if synthetics.attributes.location.privateLocation}}Private{{else}}Managed{{/if}} Location {{synthetics.attributes.location.id}}.
{{! If alert, provide details about the failure }}
@@ -98,7 +101,9 @@ The test successfully extracted the following variables:
{{#each synthetics.attributes.result.variables.extracted}}
* **Name:** \`{{name}}\` (Global Variable ID: \`{{id}}\`)
Value: {{#if secure}}*Obfuscated (value hidden)*{{else}}\`{{{val}}}\`{{/if}}{{/each}}
-{{/if}}`,
+{{/if}}
+
+${notify({ pages: true })}`,
},
- { protect: true }
+ { protect: true, dependsOn: goalertWebhooks }
);
diff --git a/infra/stacks/observability/AGENTS.md b/infra/stacks/observability/AGENTS.md
new file mode 100644
index 0000000000..d995967ba9
--- /dev/null
+++ b/infra/stacks/observability/AGENTS.md
@@ -0,0 +1,48 @@
+# observability
+
+## What it is
+
+The Ohio (us-east-2) Grafana host: Grafana, Loki, Tempo, Prometheus, Alloy and
+the GoAlert on-call pager on one NixOS EC2 instance, separate from production in
+us-east-1. `README.md` is the full operations guide.
+
+## Start here
+
+- `nixos/host.nix`: boot order, retained volume, stable uids, service hardening.
+- `nixos/services.nix`, `nixos/goalert.nix`: the systemd services and credentials.
+- `index.ts`: VPC, ALB, certificate, DNS, alarms and EC2 user data.
+
+## Recipes
+
+- Change a Grafana prod alert rule: edit `prodRules` in `nixos/alerting.nix`, and
+ its Datadog copy in `infra/stacks/monitoring/monitors/` by hand.
+- Add a runtime setting: `settings.ts` → `render.ts` (bump the schema version) →
+ `nixos/read-user-data.py` (keys, pattern, version) → the consuming `.nix` file.
+- Add a secret: `assets/refresh-secrets.py` (validation) → `LoadCredential` in
+ the unit → README secret list. Add the unit to `secretConsumers` in `host.nix`.
+- Upgrade GoAlert: bump `version` and the hashes in `nixos/goalert/package.nix`.
+
+## Checks
+
+From this directory:
+- `nix build --dry-run .#nixosConfigurations.observability.config.system.build.toplevel`
+- `nix fmt -- <changed .nix files>`; `python3 -m py_compile` on changed Python.
+
+From `infra/`: `bun run check`, then Biome on changed `.ts`.
+
+## Gotchas
+
+- Changes ship only as a new AMI plus an approved deploy; never `pulumi up` or
+ publish an image without explicit permission.
+- The host reads secrets from the stack's own Secrets Manager secret in
+ us-east-2, so it never depends on production to boot. Doppler
+ `observability/prd` holds the values; copy changes into the secret.
+- A service listed in `host.nix` `services` gets `ProtectSystem=strict`, metadata
+ blocking and the bootstrap dependency; one that needs the instance role must be
+ in the `IPAddressDeny` exception list or run as its own unit.
+- GoAlert's application config lives in its database; `goalert/apply-config.py`
+ overwrites it on every start from `goalert.nix`. `set-config` accepts invalid
+ values silently, so check the journal for `validate config`.
+- Grafana provisioning expands `$VAR` in contact point settings, but not in rule
+ queries or annotations, so `$B` and `$values` work there and secrets must come
+ in through the Grafana unit's environment.
diff --git a/infra/stacks/observability/CLAUDE.md b/infra/stacks/observability/CLAUDE.md
new file mode 120000
index 0000000000..47dc3e3d86
--- /dev/null
+++ b/infra/stacks/observability/CLAUDE.md
@@ -0,0 +1 @@
+AGENTS.md
\ No newline at end of file
diff --git a/infra/stacks/observability/Pulumi.prod.yaml b/infra/stacks/observability/Pulumi.prod.yaml
index aff03f765a..e233257080 100644
--- a/infra/stacks/observability/Pulumi.prod.yaml
+++ b/infra/stacks/observability/Pulumi.prod.yaml
@@ -2,5 +2,5 @@ config:
aws:region: us-east-2
observability:ciIngestSecretArn: arn:aws:secretsmanager:us-east-1:569036502058:secret:observability/dev-ingest-7UhJbL
observability:alarmTopicArn: arn:aws:sns:us-east-2:569036502058:observability-prod-alerts
- observability:amiId: ami-0e88a39db857c8c52
+ observability:amiId: ami-0e775059ad164977c
observability:secretArn: arn:aws:secretsmanager:us-east-2:569036502058:secret:observability/prod-TwQpBJ
diff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.md
index d51de3801d..43fd8ccde9 100644
--- a/infra/stacks/observability/README.md
+++ b/infra/stacks/observability/README.md
@@ -62,7 +62,8 @@ historical backfill, or CI alert notifications. Validate a successful and failed
workflow, a rerun, and a multi-job trace after rollout before relying on the dashboard.
This stack starts Grafana alongside Datadog on one private EC2 instance. It
-runs Grafana, Loki, Tempo, Prometheus, Alloy and nginx as native systemd services
+runs Grafana, Loki, Tempo, Prometheus, Alloy, nginx and the GoAlert on-call
+pager (with its own Postgres) as native systemd services
on a prebuilt NixOS image. There is no Docker or Compose dependency. Nix owns
packages, configuration, service users, resource limits and startup dependencies:
@@ -73,6 +74,9 @@ packages, configuration, service users, resource limits and startup dependencies
- `nixos/host-telemetry.nix` and `nixos/host.alloy`: host metrics and journal logs.
- `nixos/cloudwatch.nix`: independent disk and memory health metrics.
- `nixos/proxy.nix`: native nginx virtual hosts and query-only backend gateway.
+- `nixos/goalert.nix` and `nixos/goalert/`: GoAlert, its Postgres, backups and
+ the Macro Alerts relay (see [On-call paging](#on-call-paging-goalert)).
+- `nixos/alerting.nix`: Grafana alert rules, contact points and routing.
- `nixos/host.nix`: boot dependencies, retained storage and stable service identities.
NixOS modules generate application configuration from Nix attributes. Alloy's
@@ -90,6 +94,7 @@ OTEL exporter -- HTTPS + bearer token --> ALB --> nginx --> Alloy
--> Loki --> S3 logs
--> Tempo --> S3 traces
--> Prometheus --> EBS
+Datadog, on-call browsers -- HTTPS --> ALB --> nginx --> GoAlert --> Postgres --> EBS
```
- One `m7i.xlarge` (4 vCPU / 16 GiB) in a new VPC's private `us-east-2a`
@@ -111,7 +116,8 @@ OTEL exporter -- HTTPS + bearer token --> ALB --> nginx --> Alloy
an application-consistent or cross-region backup. Snapshot recovery can lose
up to a day of local state. S3 does not protect telemetry still buffered locally.
- No RDS, shared filesystem, Kafka or cluster. Grafana's SQLite stores users,
- dashboards and settings, not log/trace/metric history.
+ dashboards and settings, not log/trace/metric history. GoAlert's schedules,
+ users and alerts live in a local Postgres 17 on the same volume.
- All service and OS versions are pinned by this stack's `flake.lock` on NixOS
26.05. Grafana plugin auto-install/update is disabled. Reviewed lock updates
and replacement AMIs deliver upgrades; there is no installation or build at boot.
@@ -230,10 +236,13 @@ DNS-validated regional ACM certificate. Region validation rejects `us-east-1`.
and secrets per environment. Confirm Workspace MFA enforcement.
2. Through the approved secret-management process, create a Secrets Manager JSON
secret in **us-east-2** containing `google_client_id`, `google_client_secret`,
- `grafana_secret_key`, and `otlp_token`. Generate independent cryptographically
- random values of at least 32 characters for the last two. Preserve
- `grafana_secret_key` through recovery; changing it can make stored credentials
- unreadable. Never paste secret values into shell commands, this file or PRs.
+ `grafana_secret_key`, and `otlp_token`, plus the GoAlert fields listed under
+ [On-call paging](#on-call-paging-goalert). Generate independent cryptographically
+ random values of at least 32 characters for `grafana_secret_key`, `otlp_token`
+ and `goalert_encryption_key`. Preserve `grafana_secret_key` and
+ `goalert_encryption_key` through recovery; changing them makes stored
+ credentials unreadable. Never paste secret values into shell commands, this
+ file or PRs.
3. Select an existing **us-east-2** SNS topic
with a confirmed, monitored subscription for infrastructure alarms. Its
delivery destination must remain accessible during a production outage.
@@ -426,8 +435,24 @@ models. Grafana uses the EC2 role for regional metric reads and publishing to th
configured alarm SNS topic. Other unprivileged HTTP/ingest services still cannot
access instance metadata.
-`nixos/alerting.nix` starts with the three dev monitors declared in the monitoring
-stack: DSS errors (1013923), channel-invite spam (4876245), and document text
+`prodRules` in `nixos/alerting.nix` holds the prod CloudWatch rules: macrodb
+memory, swap, load, CPU, connections, replica lag, read latency and storage;
+Redis memory and evictions; SQS queue age. Each is a warning and a critical rule
+in the `Prod` group, reading CloudWatch in Virginia directly. Each has a Datadog
+copy in `infra/stacks/monitoring/monitors/`; keep thresholds and windows in sync
+by hand. Forecasts, anomalies, Database Monitoring, APM and log rules are
+Datadog-only.
+
+Every alert posts to the `Macro Alerts` contact point through the loopback
+relay; critical ones renotify hourly, others every four hours. Critical alerts
+of rules with `pages = true` also go to the `GoAlert` contact point over
+loopback. GoAlert owns the alert from there: it opens, acknowledges and closes
+it, and its escalation posts each change to the channel. Their Datadog copies
+only notify, so an incident pages once.
+
+The dev rules below keep their own route to `Macro team email`.
+
+The three dev monitors declared in the monitoring stack: DSS errors (1013923), channel-invite spam (4876245), and document text
extractor failure ratio (1291772). Warning and critical rules are separate and
mutually exclusive. Their windows and thresholds come from those definitions;
the Lambda ratio sums four hours ending fifteen minutes ago. No-data is OK for
@@ -447,8 +472,82 @@ without SMTP credentials. The configured topic already has [email protected] and
[email protected] subscriptions; each recipient must confirm their SNS subscription.
Test the contact point and verify receipt before relying on notifications.
-This release adds `alarmTopicArn` to user-data schema 5. Deploy it together with
-the new reviewed AMI; a schema-4 image cannot boot schema-5 user data.
+User-data schema 6 adds `oncallHost`. Deploy it together with the new reviewed
+AMI; a schema-5 image cannot boot schema-6 user data.
+
+## On-call paging (GoAlert)
+
+GoAlert is the source of truth for open alerts: Grafana sends it critical
+alerts, and on-call engineers acknowledge or close them in its UI at
+`https://oncall.macro-internal.com`. It runs on this host so paging keeps working
+while production, its database or Macro itself is down. Today it pages by
+posting to the Macro Alerts channel; phone paging (Twilio), Google login and a
+rotation are follow-ups.
+
+- `nixos/goalert/package.nix` builds GoAlert from source with `buildGoModule`,
+ embedding its web UI. Bump `version` and the hashes together.
+- `nixos/goalert.nix` runs Postgres 17 on a Unix socket only (data in
+ `/srv/observability/postgres`), and GoAlert on `127.0.0.1:3030`. Its
+ application config is a Nix attrset. GoAlert stores that config only in its
+ database, so `apply-config.py` migrates the schema, runs `goalert set-config`
+ and creates the `admin` login before every start. Edits in GoAlert's admin
+ config page are overwritten on the next restart. `set-config` stores invalid
+ values without failing; GoAlert logs `validate config` errors when it loads
+ them, so check the journal after changing config.
+- A nightly timer dumps the database to `s3://<logs bucket>/goalert/`, one object
+ per weekday. The dump goes to a file first, so a failed dump never replaces a
+ good backup.
+- `macro-alerts-relay` on `127.0.0.1:8090` turns GoAlert (`/goalert`) and Grafana
+ (`/grafana`) webhooks into Macro channel posts and adds the channel's bot token.
+ GoAlert webhooks cannot set headers, and the token stays in one service.
+- nginx serves `oncall.macro-internal.com` from a runtime server block. Only
+ `/health`, `/api/v2/generic/incoming` and `/api/v2/heartbeat/` are reachable
+ without a session (GoAlert checks the integration key); other inbound
+ integrations return 404. The rest is GoAlert's UI behind its login.
+- The ALB sends the on-call host to its own target group, health-checked at
+ `/goalert-healthz`. Alarm `observability-goalert` publishes to `alarmTopicArn`
+ when it fails, so a GoAlert outage is reported without GoAlert.
+
+### Secrets
+
+Doppler `observability/prd` is the source of truth; copy the values into the
+us-east-2 Secrets Manager secret the host reads, under the lowercase names:
+`goalert_encryption_key` (≥32 random characters), `goalert_admin_password`
+(≥32 random characters, the `admin` login), `macro_alerts_webhook_url` (HTTPS)
+and `macro_alerts_bot_token` (the Alerts channel's webhook and bot token).
+`goalert_grafana_integration_key` is optional until GoAlert has created it;
+without it Grafana's pages are rejected with 401 and the GoAlert contact point
+shows as failing.
+
+### First deployment
+
+1. Add the secrets above, then build, publish and deploy the AMI as usual. The
+ certificate gains the on-call name, so Pulumi replaces the ACM certificate and
+ updates the listener.
+2. Sign in as `admin`. Create the escalation policy and service below, and a
+ Grafana integration key on `Grafana critical`. Put the key in Doppler and the
+ secret as `goalert_grafana_integration_key` and restart
+ `observability-secrets`.
+3. Send a test alert through the service and check that it reaches the channel
+ and can be acknowledged and closed.
+
+### Escalation
+
+- Escalation policy `Macro prod`, one step: webhook
+ `http://127.0.0.1:8090/goalert`. The channel sees each alert, acknowledgement
+ and close.
+- Service `Grafana critical` on `Macro prod`. Warnings never page; they only
+ reach the channel.
+- Acknowledge an alert to stop its escalation, close it when handled; Grafana
+ closes it on recovery. To silence a whole service for a while, use its
+ maintenance mode.
+
+### Recovery
+
+GoAlert's state is on the retained volume, so a replacement instance resumes it.
+For a lost volume, restore the latest dump with
+`pg_restore --clean -d goalert` before starting `goalert`, using the same
+`goalert_encryption_key`.
## Dev dual export
diff --git a/infra/stacks/observability/assets/refresh-secrets.py b/infra/stacks/observability/assets/refresh-secrets.py
index dde22d99f0..2e4b7e2951 100644
--- a/infra/stacks/observability/assets/refresh-secrets.py
+++ b/infra/stacks/observability/assets/refresh-secrets.py
@@ -3,6 +3,7 @@
import json
import os
from pathlib import Path
+import re
import subprocess
config = json.loads(Path('/opt/observability/bootstrap.json').read_text())
@@ -12,7 +13,11 @@ result = subprocess.run(
check=True, capture_output=True, text=True,
)
values = json.loads(json.loads(result.stdout)['SecretString'])
-keys = ['google_client_id', 'google_client_secret', 'grafana_secret_key', 'otlp_token']
+keys = [
+ 'google_client_id', 'google_client_secret', 'grafana_secret_key', 'otlp_token',
+ 'goalert_encryption_key', 'goalert_admin_password',
+ 'macro_alerts_webhook_url', 'macro_alerts_bot_token',
+]
# Validate the entire bundle before writing any file. Token is at least 32 bytes.
for key in keys:
value = values.get(key)
@@ -20,16 +25,27 @@ for key in keys:
raise SystemExit(f'Missing or invalid secret field: {key}')
if '\n' in value or '\r' in value:
raise SystemExit(f'Secret must be a single line: {key}')
-for key in ['grafana_secret_key', 'otlp_token']:
+for key in ['grafana_secret_key', 'otlp_token', 'goalert_encryption_key', 'goalert_admin_password']:
if len(values[key]) < 32:
raise SystemExit(f'Secret too short: {key}')
+if not values['macro_alerts_webhook_url'].startswith('https://'):
+ raise SystemExit('macro_alerts_webhook_url must use HTTPS')
+# GoAlert creates its Grafana integration key, so it can only be added after the
+# first boot. Until then GoAlert rejects Grafana's pages with 401, and Grafana
+# reports the GoAlert contact point as failing.
+grafana_key = values.get('goalert_grafana_integration_key', '')
+if not re.fullmatch(r'[0-9a-f-]{36}|', grafana_key):
+ raise SystemExit('goalert_grafana_integration_key must be a GoAlert key UUID')
+if not grafana_key:
+ print('goalert_grafana_integration_key is not set; Grafana cannot page through GoAlert')
+values['grafana-goalert.env'] = f'GOALERT_GRAFANA_KEY={grafana_key}\n'
# Native remote-write uses HTTP Basic over TLS. Only the password hash reaches nginx.
hashed = subprocess.run(
['openssl', 'passwd', '-6', '-stdin'], input=values['otlp_token'] + '\n',
check=True, capture_output=True, text=True,
).stdout.strip()
values['metrics.htpasswd'] = 'metrics:' + hashed + '\n'
-keys.append('metrics.htpasswd')
+keys += ['metrics.htpasswd', 'grafana-goalert.env']
directory = Path('/run/macro-observability')
directory.mkdir(mode=0o700, exist_ok=True)
os.chown(directory, 0, 0)
diff --git a/infra/stacks/observability/index.ts b/infra/stacks/observability/index.ts
index b9e110520e..def1a85f02 100644
--- a/infra/stacks/observability/index.ts
+++ b/infra/stacks/observability/index.ts
@@ -20,6 +20,7 @@ const baseDomain = 'macro-internal.com';
const suffix = stack === 'prod' ? '' : '-dev';
const grafanaHost = `grafana${suffix}.${baseDomain}`;
const otlpHost = `otlp${suffix}.${baseDomain}`;
+const oncallHost = `oncall${suffix}.${baseDomain}`;
const tags = { project: 'observability', environment: stack };
const durable = { protect: true, retainOnDelete: true };
const secretArn = config.require('secretArn');
@@ -34,6 +35,7 @@ validateSettings({
region,
grafanaHost,
otlpHost,
+ oncallHost,
secretArn,
alarmTopicArn,
volumeId: 'vol-0',
@@ -59,24 +61,26 @@ const zone = aws.route53.getZoneOutput({
});
const certificate = new aws.acm.Certificate('observability', {
domainName: grafanaHost,
- subjectAlternativeNames: [otlpHost],
+ subjectAlternativeNames: [otlpHost, oncallHost],
validationMethod: 'DNS',
tags,
});
-const validationRecords = [grafanaHost, otlpHost].map((hostname) => {
- const option = certificate.domainValidationOptions.apply((options) => {
- const match = options.find((entry) => entry.domainName === hostname);
- if (!match) throw new Error(`Missing ACM validation for ${hostname}`);
- return match;
- });
- return new aws.route53.Record(`observability-certificate-${hostname}`, {
- zoneId: zone.zoneId,
- name: option.resourceRecordName,
- type: option.resourceRecordType,
- records: [option.resourceRecordValue],
- ttl: 300,
- });
-});
+const validationRecords = [grafanaHost, otlpHost, oncallHost].map(
+ (hostname) => {
+ const option = certificate.domainValidationOptions.apply((options) => {
+ const match = options.find((entry) => entry.domainName === hostname);
+ if (!match) throw new Error(`Missing ACM validation for ${hostname}`);
+ return match;
+ });
+ return new aws.route53.Record(`observability-certificate-${hostname}`, {
+ zoneId: zone.zoneId,
+ name: option.resourceRecordName,
+ type: option.resourceRecordType,
+ records: [option.resourceRecordValue],
+ ttl: 300,
+ });
+ }
+);
const validatedCertificate = new aws.acm.CertificateValidation(
'observability',
{
@@ -281,6 +285,7 @@ const instance = new aws.ec2.Instance(
region,
grafanaHost,
otlpHost,
+ oncallHost,
secretArn,
alarmTopicArn,
volumeId,
@@ -346,7 +351,29 @@ new aws.lb.ListenerRule('observability-hosts', {
conditions: [{ hostHeader: { values: [grafanaHost, otlpHost] } }],
actions: [{ type: 'forward', targetGroupArn: target.arn }],
});
-for (const hostname of [grafanaHost, otlpHost]) {
+// Same nginx port, but its own health check so a GoAlert outage alarms
+// separately from Grafana.
+const oncallTarget = new aws.lb.TargetGroup('observability-oncall', {
+ port: 8080,
+ protocol: 'HTTP',
+ targetType: 'instance',
+ vpcId: vpc.vpcId,
+ healthCheck: { path: '/goalert-healthz', matcher: '200', interval: 30 },
+ deregistrationDelay: 30,
+ tags,
+});
+new aws.lb.TargetGroupAttachment('observability-oncall', {
+ targetGroupArn: oncallTarget.arn,
+ targetId: instance.id,
+ port: 8080,
+});
+new aws.lb.ListenerRule('observability-oncall', {
+ listenerArn: listener.arn,
+ priority: 2,
+ conditions: [{ hostHeader: { values: [oncallHost] } }],
+ actions: [{ type: 'forward', targetGroupArn: oncallTarget.arn }],
+});
+for (const hostname of [grafanaHost, otlpHost, oncallHost]) {
new aws.route53.Record(hostname, {
zoneId: zone.zoneId,
name: hostname,
@@ -418,6 +445,20 @@ new aws.cloudwatch.MetricAlarm('observability-ui', {
dimensions: { LoadBalancer: alb.arnSuffix, TargetGroup: target.arnSuffix },
treatMissingData: 'breaching',
});
+// Paging must not depend on GoAlert to report that GoAlert is down. On-call
+// phones subscribe to this topic by SMS directly (see README).
+new aws.cloudwatch.MetricAlarm('observability-goalert', {
+ ...alarm,
+ alarmDescription: `GoAlert at ${oncallHost} is failing health checks; pages are not being sent.`,
+ namespace: 'AWS/ApplicationELB',
+ metricName: 'UnHealthyHostCount',
+ threshold: 0,
+ dimensions: {
+ LoadBalancer: alb.arnSuffix,
+ TargetGroup: oncallTarget.arnSuffix,
+ },
+ treatMissingData: 'breaching',
+});
for (const [name, path] of [
['observability-disk', '/srv/observability'],
['observability-root-disk', '/'],
@@ -444,6 +485,7 @@ new aws.cloudwatch.MetricAlarm('observability-memory', {
export const grafanaUrl = `https://${grafanaHost}`;
export const otlpHttpEndpoint = `https://${otlpHost}`;
+export const goalertUrl = `https://${oncallHost}`;
export const instanceId = instance.id;
export const dataVolumeId = data.id;
export const logsBucket = logs.bucket;
diff --git a/infra/stacks/observability/nixos/alerting.nix b/infra/stacks/observability/nixos/alerting.nix
index dacf000b2e..e761fa4a35 100644
--- a/infra/stacks/observability/nixos/alerting.nix
+++ b/infra/stacks/observability/nixos/alerting.nix
@@ -1,4 +1,4 @@
-{ ... }:
+{ lib, ... }:
let
lokiQuery = expr: {
refId = "A";
@@ -33,36 +33,527 @@ let
datadog_url = "https://us5.datadoghq.com/monitors/${monitorId}";
description = "Migrated threshold and window; Datadog remains active during validation.";
};
- data = query ++ [
- {
+ data = query ++ expressions "last" condition;
+ };
+ # Reduce query A to one value per series (B), then compare it (C).
+ expressions = reducer: condition: [
+ {
+ refId = "B";
+ datasourceUid = "__expr__";
+ relativeTimeRange = {
+ from = 0;
+ to = 0;
+ };
+ model = {
refId = "B";
- datasourceUid = "__expr__";
- relativeTimeRange = {
- from = 0;
- to = 0;
- };
- model = {
- refId = "B";
- type = "reduce";
- expression = "A";
- reducer = "last";
- };
- }
- {
+ type = "reduce";
+ expression = "A";
+ inherit reducer;
+ };
+ }
+ {
+ refId = "C";
+ datasourceUid = "__expr__";
+ relativeTimeRange = {
+ from = 0;
+ to = 0;
+ };
+ model = {
refId = "C";
- datasourceUid = "__expr__";
+ type = "math";
+ expression = condition;
+ };
+ }
+ ];
+
+ # Prod rules. Each has a matching Datadog monitor in
+ # infra/stacks/monitoring/monitors/{macrodb,redis,queues}.ts; keep thresholds
+ # and windows in sync by hand. Grafana pages for the rules marked `pages`, so
+ # their Datadog copies only notify.
+ cloudwatch =
+ {
+ namespace,
+ metricName,
+ dimensions,
+ statistic,
+ }:
+ window: [
+ {
+ refId = "A";
+ datasourceUid = "cloudwatch";
relativeTimeRange = {
- from = 0;
+ from = window;
to = 0;
};
model = {
- refId = "C";
- type = "math";
- expression = condition;
+ refId = "A";
+ id = "a";
+ region = "us-east-1";
+ inherit
+ namespace
+ metricName
+ dimensions
+ statistic
+ ;
+ period = "60";
+ matchExact = true;
+ metricQueryType = 0;
+ metricEditorMode = 0;
+ queryMode = "Metrics";
};
}
];
+ # `window` is in seconds; `condition` compares the reduced value $B.
+ prodRule =
+ {
+ uid,
+ title,
+ severity,
+ service,
+ pages ? false,
+ metric,
+ window,
+ reducer,
+ condition,
+ description,
+ runbook,
+ }:
+ {
+ inherit uid;
+ title = "[PROD] ${title} (${severity})";
+ condition = "C";
+ for = "0s";
+ # Sparse CloudWatch counters have no datapoints when idle.
+ noDataState = "OK";
+ execErrState = "Error";
+ isPaused = false;
+ labels = {
+ environment = "prod";
+ inherit severity service;
+ # Only critical alerts page, through GoAlert.
+ pages = lib.boolToString (pages && severity == "critical");
+ };
+ annotations = {
+ summary = title;
+ inherit description;
+ runbook_url = "https://github.com/macro-inc/macro/blob/main/docs/runbooks/${runbook}";
+ };
+ data = cloudwatch metric window ++ expressions reducer condition;
+ };
+
+ rds = metricName: instances: statistic: {
+ namespace = "AWS/RDS";
+ inherit metricName statistic;
+ dimensions.DBInstanceIdentifier = instances;
+ };
+ bothInstances = [
+ "macro-db-prod"
+ "macro-db-prod-read-replica"
+ ];
+ redisNodes = [
+ "macro-cache-prod-001"
+ "macro-cache-prod-002"
+ "email-service-redis-prod-c862c10"
+ "calendar-service-redis-prod-aee80d2"
+ "connection-gateway-redis-prod-f9d8cd0"
+ ];
+ redis = metricName: statistic: {
+ namespace = "AWS/ElastiCache";
+ inherit metricName statistic;
+ dimensions.CacheClusterId = redisNodes;
+ };
+ queueAge = queues: {
+ namespace = "AWS/SQS";
+ metricName = "ApproximateAgeOfOldestMessage";
+ statistic = "Maximum";
+ dimensions.QueueName = queues;
+ };
+ workQueues = [
+ "email-service-refresh-queue-prod"
+ "convert-service-queue-prod"
+ "ai-projection-queue-prod"
+ "email-service-gmail-webhook-queue-prod"
+ "notification-ingress-queue-prod"
+ ];
+
+ primaryMemory = {
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "FreeableMemory" [ "macro-db-prod" ] "Average";
+ window = 900;
+ reducer = "mean";
+ description = "Freeable memory is {{ $values.B }} bytes. Normal is 7-18 GB; it fell to about 0.45 GB during the October overload. Each connection holds about 14 MB of private memory, so check connection count and swap first.";
+ runbook = "macrodb.md#memory";
+ };
+ replicaMemory = {
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "FreeableMemory" [ "macro-db-prod-read-replica" ] "Average";
+ window = 900;
+ reducer = "mean";
+ description = "Replica freeable memory is {{ $values.B }} bytes. Normal minimum is about 3 GB. The replica serves GraphQL reads, so memory pressure shows up as slow lists.";
+ runbook = "macrodb.md#memory";
+ };
+ swap = {
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "SwapUsage" bothInstances "Maximum";
+ window = 900;
+ reducer = "mean";
+ description = "{{ $labels.DBInstanceIdentifier }} is using {{ $values.B }} bytes of swap. Normal is under 0.6 GB. Swap above 5 GB preceded both October overloads by hours.";
+ runbook = "macrodb.md#memory";
+ };
+ load = {
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "DBLoadRelativeToNumVCPUs" bothInstances "Average";
+ window = 900;
+ reducer = "mean";
+ description = "{{ $labels.DBInstanceIdentifier }} has {{ $values.B }}x as many active sessions as vCPUs. Normal is under 1.3; the October overload reached 8. Look at Performance Insights top waits (locks, IO) and the services holding the most connections.";
+ runbook = "macrodb.md#load";
+ };
+ cpu = {
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "CPUUtilization" bothInstances "Average";
+ window = 900;
+ reducer = "mean";
+ description = "{{ $labels.DBInstanceIdentifier }} CPU averaged {{ $values.B }}% over 15 minutes. Normal peaks are about 30%.";
+ runbook = "macrodb.md#load";
+ };
+ primaryConnections = {
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "DatabaseConnections" [ "macro-db-prod" ] "Maximum";
+ window = 600;
+ reducer = "mean";
+ description = "macro-db-prod has {{ $values.B }} connections. Normal 15-minute peaks are about 1,100; max_connections is 3,439. Each connection costs about 14 MB, so this also drains freeable memory.";
+ runbook = "macrodb.md#connections";
+ };
+ replicaConnections = {
+ service = "macro-db-prod";
+ metric = rds "DatabaseConnections" [ "macro-db-prod-read-replica" ] "Maximum";
+ window = 600;
+ reducer = "mean";
+ description = "The read replica has {{ $values.B }} connections. Normal peaks are about 460.";
+ runbook = "macrodb.md#connections";
};
+ replicaLag = {
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "ReplicaLag" [ "macro-db-prod-read-replica" ] "Maximum";
+ window = 600;
+ reducer = "mean";
+ description = "The read replica is {{ $values.B }} seconds behind the primary. Normal is under 6 seconds. GraphQL reads use the replica, so users see stale data and missing new items.";
+ runbook = "macrodb.md#replica-lag";
+ };
+ readLatency = {
+ service = "macro-db-prod";
+ metric = rds "ReadLatency" bothInstances "Average";
+ window = 900;
+ reducer = "mean";
+ description = "{{ $labels.DBInstanceIdentifier }} disk reads take {{ $values.B }} seconds on average. Normal is about 0.6 ms.";
+ runbook = "macrodb.md#load";
+ };
+ freeStorage = {
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "FreeStorageSpace" bothInstances "Minimum";
+ window = 3600;
+ reducer = "min";
+ description = "{{ $labels.DBInstanceIdentifier }} has {{ $values.B }} bytes of free storage out of 5.37 TB.";
+ runbook = "macrodb.md#storage";
+ };
+ redisMemory = {
+ service = "elasticache";
+ pages = true;
+ metric = redis "DatabaseMemoryUsagePercentage" "Maximum";
+ window = 600;
+ reducer = "mean";
+ description = "{{ $labels.CacheClusterId }} is using {{ $values.B }}% of maxmemory. Keys without a TTL are never evicted under volatile-lru, so at 100% writes fail.";
+ runbook = "redis.md#memory";
+ };
+ redisEvictions = {
+ service = "elasticache";
+ metric = redis "Evictions" "Sum";
+ window = 600;
+ reducer = "sum";
+ description = "{{ $labels.CacheClusterId }} evicted {{ $values.B }} keys in 10 minutes. Prod caches normally evict nothing.";
+ runbook = "redis.md#memory";
+ };
+ backfillAge = {
+ service = "email-service";
+ metric = queueAge [ "email-service-backfill-queue-prod" ];
+ window = 900;
+ reducer = "mean";
+ description = "The oldest message in email-service-backfill-queue-prod is {{ $values.B }} seconds old. Normal peaks are about 50 minutes; new users wait this long for their mailbox to fill.";
+ runbook = "queues.md#backlog";
+ };
+ workQueueAge = {
+ service = "sqs";
+ pages = true;
+ metric = queueAge workQueues;
+ window = 900;
+ reducer = "mean";
+ description = "The oldest message in {{ $labels.QueueName }} is {{ $values.B }} seconds old. These queues normally drain within minutes.";
+ runbook = "queues.md#backlog";
+ };
+
+ # Warning and critical are mutually exclusive bands, as in the dev rules.
+ prodRules = map prodRule [
+ (
+ primaryMemory
+ // {
+ uid = "macrodb-primary-freeable-memory-crit";
+ title = "macro-db-prod freeable memory low";
+ severity = "critical";
+ condition = "$B < 1000000000";
+ }
+ )
+ (
+ primaryMemory
+ // {
+ uid = "macrodb-primary-freeable-memory-warn";
+ title = "macro-db-prod freeable memory low";
+ severity = "warning";
+ condition = "$B < 2000000000 && $B >= 1000000000";
+ }
+ )
+ (
+ replicaMemory
+ // {
+ uid = "macrodb-replica-freeable-memory-crit";
+ title = "macro-db-prod-read-replica freeable memory low";
+ severity = "critical";
+ condition = "$B < 750000000";
+ }
+ )
+ (
+ replicaMemory
+ // {
+ uid = "macrodb-replica-freeable-memory-warn";
+ title = "macro-db-prod-read-replica freeable memory low";
+ severity = "warning";
+ condition = "$B < 1500000000 && $B >= 750000000";
+ }
+ )
+ (
+ swap
+ // {
+ uid = "macrodb-swap-crit";
+ title = "macro-db-prod swapping";
+ severity = "critical";
+ condition = "$B > 5000000000";
+ }
+ )
+ (
+ swap
+ // {
+ uid = "macrodb-swap-warn";
+ title = "macro-db-prod swapping";
+ severity = "warning";
+ condition = "$B > 2000000000 && $B <= 5000000000";
+ }
+ )
+ (
+ load
+ // {
+ uid = "macrodb-load-crit";
+ title = "macro-db-prod database load above vCPU count";
+ severity = "critical";
+ condition = "$B > 4";
+ }
+ )
+ (
+ load
+ // {
+ uid = "macrodb-load-warn";
+ title = "macro-db-prod database load above vCPU count";
+ severity = "warning";
+ condition = "$B > 2 && $B <= 4";
+ }
+ )
+ (
+ cpu
+ // {
+ uid = "macrodb-cpu-crit";
+ title = "macro-db-prod CPU high";
+ severity = "critical";
+ condition = "$B > 85";
+ }
+ )
+ (
+ cpu
+ // {
+ uid = "macrodb-cpu-warn";
+ title = "macro-db-prod CPU high";
+ severity = "warning";
+ condition = "$B > 70 && $B <= 85";
+ }
+ )
+ (
+ primaryConnections
+ // {
+ uid = "macrodb-primary-connections-crit";
+ title = "macro-db-prod connections high";
+ severity = "critical";
+ condition = "$B > 2200";
+ }
+ )
+ (
+ primaryConnections
+ // {
+ uid = "macrodb-primary-connections-warn";
+ title = "macro-db-prod connections high";
+ severity = "warning";
+ condition = "$B > 1700 && $B <= 2200";
+ }
+ )
+ (
+ replicaConnections
+ // {
+ uid = "macrodb-replica-connections-crit";
+ title = "macro-db-prod-read-replica connections high";
+ severity = "critical";
+ condition = "$B > 700";
+ }
+ )
+ (
+ replicaConnections
+ // {
+ uid = "macrodb-replica-connections-warn";
+ title = "macro-db-prod-read-replica connections high";
+ severity = "warning";
+ condition = "$B > 550 && $B <= 700";
+ }
+ )
+ (
+ replicaLag
+ // {
+ uid = "macrodb-replica-lag-crit";
+ title = "macro-db-prod-read-replica lagging";
+ severity = "critical";
+ condition = "$B > 300";
+ }
+ )
+ (
+ replicaLag
+ // {
+ uid = "macrodb-replica-lag-warn";
+ title = "macro-db-prod-read-replica lagging";
+ severity = "warning";
+ condition = "$B > 60 && $B <= 300";
+ }
+ )
+ (
+ readLatency
+ // {
+ uid = "macrodb-read-latency-crit";
+ title = "macro-db-prod disk read latency high";
+ severity = "critical";
+ condition = "$B > 0.01";
+ }
+ )
+ (
+ readLatency
+ // {
+ uid = "macrodb-read-latency-warn";
+ title = "macro-db-prod disk read latency high";
+ severity = "warning";
+ condition = "$B > 0.005 && $B <= 0.01";
+ }
+ )
+ (
+ freeStorage
+ // {
+ uid = "macrodb-free-storage-crit";
+ title = "macro-db-prod free storage low";
+ severity = "critical";
+ condition = "$B < 1000000000000";
+ }
+ )
+ (
+ freeStorage
+ // {
+ uid = "macrodb-free-storage-warn";
+ title = "macro-db-prod free storage low";
+ severity = "warning";
+ condition = "$B < 1500000000000 && $B >= 1000000000000";
+ }
+ )
+ (
+ redisMemory
+ // {
+ uid = "redis-memory-crit";
+ title = "Redis memory high";
+ severity = "critical";
+ condition = "$B > 90";
+ }
+ )
+ (
+ redisMemory
+ // {
+ uid = "redis-memory-warn";
+ title = "Redis memory high";
+ severity = "warning";
+ condition = "$B > 75 && $B <= 90";
+ }
+ )
+ (
+ redisEvictions
+ // {
+ uid = "redis-evictions-crit";
+ title = "Redis evicting keys";
+ severity = "critical";
+ condition = "$B > 1000";
+ }
+ )
+ (
+ redisEvictions
+ // {
+ uid = "redis-evictions-warn";
+ title = "Redis evicting keys";
+ severity = "warning";
+ condition = "$B > 0 && $B <= 1000";
+ }
+ )
+ (
+ backfillAge
+ // {
+ uid = "sqs-email-backfill-age-crit";
+ title = "Email backfill queue falling behind";
+ severity = "critical";
+ condition = "$B > 10800";
+ }
+ )
+ (
+ backfillAge
+ // {
+ uid = "sqs-email-backfill-age-warn";
+ title = "Email backfill queue falling behind";
+ severity = "warning";
+ condition = "$B > 3600 && $B <= 10800";
+ }
+ )
+ (
+ workQueueAge
+ // {
+ uid = "sqs-work-queue-age-crit";
+ title = "Prod work queue falling behind";
+ severity = "critical";
+ condition = "$B > 7200";
+ }
+ )
+ (
+ workQueueAge
+ // {
+ uid = "sqs-work-queue-age-warn";
+ title = "Prod work queue falling behind";
+ severity = "warning";
+ condition = "$B > 1800 && $B <= 7200";
+ }
+ )
+ ];
dss = [
(lokiQuery ''sum(count_over_time({service_name="cloud-storage-service",deployment_environment="dev"} |~ "(?i)\\berror\\w*" [5m]))'')
];
@@ -133,6 +624,41 @@ in
}
];
}
+ {
+ orgId = 1;
+ name = "Macro Alerts";
+ receivers = [
+ {
+ uid = "macro-alerts";
+ type = "webhook";
+ disableResolveMessage = false;
+ # The loopback relay adds the channel token; see goalert.nix.
+ settings = {
+ url = "http://127.0.0.1:8090/grafana";
+ httpMethod = "POST";
+ title = ''{{ template "default.title" . }}'';
+ message = ''{{ template "default.message" . }}'';
+ };
+ }
+ ];
+ }
+ {
+ orgId = 1;
+ name = "GoAlert";
+ receivers = [
+ {
+ uid = "goalert";
+ type = "webhook";
+ disableResolveMessage = false;
+ # Loopback, so paging does not depend on the ALB. GoAlert closes the
+ # alert when Grafana sends the resolved notification.
+ settings = {
+ url = "http://127.0.0.1:3030/api/v2/grafana/incoming?token=$GOALERT_GRAFANA_KEY";
+ httpMethod = "POST";
+ };
+ }
+ ];
+ }
];
};
policies.settings = {
@@ -140,21 +666,71 @@ in
policies = [
{
orgId = 1;
- receiver = "Macro team email";
+ receiver = "Macro Alerts";
group_by = [
"grafana_folder"
"alertname"
"severity"
+ "pages"
];
group_wait = "30s";
group_interval = "5m";
repeat_interval = "4h";
+ routes = [
+ {
+ receiver = "Macro team email";
+ object_matchers = [
+ [
+ "environment"
+ "="
+ "dev"
+ ]
+ ];
+ }
+ # Paging prod rules; `continue` lets the next routes still post them
+ # to Macro Alerts.
+ {
+ receiver = "GoAlert";
+ object_matchers = [
+ [
+ "pages"
+ "="
+ "true"
+ ]
+ [
+ "severity"
+ "="
+ "critical"
+ ]
+ ];
+ continue = true;
+ }
+ {
+ receiver = "Macro Alerts";
+ object_matchers = [
+ [
+ "severity"
+ "="
+ "critical"
+ ]
+ ];
+ repeat_interval = "1h";
+ }
+ { receiver = "Macro Alerts"; }
+ ];
}
];
};
rules.settings = {
apiVersion = 1;
groups = [
+ {
+ orgId = 1;
+ name = "Prod";
+ folder = "Macro";
+ interval = "1m";
+ rules = prodRules;
+ }
{
orgId = 1;
name = "Datadog dev migration";
diff --git a/infra/stacks/observability/nixos/goalert.nix b/infra/stacks/observability/nixos/goalert.nix
new file mode 100644
index 0000000000..480fb341af
--- /dev/null
+++ b/infra/stacks/observability/nixos/goalert.nix
@@ -0,0 +1,150 @@
+# GoAlert on-call paging, its Postgres, and the Macro Alerts relay.
+#
+# Everything here runs in Ohio and talks only to loopback, so paging keeps
+# working while production or its database is down. Pages reach the Macro
+# Alerts channel through the relay; phone paging (Twilio) and Google login are
+# not set up yet.
+{
+ config,
+ lib,
+ pkgs,
+ ...
+}:
+let
+ goalert = pkgs.callPackage ./goalert/package.nix { };
+ credential = name: "${name}:/run/macro-observability/${name}";
+ needsSecrets = {
+ requires = [ "observability-secrets.service" ];
+ after = [ "observability-secrets.service" ];
+ partOf = [ "observability-secrets.service" ];
+ };
+ relayUrl = "http://127.0.0.1:8090/goalert";
+ # GoAlert's config.Config, applied with `goalert set-config` before every
+ # start. `@name@` strings are replaced with the credential of that name.
+ goalertConfig = {
+ General.ApplicationName = "Macro On-Call";
+ # One `admin` login, created by apply-config.py from goalert_admin_password.
+ Auth.DisableBasic = false;
+ Webhook = {
+ Enable = true;
+ AllowedURLs = [ relayUrl ];
+ };
+ Maintenance = {
+ AlertCleanupDays = 180;
+ AlertAutoCloseDays = 0;
+ };
+ };
+ dbUrl = "postgres://goalert@/goalert?host=/run/postgresql&sslmode=disable";
+ goalertEnvironment = {
+ GOALERT_DB_URL = dbUrl;
+ GOALERT_LISTEN = "127.0.0.1:3030";
+ GOMEMLIMIT = "600MiB";
+ };
+ # The encryption key only ever lives in this process's environment.
+ start = pkgs.writeShellScript "goalert-start" ''
+ GOALERT_DATA_ENCRYPTION_KEY="$(< "$CREDENTIALS_DIRECTORY/goalert_encryption_key")"
+ export GOALERT_DATA_ENCRYPTION_KEY
+ exec ${lib.getExe goalert}
+ '';
+ backup = pkgs.writeShellScript "goalert-backup" ''
+ set -euo pipefail
+ # Dump to a file first, so a failed dump never replaces a good backup.
+ dump="$(mktemp)"
+ ${config.services.postgresql.package}/bin/pg_dump --format=custom --file="$dump" goalert
+ # One object per weekday keeps a rolling week without a lifecycle rule on
+ # a bucket whose retention Loki owns.
+ ${lib.getExe pkgs.awscli2} s3 cp --only-show-errors "$dump" \
+ "s3://$LOGS_BUCKET/goalert/goalert-$(date +%a).dump"
+ '';
+in
+{
+ services.postgresql = {
+ enable = true;
+ package = pkgs.postgresql_17;
+ dataDir = "/srv/observability/postgres";
+ enableTCPIP = false;
+ settings.listen_addresses = lib.mkForce "";
+ ensureDatabases = [ "goalert" ];
+ ensureUsers = [
+ {
+ name = "goalert";
+ ensureDBOwnership = true;
+ }
+ ];
+ };
+
+ systemd.services = {
+ postgresql = {
+ serviceConfig.MemoryMax = "512M";
+ # Like the other services, retry forever instead of the module's limit.
+ unitConfig.StartLimitIntervalSec = lib.mkForce 0;
+ };
+
+ goalert = needsSecrets // {
+ description = "GoAlert on-call paging";
+ wantedBy = [ "multi-user.target" ];
+ requires = needsSecrets.requires ++ [ "postgresql.target" ];
+ after = needsSecrets.after ++ [ "postgresql.target" ];
+ environment = goalertEnvironment;
+ serviceConfig = {
+ User = "goalert";
+ Group = "goalert";
+ EnvironmentFile = "/opt/observability/runtime.env";
+ LoadCredential = map credential [
+ "goalert_encryption_key"
+ "goalert_admin_password"
+ ];
+ ExecStartPre = [
+ "${pkgs.python3}/bin/python3 ${./goalert/apply-config.py} ${lib.getExe goalert} ${pkgs.writeText "goalert-config.json" (builtins.toJSON goalertConfig)} ${config.services.postgresql.package}/bin/psql"
+ ];
+ ExecStart = start;
+ MemoryMax = "768M";
+ CapabilityBoundingSet = [ "" ];
+ PrivateDevices = true;
+ };
+ };
+
+ macro-alerts-relay = needsSecrets // {
+ description = "Forward GoAlert and Grafana notifications to Macro Alerts";
+ wantedBy = [ "multi-user.target" ];
+ serviceConfig = {
+ DynamicUser = true;
+ LoadCredential = map credential [
+ "macro_alerts_webhook_url"
+ "macro_alerts_bot_token"
+ ];
+ ExecStart = "${pkgs.python3}/bin/python3 ${./goalert/macro-alerts-relay.py}";
+ MemoryMax = "128M";
+ CapabilityBoundingSet = [ "" ];
+ PrivateDevices = true;
+ };
+ };
+
+ goalert-backup = {
+ description = "Dump the GoAlert database to S3";
+ requires = [ "postgresql.target" ];
+ after = [ "postgresql.target" ];
+ serviceConfig = {
+ Type = "oneshot";
+ User = "goalert";
+ Group = "goalert";
+ EnvironmentFile = "/opt/observability/runtime.env";
+ ExecStart = backup;
+ PrivateTmp = true;
+ ProtectSystem = "strict";
+ ProtectHome = true;
+ NoNewPrivileges = true;
+ # The AWS CLI writes its cache under HOME.
+ Environment = "HOME=/tmp";
+ };
+ };
+ };
+
+ systemd.timers.goalert-backup = {
+ wantedBy = [ "timers.target" ];
+ timerConfig = {
+ OnCalendar = "*-*-* 05:00:00";
+ Persistent = true;
+ };
+ };
+}
diff --git a/infra/stacks/observability/nixos/goalert/apply-config.py b/infra/stacks/observability/nixos/goalert/apply-config.py
new file mode 100644
index 0000000000..2180aa71e5
--- /dev/null
+++ b/infra/stacks/observability/nixos/goalert/apply-config.py
@@ -0,0 +1,48 @@
+"""Migrate GoAlert's database, replace its config with the Nix-declared one, and
+create the `admin` login if it does not exist.
+
+GoAlert keeps application config only in Postgres. Running this before every
+start makes the Nix attrset the source of truth: edits in the admin UI last
+only until the next restart. Secret placeholders such as `@goalert_admin_password@`
+are replaced with credential files; values never pass through a shell.
+"""
+import json
+import os
+from pathlib import Path
+import subprocess
+import sys
+
+goalert, template_path, psql = sys.argv[1], sys.argv[2], sys.argv[3]
+credentials = Path(os.environ['CREDENTIALS_DIRECTORY'])
+
+
+def substitute(value):
+ if isinstance(value, dict):
+ return {key: substitute(item) for key, item in value.items()}
+ if isinstance(value, list):
+ return [substitute(item) for item in value]
+ if isinstance(value, str) and value.startswith('@') and value.endswith('@'):
+ return (credentials / value[1:-1]).read_text()
+ return value
+
+
+config = substitute(json.loads(Path(template_path).read_text()))
+environment = {
+ **os.environ,
+ 'GOALERT_DATA_ENCRYPTION_KEY': (credentials / 'goalert_encryption_key').read_text(),
+}
+subprocess.run([goalert, 'migrate'], env=environment, check=True)
+subprocess.run(
+ [goalert, 'set-config'], env=environment, input=json.dumps(config), text=True, check=True,
+)
+admin_exists = subprocess.run(
+ [psql, '-tAc', "select 1 from auth_basic_users where username = 'admin'", environment['GOALERT_DB_URL']],
+ env=environment, capture_output=True, text=True, check=True,
+).stdout.strip()
+if not admin_exists:
+ # `add-user` takes the password only as an argument; this runs once, on first boot.
+ subprocess.run(
+ [goalert, 'add-user', '--admin', '--user', 'admin', '--email', '[email protected]',
+ '--pass', (credentials / 'goalert_admin_password').read_text()],
+ env=environment, check=True,
+ )
diff --git a/infra/stacks/observability/nixos/goalert/macro-alerts-relay.py b/infra/stacks/observability/nixos/goalert/macro-alerts-relay.py
new file mode 100644
index 0000000000..6e203a9b86
--- /dev/null
+++ b/infra/stacks/observability/nixos/goalert/macro-alerts-relay.py
@@ -0,0 +1,102 @@
+"""Forward GoAlert and Grafana webhooks to the Macro Alerts channel.
+
+Macro's channel webhook takes `{"content": ...}` and a bot-token header. GoAlert
+cannot set headers, so both senders post here over loopback and the token stays
+in this one service. Each request is forwarded once; the sender owns retries.
+"""
+import json
+import logging
+import os
+import time
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+import urllib.request
+
+CREDENTIALS = Path(os.environ['CREDENTIALS_DIRECTORY'])
+WEBHOOK_URL = (CREDENTIALS / 'macro_alerts_webhook_url').read_text()
+BOT_TOKEN = (CREDENTIALS / 'macro_alerts_bot_token').read_text()
+# GoAlert abandons a webhook after three seconds.
+FORWARD_TIMEOUT_SECONDS = 2.5
+MAX_BODY_BYTES = 256 * 1024
+MAX_CONTENT_CHARS = 4000
+
+
+def goalert_content(payload: dict) -> str | None:
+ kind = payload.get('Type')
+ link = payload.get('GoAlertURL', '')
+ if kind == 'Alert':
+ return f"🚨 Paging on-call: {payload.get('Summary', '')}\n{link}"
+ if kind == 'AlertStatus':
+ return f"{payload.get('LogEntry', '')}: {payload.get('Summary', '')}\n{link}"
+ if kind == 'AlertBundle':
+ return f"🚨 Paging on-call: {payload.get('Count')} alerts for {payload.get('ServiceName', '')}\n{link}"
+ if kind == 'AlertStatusBundle':
+ return f"{payload.get('LogEntry', '')} ({payload.get('Count')} alerts)\n{link}"
+ if kind == 'ScheduleOnCallUsers':
+ names = ', '.join(user.get('Name', '') for user in payload.get('Users', []))
+ return f"On call for {payload.get('ScheduleName', '')}: {names or 'nobody'}\n{payload.get('ScheduleURL', '')}"
+ if kind in ('Test', 'Verification'):
+ return f"GoAlert {kind.lower()} message for the Macro Alerts webhook."
+ return None
+
+
+def grafana_content(payload: dict) -> str | None:
+ title = payload.get('title')
+ message = payload.get('message')
+ if not isinstance(title, str) or not isinstance(message, str):
+ return None
+ return f"{title}\n{message}"
+
+
+SOURCES = {'/goalert': goalert_content, '/grafana': grafana_content}
+
+
+class Relay(BaseHTTPRequestHandler):
+ def do_POST(self) -> None:
+ format_content = SOURCES.get(self.path)
+ length = int(self.headers.get('Content-Length', '0'))
+ if format_content is None or length <= 0 or length > MAX_BODY_BYTES:
+ self.send_error(404 if format_content is None else 400)
+ return
+ try:
+ payload = json.loads(self.rfile.read(length))
+ except json.JSONDecodeError:
+ self.send_error(400)
+ return
+ content = format_content(payload) if isinstance(payload, dict) else None
+ if content is None:
+ logging.warning('unrecognised %s payload', self.path)
+ self.send_error(422)
+ return
+ request = urllib.request.Request(
+ WEBHOOK_URL,
+ data=json.dumps({'content': content[:MAX_CONTENT_CHARS]}).encode(),
+ headers={'Content-Type': 'application/json', 'x-macro-channel-bot-token': BOT_TOKEN},
+ method='POST',
+ )
+ started = time.monotonic()
+ try:
+ with urllib.request.urlopen(request, timeout=FORWARD_TIMEOUT_SECONDS) as response:
+ status = response.status
+ except OSError as error:
+ logging.warning(
+ 'forward source=%s outcome=failed elapsed_ms=%d error=%s',
+ self.path, (time.monotonic() - started) * 1000, error,
+ )
+ self.send_error(502)
+ return
+ logging.info(
+ 'forward source=%s status=%d elapsed_ms=%d',
+ self.path, status, (time.monotonic() - started) * 1000,
+ )
+ self.send_response(204 if 200 <= status < 300 else 502)
+ self.end_headers()
+
+ def log_message(self, format: str, *args: object) -> None:
+ # Request lines carry no secrets, but errors are logged explicitly above.
+ pass
+
+
+if __name__ == '__main__':
+ logging.basicConfig(level=logging.INFO, format='%(levelname)s %(message)s')
+ ThreadingHTTPServer(('127.0.0.1', 8090), Relay).serve_forever()
diff --git a/infra/stacks/observability/nixos/goalert/package.nix b/infra/stacks/observability/nixos/goalert/package.nix
new file mode 100644
index 0000000000..f22931e8ca
--- /dev/null
+++ b/infra/stacks/observability/nixos/goalert/package.nix
@@ -0,0 +1,95 @@
+# GoAlert is not in nixpkgs, so build it here from the tagged source. The Go
+# binary embeds the web UI (web/src/build), which is bundled with bun first.
+{
+ lib,
+ stdenvNoCC,
+ buildGoModule,
+ fetchFromGitHub,
+ bun,
+}:
+let
+ version = "0.35.0";
+ # `git rev-parse v0.35.0` and its commit date, for `goalert version`.
+ commit = "db9829187a1c1f96a7c57112b74dba54442d7c6b";
+ commitDate = "2026-09-15T21:24:20Z";
+
+ src = fetchFromGitHub {
+ owner = "target";
+ repo = "goalert";
+ tag = "v${version}";
+ hash = "sha256-huzessc9Iy/j689miscZJ6/MZIClA+qG3EX1tOokk54=";
+ };
+
+ nodeModules = stdenvNoCC.mkDerivation {
+ pname = "goalert-node-modules";
+ inherit version src;
+ nativeBuildInputs = [ bun ];
+ dontConfigure = true;
+ dontFixup = true;
+ buildPhase = ''
+ runHook preBuild
+ export HOME=$TMPDIR
+ bun install --frozen-lockfile --ignore-scripts --no-progress
+ runHook postBuild
+ '';
+ installPhase = ''
+ runHook preInstall
+ cp -R node_modules $out
+ runHook postInstall
+ '';
+ outputHash = "sha256-ECi5MztQ+pavprPftLFHu4q9wxSg9SZywFecRD4WFJI=";
+ outputHashMode = "recursive";
+ };
+
+ # Mirrors the Makefile's web/src/build/static/app.js target.
+ ui = stdenvNoCC.mkDerivation {
+ pname = "goalert-ui";
+ inherit version src;
+ nativeBuildInputs = [ bun ];
+ dontConfigure = true;
+ buildPhase = ''
+ runHook preBuild
+ cp -R ${nodeModules} node_modules
+ chmod -R u+w node_modules
+ patchShebangs node_modules
+ mkdir -p web/src/build/static
+ cp web/src/app/public/icons/favicon-* web/src/app/public/logos/lightmode_* \
+ web/src/app/public/logos/darkmode_* web/src/app/public/logos/goalert-alt-logo.png \
+ web/src/build/static/
+ GOALERT_VERSION=v${version} bun web/src/esbuild.config.js --prod
+ runHook postBuild
+ '';
+ installPhase = ''
+ runHook preInstall
+ cp -R web/src/build $out
+ runHook postInstall
+ '';
+ };
+in
+buildGoModule {
+ pname = "goalert";
+ inherit version src;
+ vendorHash = "sha256-htUZy6mHcuINTa6ih/cD2hF5pHjZWPqX1vuiJAu1zKQ=";
+ subPackages = [ "cmd/goalert" ];
+ preBuild = ''
+ rm -rf web/src/build
+ cp -R ${ui} web/src/build
+ '';
+ ldflags = [
+ "-s"
+ "-w"
+ "-X github.com/target/goalert/version.gitVersion=v${version}"
+ "-X github.com/target/goalert/version.gitCommit=${commit}"
+ "-X github.com/target/goalert/version.gitTreeState=clean"
+ "-X github.com/target/goalert/version.buildDate=${commitDate}"
+ ];
+ # The test suite needs a Postgres database.
+ doCheck = false;
+ meta = {
+ description = "On-call scheduling, automated escalations and notifications";
+ homepage = "https://github.com/target/goalert";
+ license = lib.licenses.asl20;
+ mainProgram = "goalert";
+ platforms = [ "x86_64-linux" ];
+ };
+}
diff --git a/infra/stacks/observability/nixos/host.nix b/infra/stacks/observability/nixos/host.nix
index 5be83e4920..ba1838783b 100644
--- a/infra/stacks/observability/nixos/host.nix
+++ b/infra/stacks/observability/nixos/host.nix
@@ -16,11 +16,16 @@ let
"alloy"
"alloy-ingest"
"nginx"
+ "postgresql"
+ "goalert"
+ "macro-alerts-relay"
];
secretConsumers = [
"grafana.service"
"alloy-ingest.service"
"nginx.service"
+ "goalert.service"
+ "macro-alerts-relay.service"
];
dataUsers = {
grafana = 20001;
@@ -29,6 +34,8 @@ let
prometheus = 20004;
alloy = 20005;
alloy-ingest = 20006;
+ postgres = 20007;
+ goalert = 20008;
};
in
{
diff --git a/infra/stacks/observability/nixos/proxy.nix b/infra/stacks/observability/nixos/proxy.nix
index 72f8e34126..bb0eea6a94 100644
--- a/infra/stacks/observability/nixos/proxy.nix
+++ b/infra/stacks/observability/nixos/proxy.nix
@@ -3,6 +3,20 @@ let
deny = {
extraConfig = "return 404;";
};
+ # TLS ends at the ALB; GoAlert redirects plain-HTTP requests without this.
+ goalertProxy = ''
+ proxy_pass http://127.0.0.1:3030;
+ proxy_set_header Host $host;
+ proxy_set_header X-Forwarded-Proto https;
+ proxy_set_header X-Forwarded-For $remote_addr;
+ '';
+ # Unauthenticated paging endpoints. GoAlert checks the integration key on each.
+ pagingIngress = ''
+ limit_except POST { deny all; }
+ limit_req zone=paging burst=50 nodelay;
+ limit_req_status 429;
+ ${goalertProxy}
+ '';
websocketHeaders = ''
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
@@ -28,7 +42,9 @@ in
log_format safe '$remote_addr $host $request_method $uri $status';
access_log syslog:server=unix:/dev/log,facility=local7,tag=nginx_access,severity=info safe;
limit_req_zone $server_name zone=ingest:1m rate=100r/s;
+ limit_req_zone $server_name zone=paging:1m rate=20r/s;
map $http_upgrade $connection_upgrade { default upgrade; ''' close; }
+ include /opt/observability/oncall-server.conf;
'';
virtualHosts.public = {
listen = [
@@ -45,6 +61,11 @@ in
proxyPass = "http://127.0.0.1:3000/api/health";
extraConfig = "access_log off; proxy_set_header Host $grafana_host;";
};
+ # The on-call target group's health check arrives without the on-call host.
+ "= /goalert-healthz" = {
+ proxyPass = "http://127.0.0.1:3030/health";
+ extraConfig = "access_log off; proxy_set_header X-Forwarded-Proto https;";
+ };
"/" = {
proxyPass = "http://127.0.0.1:3000";
extraConfig = ''
@@ -127,5 +148,24 @@ in
};
};
};
+ # Included by the runtime on-call server block, which carries the hostname.
+ environment.etc."nginx/oncall-locations.conf".text = ''
+ add_header Strict-Transport-Security 'max-age=31536000' always;
+ location = /health {
+ access_log off;
+ limit_except GET { deny all; }
+ ${goalertProxy}
+ }
+ location ^~ /api/v2/generic/incoming { ${pagingIngress} }
+ location ^~ /api/v2/heartbeat/ { ${pagingIngress} }
+ # Grafana pages GoAlert over loopback; other inbound integrations are unused.
+ location ~ ^/(api/v2/(grafana|prometheusalertmanager|site24x7|mailgun|uik|twilio)|v1/webhooks|health/) {
+ return 404;
+ }
+ location / {
+ ${goalertProxy}
+ ${websocketHeaders}
+ }
+ '';
systemd.services.nginx.serviceConfig.MemoryMax = "256M";
}
diff --git a/infra/stacks/observability/nixos/read-user-data.py b/infra/stacks/observability/nixos/read-user-data.py
index a6cd081a08..d0d381b40f 100644
--- a/infra/stacks/observability/nixos/read-user-data.py
+++ b/infra/stacks/observability/nixos/read-user-data.py
@@ -7,13 +7,13 @@ import urllib.request
CONFIG_ROOT = Path('/opt/observability')
SETTING_KEYS = {
- 'region', 'grafanaHost', 'otlpHost',
+ 'region', 'grafanaHost', 'otlpHost', 'oncallHost',
'secretArn', 'alarmTopicArn', 'volumeId', 'logsBucket', 'tracesBucket',
}
def validate_payload(payload):
- if not isinstance(payload, dict) or set(payload) != {'version', 'settings'} or payload['version'] != 5:
+ if not isinstance(payload, dict) or set(payload) != {'version', 'settings'} or payload['version'] != 6:
raise ValueError('Unsupported observability user-data schema')
settings = payload['settings']
if not isinstance(settings, dict) or set(settings) != SETTING_KEYS:
@@ -22,6 +22,7 @@ def validate_payload(payload):
'region': r'us-east-2',
'grafanaHost': r'[a-z0-9-]+\.macro-internal\.com',
'otlpHost': r'[a-z0-9-]+\.macro-internal\.com',
+ 'oncallHost': r'[a-z0-9-]+\.macro-internal\.com',
'alarmTopicArn': r'arn:aws:sns:us-east-2:\d{12}:[a-zA-Z0-9_-]+',
'secretArn': r'arn:aws:secretsmanager:us-east-2:\d{12}:secret:[a-zA-Z0-9/_+=.@-]+',
'volumeId': r'vol-[a-f0-9]+',
@@ -31,8 +32,9 @@ def validate_payload(payload):
for key, pattern in patterns.items():
if not isinstance(settings[key], str) or not re.fullmatch(pattern, settings[key]):
raise ValueError(f'Invalid runtime setting: {key}')
- if settings['grafanaHost'] == settings['otlpHost']:
- raise ValueError('UI and ingestion require separate hostnames')
+ hosts = [settings['grafanaHost'], settings['otlpHost'], settings['oncallHost']]
+ if len(set(hosts)) != len(hosts):
+ raise ValueError('Grafana, ingestion and on-call require separate hostnames')
return settings
@@ -43,6 +45,7 @@ def write_runtime(payload, output_root):
'GRAFANA_ALERT_TOPIC_ARN': settings['alarmTopicArn'],
'GRAFANA_HOST': settings['grafanaHost'],
'GRAFANA_ROOT_URL': 'https://' + settings['grafanaHost'] + '/',
+ 'GOALERT_PUBLIC_URL': 'https://' + settings['oncallHost'],
'LOGS_BUCKET': settings['logsBucket'],
'TRACES_BUCKET': settings['tracesBucket'],
}
@@ -56,6 +59,12 @@ def write_runtime(payload, output_root):
'}\nmap "" $grafana_host {\n'
f" default {settings['grafanaHost']};\n" + '}\n'
),
+ # GoAlert gets its own server so its paths never overlap Grafana's.
+ 'oncall-server.conf': (
+ 'server {\n listen 0.0.0.0:8080;\n'
+ f" server_name {settings['oncallHost']};\n"
+ ' include /etc/nginx/oncall-locations.conf;\n}\n'
+ ),
'bootstrap.json': json.dumps({'region': settings['region'], 'secretArn': settings['secretArn']}),
'volume-id': settings['volumeId'],
}
diff --git a/infra/stacks/observability/nixos/services.nix b/infra/stacks/observability/nixos/services.nix
index c10a6932b8..e7b15641d9 100644
--- a/infra/stacks/observability/nixos/services.nix
+++ b/infra/stacks/observability/nixos/services.nix
@@ -20,6 +20,7 @@ in
./database-dashboard.nix
./database-queries-dashboard.nix
./alerting.nix
+ ./goalert.nix
];
services.grafana = {
enable = true;
@@ -64,7 +65,11 @@ in
};
grafana = needsSecrets // {
serviceConfig = {
- EnvironmentFile = "/opt/observability/runtime.env";
+ EnvironmentFile = [
+ "/opt/observability/runtime.env"
+ # Interpolated into the GoAlert contact point by alerting provisioning.
+ "/run/macro-observability/grafana-goalert.env"
+ ];
LoadCredential = map credential [
"google_client_id"
"google_client_secret"
diff --git a/infra/stacks/observability/render.ts b/infra/stacks/observability/render.ts
index 0d1a10d3ef..3eb8485611 100644
--- a/infra/stacks/observability/render.ts
+++ b/infra/stacks/observability/render.ts
@@ -4,7 +4,7 @@ import { type Settings, validateSettings } from './settings';
// Nix owns service configuration. User data carries only validated runtime values.
export function renderUserData(settings: Settings): string {
validateSettings(settings);
- const compressed = gzipSync(JSON.stringify({ version: 5, settings }));
+ const compressed = gzipSync(JSON.stringify({ version: 6, settings }));
if (compressed.length > 16 * 1024) {
throw new Error('Configuration exceeds the EC2 user-data limit');
}
diff --git a/infra/stacks/observability/settings.ts b/infra/stacks/observability/settings.ts
index 9f65d5ba3e..7af8bcf53a 100644
--- a/infra/stacks/observability/settings.ts
+++ b/infra/stacks/observability/settings.ts
@@ -2,6 +2,7 @@ export interface Settings {
region: string;
grafanaHost: string;
otlpHost: string;
+ oncallHost: string;
secretArn: string;
alarmTopicArn: string;
volumeId: string;
@@ -38,15 +39,18 @@ export function validateRegionalArn(
// Validate runtime values before they reach the NixOS configuration renderer.
export function validateSettings(settings: Settings): void {
- for (const host of [settings.grafanaHost, settings.otlpHost]) {
+ const hosts = [settings.grafanaHost, settings.otlpHost, settings.oncallHost];
+ for (const host of hosts) {
if (!/^[a-z0-9-]+\.macro-internal\.com$/.test(host)) {
throw new Error(
'Observability hostnames must be subdomains of macro-internal.com'
);
}
}
- if (settings.grafanaHost === settings.otlpHost) {
- throw new Error('UI and ingestion require separate hostnames');
+ if (new Set(hosts).size !== hosts.length) {
+ throw new Error(
+ 'Grafana, ingestion and on-call require separate hostnames'
+ );
}
if (
!/^arn:aws:secretsmanager:[a-z0-9-]+:\d{12}:secret:[a-zA-Z0-9/_+=.@-]+$/.test(