Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
feat(observability): Google login for GoAlert and a graph for every prod alert (#9089)
Wolf Mermelstein410f56ce1203
diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.mdindex 39999c4aa3..f89dd81d55 100644--- a/infra/stacks/monitoring/AGENTS.md+++ b/infra/stacks/monitoring/AGENTS.md@@ -5,7 +5,8 @@Org-global Datadog monitors, synthetics, and rollout dashboards. This stack ismanual and has only a prod stack, even for monitors covering both environments.Prod threshold and anomaly monitors are in `monitors/{macrodb,redis,queues,regressions}.ts`-(`prodMonitor()` in `prod-monitor.ts`); every monitor notifies through `notify.ts`.+(`prodMonitor()` in `prod-monitor.ts`), each file with a dashboard graphing its+monitors that their messages link to; every monitor notifies through `notify.ts`.## Checks@@ -23,12 +24,14 @@ From `infra/`:- The CloudWatch-based monitors in `macrodb.ts`, `redis.ts` and `queues.ts`have hand-written Grafana copies in- `infra/stacks/observability/nixos/alerting.nix`. Change thresholds and windows+ `infra/stacks/observability/nixos/prod-alerts.nix`. Change thresholds and windowsin both. Grafana pages for those, so their Datadog copies use `pages: false`.- Read `README.md` before deployment: imported resources must be adopted firstto avoid duplicate alerts. Never deploy without explicit approval.- Deploying needs the `goalertIntegrationKey` secret in stack config, andGoAlert (infra/stacks/observability) reachable at oncall.macro-internal.com.+- A prod monitor's graph on its file's dashboard repeats its metric and+ thresholds by hand; change both. `routing.test.ts` matches them by title.- Never mention a Slack handle or email address in a monitor message; end itwith `notify({ pages })`. `routing.test.ts` enforces it.- Notification retention adds nine monitors and one dashboard. Deploy DSS healthdiff --git a/infra/stacks/monitoring/Pulumi.prod.yaml b/infra/stacks/monitoring/Pulumi.prod.yamlindex 138f62896b..c565ca2d7f 100644--- a/infra/stacks/monitoring/Pulumi.prod.yaml+++ b/infra/stacks/monitoring/Pulumi.prod.yaml@@ -1,4 +1,6 @@# Datadog monitors are org-global, so this project uses only the prod stack.# The provider reads credentials and the Datadog site from DD_API_KEY,# DD_APP_KEY, and DD_HOST, supplied locally or by the deployment workflow.-config: {}+config:+ monitoring:goalertIntegrationKey:+ secure: AAABAKEDvnVx7IjOr5rgXFTFIDwyqUy8gmBPo/bVOSKuV6DsgerGZ6IddCTt5xiG4VtYCz39X2aI4B5bUVXb+WSz0cU=diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.mdindex 2a233b8558..caef04f3ab 100644--- a/infra/stacks/monitoring/README.md+++ b/infra/stacks/monitoring/README.md@@ -6,7 +6,7 @@ Datadog monitors and synthetic tests as code.Datadog. This stack **adopts** them — it does not recreate them. `import.json`lists every one by id. Everything else is declared here first and is createdon deploy: the prod threshold and anomaly monitors (`monitors/macrodb.ts`,-`redis.ts`, `queues.ts`, `regressions.ts`), the rollout monitors, the GoAlert webhooks and health check, and one log-based metric.+`redis.ts`, `queues.ts`, `regressions.ts`) and their dashboards, the rollout monitors, the GoAlert webhooks and health check, and one log-based metric.## Routing@@ -40,12 +40,19 @@ its thresholds many times a day.`monitors/macrodb.ts`, `redis.ts`, `queues.ts` and `regressions.ts` declarethe prod threshold, forecast and anomaly monitors with `prodMonitor()`. The oneson CloudWatch metrics have hand-written copies in Grafana-(`infra/stacks/observability/nixos/alerting.nix`), and Grafana pages on-call for+(`infra/stacks/observability/nixos/prod-alerts.nix`), and Grafana pages on-call forthe critical ones through GoAlert. Their Datadog copies only notify, so oneincident pages once. Keep thresholds and windows in sync by hand. The error-logsanomaly monitor reads `macro.logs.errors`, a log-based metric declared in`regressions.ts`, because log monitors have no anomaly function.+Each of those files also declares a dashboard (MacroDB, Redis, Queue and+Regression alerts) with one graph per monitor, titled like the monitor, its+critical and warning thresholds drawn as markers; anomaly graphs show the band+instead. Every prod monitor's message links to its dashboard. Change a query or+threshold in the graph too; `routing.test.ts` checks that each monitor has a+graph with matching markers.+## Read this before deploying**Set the GoAlert integration key and deploy GoAlert first.** The webhooks@@ -88,10 +95,10 @@ The gate is that preview:- **35 monitors + 1 synthetic test updated, in `message` and`renotifyInterval` only** — the routing move. The two AI-editing monitors,already deployed from this stack, update the same way.-- **69 resources to create**: 64 monitors (25 prod threshold and anomaly, 19 GraphQL soup, 9+- **73 resources to create**: 64 monitors (25 prod threshold and anomaly, 19 GraphQL soup, 9notification retention, 10 WhatsApp session, 1 native call push), 2 webhooks,- 1 log-based metric, the GoAlert synthetic test, and the notification- retention dashboard.+ 1 log-based metric, the GoAlert synthetic test, the four prod alert+ dashboards, and the notification retention dashboard.- **Any other update, a replace, or a delete** — the hand-written code in`monitors/*.ts` drifted from the live monitor. Reconcile against`imported.generated.ts` before deploying.@@ -158,7 +165,7 @@ is applied. Reach for `restrictedRoles` if convention stops being enough.## Adding a monitorAdd a prod threshold or anomaly monitor with `prodMonitor()` in the matching-`monitors/*.ts`, and its Grafana copy in `alerting.nix` when the metric is in+`monitors/*.ts`, a `thresholdGraph()` for it on that file's dashboard, and its Grafana copy in `prod-alerts.nix` when the metric is inCloudWatch. Declare anything else with `new datadog.Monitor(...)`, ending itsmessage with `notify({ pages })` and setting `renotifyInterval: RENOTIFY_MINUTES`. Use the `adopted()` helper onlyfor monitors that already exist in Datadog — it sets `protect`, which is therediff --git a/infra/stacks/monitoring/monitors/macrodb.ts b/infra/stacks/monitoring/monitors/macrodb.tsindex b191b943c1..bcc9020726 100644--- a/infra/stacks/monitoring/monitors/macrodb.ts+++ b/infra/stacks/monitoring/monitors/macrodb.ts@@ -2,9 +2,120 @@//// Thresholds come from two weeks of 5-minute data before the October overload.// Monitors on CloudWatch metrics also exist in Grafana-// (infra/stacks/observability/nixos/alerting.nix); change both together.+// (infra/stacks/observability/nixos/prod-alerts.nix); change both together.// Grafana pages on-call for those through GoAlert, so they only notify here.-import { prodMonitor } from '../prod-monitor';+import * as datadog from '@pulumi/datadog';+import { graphGroup, prodMonitor, thresholdGraph } from '../prod-monitor';++// One graph per monitor below, titled like it, with its thresholds.+const dashboard = new datadog.Dashboard('macrodb-alerts', {+ title: 'MacroDB alerts',+ description:+ 'Every prod MacroDB monitor against its thresholds. Declared in infra/stacks/monitoring/monitors/macrodb.ts; see docs/runbooks/macrodb.md.',+ layoutType: 'ordered',+ widgets: [+ graphGroup('Memory', [+ thresholdGraph({+ title: 'macro-db-prod freeable memory low',+ query:+ 'avg:aws.rds.freeable_memory{dbinstanceidentifier:macro-db-prod}',+ critical: '1000000000',+ warning: '2000000000',+ }),+ thresholdGraph({+ title: 'macro-db-prod-read-replica freeable memory low',+ query:+ 'avg:aws.rds.freeable_memory{dbinstanceidentifier:macro-db-prod-read-replica}',+ critical: '750000000',+ warning: '1500000000',+ }),+ thresholdGraph({+ title: 'macro-db-prod swapping',+ query:+ 'max:aws.rds.swap_usage{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}',+ critical: '5000000000',+ warning: '2000000000',+ }),+ ]),+ graphGroup('Load', [+ thresholdGraph({+ title: 'macro-db-prod database load above vCPU count',+ query:+ 'avg:aws.rds.dbload_relative_to_num_vcpus{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}',+ critical: '4',+ warning: '2',+ }),+ thresholdGraph({+ title: 'macro-db-prod CPU high',+ query:+ 'avg:aws.rds.cpuutilization{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}',+ critical: '85',+ warning: '70',+ }),+ thresholdGraph({+ title: 'macro-db-prod disk read latency high',+ query:+ 'avg:aws.rds.read_latency{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}',+ critical: '0.01',+ warning: '0.005',+ }),+ ]),+ graphGroup('Connections and replication', [+ thresholdGraph({+ title: 'macro-db-prod connections high',+ query:+ 'max:aws.rds.database_connections{dbinstanceidentifier:macro-db-prod}',+ critical: '2200',+ warning: '1700',+ }),+ thresholdGraph({+ title: 'macro-db-prod-read-replica connections high',+ query:+ 'max:aws.rds.database_connections{dbinstanceidentifier:macro-db-prod-read-replica}',+ critical: '700',+ warning: '550',+ }),+ thresholdGraph({+ title: 'macro-db-prod-read-replica lagging',+ query:+ 'max:aws.rds.replica_lag{dbinstanceidentifier:macro-db-prod-read-replica}',+ critical: '300',+ warning: '60',+ }),+ ]),+ graphGroup('Locks', [+ thresholdGraph({+ title: 'macro-db-prod connections blocked on locks',+ query:+ 'max:postgresql.activity.blocked_connections{host:macro-db-prod*} by {host}',+ critical: '50',+ warning: '20',+ }),+ thresholdGraph({+ title: 'macro-db-prod long-running transaction',+ query:+ 'max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host}',+ critical: '1800',+ warning: '600',+ }),+ ]),+ graphGroup('Storage', [+ thresholdGraph({+ title: 'macro-db-prod free storage low',+ query:+ 'min:aws.rds.free_storage_space{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}',+ critical: '1000000000000',+ warning: '1500000000000',+ }),+ thresholdGraph({+ title: 'macro-db-prod free storage forecast to run low within a week',+ query:+ "forecast(min:aws.rds.free_storage_space{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}, 'linear', 1)",+ critical: '500000000000',+ }),+ ]),+ ],+});prodMonitor('macrodb-primary-freeable-memory', {title: 'macro-db-prod freeable memory low',@@ -15,6 +126,7 @@ prodMonitor('macrodb-primary-freeable-memory', {pages: false,description:'Freeable memory is {{value}} bytes. Normal is 7-18 GB; it fell to about 0.45 GB during the October overload. Each connection holds about 14 MB of private memory, so check connection count and swap first.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#memory',});@@ -27,6 +139,7 @@ prodMonitor('macrodb-replica-freeable-memory', {pages: false,description:'Replica freeable memory is {{value}} bytes. Normal minimum is about 3 GB. The replica serves GraphQL reads, so memory pressure shows up as slow lists.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#memory',});@@ -39,6 +152,7 @@ prodMonitor('macrodb-swap', {pages: false,description:'{{dbinstanceidentifier.name}} is using {{value}} bytes of swap. Normal is under 0.6 GB. Swap above 5 GB preceded both October overloads by hours.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#memory',});@@ -51,6 +165,7 @@ prodMonitor('macrodb-load', {pages: false,description:'{{dbinstanceidentifier.name}} has {{value}}x as many active sessions as vCPUs. Normal is under 1.3; the October overload reached 8. Look at Performance Insights top waits (locks, IO) and the services holding the most connections.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#load',});@@ -63,6 +178,7 @@ prodMonitor('macrodb-cpu', {pages: false,description:'{{dbinstanceidentifier.name}} CPU averaged {{value}}% over 15 minutes. Normal peaks are about 30%.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#load',});@@ -75,6 +191,7 @@ prodMonitor('macrodb-primary-connections', {pages: false,description:'macro-db-prod has {{value}} connections. Normal 15-minute peaks are about 1,100; max_connections is 3,439. Each connection costs about 14 MB, so this also drains freeable memory.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#connections',});@@ -87,6 +204,7 @@ prodMonitor('macrodb-replica-connections', {pages: false,description:'The read replica has {{value}} connections. Normal peaks are about 460.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#connections',});@@ -99,6 +217,7 @@ prodMonitor('macrodb-replica-lag', {pages: false,description:'The read replica is {{value}} seconds behind the primary. Normal is under 6 seconds. GraphQL reads use the replica, so users see stale data and missing new items.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#replica-lag',});@@ -111,6 +230,7 @@ prodMonitor('macrodb-read-latency', {pages: false,description:'{{dbinstanceidentifier.name}} disk reads take {{value}} seconds on average. Normal is about 0.6 ms.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#load',});@@ -123,6 +243,7 @@ prodMonitor('macrodb-free-storage', {pages: false,description:'{{dbinstanceidentifier.name}} has {{value}} bytes of free storage out of 5.37 TB.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#storage',});@@ -136,6 +257,7 @@ prodMonitor('macrodb-free-storage-forecast', {pages: false,description:'At the current write rate {{dbinstanceidentifier.name}} drops below 500 GB free within a week. Find what started writing; recent growth was 100-200 GB a day.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#storage',});@@ -150,6 +272,7 @@ prodMonitor('macrodb-blocked-connections', {pages: true,description:"{{host.name}} has {{value}} connections waiting on locks. Normal is under 7. Advisory and tuple locks dominated the October overload; see Database Monitoring's blocking queries view.",+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#locks',});@@ -162,5 +285,6 @@ prodMonitor('macrodb-long-transactions', {pages: false,description:'The oldest open transaction on {{host.name}} is {{value}} seconds old. Long transactions hold locks and stop vacuum.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#locks',});diff --git a/infra/stacks/monitoring/monitors/queues.ts b/infra/stacks/monitoring/monitors/queues.tsindex 393de6b269..3d00a7181d 100644--- a/infra/stacks/monitoring/monitors/queues.ts+++ b/infra/stacks/monitoring/monitors/queues.ts@@ -1,9 +1,51 @@// Prod SQS queues: dead letters, backlog age and backlog growth.//// The two queue-age monitors also exist in Grafana-// (infra/stacks/observability/nixos/alerting.nix); change both together.+// (infra/stacks/observability/nixos/prod-alerts.nix); change both together.// Grafana pages on-call for the work queues, so they only notify here.-import { prodMonitor } from '../prod-monitor';+import * as datadog from '@pulumi/datadog';+import { graphGroup, prodMonitor, thresholdGraph } from '../prod-monitor';++// One graph per monitor below, titled like it, with its thresholds.+const dashboard = new datadog.Dashboard('queue-alerts', {+ title: 'Queue alerts',+ description:+ 'Every prod SQS monitor against its thresholds. Declared in infra/stacks/monitoring/monitors/queues.ts; see docs/runbooks/queues.md.',+ layoutType: 'ordered',+ widgets: [+ graphGroup('Dead letters', [+ thresholdGraph({+ // Thirty-minute sums, as the monitor evaluates them.+ title: 'Messages landing in a prod dead-letter queue',+ query:+ 'sum:aws.sqs.number_of_messages_sent{queuename:*-dlq-prod} by {queuename}.as_count().rollup(sum, 1800)',+ critical: '50',+ warning: '0',+ }),+ ]),+ graphGroup('Backlog', [+ thresholdGraph({+ title: 'Email backfill queue falling behind',+ query:+ 'max:aws.sqs.approximate_age_of_oldest_message{queuename:email-service-backfill-queue-prod}',+ critical: '10800',+ warning: '3600',+ }),+ thresholdGraph({+ title: 'Prod work queue falling behind',+ query:+ 'max:aws.sqs.approximate_age_of_oldest_message{queuename:email-service-refresh-queue-prod OR queuename:convert-service-queue-prod OR queuename:ai-projection-queue-prod OR queuename:email-service-gmail-webhook-queue-prod OR queuename:notification-ingress-queue-prod} by {queuename}',+ critical: '7200',+ warning: '1800',+ }),+ thresholdGraph({+ title: 'Email backfill backlog growing abnormally',+ query:+ "anomalies(max:aws.sqs.approximate_number_of_messages_visible{queuename:email-service-backfill-queue-prod}, 'agile', 3)",+ }),+ ]),+ ],+});// Datadog only: CloudWatch dimensions can't be filtered by name pattern.prodMonitor('sqs-dlq-inflow', {@@ -15,6 +57,7 @@ prodMonitor('sqs-dlq-inflow', {pages: false,description:'{{value}} messages moved to {{queuename.name}} in 30 minutes. Most DLQs hold old messages permanently, so this alerts on new arrivals only.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/queues.md#dead-letters',});@@ -27,6 +70,7 @@ prodMonitor('sqs-email-backfill-age', {pages: false,description:'The oldest message in email-service-backfill-queue-prod is {{value}} seconds old. Normal peaks are about 50 minutes; new users wait this long for their mailbox to fill.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/queues.md#backlog',});@@ -39,6 +83,7 @@ prodMonitor('sqs-work-queue-age', {pages: false,description:'The oldest message in {{queuename.name}} is {{value}} seconds old. These queues normally drain within minutes.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/queues.md#backlog',});@@ -57,5 +102,6 @@ prodMonitor('sqs-email-backfill-backlog-anomaly', {pages: false,description:'Visible messages in email-service-backfill-queue-prod are above their usual band. A jump in enqueues loads macrodb through the backfill workers.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/queues.md#backlog',});diff --git a/infra/stacks/monitoring/monitors/redis.ts b/infra/stacks/monitoring/monitors/redis.tsindex 5165ac7afe..7fde9b118d 100644--- a/infra/stacks/monitoring/monitors/redis.ts+++ b/infra/stacks/monitoring/monitors/redis.ts@@ -2,9 +2,45 @@//// macro-cache-prod climbed from 60% to 100% over a month before writes failed.// Memory and evictions also exist in Grafana-// (infra/stacks/observability/nixos/alerting.nix); change both together.+// (infra/stacks/observability/nixos/prod-alerts.nix); change both together.// Grafana pages on-call for memory, so it only notifies here.-import { prodMonitor } from '../prod-monitor';+import * as datadog from '@pulumi/datadog';+import { graphGroup, prodMonitor, thresholdGraph } from '../prod-monitor';++// One graph per monitor below, titled like it, with its thresholds.+const dashboard = new datadog.Dashboard('redis-alerts', {+ title: 'Redis alerts',+ description:+ 'Every prod Redis monitor against its thresholds. Declared in infra/stacks/monitoring/monitors/redis.ts; see docs/runbooks/redis.md.',+ layoutType: 'ordered',+ widgets: [+ graphGroup('Memory', [+ thresholdGraph({+ title: 'Redis memory high',+ query:+ 'max:aws.elasticache.database_memory_usage_percentage{cacheclusterid:*prod*} by {cacheclusterid}',+ critical: '90',+ warning: '75',+ }),+ thresholdGraph({+ title: 'Redis memory forecast to fill within three days',+ query:+ "forecast(max:aws.elasticache.database_memory_usage_percentage{cacheclusterid:*prod*} by {cacheclusterid}, 'linear', 1)",+ critical: '95',+ }),+ ]),+ graphGroup('Evictions', [+ thresholdGraph({+ // Ten-minute sums, as the monitor evaluates them.+ title: 'Redis evicting keys',+ query:+ 'sum:aws.elasticache.evictions{cacheclusterid:*prod*} by {cacheclusterid}.as_count().rollup(sum, 600)',+ critical: '1000',+ warning: '0',+ }),+ ]),+ ],+});prodMonitor('redis-memory', {title: 'Redis memory high',@@ -15,6 +51,7 @@ prodMonitor('redis-memory', {pages: false,description:'{{cacheclusterid.name}} is using {{value}}% of maxmemory. Keys without a TTL are never evicted under volatile-lru, so at 100% writes fail.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/redis.md#memory',});@@ -28,6 +65,7 @@ prodMonitor('redis-memory-forecast', {pages: false,description:'{{cacheclusterid.name}} memory is growing fast enough to pass 95% within three days. macro-cache-prod climbed from 60% to 100% over a month before writes failed.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/redis.md#memory',});@@ -40,5 +78,6 @@ prodMonitor('redis-evictions', {pages: false,description:'{{cacheclusterid.name}} evicted {{value}} keys in 10 minutes. Prod caches normally evict nothing.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/redis.md#memory',});diff --git a/infra/stacks/monitoring/monitors/regressions.ts b/infra/stacks/monitoring/monitors/regressions.tsindex 86ff6c09e8..75d8579a7b 100644--- a/infra/stacks/monitoring/monitors/regressions.ts+++ b/infra/stacks/monitoring/monitors/regressions.ts@@ -4,7 +4,7 @@// Datadog only: prod traces and logs don't reach Grafana, and Grafana alerting// has no anomaly detection.import * as datadog from '@pulumi/datadog';-import { prodMonitor } from '../prod-monitor';+import { graphGroup, prodMonitor, thresholdGraph } from '../prod-monitor';// Datadog has no anomaly function for log monitors, so the error-logs anomaly// monitor reads this log-based metric instead. Counts every prod error log@@ -16,6 +16,46 @@ new datadog.LogsMetric('macro-logs-errors', {groupBies: [{ path: 'service', tagName: 'service' }],});+// One graph per monitor below, titled like it, with the anomaly band each+// monitor alerts outside of.+const dashboard = new datadog.Dashboard('regression-alerts', {+ title: 'Regression alerts',+ description:+ 'Every prod error and latency anomaly monitor with its band. Declared in infra/stacks/monitoring/monitors/regressions.ts; see docs/runbooks/regressions.md.',+ layoutType: 'ordered',+ widgets: [+ graphGroup('Errors', [+ thresholdGraph({+ title: 'HTTP server errors abnormally high',+ query:+ "anomalies(sum:trace.http.server.request.errors{env:prod} by {service}.as_count(), 'agile', 3)",+ }),+ thresholdGraph({+ title: 'Internal span errors abnormally high',+ query:+ "anomalies(sum:trace.Internal.errors{env:prod} by {service}.as_count(), 'agile', 3)",+ }),+ thresholdGraph({+ title: 'Queue consumer errors abnormally high',+ query:+ "anomalies(sum:trace.Consumer.errors{env:prod} by {service}.as_count(), 'agile', 3)",+ }),+ thresholdGraph({+ title: 'Error logs abnormally high',+ query:+ "anomalies(sum:macro.logs.errors{*} by {service}.as_count(), 'agile', 3)",+ }),+ ]),+ graphGroup('Latency', [+ thresholdGraph({+ title: 'HTTP latency abnormally high',+ query:+ "anomalies(p95:trace.http.server.request{env:prod AND (service:cloud-storage-service OR service:authentication-service)} by {service}, 'agile', 3)",+ }),+ ]),+ ],+});+// An anomaly monitor recovers once no point in its window is outside the band.const ANOMALY_THRESHOLDS = { critical: '1', criticalRecovery: '0' };const ANOMALY_WINDOWS = {@@ -33,6 +73,7 @@ prodMonitor('apm-server-error-anomaly', {pages: false,description:'{{service.name}} is returning more HTTP server errors than its usual band. Check the latest deploy of that service first.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/regressions.md#errors',});@@ -46,6 +87,7 @@ prodMonitor('apm-internal-error-anomaly', {pages: false,description:'{{service.name}} is recording more failed internal spans (background jobs, workers, browser operations) than its usual band. Check the latest deploy of that service first.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/regressions.md#errors',});@@ -59,6 +101,7 @@ prodMonitor('apm-consumer-error-anomaly', {pages: false,description:'{{service.name}} is failing more queue and Kafka messages than its usual band.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/regressions.md#errors',});@@ -71,6 +114,7 @@ prodMonitor('apm-latency-anomaly', {monitorThresholdWindows: ANOMALY_WINDOWS,pages: false,description: '{{service.name}} p95 HTTP latency is above its usual band.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/regressions.md#latency',});@@ -84,5 +128,6 @@ prodMonitor('error-logs-anomaly', {pages: false,description:'{{service.name}} is logging more errors than its usual band. Check the latest deploy of that service first.',+ dashboard: dashboard.url,runbook: 'docs/runbooks/regressions.md#errors',});diff --git a/infra/stacks/monitoring/notify.ts b/infra/stacks/monitoring/notify.tsindex 4a4380d61e..83234c16a3 100644--- a/infra/stacks/monitoring/notify.ts+++ b/infra/stacks/monitoring/notify.ts@@ -1,3 +1,5 @@+import * as pulumi from '@pulumi/pulumi';+// Where every monitor in this stack notifies.//// All monitors post to the Macro Alerts channel. Paging monitors also open a@@ -24,6 +26,9 @@ export function notify({ pages }: { pages: boolean }): string {${MACRO_ALERTS}`;}+// Datadog's site: dashboard `url`s are paths under it.+export const DATADOG_SITE = 'https://us5.datadoghq.com';+/*** A monitor message: the state and value against its threshold, what it* means, where to start, and the notification handles.@@ -31,17 +36,26 @@ ${MACRO_ALERTS}`;export function alertMessage({description,runbook,+ dashboard,pages,}: {description: string;runbook: string;+ /** A dashboard's `url`, a path such as `/dashboard/abc-def-ghi/title`. */+ dashboard?: pulumi.Input<string>;pages: boolean;-}): string {- return `{{#is_alert}}🔴 CRITICAL: {{value}} (threshold {{threshold}}){{/is_alert}}{{#is_warning}}🟠 WARNING: {{value}} (warning threshold {{warn_threshold}}){{/is_warning}}{{#is_recovery}}✅ RECOVERED: {{value}}{{/is_recovery}}+}): pulumi.Output<string> {+ return pulumi.output(dashboard).apply((path) => {+ const links = [+ `Runbook: https://github.com/macro-inc/macro/blob/main/${runbook}`,+ ];+ if (path !== undefined) links.push(`Dashboard: ${DATADOG_SITE}${path}`);+ return `{{#is_alert}}🔴 CRITICAL: {{value}} (threshold {{threshold}}){{/is_alert}}{{#is_warning}}🟠 WARNING: {{value}} (warning threshold {{warn_threshold}}){{/is_warning}}{{#is_recovery}}✅ RECOVERED: {{value}}{{/is_recovery}}${description}-Runbook: https://github.com/macro-inc/macro/blob/main/${runbook}+${links.join('\n')}${notify({ pages })}`;+ });}diff --git a/infra/stacks/monitoring/prod-monitor.ts b/infra/stacks/monitoring/prod-monitor.tsindex eca574ef86..ec8fd8c415 100644--- a/infra/stacks/monitoring/prod-monitor.ts+++ b/infra/stacks/monitoring/prod-monitor.ts@@ -1,10 +1,11 @@import * as datadog from '@pulumi/datadog';+import type * as pulumi from '@pulumi/pulumi';import { goalertWebhooks } from './goalert';import { alertMessage, RENOTIFY_MINUTES } from './notify';/*** A prod monitor declared in this stack. The rules with a CloudWatch metric- * also exist in Grafana (infra/stacks/observability/nixos/alerting.nix); keep+ * also exist in Grafana (infra/stacks/observability/nixos/prod-alerts.nix); keep* their thresholds and windows in sync there by hand.*/export function prodMonitor(@@ -21,6 +22,8 @@ export function prodMonitor(description: string;/** Repository path with a heading anchor, e.g. `docs/runbooks/macrodb.md#memory`. */runbook: string;+ /** `url` of the dashboard graphing this monitor, under a `thresholdGraph` titled `title`. */+ dashboard?: pulumi.Input<string>;}): datadog.Monitor {return new datadog.Monitor(@@ -32,6 +35,7 @@ export function prodMonitor(message: alertMessage({description: args.description,runbook: args.runbook,+ dashboard: args.dashboard,pages: args.pages,}),tags: ['env:prod', `service:${args.service}`],@@ -47,3 +51,47 @@ export function prodMonitor({ dependsOn: args.pages ? goalertWebhooks : [] });}++/**+ * A line graph of one prod monitor's metric, titled like the monitor, with+ * its critical and warning thresholds drawn as markers. Anomaly graphs have no+ * markers: the band is the threshold.+ */+export function thresholdGraph(args: {+ title: string;+ query: string;+ critical?: string;+ warning?: string;+}): datadog.types.input.DashboardWidget {+ const markers: datadog.types.input.DashboardWidgetTimeseriesDefinitionMarker[] =+ [];+ if (args.critical !== undefined) {+ markers.push({+ label: 'critical',+ value: `y = ${args.critical}`,+ displayType: 'error dashed',+ });+ }+ if (args.warning !== undefined) {+ markers.push({+ label: 'warning',+ value: `y = ${args.warning}`,+ displayType: 'warning dashed',+ });+ }+ return {+ timeseriesDefinition: {+ title: args.title,+ requests: [{ q: args.query, displayType: 'line' }],+ markers,+ },+ };+}++/** A titled group of related graphs on a dashboard. */+export function graphGroup(+ title: string,+ widgets: datadog.types.input.DashboardWidget[]+): datadog.types.input.DashboardWidget {+ return { groupDefinition: { title, layoutType: 'ordered', widgets } };+}diff --git a/infra/stacks/monitoring/routing.test.ts b/infra/stacks/monitoring/routing.test.tsindex b140d1d5e0..3d1810571c 100644--- a/infra/stacks/monitoring/routing.test.ts+++ b/infra/stacks/monitoring/routing.test.ts@@ -12,7 +12,12 @@ beforeAll(async () => {{newResource(args) {resources.push(args);- return { id: args.name, state: args.inputs };+ // Datadog returns a dashboard's url as a path under the site.+ const state =+ args.type === 'datadog:index/dashboard:Dashboard'+ ? { ...args.inputs, url: `/dashboard/${args.name}/title` }+ : args.inputs;+ return { id: args.name, state };},call(args) {return args.inputs;@@ -28,6 +33,8 @@ beforeAll(async () => {const monitors = () =>resources.filter((r) => r.type === 'datadog:index/monitor:Monitor');+const dashboards = () =>+ resources.filter((r) => r.type === 'datadog:index/dashboard:Dashboard');const synthetics = () =>resources.filter((r) => r.type === 'datadog:index/syntheticsTest:SyntheticsTest'@@ -116,3 +123,46 @@ test('anomaly rules recover at zero and carry their alert window', () => {recoveryWindow: 'last_15m',});});++test('a prod monitor links to its dashboard', () => {+ const cpu = monitors().find((r) => r.name === 'macrodb-cpu');+ expect(cpu?.inputs.message).toContain(+ 'Runbook: https://github.com/macro-inc/macro/blob/main/docs/runbooks/macrodb.md#load\nDashboard: https://us5.datadoghq.com/dashboard/macrodb-alerts/title'+ );+});++test('every prod monitor is graphed on its dashboard with its thresholds', () => {+ const prod = monitors().filter((r) =>+ r.inputs.message.includes('Runbook: https://github.com/macro-inc/macro/')+ );+ expect(prod).toHaveLength(25);+ for (const monitor of prod) {+ const linked = monitor.inputs.message.match(+ /Dashboard: https:\/\/us5\.datadoghq\.com\/dashboard\/([\w-]+)\//+ )?.[1];+ const dashboard = dashboards().find((d) => d.name === linked);+ const title = monitor.inputs.name.replace('[PROD] ', '');+ const graph = dashboard?.inputs.widgets+ .flatMap(+ (group: { groupDefinition: { widgets: unknown[] } }) =>+ group.groupDefinition.widgets+ )+ .find(+ (widget: { timeseriesDefinition: { title: string } }) =>+ widget.timeseriesDefinition.title === title+ );+ const { critical, warning } = monitor.inputs.monitorThresholds;+ const anomaly = monitor.inputs.query.includes('anomalies(');+ expect({+ name: monitor.name,+ markers: graph?.timeseriesDefinition.markers.map(+ (marker: { value: string }) => marker.value+ ),+ }).toEqual({+ name: monitor.name,+ markers: anomaly+ ? []+ : [`y = ${critical}`, ...(warning ? [`y = ${warning}`] : [])],+ });+ }+});diff --git a/infra/stacks/observability/AGENTS.md b/infra/stacks/observability/AGENTS.mdindex d995967ba9..a0364f759c 100644--- a/infra/stacks/observability/AGENTS.md+++ b/infra/stacks/observability/AGENTS.md@@ -14,8 +14,10 @@ us-east-1. `README.md` is the full operations guide.## Recipes-- Change a Grafana prod alert rule: edit `prodRules` in `nixos/alerting.nix`, and- its Datadog copy in `infra/stacks/monitoring/monitors/` by hand.+- Change a Grafana prod alert rule: edit its entry in `nixos/prod-alerts.nix`+ (rules and the `Prod alerts` dashboard panel both read it), and its Datadog+ copy in `infra/stacks/monitoring/monitors/` by hand. A new entry takes the next+ free `panelId`; never renumber one, since sent notifications link to it.- Add a runtime setting: `settings.ts` → `render.ts` (bump the schema version) →`nixos/read-user-data.py` (keys, pattern, version) → the consuming `.nix` file.- Add a secret: `assets/refresh-secrets.py` (validation) → `LoadCredential` indiff --git a/infra/stacks/observability/Pulumi.prod.yaml b/infra/stacks/observability/Pulumi.prod.yamlindex e233257080..5de60a5f49 100644--- a/infra/stacks/observability/Pulumi.prod.yaml+++ b/infra/stacks/observability/Pulumi.prod.yaml@@ -2,5 +2,5 @@ config:aws:region: us-east-2observability:ciIngestSecretArn: arn:aws:secretsmanager:us-east-1:569036502058:secret:observability/dev-ingest-7UhJbLobservability:alarmTopicArn: arn:aws:sns:us-east-2:569036502058:observability-prod-alerts- observability:amiId: ami-0e775059ad164977c+ observability:amiId: ami-0af944c17d63ec7c4observability:secretArn: arn:aws:secretsmanager:us-east-2:569036502058:secret:observability/prod-TwQpBJdiff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.mdindex 43fd8ccde9..ff13d8f15d 100644--- a/infra/stacks/observability/README.md+++ b/infra/stacks/observability/README.md@@ -77,6 +77,8 @@ packages, configuration, service users, resource limits and startup dependencies- `nixos/goalert.nix` and `nixos/goalert/`: GoAlert, its Postgres, backups andthe Macro Alerts relay (see [On-call paging](#on-call-paging-goalert)).- `nixos/alerting.nix`: Grafana alert rules, contact points and routing.+- `nixos/prod-alerts.nix`: prod alert queries and thresholds, shared by the+ rules and `nixos/prod-alerts-dashboard.nix`.- `nixos/host.nix`: boot dependencies, retained storage and stable service identities.NixOS modules generate application configuration from Nix attributes. Alloy's@@ -435,20 +437,27 @@ models. Grafana uses the EC2 role for regional metric reads and publishing to thconfigured alarm SNS topic. Other unprivileged HTTP/ingest services still cannotaccess instance metadata.-`prodRules` in `nixos/alerting.nix` holds the prod CloudWatch rules: macrodb-memory, swap, load, CPU, connections, replica lag, read latency and storage;-Redis memory and evictions; SQS queue age. Each is a warning and a critical rule-in the `Prod` group, reading CloudWatch in Virginia directly. Each has a Datadog-copy in `infra/stacks/monitoring/monitors/`; keep thresholds and windows in sync-by hand. Forecasts, anomalies, Database Monitoring, APM and log rules are-Datadog-only.+`nixos/prod-alerts.nix` lists the prod CloudWatch alerts: macrodb memory, swap,+load, CPU, connections, replica lag, read latency and storage; Redis memory and+evictions; SQS queue age. Each entry holds its query, thresholds, window and+text once. `nixos/alerting.nix` turns each into a warning and a critical rule in+the `Prod` group, reading CloudWatch in Virginia directly, and+`nixos/prod-alerts-dashboard.nix` draws it as one panel with both thresholds on+the `Prod alerts` dashboard (`/d/prod-alerts`, rows macrodb, Redis and Queues).+Each rule carries its panel as `__dashboardUid__`/`__panelId__`, so the rule+view and every notification link to the graph of the metric that fired. Each+has a Datadog copy in `infra/stacks/monitoring/monitors/`; keep thresholds and+windows in sync by hand. Forecasts, anomalies, Database Monitoring, APM and log+rules are Datadog-only.Every alert posts to the `Macro Alerts` contact point through the loopback-relay; critical ones renotify hourly, others every four hours. Critical alerts-of rules with `pages = true` also go to the `GoAlert` contact point over-loopback. GoAlert owns the alert from there: it opens, acknowledges and closes-it, and its escalation posts each change to the channel. Their Datadog copies-only notify, so an incident pages once.+relay; critical ones renotify hourly, others every four hours. Grafana's+`default.message` lists each alert's annotations (summary, description,+runbook) and a `Panel:` link to its graph, opened an hour before the alert+started. Critical alerts of rules with `pages = true` also go to the `GoAlert`+contact point over loopback. GoAlert owns the alert from there: it opens,+acknowledges and closes it, and its escalation posts each change to the+channel. Their Datadog copies only notify, so an incident pages once.The dev rules below keep their own route to `Macro team email`.@@ -479,10 +488,10 @@ AMI; a schema-5 image cannot boot schema-6 user data.GoAlert is the source of truth for open alerts: Grafana sends it criticalalerts, and on-call engineers acknowledge or close them in its UI at-`https://oncall.macro-internal.com`. It runs on this host so paging keeps working-while production, its database or Macro itself is down. Today it pages by-posting to the Macro Alerts channel; phone paging (Twilio), Google login and a-rotation are follow-ups.+`https://oncall.macro-internal.com`, signing in with Google. It runs on this host+so paging keeps working while production, its database or Macro itself is down.+Today it pages by posting to the Macro Alerts channel; phone paging (Twilio) and+a rotation are follow-ups.- `nixos/goalert/package.nix` builds GoAlert from source with `buildGoModule`,embedding its web UI. Bump `version` and the hashes together.@@ -504,6 +513,10 @@ rotation are follow-ups.`/health`, `/api/v2/generic/incoming` and `/api/v2/heartbeat/` are reachablewithout a session (GoAlert checks the integration key); other inboundintegrations return 404. The rest is GoAlert's UI behind its login.+- Login is Google OIDC with Grafana's OAuth client, whose redirect URIs include+ `https://oncall.macro-internal.com/api/v2/identity/providers/oidc/callback`.+ GoAlert cannot restrict OIDC to a domain, so that client must keep the+ Internal audience. The basic-auth `admin` login is the break-glass account.- The ALB sends the on-call host to its own target group, health-checked at`/goalert-healthz`. Alarm `observability-goalert` publishes to `alarmTopicArn`when it fails, so a GoAlert outage is reported without GoAlert.diff --git a/infra/stacks/observability/nixos/alerting.nix b/infra/stacks/observability/nixos/alerting.nixindex e761fa4a35..4babf369ae 100644--- a/infra/stacks/observability/nixos/alerting.nix+++ b/infra/stacks/observability/nixos/alerting.nix@@ -66,61 +66,18 @@ let}];- # Prod rules. Each has a matching Datadog monitor in- # infra/stacks/monitoring/monitors/{macrodb,redis,queues}.ts; keep thresholds- # and windows in sync by hand. Grafana pages for the rules marked `pages`, so- # their Datadog copies only notify.- cloudwatch =- {- namespace,- metricName,- dimensions,- statistic,- }:- window: [- {- refId = "A";- datasourceUid = "cloudwatch";- relativeTimeRange = {- from = window;- to = 0;- };- model = {- refId = "A";- id = "a";- region = "us-east-1";- inherit- namespace- metricName- dimensions- statistic- ;- period = "60";- matchExact = true;- metricQueryType = 0;- metricEditorMode = 0;- queryMode = "Metrics";- };- }- ];- # `window` is in seconds; `condition` compares the reduced value $B.+ # Prod rules: a critical and a warning rule per entry in prod-alerts.nix,+ # each linked to that entry's panel on the "Prod alerts" dashboard.+ prodAlerts = import ./prod-alerts.nix;prodRule =+ alert:{- uid,- title,severity,- service,- pages ? false,- metric,- window,- reducer,condition,- description,- runbook,}:{- inherit uid;- title = "[PROD] ${title} (${severity})";+ uid = "${alert.uid}-${if severity == "critical" then "crit" else "warn"}";+ title = "[PROD] ${alert.title} (${severity})";condition = "C";for = "0s";# Sparse CloudWatch counters have no datapoints when idle.@@ -129,431 +86,63 @@ letisPaused = false;labels = {environment = "prod";- inherit severity service;+ inherit severity;+ inherit (alert) service;# Only critical alerts page, through GoAlert.- pages = lib.boolToString (pages && severity == "critical");+ pages = lib.boolToString ((alert.pages or false) && severity == "critical");};+ # Grafana turns the dashboard and panel annotations into the panel link+ # in notifications and the rule view.annotations = {- summary = title;- inherit description;- runbook_url = "https://github.com/macro-inc/macro/blob/main/docs/runbooks/${runbook}";+ summary = alert.title;+ inherit (alert) description;+ runbook_url = "https://github.com/macro-inc/macro/blob/main/docs/runbooks/${alert.runbook}";+ __dashboardUid__ = prodAlerts.dashboardUid;+ __panelId__ = toString alert.panelId;};- data = cloudwatch metric window ++ expressions reducer condition;+ data = [+ {+ refId = "A";+ datasourceUid = "cloudwatch";+ relativeTimeRange = {+ from = alert.window;+ to = 0;+ };+ model = prodAlerts.cloudwatchQuery alert.metric;+ }+ ]+ ++ expressions alert.reducer condition;};-- rds = metricName: instances: statistic: {- namespace = "AWS/RDS";- inherit metricName statistic;- dimensions.DBInstanceIdentifier = instances;- };- bothInstances = [- "macro-db-prod"- "macro-db-prod-read-replica"- ];- redisNodes = [- "macro-cache-prod-001"- "macro-cache-prod-002"- "email-service-redis-prod-c862c10"- "calendar-service-redis-prod-aee80d2"- "connection-gateway-redis-prod-f9d8cd0"- ];- redis = metricName: statistic: {- namespace = "AWS/ElastiCache";- inherit metricName statistic;- dimensions.CacheClusterId = redisNodes;- };- queueAge = queues: {- namespace = "AWS/SQS";- metricName = "ApproximateAgeOfOldestMessage";- statistic = "Maximum";- dimensions.QueueName = queues;- };- workQueues = [- "email-service-refresh-queue-prod"- "convert-service-queue-prod"- "ai-projection-queue-prod"- "email-service-gmail-webhook-queue-prod"- "notification-ingress-queue-prod"- ];-- primaryMemory = {- service = "macro-db-prod";- pages = true;- metric = rds "FreeableMemory" [ "macro-db-prod" ] "Average";- window = 900;- reducer = "mean";- description = "Freeable memory is {{ $values.B }} bytes. Normal is 7-18 GB; it fell to about 0.45 GB during the October overload. Each connection holds about 14 MB of private memory, so check connection count and swap first.";- runbook = "macrodb.md#memory";- };- replicaMemory = {- service = "macro-db-prod";- pages = true;- metric = rds "FreeableMemory" [ "macro-db-prod-read-replica" ] "Average";- window = 900;- reducer = "mean";- description = "Replica freeable memory is {{ $values.B }} bytes. Normal minimum is about 3 GB. The replica serves GraphQL reads, so memory pressure shows up as slow lists.";- runbook = "macrodb.md#memory";- };- swap = {- service = "macro-db-prod";- pages = true;- metric = rds "SwapUsage" bothInstances "Maximum";- window = 900;- reducer = "mean";- description = "{{ $labels.DBInstanceIdentifier }} is using {{ $values.B }} bytes of swap. Normal is under 0.6 GB. Swap above 5 GB preceded both October overloads by hours.";- runbook = "macrodb.md#memory";- };- load = {- service = "macro-db-prod";- pages = true;- metric = rds "DBLoadRelativeToNumVCPUs" bothInstances "Average";- window = 900;- reducer = "mean";- description = "{{ $labels.DBInstanceIdentifier }} has {{ $values.B }}x as many active sessions as vCPUs. Normal is under 1.3; the October overload reached 8. Look at Performance Insights top waits (locks, IO) and the services holding the most connections.";- runbook = "macrodb.md#load";- };- cpu = {- service = "macro-db-prod";- pages = true;- metric = rds "CPUUtilization" bothInstances "Average";- window = 900;- reducer = "mean";- description = "{{ $labels.DBInstanceIdentifier }} CPU averaged {{ $values.B }}% over 15 minutes. Normal peaks are about 30%.";- runbook = "macrodb.md#load";- };- primaryConnections = {- service = "macro-db-prod";- pages = true;- metric = rds "DatabaseConnections" [ "macro-db-prod" ] "Maximum";- window = 600;- reducer = "mean";- description = "macro-db-prod has {{ $values.B }} connections. Normal 15-minute peaks are about 1,100; max_connections is 3,439. Each connection costs about 14 MB, so this also drains freeable memory.";- runbook = "macrodb.md#connections";- };- replicaConnections = {- service = "macro-db-prod";- metric = rds "DatabaseConnections" [ "macro-db-prod-read-replica" ] "Maximum";- window = 600;- reducer = "mean";- description = "The read replica has {{ $values.B }} connections. Normal peaks are about 460.";- runbook = "macrodb.md#connections";- };- replicaLag = {- service = "macro-db-prod";- pages = true;- metric = rds "ReplicaLag" [ "macro-db-prod-read-replica" ] "Maximum";- window = 600;- reducer = "mean";- description = "The read replica is {{ $values.B }} seconds behind the primary. Normal is under 6 seconds. GraphQL reads use the replica, so users see stale data and missing new items.";- runbook = "macrodb.md#replica-lag";- };- readLatency = {- service = "macro-db-prod";- metric = rds "ReadLatency" bothInstances "Average";- window = 900;- reducer = "mean";- description = "{{ $labels.DBInstanceIdentifier }} disk reads take {{ $values.B }} seconds on average. Normal is about 0.6 ms.";- runbook = "macrodb.md#load";- };- freeStorage = {- service = "macro-db-prod";- pages = true;- metric = rds "FreeStorageSpace" bothInstances "Minimum";- window = 3600;- reducer = "min";- description = "{{ $labels.DBInstanceIdentifier }} has {{ $values.B }} bytes of free storage out of 5.37 TB.";- runbook = "macrodb.md#storage";- };- redisMemory = {- service = "elasticache";- pages = true;- metric = redis "DatabaseMemoryUsagePercentage" "Maximum";- window = 600;- reducer = "mean";- description = "{{ $labels.CacheClusterId }} is using {{ $values.B }}% of maxmemory. Keys without a TTL are never evicted under volatile-lru, so at 100% writes fail.";- runbook = "redis.md#memory";- };- redisEvictions = {- service = "elasticache";- metric = redis "Evictions" "Sum";- window = 600;- reducer = "sum";- description = "{{ $labels.CacheClusterId }} evicted {{ $values.B }} keys in 10 minutes. Prod caches normally evict nothing.";- runbook = "redis.md#memory";- };- backfillAge = {- service = "email-service";- metric = queueAge [ "email-service-backfill-queue-prod" ];- window = 900;- reducer = "mean";- description = "The oldest message in email-service-backfill-queue-prod is {{ $values.B }} seconds old. Normal peaks are about 50 minutes; new users wait this long for their mailbox to fill.";- runbook = "queues.md#backlog";- };- workQueueAge = {- service = "sqs";- pages = true;- metric = queueAge workQueues;- window = 900;- reducer = "mean";- description = "The oldest message in {{ $labels.QueueName }} is {{ $values.B }} seconds old. These queues normally drain within minutes.";- runbook = "queues.md#backlog";- };-# Warning and critical are mutually exclusive bands, as in the dev rules.- prodRules = map prodRule [- (- primaryMemory- // {- uid = "macrodb-primary-freeable-memory-crit";- title = "macro-db-prod freeable memory low";- severity = "critical";- condition = "$B < 1000000000";- }- )- (- primaryMemory- // {- uid = "macrodb-primary-freeable-memory-warn";- title = "macro-db-prod freeable memory low";- severity = "warning";- condition = "$B < 2000000000 && $B >= 1000000000";- }- )- (- replicaMemory- // {- uid = "macrodb-replica-freeable-memory-crit";- title = "macro-db-prod-read-replica freeable memory low";- severity = "critical";- condition = "$B < 750000000";- }- )- (- replicaMemory- // {- uid = "macrodb-replica-freeable-memory-warn";- title = "macro-db-prod-read-replica freeable memory low";- severity = "warning";- condition = "$B < 1500000000 && $B >= 750000000";- }- )- (- swap- // {- uid = "macrodb-swap-crit";- title = "macro-db-prod swapping";- severity = "critical";- condition = "$B > 5000000000";- }- )- (- swap- // {- uid = "macrodb-swap-warn";- title = "macro-db-prod swapping";- severity = "warning";- condition = "$B > 2000000000 && $B <= 5000000000";- }- )- (- load- // {- uid = "macrodb-load-crit";- title = "macro-db-prod database load above vCPU count";- severity = "critical";- condition = "$B > 4";- }- )- (- load- // {- uid = "macrodb-load-warn";- title = "macro-db-prod database load above vCPU count";- severity = "warning";- condition = "$B > 2 && $B <= 4";- }- )- (- cpu- // {- uid = "macrodb-cpu-crit";- title = "macro-db-prod CPU high";- severity = "critical";- condition = "$B > 85";- }- )- (- cpu- // {- uid = "macrodb-cpu-warn";- title = "macro-db-prod CPU high";- severity = "warning";- condition = "$B > 70 && $B <= 85";- }- )- (- primaryConnections- // {- uid = "macrodb-primary-connections-crit";- title = "macro-db-prod connections high";- severity = "critical";- condition = "$B > 2200";- }- )- (- primaryConnections- // {- uid = "macrodb-primary-connections-warn";- title = "macro-db-prod connections high";- severity = "warning";- condition = "$B > 1700 && $B <= 2200";- }- )- (- replicaConnections- // {- uid = "macrodb-replica-connections-crit";- title = "macro-db-prod-read-replica connections high";- severity = "critical";- condition = "$B > 700";- }- )- (- replicaConnections- // {- uid = "macrodb-replica-connections-warn";- title = "macro-db-prod-read-replica connections high";- severity = "warning";- condition = "$B > 550 && $B <= 700";- }- )- (- replicaLag- // {- uid = "macrodb-replica-lag-crit";- title = "macro-db-prod-read-replica lagging";- severity = "critical";- condition = "$B > 300";- }- )- (- replicaLag- // {- uid = "macrodb-replica-lag-warn";- title = "macro-db-prod-read-replica lagging";- severity = "warning";- condition = "$B > 60 && $B <= 300";- }- )- (- readLatency- // {- uid = "macrodb-read-latency-crit";- title = "macro-db-prod disk read latency high";- severity = "critical";- condition = "$B > 0.01";- }- )- (- readLatency- // {- uid = "macrodb-read-latency-warn";- title = "macro-db-prod disk read latency high";- severity = "warning";- condition = "$B > 0.005 && $B <= 0.01";- }- )- (- freeStorage- // {- uid = "macrodb-free-storage-crit";- title = "macro-db-prod free storage low";- severity = "critical";- condition = "$B < 1000000000000";- }- )- (- freeStorage- // {- uid = "macrodb-free-storage-warn";- title = "macro-db-prod free storage low";- severity = "warning";- condition = "$B < 1500000000000 && $B >= 1000000000000";- }- )- (- redisMemory- // {- uid = "redis-memory-crit";- title = "Redis memory high";- severity = "critical";- condition = "$B > 90";- }- )- (- redisMemory- // {- uid = "redis-memory-warn";- title = "Redis memory high";- severity = "warning";- condition = "$B > 75 && $B <= 90";- }- )- (- redisEvictions- // {- uid = "redis-evictions-crit";- title = "Redis evicting keys";- severity = "critical";- condition = "$B > 1000";- }- )- (- redisEvictions- // {- uid = "redis-evictions-warn";- title = "Redis evicting keys";- severity = "warning";- condition = "$B > 0 && $B <= 1000";- }- )- (- backfillAge- // {- uid = "sqs-email-backfill-age-crit";- title = "Email backfill queue falling behind";- severity = "critical";- condition = "$B > 10800";- }- )- (- backfillAge- // {- uid = "sqs-email-backfill-age-warn";- title = "Email backfill queue falling behind";- severity = "warning";- condition = "$B > 3600 && $B <= 10800";- }- )- (- workQueueAge- // {- uid = "sqs-work-queue-age-crit";- title = "Prod work queue falling behind";- severity = "critical";- condition = "$B > 7200";- }- )- (- workQueueAge- // {- uid = "sqs-work-queue-age-warn";- title = "Prod work queue falling behind";- severity = "warning";- condition = "$B > 1800 && $B <= 7200";- }- )- ];+ prodRulePair =+ alert:+ let+ critical = builtins.toJSON alert.critical;+ warning = builtins.toJSON alert.warning;+ in+ if alert.firesWhen == "above" then+ [+ (prodRule alert {+ severity = "critical";+ condition = "$B > ${critical}";+ })+ (prodRule alert {+ severity = "warning";+ condition = "$B > ${warning} && $B <= ${critical}";+ })+ ]+ else+ [+ (prodRule alert {+ severity = "critical";+ condition = "$B < ${critical}";+ })+ (prodRule alert {+ severity = "warning";+ condition = "$B < ${warning} && $B >= ${critical}";+ })+ ];+ prodRules = lib.concatMap (row: lib.concatMap prodRulePair row.alerts) prodAlerts.rows;dss = [(lokiQuery ''sum(count_over_time({service_name="cloud-storage-service",deployment_environment="dev"} |~ "(?i)\\berror\\w*" [5m]))'')];diff --git a/infra/stacks/observability/nixos/goalert.nix b/infra/stacks/observability/nixos/goalert.nixindex 480fb341af..fea82f3a14 100644--- a/infra/stacks/observability/nixos/goalert.nix+++ b/infra/stacks/observability/nixos/goalert.nix@@ -2,8 +2,7 @@## Everything here runs in Ohio and talks only to loopback, so paging keeps# working while production or its database is down. Pages reach the Macro-# Alerts channel through the relay; phone paging (Twilio) and Google login are-# not set up yet.+# Alerts channel through the relay; phone paging (Twilio) is not set up yet.{config,lib,@@ -23,8 +22,20 @@ let# start. `@name@` strings are replaced with the credential of that name.goalertConfig = {General.ApplicationName = "Macro On-Call";- # One `admin` login, created by apply-config.py from goalert_admin_password.+ # Google login, with Grafana's OAuth client. GoAlert cannot restrict OIDC+ # logins to a domain; the client's Internal audience keeps them to Workspace+ # accounts. Basic auth stays on for the break-glass `admin` login, created+ # by apply-config.py from goalert_admin_password.Auth.DisableBasic = false;+ OIDC = {+ Enable = true;+ NewUsers = true;+ OverrideName = "Google";+ IssuerURL = "https://accounts.google.com";+ ClientID = "@google_client_id@";+ ClientSecret = "@google_client_secret@";+ Scopes = "openid profile email";+ };Webhook = {Enable = true;AllowedURLs = [ relayUrl ];@@ -93,6 +104,8 @@ inLoadCredential = map credential ["goalert_encryption_key""goalert_admin_password"+ "google_client_id"+ "google_client_secret"];ExecStartPre = ["${pkgs.python3}/bin/python3 ${./goalert/apply-config.py} ${lib.getExe goalert} ${pkgs.writeText "goalert-config.json" (builtins.toJSON goalertConfig)} ${config.services.postgresql.package}/bin/psql"diff --git a/infra/stacks/observability/nixos/prod-alerts-dashboard.nix b/infra/stacks/observability/nixos/prod-alerts-dashboard.nixnew file mode 100644index 0000000000..314c9bf873--- /dev/null+++ b/infra/stacks/observability/nixos/prod-alerts-dashboard.nix@@ -0,0 +1,143 @@+{ lib, pkgs, ... }:+let+ prodAlerts = import ./prod-alerts.nix;+ datasource = {+ type = "cloudwatch";+ uid = "cloudwatch";+ };++ # Threshold steps colour a value from the step it reaches upwards.+ thresholdSteps =+ alert:+ if alert.firesWhen == "above" then+ [+ {+ color = "green";+ value = null;+ }+ {+ color = "orange";+ value = alert.warning;+ }+ {+ color = "red";+ value = alert.critical;+ }+ ]+ else+ [+ {+ color = "red";+ value = null;+ }+ {+ color = "orange";+ value = alert.critical;+ }+ {+ color = "green";+ value = alert.warning;+ }+ ];++ panel = alert: gridPos: {+ id = alert.panelId;+ inherit (alert) title;+ inherit datasource gridPos;+ type = "timeseries";+ description = "Alerts on the ${alert.reducer} over ${toString (alert.window / 60)} minutes: warning (orange) and critical (red). Rules ${alert.uid}-warn and ${alert.uid}-crit.";+ targets = [ (prodAlerts.cloudwatchQuery alert.metric // { inherit datasource; }) ];+ fieldConfig.defaults = {+ inherit (alert) unit;+ thresholds = {+ mode = "absolute";+ steps = thresholdSteps alert;+ };+ custom.thresholdsStyle.mode = "dashed";+ };+ options.legend = {+ displayMode = "list";+ placement = "bottom";+ showLegend = true;+ };+ };++ # A row header, then two panels per line.+ rowPanels =+ y: row:+ [+ {+ inherit (row) id title;+ type = "row";+ collapsed = false;+ panels = [ ];+ gridPos = {+ x = 0;+ inherit y;+ w = 24;+ h = 1;+ };+ }+ ]+ ++ lib.imap0 (+ index: alert:+ panel alert {+ x = lib.mod index 2 * 12;+ y = y + 1 + builtins.div index 2 * 8;+ w = 12;+ h = 8;+ }+ ) row.alerts;+ rowHeight = row: 1 + (builtins.length row.alerts + 1) / 2 * 8;+ panels =+ (lib.foldl'+ (layout: row: {+ y = layout.y + rowHeight row;+ panels = layout.panels ++ rowPanels layout.y row;+ })+ {+ y = 0;+ panels = [ ];+ }+ prodAlerts.rows+ ).panels;++ dashboard = {+ uid = prodAlerts.dashboardUid;+ title = "Prod alerts";+ schemaVersion = 39;+ version = 1;+ editable = false;+ tags = [+ "macro"+ "prod"+ "alerts"+ ];+ timezone = "browser";+ refresh = "1m";+ time = {+ from = "now-6h";+ to = "now";+ };+ description = "One panel per prod alert rule in nixos/alerting.nix, with its warning and critical thresholds. Alert notifications link to the panel that fired. Reads CloudWatch in us-east-1.";+ inherit panels;+ };+in+{+ services.grafana.provision.dashboards.settings.providers = [+ {+ name = "macro-prod-alerts";+ orgId = 1;+ folder = "Macro";+ type = "file";+ disableDeletion = true;+ allowUiUpdates = false;+ options.path = pkgs.linkFarm "macro-prod-alerts-dashboard" [+ {+ name = "prod-alerts.json";+ path = pkgs.writeText "prod-alerts.json" (builtins.toJSON dashboard);+ }+ ];+ }+ ];+}diff --git a/infra/stacks/observability/nixos/prod-alerts.nix b/infra/stacks/observability/nixos/prod-alerts.nixnew file mode 100644index 0000000000..68810ffe24--- /dev/null+++ b/infra/stacks/observability/nixos/prod-alerts.nix@@ -0,0 +1,320 @@+# Prod alert metrics, shared by the rules in alerting.nix and the panels in+# prod-alerts-dashboard.nix so a rule and its graph cannot drift apart.+#+# Each entry becomes a critical and a warning rule (`<uid>-crit`, `<uid>-warn`)+# and one dashboard panel (`panelId`). Each has a matching Datadog monitor in+# infra/stacks/monitoring/monitors/{macrodb,redis,queues}.ts; keep thresholds+# and windows in sync by hand. Grafana pages for the entries marked `pages`, so+# their Datadog copies only notify.+#+# `firesWhen` is "above" or "below": critical fires past `critical`, warning+# fires between `warning` and `critical`. `window` is in seconds; the rule+# reduces it with `reducer` before comparing.+let+ # The CloudWatch query a rule evaluates and its panel draws.+ cloudwatchQuery =+ {+ namespace,+ metricName,+ dimensions,+ statistic,+ }:+ {+ refId = "A";+ id = "a";+ region = "us-east-1";+ inherit+ namespace+ metricName+ dimensions+ statistic+ ;+ period = "60";+ matchExact = true;+ metricQueryType = 0;+ metricEditorMode = 0;+ queryMode = "Metrics";+ };++ rds = metricName: instances: statistic: {+ namespace = "AWS/RDS";+ inherit metricName statistic;+ dimensions.DBInstanceIdentifier = instances;+ };+ bothInstances = [+ "macro-db-prod"+ "macro-db-prod-read-replica"+ ];+ redisNodes = [+ "macro-cache-prod-001"+ "macro-cache-prod-002"+ "email-service-redis-prod-c862c10"+ "calendar-service-redis-prod-aee80d2"+ "connection-gateway-redis-prod-f9d8cd0"+ ];+ redis = metricName: statistic: {+ namespace = "AWS/ElastiCache";+ inherit metricName statistic;+ dimensions.CacheClusterId = redisNodes;+ };+ queueAge = queues: {+ namespace = "AWS/SQS";+ metricName = "ApproximateAgeOfOldestMessage";+ statistic = "Maximum";+ dimensions.QueueName = queues;+ };+ workQueues = [+ "email-service-refresh-queue-prod"+ "convert-service-queue-prod"+ "ai-projection-queue-prod"+ "email-service-gmail-webhook-queue-prod"+ "notification-ingress-queue-prod"+ ];+in+{+ inherit cloudwatchQuery;+ dashboardUid = "prod-alerts";++ # Dashboard rows, in order. Panel ids are stable: notification links use them.+ rows = [+ {+ id = 101;+ title = "macrodb";+ alerts = [+ {+ panelId = 1;+ uid = "macrodb-primary-freeable-memory";+ title = "macro-db-prod freeable memory low";+ service = "macro-db-prod";+ pages = true;+ metric = rds "FreeableMemory" [ "macro-db-prod" ] "Average";+ unit = "decbytes";+ window = 900;+ reducer = "mean";+ firesWhen = "below";+ warning = 2000000000;+ critical = 1000000000;+ description = "Freeable memory is {{ $values.B }} bytes. Normal is 7-18 GB; it fell to about 0.45 GB during the October overload. Each connection holds about 14 MB of private memory, so check connection count and swap first.";+ runbook = "macrodb.md#memory";+ }+ {+ panelId = 2;+ uid = "macrodb-replica-freeable-memory";+ title = "macro-db-prod-read-replica freeable memory low";+ service = "macro-db-prod";+ pages = true;+ metric = rds "FreeableMemory" [ "macro-db-prod-read-replica" ] "Average";+ unit = "decbytes";+ window = 900;+ reducer = "mean";+ firesWhen = "below";+ warning = 1500000000;+ critical = 750000000;+ description = "Replica freeable memory is {{ $values.B }} bytes. Normal minimum is about 3 GB. The replica serves GraphQL reads, so memory pressure shows up as slow lists.";+ runbook = "macrodb.md#memory";+ }+ {+ panelId = 3;+ uid = "macrodb-swap";+ title = "macro-db-prod swapping";+ service = "macro-db-prod";+ pages = true;+ metric = rds "SwapUsage" bothInstances "Maximum";+ unit = "decbytes";+ window = 900;+ reducer = "mean";+ firesWhen = "above";+ warning = 2000000000;+ critical = 5000000000;+ description = "{{ $labels.DBInstanceIdentifier }} is using {{ $values.B }} bytes of swap. Normal is under 0.6 GB. Swap above 5 GB preceded both October overloads by hours.";+ runbook = "macrodb.md#memory";+ }+ {+ panelId = 4;+ uid = "macrodb-load";+ title = "macro-db-prod database load above vCPU count";+ service = "macro-db-prod";+ pages = true;+ metric = rds "DBLoadRelativeToNumVCPUs" bothInstances "Average";+ unit = "short";+ window = 900;+ reducer = "mean";+ firesWhen = "above";+ warning = 2;+ critical = 4;+ description = "{{ $labels.DBInstanceIdentifier }} has {{ $values.B }}x as many active sessions as vCPUs. Normal is under 1.3; the October overload reached 8. Look at Performance Insights top waits (locks, IO) and the services holding the most connections.";+ runbook = "macrodb.md#load";+ }+ {+ panelId = 5;+ uid = "macrodb-cpu";+ title = "macro-db-prod CPU high";+ service = "macro-db-prod";+ pages = true;+ metric = rds "CPUUtilization" bothInstances "Average";+ unit = "percent";+ window = 900;+ reducer = "mean";+ firesWhen = "above";+ warning = 70;+ critical = 85;+ description = "{{ $labels.DBInstanceIdentifier }} CPU averaged {{ $values.B }}% over 15 minutes. Normal peaks are about 30%.";+ runbook = "macrodb.md#load";+ }+ {+ panelId = 6;+ uid = "macrodb-primary-connections";+ title = "macro-db-prod connections high";+ service = "macro-db-prod";+ pages = true;+ metric = rds "DatabaseConnections" [ "macro-db-prod" ] "Maximum";+ unit = "short";+ window = 600;+ reducer = "mean";+ firesWhen = "above";+ warning = 1700;+ critical = 2200;+ description = "macro-db-prod has {{ $values.B }} connections. Normal 15-minute peaks are about 1,100; max_connections is 3,439. Each connection costs about 14 MB, so this also drains freeable memory.";+ runbook = "macrodb.md#connections";+ }+ {+ panelId = 7;+ uid = "macrodb-replica-connections";+ title = "macro-db-prod-read-replica connections high";+ service = "macro-db-prod";+ metric = rds "DatabaseConnections" [ "macro-db-prod-read-replica" ] "Maximum";+ unit = "short";+ window = 600;+ reducer = "mean";+ firesWhen = "above";+ warning = 550;+ critical = 700;+ description = "The read replica has {{ $values.B }} connections. Normal peaks are about 460.";+ runbook = "macrodb.md#connections";+ }+ {+ panelId = 8;+ uid = "macrodb-replica-lag";+ title = "macro-db-prod-read-replica lagging";+ service = "macro-db-prod";+ pages = true;+ metric = rds "ReplicaLag" [ "macro-db-prod-read-replica" ] "Maximum";+ unit = "s";+ window = 600;+ reducer = "mean";+ firesWhen = "above";+ warning = 60;+ critical = 300;+ description = "The read replica is {{ $values.B }} seconds behind the primary. Normal is under 6 seconds. GraphQL reads use the replica, so users see stale data and missing new items.";+ runbook = "macrodb.md#replica-lag";+ }+ {+ panelId = 9;+ uid = "macrodb-read-latency";+ title = "macro-db-prod disk read latency high";+ service = "macro-db-prod";+ metric = rds "ReadLatency" bothInstances "Average";+ unit = "s";+ window = 900;+ reducer = "mean";+ firesWhen = "above";+ warning = 0.005;+ critical = 0.01;+ description = "{{ $labels.DBInstanceIdentifier }} disk reads take {{ $values.B }} seconds on average. Normal is about 0.6 ms.";+ runbook = "macrodb.md#load";+ }+ {+ panelId = 10;+ uid = "macrodb-free-storage";+ title = "macro-db-prod free storage low";+ service = "macro-db-prod";+ pages = true;+ metric = rds "FreeStorageSpace" bothInstances "Minimum";+ unit = "decbytes";+ window = 3600;+ reducer = "min";+ firesWhen = "below";+ warning = 1500000000000;+ critical = 1000000000000;+ description = "{{ $labels.DBInstanceIdentifier }} has {{ $values.B }} bytes of free storage out of 5.37 TB.";+ runbook = "macrodb.md#storage";+ }+ ];+ }+ {+ id = 102;+ title = "Redis";+ alerts = [+ {+ panelId = 11;+ uid = "redis-memory";+ title = "Redis memory high";+ service = "elasticache";+ pages = true;+ metric = redis "DatabaseMemoryUsagePercentage" "Maximum";+ unit = "percent";+ window = 600;+ reducer = "mean";+ firesWhen = "above";+ warning = 75;+ critical = 90;+ description = "{{ $labels.CacheClusterId }} is using {{ $values.B }}% of maxmemory. Keys without a TTL are never evicted under volatile-lru, so at 100% writes fail.";+ runbook = "redis.md#memory";+ }+ {+ panelId = 12;+ uid = "redis-evictions";+ title = "Redis evicting keys";+ service = "elasticache";+ metric = redis "Evictions" "Sum";+ unit = "short";+ window = 600;+ # Thresholds apply to the ten-minute sum; the panel shows per-minute sums.+ reducer = "sum";+ firesWhen = "above";+ warning = 0;+ critical = 1000;+ description = "{{ $labels.CacheClusterId }} evicted {{ $values.B }} keys in 10 minutes. Prod caches normally evict nothing.";+ runbook = "redis.md#memory";+ }+ ];+ }+ {+ id = 103;+ title = "Queues";+ alerts = [+ {+ panelId = 13;+ uid = "sqs-email-backfill-age";+ title = "Email backfill queue falling behind";+ service = "email-service";+ metric = queueAge [ "email-service-backfill-queue-prod" ];+ unit = "s";+ window = 900;+ reducer = "mean";+ firesWhen = "above";+ warning = 3600;+ critical = 10800;+ description = "The oldest message in email-service-backfill-queue-prod is {{ $values.B }} seconds old. Normal peaks are about 50 minutes; new users wait this long for their mailbox to fill.";+ runbook = "queues.md#backlog";+ }+ {+ panelId = 14;+ uid = "sqs-work-queue-age";+ title = "Prod work queue falling behind";+ service = "sqs";+ pages = true;+ metric = queueAge workQueues;+ unit = "s";+ window = 900;+ reducer = "mean";+ firesWhen = "above";+ warning = 1800;+ critical = 7200;+ description = "The oldest message in {{ $labels.QueueName }} is {{ $values.B }} seconds old. These queues normally drain within minutes.";+ runbook = "queues.md#backlog";+ }+ ];+ }+ ];+}diff --git a/infra/stacks/observability/nixos/services.nix b/infra/stacks/observability/nixos/services.nixindex e7b15641d9..c10cf01397 100644--- a/infra/stacks/observability/nixos/services.nix+++ b/infra/stacks/observability/nixos/services.nix@@ -19,6 +19,7 @@ in./ci-dashboard.nix./database-dashboard.nix./database-queries-dashboard.nix+ ./prod-alerts-dashboard.nix./alerting.nix./goalert.nix];