Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(observability): Google login for GoAlert and a graph for every prod alert (#9089)

Wolf Mermelstein410f56ce1203
diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.md
index 39999c4aa3..f89dd81d55 100644
--- a/infra/stacks/monitoring/AGENTS.md
+++ b/infra/stacks/monitoring/AGENTS.md
@@ -5,7 +5,8 @@
Org-global Datadog monitors, synthetics, and rollout dashboards. This stack is
manual and has only a prod stack, even for monitors covering both environments.
Prod threshold and anomaly monitors are in `monitors/{macrodb,redis,queues,regressions}.ts`
-(`prodMonitor()` in `prod-monitor.ts`); every monitor notifies through `notify.ts`.
+(`prodMonitor()` in `prod-monitor.ts`), each file with a dashboard graphing its
+monitors that their messages link to; every monitor notifies through `notify.ts`.
## Checks
@@ -23,12 +24,14 @@ From `infra/`:
- The CloudWatch-based monitors in `macrodb.ts`, `redis.ts` and `queues.ts`
have hand-written Grafana copies in
- `infra/stacks/observability/nixos/alerting.nix`. Change thresholds and windows
+ `infra/stacks/observability/nixos/prod-alerts.nix`. Change thresholds and windows
in both. Grafana pages for those, so their Datadog copies use `pages: false`.
- Read `README.md` before deployment: imported resources must be adopted first
to avoid duplicate alerts. Never deploy without explicit approval.
- Deploying needs the `goalertIntegrationKey` secret in stack config, and
GoAlert (infra/stacks/observability) reachable at oncall.macro-internal.com.
+- A prod monitor's graph on its file's dashboard repeats its metric and
+ thresholds by hand; change both. `routing.test.ts` matches them by title.
- Never mention a Slack handle or email address in a monitor message; end it
with `notify({ pages })`. `routing.test.ts` enforces it.
- Notification retention adds nine monitors and one dashboard. Deploy DSS health
diff --git a/infra/stacks/monitoring/Pulumi.prod.yaml b/infra/stacks/monitoring/Pulumi.prod.yaml
index 138f62896b..c565ca2d7f 100644
--- a/infra/stacks/monitoring/Pulumi.prod.yaml
+++ b/infra/stacks/monitoring/Pulumi.prod.yaml
@@ -1,4 +1,6 @@
# Datadog monitors are org-global, so this project uses only the prod stack.
# The provider reads credentials and the Datadog site from DD_API_KEY,
# DD_APP_KEY, and DD_HOST, supplied locally or by the deployment workflow.
-config: {}
+config:
+ monitoring:goalertIntegrationKey:
+ secure: AAABAKEDvnVx7IjOr5rgXFTFIDwyqUy8gmBPo/bVOSKuV6DsgerGZ6IddCTt5xiG4VtYCz39X2aI4B5bUVXb+WSz0cU=
diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.md
index 2a233b8558..caef04f3ab 100644
--- a/infra/stacks/monitoring/README.md
+++ b/infra/stacks/monitoring/README.md
@@ -6,7 +6,7 @@ Datadog monitors and synthetic tests as code.
Datadog. This stack **adopts** them — it does not recreate them. `import.json`
lists every one by id. Everything else is declared here first and is created
on deploy: the prod threshold and anomaly monitors (`monitors/macrodb.ts`,
-`redis.ts`, `queues.ts`, `regressions.ts`), the rollout monitors, the GoAlert webhooks and health check, and one log-based metric.
+`redis.ts`, `queues.ts`, `regressions.ts`) and their dashboards, the rollout monitors, the GoAlert webhooks and health check, and one log-based metric.
## Routing
@@ -40,12 +40,19 @@ its thresholds many times a day.
`monitors/macrodb.ts`, `redis.ts`, `queues.ts` and `regressions.ts` declare
the prod threshold, forecast and anomaly monitors with `prodMonitor()`. The ones
on CloudWatch metrics have hand-written copies in Grafana
-(`infra/stacks/observability/nixos/alerting.nix`), and Grafana pages on-call for
+(`infra/stacks/observability/nixos/prod-alerts.nix`), and Grafana pages on-call for
the critical ones through GoAlert. Their Datadog copies only notify, so one
incident pages once. Keep thresholds and windows in sync by hand. The error-logs
anomaly monitor reads `macro.logs.errors`, a log-based metric declared in
`regressions.ts`, because log monitors have no anomaly function.
+Each of those files also declares a dashboard (MacroDB, Redis, Queue and
+Regression alerts) with one graph per monitor, titled like the monitor, its
+critical and warning thresholds drawn as markers; anomaly graphs show the band
+instead. Every prod monitor's message links to its dashboard. Change a query or
+threshold in the graph too; `routing.test.ts` checks that each monitor has a
+graph with matching markers.
+
## Read this before deploying
**Set the GoAlert integration key and deploy GoAlert first.** The webhooks
@@ -88,10 +95,10 @@ The gate is that preview:
- **35 monitors + 1 synthetic test updated, in `message` and
`renotifyInterval` only** — the routing move. The two AI-editing monitors,
already deployed from this stack, update the same way.
-- **69 resources to create**: 64 monitors (25 prod threshold and anomaly, 19 GraphQL soup, 9
+- **73 resources to create**: 64 monitors (25 prod threshold and anomaly, 19 GraphQL soup, 9
notification retention, 10 WhatsApp session, 1 native call push), 2 webhooks,
- 1 log-based metric, the GoAlert synthetic test, and the notification
- retention dashboard.
+ 1 log-based metric, the GoAlert synthetic test, the four prod alert
+ dashboards, and the notification retention dashboard.
- **Any other update, a replace, or a delete** — the hand-written code in
`monitors/*.ts` drifted from the live monitor. Reconcile against
`imported.generated.ts` before deploying.
@@ -158,7 +165,7 @@ is applied. Reach for `restrictedRoles` if convention stops being enough.
## Adding a monitor
Add a prod threshold or anomaly monitor with `prodMonitor()` in the matching
-`monitors/*.ts`, and its Grafana copy in `alerting.nix` when the metric is in
+`monitors/*.ts`, a `thresholdGraph()` for it on that file's dashboard, and its Grafana copy in `prod-alerts.nix` when the metric is in
CloudWatch. Declare anything else with `new datadog.Monitor(...)`, ending its
message with `notify({ pages })` and setting `renotifyInterval: RENOTIFY_MINUTES`. Use the `adopted()` helper only
for monitors that already exist in Datadog — it sets `protect`, which is there
diff --git a/infra/stacks/monitoring/monitors/macrodb.ts b/infra/stacks/monitoring/monitors/macrodb.ts
index b191b943c1..bcc9020726 100644
--- a/infra/stacks/monitoring/monitors/macrodb.ts
+++ b/infra/stacks/monitoring/monitors/macrodb.ts
@@ -2,9 +2,120 @@
//
// Thresholds come from two weeks of 5-minute data before the October overload.
// Monitors on CloudWatch metrics also exist in Grafana
-// (infra/stacks/observability/nixos/alerting.nix); change both together.
+// (infra/stacks/observability/nixos/prod-alerts.nix); change both together.
// Grafana pages on-call for those through GoAlert, so they only notify here.
-import { prodMonitor } from '../prod-monitor';
+import * as datadog from '@pulumi/datadog';
+import { graphGroup, prodMonitor, thresholdGraph } from '../prod-monitor';
+
+// One graph per monitor below, titled like it, with its thresholds.
+const dashboard = new datadog.Dashboard('macrodb-alerts', {
+ title: 'MacroDB alerts',
+ description:
+ 'Every prod MacroDB monitor against its thresholds. Declared in infra/stacks/monitoring/monitors/macrodb.ts; see docs/runbooks/macrodb.md.',
+ layoutType: 'ordered',
+ widgets: [
+ graphGroup('Memory', [
+ thresholdGraph({
+ title: 'macro-db-prod freeable memory low',
+ query:
+ 'avg:aws.rds.freeable_memory{dbinstanceidentifier:macro-db-prod}',
+ critical: '1000000000',
+ warning: '2000000000',
+ }),
+ thresholdGraph({
+ title: 'macro-db-prod-read-replica freeable memory low',
+ query:
+ 'avg:aws.rds.freeable_memory{dbinstanceidentifier:macro-db-prod-read-replica}',
+ critical: '750000000',
+ warning: '1500000000',
+ }),
+ thresholdGraph({
+ title: 'macro-db-prod swapping',
+ query:
+ 'max:aws.rds.swap_usage{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}',
+ critical: '5000000000',
+ warning: '2000000000',
+ }),
+ ]),
+ graphGroup('Load', [
+ thresholdGraph({
+ title: 'macro-db-prod database load above vCPU count',
+ query:
+ 'avg:aws.rds.dbload_relative_to_num_vcpus{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}',
+ critical: '4',
+ warning: '2',
+ }),
+ thresholdGraph({
+ title: 'macro-db-prod CPU high',
+ query:
+ 'avg:aws.rds.cpuutilization{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}',
+ critical: '85',
+ warning: '70',
+ }),
+ thresholdGraph({
+ title: 'macro-db-prod disk read latency high',
+ query:
+ 'avg:aws.rds.read_latency{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}',
+ critical: '0.01',
+ warning: '0.005',
+ }),
+ ]),
+ graphGroup('Connections and replication', [
+ thresholdGraph({
+ title: 'macro-db-prod connections high',
+ query:
+ 'max:aws.rds.database_connections{dbinstanceidentifier:macro-db-prod}',
+ critical: '2200',
+ warning: '1700',
+ }),
+ thresholdGraph({
+ title: 'macro-db-prod-read-replica connections high',
+ query:
+ 'max:aws.rds.database_connections{dbinstanceidentifier:macro-db-prod-read-replica}',
+ critical: '700',
+ warning: '550',
+ }),
+ thresholdGraph({
+ title: 'macro-db-prod-read-replica lagging',
+ query:
+ 'max:aws.rds.replica_lag{dbinstanceidentifier:macro-db-prod-read-replica}',
+ critical: '300',
+ warning: '60',
+ }),
+ ]),
+ graphGroup('Locks', [
+ thresholdGraph({
+ title: 'macro-db-prod connections blocked on locks',
+ query:
+ 'max:postgresql.activity.blocked_connections{host:macro-db-prod*} by {host}',
+ critical: '50',
+ warning: '20',
+ }),
+ thresholdGraph({
+ title: 'macro-db-prod long-running transaction',
+ query:
+ 'max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host}',
+ critical: '1800',
+ warning: '600',
+ }),
+ ]),
+ graphGroup('Storage', [
+ thresholdGraph({
+ title: 'macro-db-prod free storage low',
+ query:
+ 'min:aws.rds.free_storage_space{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}',
+ critical: '1000000000000',
+ warning: '1500000000000',
+ }),
+ thresholdGraph({
+ title: 'macro-db-prod free storage forecast to run low within a week',
+ query:
+ "forecast(min:aws.rds.free_storage_space{dbinstanceidentifier:macro-db-prod*} by {dbinstanceidentifier}, 'linear', 1)",
+ critical: '500000000000',
+ }),
+ ]),
+ ],
+});
prodMonitor('macrodb-primary-freeable-memory', {
title: 'macro-db-prod freeable memory low',
@@ -15,6 +126,7 @@ prodMonitor('macrodb-primary-freeable-memory', {
pages: false,
description:
'Freeable memory is {{value}} bytes. Normal is 7-18 GB; it fell to about 0.45 GB during the October overload. Each connection holds about 14 MB of private memory, so check connection count and swap first.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#memory',
});
@@ -27,6 +139,7 @@ prodMonitor('macrodb-replica-freeable-memory', {
pages: false,
description:
'Replica freeable memory is {{value}} bytes. Normal minimum is about 3 GB. The replica serves GraphQL reads, so memory pressure shows up as slow lists.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#memory',
});
@@ -39,6 +152,7 @@ prodMonitor('macrodb-swap', {
pages: false,
description:
'{{dbinstanceidentifier.name}} is using {{value}} bytes of swap. Normal is under 0.6 GB. Swap above 5 GB preceded both October overloads by hours.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#memory',
});
@@ -51,6 +165,7 @@ prodMonitor('macrodb-load', {
pages: false,
description:
'{{dbinstanceidentifier.name}} has {{value}}x as many active sessions as vCPUs. Normal is under 1.3; the October overload reached 8. Look at Performance Insights top waits (locks, IO) and the services holding the most connections.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#load',
});
@@ -63,6 +178,7 @@ prodMonitor('macrodb-cpu', {
pages: false,
description:
'{{dbinstanceidentifier.name}} CPU averaged {{value}}% over 15 minutes. Normal peaks are about 30%.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#load',
});
@@ -75,6 +191,7 @@ prodMonitor('macrodb-primary-connections', {
pages: false,
description:
'macro-db-prod has {{value}} connections. Normal 15-minute peaks are about 1,100; max_connections is 3,439. Each connection costs about 14 MB, so this also drains freeable memory.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#connections',
});
@@ -87,6 +204,7 @@ prodMonitor('macrodb-replica-connections', {
pages: false,
description:
'The read replica has {{value}} connections. Normal peaks are about 460.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#connections',
});
@@ -99,6 +217,7 @@ prodMonitor('macrodb-replica-lag', {
pages: false,
description:
'The read replica is {{value}} seconds behind the primary. Normal is under 6 seconds. GraphQL reads use the replica, so users see stale data and missing new items.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#replica-lag',
});
@@ -111,6 +230,7 @@ prodMonitor('macrodb-read-latency', {
pages: false,
description:
'{{dbinstanceidentifier.name}} disk reads take {{value}} seconds on average. Normal is about 0.6 ms.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#load',
});
@@ -123,6 +243,7 @@ prodMonitor('macrodb-free-storage', {
pages: false,
description:
'{{dbinstanceidentifier.name}} has {{value}} bytes of free storage out of 5.37 TB.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#storage',
});
@@ -136,6 +257,7 @@ prodMonitor('macrodb-free-storage-forecast', {
pages: false,
description:
'At the current write rate {{dbinstanceidentifier.name}} drops below 500 GB free within a week. Find what started writing; recent growth was 100-200 GB a day.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#storage',
});
@@ -150,6 +272,7 @@ prodMonitor('macrodb-blocked-connections', {
pages: true,
description:
"{{host.name}} has {{value}} connections waiting on locks. Normal is under 7. Advisory and tuple locks dominated the October overload; see Database Monitoring's blocking queries view.",
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#locks',
});
@@ -162,5 +285,6 @@ prodMonitor('macrodb-long-transactions', {
pages: false,
description:
'The oldest open transaction on {{host.name}} is {{value}} seconds old. Long transactions hold locks and stop vacuum.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#locks',
});
diff --git a/infra/stacks/monitoring/monitors/queues.ts b/infra/stacks/monitoring/monitors/queues.ts
index 393de6b269..3d00a7181d 100644
--- a/infra/stacks/monitoring/monitors/queues.ts
+++ b/infra/stacks/monitoring/monitors/queues.ts
@@ -1,9 +1,51 @@
// Prod SQS queues: dead letters, backlog age and backlog growth.
//
// The two queue-age monitors also exist in Grafana
-// (infra/stacks/observability/nixos/alerting.nix); change both together.
+// (infra/stacks/observability/nixos/prod-alerts.nix); change both together.
// Grafana pages on-call for the work queues, so they only notify here.
-import { prodMonitor } from '../prod-monitor';
+import * as datadog from '@pulumi/datadog';
+import { graphGroup, prodMonitor, thresholdGraph } from '../prod-monitor';
+
+// One graph per monitor below, titled like it, with its thresholds.
+const dashboard = new datadog.Dashboard('queue-alerts', {
+ title: 'Queue alerts',
+ description:
+ 'Every prod SQS monitor against its thresholds. Declared in infra/stacks/monitoring/monitors/queues.ts; see docs/runbooks/queues.md.',
+ layoutType: 'ordered',
+ widgets: [
+ graphGroup('Dead letters', [
+ thresholdGraph({
+ // Thirty-minute sums, as the monitor evaluates them.
+ title: 'Messages landing in a prod dead-letter queue',
+ query:
+ 'sum:aws.sqs.number_of_messages_sent{queuename:*-dlq-prod} by {queuename}.as_count().rollup(sum, 1800)',
+ critical: '50',
+ warning: '0',
+ }),
+ ]),
+ graphGroup('Backlog', [
+ thresholdGraph({
+ title: 'Email backfill queue falling behind',
+ query:
+ 'max:aws.sqs.approximate_age_of_oldest_message{queuename:email-service-backfill-queue-prod}',
+ critical: '10800',
+ warning: '3600',
+ }),
+ thresholdGraph({
+ title: 'Prod work queue falling behind',
+ query:
+ 'max:aws.sqs.approximate_age_of_oldest_message{queuename:email-service-refresh-queue-prod OR queuename:convert-service-queue-prod OR queuename:ai-projection-queue-prod OR queuename:email-service-gmail-webhook-queue-prod OR queuename:notification-ingress-queue-prod} by {queuename}',
+ critical: '7200',
+ warning: '1800',
+ }),
+ thresholdGraph({
+ title: 'Email backfill backlog growing abnormally',
+ query:
+ "anomalies(max:aws.sqs.approximate_number_of_messages_visible{queuename:email-service-backfill-queue-prod}, 'agile', 3)",
+ }),
+ ]),
+ ],
+});
// Datadog only: CloudWatch dimensions can't be filtered by name pattern.
prodMonitor('sqs-dlq-inflow', {
@@ -15,6 +57,7 @@ prodMonitor('sqs-dlq-inflow', {
pages: false,
description:
'{{value}} messages moved to {{queuename.name}} in 30 minutes. Most DLQs hold old messages permanently, so this alerts on new arrivals only.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/queues.md#dead-letters',
});
@@ -27,6 +70,7 @@ prodMonitor('sqs-email-backfill-age', {
pages: false,
description:
'The oldest message in email-service-backfill-queue-prod is {{value}} seconds old. Normal peaks are about 50 minutes; new users wait this long for their mailbox to fill.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/queues.md#backlog',
});
@@ -39,6 +83,7 @@ prodMonitor('sqs-work-queue-age', {
pages: false,
description:
'The oldest message in {{queuename.name}} is {{value}} seconds old. These queues normally drain within minutes.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/queues.md#backlog',
});
@@ -57,5 +102,6 @@ prodMonitor('sqs-email-backfill-backlog-anomaly', {
pages: false,
description:
'Visible messages in email-service-backfill-queue-prod are above their usual band. A jump in enqueues loads macrodb through the backfill workers.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/queues.md#backlog',
});
diff --git a/infra/stacks/monitoring/monitors/redis.ts b/infra/stacks/monitoring/monitors/redis.ts
index 5165ac7afe..7fde9b118d 100644
--- a/infra/stacks/monitoring/monitors/redis.ts
+++ b/infra/stacks/monitoring/monitors/redis.ts
@@ -2,9 +2,45 @@
//
// macro-cache-prod climbed from 60% to 100% over a month before writes failed.
// Memory and evictions also exist in Grafana
-// (infra/stacks/observability/nixos/alerting.nix); change both together.
+// (infra/stacks/observability/nixos/prod-alerts.nix); change both together.
// Grafana pages on-call for memory, so it only notifies here.
-import { prodMonitor } from '../prod-monitor';
+import * as datadog from '@pulumi/datadog';
+import { graphGroup, prodMonitor, thresholdGraph } from '../prod-monitor';
+
+// One graph per monitor below, titled like it, with its thresholds.
+const dashboard = new datadog.Dashboard('redis-alerts', {
+ title: 'Redis alerts',
+ description:
+ 'Every prod Redis monitor against its thresholds. Declared in infra/stacks/monitoring/monitors/redis.ts; see docs/runbooks/redis.md.',
+ layoutType: 'ordered',
+ widgets: [
+ graphGroup('Memory', [
+ thresholdGraph({
+ title: 'Redis memory high',
+ query:
+ 'max:aws.elasticache.database_memory_usage_percentage{cacheclusterid:*prod*} by {cacheclusterid}',
+ critical: '90',
+ warning: '75',
+ }),
+ thresholdGraph({
+ title: 'Redis memory forecast to fill within three days',
+ query:
+ "forecast(max:aws.elasticache.database_memory_usage_percentage{cacheclusterid:*prod*} by {cacheclusterid}, 'linear', 1)",
+ critical: '95',
+ }),
+ ]),
+ graphGroup('Evictions', [
+ thresholdGraph({
+ // Ten-minute sums, as the monitor evaluates them.
+ title: 'Redis evicting keys',
+ query:
+ 'sum:aws.elasticache.evictions{cacheclusterid:*prod*} by {cacheclusterid}.as_count().rollup(sum, 600)',
+ critical: '1000',
+ warning: '0',
+ }),
+ ]),
+ ],
+});
prodMonitor('redis-memory', {
title: 'Redis memory high',
@@ -15,6 +51,7 @@ prodMonitor('redis-memory', {
pages: false,
description:
'{{cacheclusterid.name}} is using {{value}}% of maxmemory. Keys without a TTL are never evicted under volatile-lru, so at 100% writes fail.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/redis.md#memory',
});
@@ -28,6 +65,7 @@ prodMonitor('redis-memory-forecast', {
pages: false,
description:
'{{cacheclusterid.name}} memory is growing fast enough to pass 95% within three days. macro-cache-prod climbed from 60% to 100% over a month before writes failed.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/redis.md#memory',
});
@@ -40,5 +78,6 @@ prodMonitor('redis-evictions', {
pages: false,
description:
'{{cacheclusterid.name}} evicted {{value}} keys in 10 minutes. Prod caches normally evict nothing.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/redis.md#memory',
});
diff --git a/infra/stacks/monitoring/monitors/regressions.ts b/infra/stacks/monitoring/monitors/regressions.ts
index 86ff6c09e8..75d8579a7b 100644
--- a/infra/stacks/monitoring/monitors/regressions.ts
+++ b/infra/stacks/monitoring/monitors/regressions.ts
@@ -4,7 +4,7 @@
// Datadog only: prod traces and logs don't reach Grafana, and Grafana alerting
// has no anomaly detection.
import * as datadog from '@pulumi/datadog';
-import { prodMonitor } from '../prod-monitor';
+import { graphGroup, prodMonitor, thresholdGraph } from '../prod-monitor';
// Datadog has no anomaly function for log monitors, so the error-logs anomaly
// monitor reads this log-based metric instead. Counts every prod error log
@@ -16,6 +16,46 @@ new datadog.LogsMetric('macro-logs-errors', {
groupBies: [{ path: 'service', tagName: 'service' }],
});
+// One graph per monitor below, titled like it, with the anomaly band each
+// monitor alerts outside of.
+const dashboard = new datadog.Dashboard('regression-alerts', {
+ title: 'Regression alerts',
+ description:
+ 'Every prod error and latency anomaly monitor with its band. Declared in infra/stacks/monitoring/monitors/regressions.ts; see docs/runbooks/regressions.md.',
+ layoutType: 'ordered',
+ widgets: [
+ graphGroup('Errors', [
+ thresholdGraph({
+ title: 'HTTP server errors abnormally high',
+ query:
+ "anomalies(sum:trace.http.server.request.errors{env:prod} by {service}.as_count(), 'agile', 3)",
+ }),
+ thresholdGraph({
+ title: 'Internal span errors abnormally high',
+ query:
+ "anomalies(sum:trace.Internal.errors{env:prod} by {service}.as_count(), 'agile', 3)",
+ }),
+ thresholdGraph({
+ title: 'Queue consumer errors abnormally high',
+ query:
+ "anomalies(sum:trace.Consumer.errors{env:prod} by {service}.as_count(), 'agile', 3)",
+ }),
+ thresholdGraph({
+ title: 'Error logs abnormally high',
+ query:
+ "anomalies(sum:macro.logs.errors{*} by {service}.as_count(), 'agile', 3)",
+ }),
+ ]),
+ graphGroup('Latency', [
+ thresholdGraph({
+ title: 'HTTP latency abnormally high',
+ query:
+ "anomalies(p95:trace.http.server.request{env:prod AND (service:cloud-storage-service OR service:authentication-service)} by {service}, 'agile', 3)",
+ }),
+ ]),
+ ],
+});
+
// An anomaly monitor recovers once no point in its window is outside the band.
const ANOMALY_THRESHOLDS = { critical: '1', criticalRecovery: '0' };
const ANOMALY_WINDOWS = {
@@ -33,6 +73,7 @@ prodMonitor('apm-server-error-anomaly', {
pages: false,
description:
'{{service.name}} is returning more HTTP server errors than its usual band. Check the latest deploy of that service first.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/regressions.md#errors',
});
@@ -46,6 +87,7 @@ prodMonitor('apm-internal-error-anomaly', {
pages: false,
description:
'{{service.name}} is recording more failed internal spans (background jobs, workers, browser operations) than its usual band. Check the latest deploy of that service first.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/regressions.md#errors',
});
@@ -59,6 +101,7 @@ prodMonitor('apm-consumer-error-anomaly', {
pages: false,
description:
'{{service.name}} is failing more queue and Kafka messages than its usual band.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/regressions.md#errors',
});
@@ -71,6 +114,7 @@ prodMonitor('apm-latency-anomaly', {
monitorThresholdWindows: ANOMALY_WINDOWS,
pages: false,
description: '{{service.name}} p95 HTTP latency is above its usual band.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/regressions.md#latency',
});
@@ -84,5 +128,6 @@ prodMonitor('error-logs-anomaly', {
pages: false,
description:
'{{service.name}} is logging more errors than its usual band. Check the latest deploy of that service first.',
+ dashboard: dashboard.url,
runbook: 'docs/runbooks/regressions.md#errors',
});
diff --git a/infra/stacks/monitoring/notify.ts b/infra/stacks/monitoring/notify.ts
index 4a4380d61e..83234c16a3 100644
--- a/infra/stacks/monitoring/notify.ts
+++ b/infra/stacks/monitoring/notify.ts
@@ -1,3 +1,5 @@
+import * as pulumi from '@pulumi/pulumi';
+
// Where every monitor in this stack notifies.
//
// All monitors post to the Macro Alerts channel. Paging monitors also open a
@@ -24,6 +26,9 @@ export function notify({ pages }: { pages: boolean }): string {
${MACRO_ALERTS}`;
}
+// Datadog's site: dashboard `url`s are paths under it.
+export const DATADOG_SITE = 'https://us5.datadoghq.com';
+
/**
* A monitor message: the state and value against its threshold, what it
* means, where to start, and the notification handles.
@@ -31,17 +36,26 @@ ${MACRO_ALERTS}`;
export function alertMessage({
description,
runbook,
+ dashboard,
pages,
}: {
description: string;
runbook: string;
+ /** A dashboard's `url`, a path such as `/dashboard/abc-def-ghi/title`. */
+ dashboard?: pulumi.Input<string>;
pages: boolean;
-}): string {
- return `{{#is_alert}}🔴 CRITICAL: {{value}} (threshold {{threshold}}){{/is_alert}}{{#is_warning}}🟠 WARNING: {{value}} (warning threshold {{warn_threshold}}){{/is_warning}}{{#is_recovery}}✅ RECOVERED: {{value}}{{/is_recovery}}
+}): pulumi.Output<string> {
+ return pulumi.output(dashboard).apply((path) => {
+ const links = [
+ `Runbook: https://github.com/macro-inc/macro/blob/main/${runbook}`,
+ ];
+ if (path !== undefined) links.push(`Dashboard: ${DATADOG_SITE}${path}`);
+ return `{{#is_alert}}🔴 CRITICAL: {{value}} (threshold {{threshold}}){{/is_alert}}{{#is_warning}}🟠 WARNING: {{value}} (warning threshold {{warn_threshold}}){{/is_warning}}{{#is_recovery}}✅ RECOVERED: {{value}}{{/is_recovery}}
${description}
-Runbook: https://github.com/macro-inc/macro/blob/main/${runbook}
+${links.join('\n')}
${notify({ pages })}`;
+ });
}
diff --git a/infra/stacks/monitoring/prod-monitor.ts b/infra/stacks/monitoring/prod-monitor.ts
index eca574ef86..ec8fd8c415 100644
--- a/infra/stacks/monitoring/prod-monitor.ts
+++ b/infra/stacks/monitoring/prod-monitor.ts
@@ -1,10 +1,11 @@
import * as datadog from '@pulumi/datadog';
+import type * as pulumi from '@pulumi/pulumi';
import { goalertWebhooks } from './goalert';
import { alertMessage, RENOTIFY_MINUTES } from './notify';
/**
* A prod monitor declared in this stack. The rules with a CloudWatch metric
- * also exist in Grafana (infra/stacks/observability/nixos/alerting.nix); keep
+ * also exist in Grafana (infra/stacks/observability/nixos/prod-alerts.nix); keep
* their thresholds and windows in sync there by hand.
*/
export function prodMonitor(
@@ -21,6 +22,8 @@ export function prodMonitor(
description: string;
/** Repository path with a heading anchor, e.g. `docs/runbooks/macrodb.md#memory`. */
runbook: string;
+ /** `url` of the dashboard graphing this monitor, under a `thresholdGraph` titled `title`. */
+ dashboard?: pulumi.Input<string>;
}
): datadog.Monitor {
return new datadog.Monitor(
@@ -32,6 +35,7 @@ export function prodMonitor(
message: alertMessage({
description: args.description,
runbook: args.runbook,
+ dashboard: args.dashboard,
pages: args.pages,
}),
tags: ['env:prod', `service:${args.service}`],
@@ -47,3 +51,47 @@ export function prodMonitor(
{ dependsOn: args.pages ? goalertWebhooks : [] }
);
}
+
+/**
+ * A line graph of one prod monitor's metric, titled like the monitor, with
+ * its critical and warning thresholds drawn as markers. Anomaly graphs have no
+ * markers: the band is the threshold.
+ */
+export function thresholdGraph(args: {
+ title: string;
+ query: string;
+ critical?: string;
+ warning?: string;
+}): datadog.types.input.DashboardWidget {
+ const markers: datadog.types.input.DashboardWidgetTimeseriesDefinitionMarker[] =
+ [];
+ if (args.critical !== undefined) {
+ markers.push({
+ label: 'critical',
+ value: `y = ${args.critical}`,
+ displayType: 'error dashed',
+ });
+ }
+ if (args.warning !== undefined) {
+ markers.push({
+ label: 'warning',
+ value: `y = ${args.warning}`,
+ displayType: 'warning dashed',
+ });
+ }
+ return {
+ timeseriesDefinition: {
+ title: args.title,
+ requests: [{ q: args.query, displayType: 'line' }],
+ markers,
+ },
+ };
+}
+
+/** A titled group of related graphs on a dashboard. */
+export function graphGroup(
+ title: string,
+ widgets: datadog.types.input.DashboardWidget[]
+): datadog.types.input.DashboardWidget {
+ return { groupDefinition: { title, layoutType: 'ordered', widgets } };
+}
diff --git a/infra/stacks/monitoring/routing.test.ts b/infra/stacks/monitoring/routing.test.ts
index b140d1d5e0..3d1810571c 100644
--- a/infra/stacks/monitoring/routing.test.ts
+++ b/infra/stacks/monitoring/routing.test.ts
@@ -12,7 +12,12 @@ beforeAll(async () => {
{
newResource(args) {
resources.push(args);
- return { id: args.name, state: args.inputs };
+ // Datadog returns a dashboard's url as a path under the site.
+ const state =
+ args.type === 'datadog:index/dashboard:Dashboard'
+ ? { ...args.inputs, url: `/dashboard/${args.name}/title` }
+ : args.inputs;
+ return { id: args.name, state };
},
call(args) {
return args.inputs;
@@ -28,6 +33,8 @@ beforeAll(async () => {
const monitors = () =>
resources.filter((r) => r.type === 'datadog:index/monitor:Monitor');
+const dashboards = () =>
+ resources.filter((r) => r.type === 'datadog:index/dashboard:Dashboard');
const synthetics = () =>
resources.filter(
(r) => r.type === 'datadog:index/syntheticsTest:SyntheticsTest'
@@ -116,3 +123,46 @@ test('anomaly rules recover at zero and carry their alert window', () => {
recoveryWindow: 'last_15m',
});
});
+
+test('a prod monitor links to its dashboard', () => {
+ const cpu = monitors().find((r) => r.name === 'macrodb-cpu');
+ expect(cpu?.inputs.message).toContain(
+ 'Runbook: https://github.com/macro-inc/macro/blob/main/docs/runbooks/macrodb.md#load\nDashboard: https://us5.datadoghq.com/dashboard/macrodb-alerts/title'
+ );
+});
+
+test('every prod monitor is graphed on its dashboard with its thresholds', () => {
+ const prod = monitors().filter((r) =>
+ r.inputs.message.includes('Runbook: https://github.com/macro-inc/macro/')
+ );
+ expect(prod).toHaveLength(25);
+ for (const monitor of prod) {
+ const linked = monitor.inputs.message.match(
+ /Dashboard: https:\/\/us5\.datadoghq\.com\/dashboard\/([\w-]+)\//
+ )?.[1];
+ const dashboard = dashboards().find((d) => d.name === linked);
+ const title = monitor.inputs.name.replace('[PROD] ', '');
+ const graph = dashboard?.inputs.widgets
+ .flatMap(
+ (group: { groupDefinition: { widgets: unknown[] } }) =>
+ group.groupDefinition.widgets
+ )
+ .find(
+ (widget: { timeseriesDefinition: { title: string } }) =>
+ widget.timeseriesDefinition.title === title
+ );
+ const { critical, warning } = monitor.inputs.monitorThresholds;
+ const anomaly = monitor.inputs.query.includes('anomalies(');
+ expect({
+ name: monitor.name,
+ markers: graph?.timeseriesDefinition.markers.map(
+ (marker: { value: string }) => marker.value
+ ),
+ }).toEqual({
+ name: monitor.name,
+ markers: anomaly
+ ? []
+ : [`y = ${critical}`, ...(warning ? [`y = ${warning}`] : [])],
+ });
+ }
+});
diff --git a/infra/stacks/observability/AGENTS.md b/infra/stacks/observability/AGENTS.md
index d995967ba9..a0364f759c 100644
--- a/infra/stacks/observability/AGENTS.md
+++ b/infra/stacks/observability/AGENTS.md
@@ -14,8 +14,10 @@ us-east-1. `README.md` is the full operations guide.
## Recipes
-- Change a Grafana prod alert rule: edit `prodRules` in `nixos/alerting.nix`, and
- its Datadog copy in `infra/stacks/monitoring/monitors/` by hand.
+- Change a Grafana prod alert rule: edit its entry in `nixos/prod-alerts.nix`
+ (rules and the `Prod alerts` dashboard panel both read it), and its Datadog
+ copy in `infra/stacks/monitoring/monitors/` by hand. A new entry takes the next
+ free `panelId`; never renumber one, since sent notifications link to it.
- Add a runtime setting: `settings.ts` → `render.ts` (bump the schema version) →
`nixos/read-user-data.py` (keys, pattern, version) → the consuming `.nix` file.
- Add a secret: `assets/refresh-secrets.py` (validation) → `LoadCredential` in
diff --git a/infra/stacks/observability/Pulumi.prod.yaml b/infra/stacks/observability/Pulumi.prod.yaml
index e233257080..5de60a5f49 100644
--- a/infra/stacks/observability/Pulumi.prod.yaml
+++ b/infra/stacks/observability/Pulumi.prod.yaml
@@ -2,5 +2,5 @@ config:
aws:region: us-east-2
observability:ciIngestSecretArn: arn:aws:secretsmanager:us-east-1:569036502058:secret:observability/dev-ingest-7UhJbL
observability:alarmTopicArn: arn:aws:sns:us-east-2:569036502058:observability-prod-alerts
- observability:amiId: ami-0e775059ad164977c
+ observability:amiId: ami-0af944c17d63ec7c4
observability:secretArn: arn:aws:secretsmanager:us-east-2:569036502058:secret:observability/prod-TwQpBJ
diff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.md
index 43fd8ccde9..ff13d8f15d 100644
--- a/infra/stacks/observability/README.md
+++ b/infra/stacks/observability/README.md
@@ -77,6 +77,8 @@ packages, configuration, service users, resource limits and startup dependencies
- `nixos/goalert.nix` and `nixos/goalert/`: GoAlert, its Postgres, backups and
the Macro Alerts relay (see [On-call paging](#on-call-paging-goalert)).
- `nixos/alerting.nix`: Grafana alert rules, contact points and routing.
+- `nixos/prod-alerts.nix`: prod alert queries and thresholds, shared by the
+ rules and `nixos/prod-alerts-dashboard.nix`.
- `nixos/host.nix`: boot dependencies, retained storage and stable service identities.
NixOS modules generate application configuration from Nix attributes. Alloy's
@@ -435,20 +437,27 @@ models. Grafana uses the EC2 role for regional metric reads and publishing to th
configured alarm SNS topic. Other unprivileged HTTP/ingest services still cannot
access instance metadata.
-`prodRules` in `nixos/alerting.nix` holds the prod CloudWatch rules: macrodb
-memory, swap, load, CPU, connections, replica lag, read latency and storage;
-Redis memory and evictions; SQS queue age. Each is a warning and a critical rule
-in the `Prod` group, reading CloudWatch in Virginia directly. Each has a Datadog
-copy in `infra/stacks/monitoring/monitors/`; keep thresholds and windows in sync
-by hand. Forecasts, anomalies, Database Monitoring, APM and log rules are
-Datadog-only.
+`nixos/prod-alerts.nix` lists the prod CloudWatch alerts: macrodb memory, swap,
+load, CPU, connections, replica lag, read latency and storage; Redis memory and
+evictions; SQS queue age. Each entry holds its query, thresholds, window and
+text once. `nixos/alerting.nix` turns each into a warning and a critical rule in
+the `Prod` group, reading CloudWatch in Virginia directly, and
+`nixos/prod-alerts-dashboard.nix` draws it as one panel with both thresholds on
+the `Prod alerts` dashboard (`/d/prod-alerts`, rows macrodb, Redis and Queues).
+Each rule carries its panel as `__dashboardUid__`/`__panelId__`, so the rule
+view and every notification link to the graph of the metric that fired. Each
+has a Datadog copy in `infra/stacks/monitoring/monitors/`; keep thresholds and
+windows in sync by hand. Forecasts, anomalies, Database Monitoring, APM and log
+rules are Datadog-only.
Every alert posts to the `Macro Alerts` contact point through the loopback
-relay; critical ones renotify hourly, others every four hours. Critical alerts
-of rules with `pages = true` also go to the `GoAlert` contact point over
-loopback. GoAlert owns the alert from there: it opens, acknowledges and closes
-it, and its escalation posts each change to the channel. Their Datadog copies
-only notify, so an incident pages once.
+relay; critical ones renotify hourly, others every four hours. Grafana's
+`default.message` lists each alert's annotations (summary, description,
+runbook) and a `Panel:` link to its graph, opened an hour before the alert
+started. Critical alerts of rules with `pages = true` also go to the `GoAlert`
+contact point over loopback. GoAlert owns the alert from there: it opens,
+acknowledges and closes it, and its escalation posts each change to the
+channel. Their Datadog copies only notify, so an incident pages once.
The dev rules below keep their own route to `Macro team email`.
@@ -479,10 +488,10 @@ AMI; a schema-5 image cannot boot schema-6 user data.
GoAlert is the source of truth for open alerts: Grafana sends it critical
alerts, and on-call engineers acknowledge or close them in its UI at
-`https://oncall.macro-internal.com`. It runs on this host so paging keeps working
-while production, its database or Macro itself is down. Today it pages by
-posting to the Macro Alerts channel; phone paging (Twilio), Google login and a
-rotation are follow-ups.
+`https://oncall.macro-internal.com`, signing in with Google. It runs on this host
+so paging keeps working while production, its database or Macro itself is down.
+Today it pages by posting to the Macro Alerts channel; phone paging (Twilio) and
+a rotation are follow-ups.
- `nixos/goalert/package.nix` builds GoAlert from source with `buildGoModule`,
embedding its web UI. Bump `version` and the hashes together.
@@ -504,6 +513,10 @@ rotation are follow-ups.
`/health`, `/api/v2/generic/incoming` and `/api/v2/heartbeat/` are reachable
without a session (GoAlert checks the integration key); other inbound
integrations return 404. The rest is GoAlert's UI behind its login.
+- Login is Google OIDC with Grafana's OAuth client, whose redirect URIs include
+ `https://oncall.macro-internal.com/api/v2/identity/providers/oidc/callback`.
+ GoAlert cannot restrict OIDC to a domain, so that client must keep the
+ Internal audience. The basic-auth `admin` login is the break-glass account.
- The ALB sends the on-call host to its own target group, health-checked at
`/goalert-healthz`. Alarm `observability-goalert` publishes to `alarmTopicArn`
when it fails, so a GoAlert outage is reported without GoAlert.
diff --git a/infra/stacks/observability/nixos/alerting.nix b/infra/stacks/observability/nixos/alerting.nix
index e761fa4a35..4babf369ae 100644
--- a/infra/stacks/observability/nixos/alerting.nix
+++ b/infra/stacks/observability/nixos/alerting.nix
@@ -66,61 +66,18 @@ let
}
];
- # Prod rules. Each has a matching Datadog monitor in
- # infra/stacks/monitoring/monitors/{macrodb,redis,queues}.ts; keep thresholds
- # and windows in sync by hand. Grafana pages for the rules marked `pages`, so
- # their Datadog copies only notify.
- cloudwatch =
- {
- namespace,
- metricName,
- dimensions,
- statistic,
- }:
- window: [
- {
- refId = "A";
- datasourceUid = "cloudwatch";
- relativeTimeRange = {
- from = window;
- to = 0;
- };
- model = {
- refId = "A";
- id = "a";
- region = "us-east-1";
- inherit
- namespace
- metricName
- dimensions
- statistic
- ;
- period = "60";
- matchExact = true;
- metricQueryType = 0;
- metricEditorMode = 0;
- queryMode = "Metrics";
- };
- }
- ];
- # `window` is in seconds; `condition` compares the reduced value $B.
+ # Prod rules: a critical and a warning rule per entry in prod-alerts.nix,
+ # each linked to that entry's panel on the "Prod alerts" dashboard.
+ prodAlerts = import ./prod-alerts.nix;
prodRule =
+ alert:
{
- uid,
- title,
severity,
- service,
- pages ? false,
- metric,
- window,
- reducer,
condition,
- description,
- runbook,
}:
{
- inherit uid;
- title = "[PROD] ${title} (${severity})";
+ uid = "${alert.uid}-${if severity == "critical" then "crit" else "warn"}";
+ title = "[PROD] ${alert.title} (${severity})";
condition = "C";
for = "0s";
# Sparse CloudWatch counters have no datapoints when idle.
@@ -129,431 +86,63 @@ let
isPaused = false;
labels = {
environment = "prod";
- inherit severity service;
+ inherit severity;
+ inherit (alert) service;
# Only critical alerts page, through GoAlert.
- pages = lib.boolToString (pages && severity == "critical");
+ pages = lib.boolToString ((alert.pages or false) && severity == "critical");
};
+ # Grafana turns the dashboard and panel annotations into the panel link
+ # in notifications and the rule view.
annotations = {
- summary = title;
- inherit description;
- runbook_url = "https://github.com/macro-inc/macro/blob/main/docs/runbooks/${runbook}";
+ summary = alert.title;
+ inherit (alert) description;
+ runbook_url = "https://github.com/macro-inc/macro/blob/main/docs/runbooks/${alert.runbook}";
+ __dashboardUid__ = prodAlerts.dashboardUid;
+ __panelId__ = toString alert.panelId;
};
- data = cloudwatch metric window ++ expressions reducer condition;
+ data = [
+ {
+ refId = "A";
+ datasourceUid = "cloudwatch";
+ relativeTimeRange = {
+ from = alert.window;
+ to = 0;
+ };
+ model = prodAlerts.cloudwatchQuery alert.metric;
+ }
+ ]
+ ++ expressions alert.reducer condition;
};
-
- rds = metricName: instances: statistic: {
- namespace = "AWS/RDS";
- inherit metricName statistic;
- dimensions.DBInstanceIdentifier = instances;
- };
- bothInstances = [
- "macro-db-prod"
- "macro-db-prod-read-replica"
- ];
- redisNodes = [
- "macro-cache-prod-001"
- "macro-cache-prod-002"
- "email-service-redis-prod-c862c10"
- "calendar-service-redis-prod-aee80d2"
- "connection-gateway-redis-prod-f9d8cd0"
- ];
- redis = metricName: statistic: {
- namespace = "AWS/ElastiCache";
- inherit metricName statistic;
- dimensions.CacheClusterId = redisNodes;
- };
- queueAge = queues: {
- namespace = "AWS/SQS";
- metricName = "ApproximateAgeOfOldestMessage";
- statistic = "Maximum";
- dimensions.QueueName = queues;
- };
- workQueues = [
- "email-service-refresh-queue-prod"
- "convert-service-queue-prod"
- "ai-projection-queue-prod"
- "email-service-gmail-webhook-queue-prod"
- "notification-ingress-queue-prod"
- ];
-
- primaryMemory = {
- service = "macro-db-prod";
- pages = true;
- metric = rds "FreeableMemory" [ "macro-db-prod" ] "Average";
- window = 900;
- reducer = "mean";
- description = "Freeable memory is {{ $values.B }} bytes. Normal is 7-18 GB; it fell to about 0.45 GB during the October overload. Each connection holds about 14 MB of private memory, so check connection count and swap first.";
- runbook = "macrodb.md#memory";
- };
- replicaMemory = {
- service = "macro-db-prod";
- pages = true;
- metric = rds "FreeableMemory" [ "macro-db-prod-read-replica" ] "Average";
- window = 900;
- reducer = "mean";
- description = "Replica freeable memory is {{ $values.B }} bytes. Normal minimum is about 3 GB. The replica serves GraphQL reads, so memory pressure shows up as slow lists.";
- runbook = "macrodb.md#memory";
- };
- swap = {
- service = "macro-db-prod";
- pages = true;
- metric = rds "SwapUsage" bothInstances "Maximum";
- window = 900;
- reducer = "mean";
- description = "{{ $labels.DBInstanceIdentifier }} is using {{ $values.B }} bytes of swap. Normal is under 0.6 GB. Swap above 5 GB preceded both October overloads by hours.";
- runbook = "macrodb.md#memory";
- };
- load = {
- service = "macro-db-prod";
- pages = true;
- metric = rds "DBLoadRelativeToNumVCPUs" bothInstances "Average";
- window = 900;
- reducer = "mean";
- description = "{{ $labels.DBInstanceIdentifier }} has {{ $values.B }}x as many active sessions as vCPUs. Normal is under 1.3; the October overload reached 8. Look at Performance Insights top waits (locks, IO) and the services holding the most connections.";
- runbook = "macrodb.md#load";
- };
- cpu = {
- service = "macro-db-prod";
- pages = true;
- metric = rds "CPUUtilization" bothInstances "Average";
- window = 900;
- reducer = "mean";
- description = "{{ $labels.DBInstanceIdentifier }} CPU averaged {{ $values.B }}% over 15 minutes. Normal peaks are about 30%.";
- runbook = "macrodb.md#load";
- };
- primaryConnections = {
- service = "macro-db-prod";
- pages = true;
- metric = rds "DatabaseConnections" [ "macro-db-prod" ] "Maximum";
- window = 600;
- reducer = "mean";
- description = "macro-db-prod has {{ $values.B }} connections. Normal 15-minute peaks are about 1,100; max_connections is 3,439. Each connection costs about 14 MB, so this also drains freeable memory.";
- runbook = "macrodb.md#connections";
- };
- replicaConnections = {
- service = "macro-db-prod";
- metric = rds "DatabaseConnections" [ "macro-db-prod-read-replica" ] "Maximum";
- window = 600;
- reducer = "mean";
- description = "The read replica has {{ $values.B }} connections. Normal peaks are about 460.";
- runbook = "macrodb.md#connections";
- };
- replicaLag = {
- service = "macro-db-prod";
- pages = true;
- metric = rds "ReplicaLag" [ "macro-db-prod-read-replica" ] "Maximum";
- window = 600;
- reducer = "mean";
- description = "The read replica is {{ $values.B }} seconds behind the primary. Normal is under 6 seconds. GraphQL reads use the replica, so users see stale data and missing new items.";
- runbook = "macrodb.md#replica-lag";
- };
- readLatency = {
- service = "macro-db-prod";
- metric = rds "ReadLatency" bothInstances "Average";
- window = 900;
- reducer = "mean";
- description = "{{ $labels.DBInstanceIdentifier }} disk reads take {{ $values.B }} seconds on average. Normal is about 0.6 ms.";
- runbook = "macrodb.md#load";
- };
- freeStorage = {
- service = "macro-db-prod";
- pages = true;
- metric = rds "FreeStorageSpace" bothInstances "Minimum";
- window = 3600;
- reducer = "min";
- description = "{{ $labels.DBInstanceIdentifier }} has {{ $values.B }} bytes of free storage out of 5.37 TB.";
- runbook = "macrodb.md#storage";
- };
- redisMemory = {
- service = "elasticache";
- pages = true;
- metric = redis "DatabaseMemoryUsagePercentage" "Maximum";
- window = 600;
- reducer = "mean";
- description = "{{ $labels.CacheClusterId }} is using {{ $values.B }}% of maxmemory. Keys without a TTL are never evicted under volatile-lru, so at 100% writes fail.";
- runbook = "redis.md#memory";
- };
- redisEvictions = {
- service = "elasticache";
- metric = redis "Evictions" "Sum";
- window = 600;
- reducer = "sum";
- description = "{{ $labels.CacheClusterId }} evicted {{ $values.B }} keys in 10 minutes. Prod caches normally evict nothing.";
- runbook = "redis.md#memory";
- };
- backfillAge = {
- service = "email-service";
- metric = queueAge [ "email-service-backfill-queue-prod" ];
- window = 900;
- reducer = "mean";
- description = "The oldest message in email-service-backfill-queue-prod is {{ $values.B }} seconds old. Normal peaks are about 50 minutes; new users wait this long for their mailbox to fill.";
- runbook = "queues.md#backlog";
- };
- workQueueAge = {
- service = "sqs";
- pages = true;
- metric = queueAge workQueues;
- window = 900;
- reducer = "mean";
- description = "The oldest message in {{ $labels.QueueName }} is {{ $values.B }} seconds old. These queues normally drain within minutes.";
- runbook = "queues.md#backlog";
- };
-
# Warning and critical are mutually exclusive bands, as in the dev rules.
- prodRules = map prodRule [
- (
- primaryMemory
- // {
- uid = "macrodb-primary-freeable-memory-crit";
- title = "macro-db-prod freeable memory low";
- severity = "critical";
- condition = "$B < 1000000000";
- }
- )
- (
- primaryMemory
- // {
- uid = "macrodb-primary-freeable-memory-warn";
- title = "macro-db-prod freeable memory low";
- severity = "warning";
- condition = "$B < 2000000000 && $B >= 1000000000";
- }
- )
- (
- replicaMemory
- // {
- uid = "macrodb-replica-freeable-memory-crit";
- title = "macro-db-prod-read-replica freeable memory low";
- severity = "critical";
- condition = "$B < 750000000";
- }
- )
- (
- replicaMemory
- // {
- uid = "macrodb-replica-freeable-memory-warn";
- title = "macro-db-prod-read-replica freeable memory low";
- severity = "warning";
- condition = "$B < 1500000000 && $B >= 750000000";
- }
- )
- (
- swap
- // {
- uid = "macrodb-swap-crit";
- title = "macro-db-prod swapping";
- severity = "critical";
- condition = "$B > 5000000000";
- }
- )
- (
- swap
- // {
- uid = "macrodb-swap-warn";
- title = "macro-db-prod swapping";
- severity = "warning";
- condition = "$B > 2000000000 && $B <= 5000000000";
- }
- )
- (
- load
- // {
- uid = "macrodb-load-crit";
- title = "macro-db-prod database load above vCPU count";
- severity = "critical";
- condition = "$B > 4";
- }
- )
- (
- load
- // {
- uid = "macrodb-load-warn";
- title = "macro-db-prod database load above vCPU count";
- severity = "warning";
- condition = "$B > 2 && $B <= 4";
- }
- )
- (
- cpu
- // {
- uid = "macrodb-cpu-crit";
- title = "macro-db-prod CPU high";
- severity = "critical";
- condition = "$B > 85";
- }
- )
- (
- cpu
- // {
- uid = "macrodb-cpu-warn";
- title = "macro-db-prod CPU high";
- severity = "warning";
- condition = "$B > 70 && $B <= 85";
- }
- )
- (
- primaryConnections
- // {
- uid = "macrodb-primary-connections-crit";
- title = "macro-db-prod connections high";
- severity = "critical";
- condition = "$B > 2200";
- }
- )
- (
- primaryConnections
- // {
- uid = "macrodb-primary-connections-warn";
- title = "macro-db-prod connections high";
- severity = "warning";
- condition = "$B > 1700 && $B <= 2200";
- }
- )
- (
- replicaConnections
- // {
- uid = "macrodb-replica-connections-crit";
- title = "macro-db-prod-read-replica connections high";
- severity = "critical";
- condition = "$B > 700";
- }
- )
- (
- replicaConnections
- // {
- uid = "macrodb-replica-connections-warn";
- title = "macro-db-prod-read-replica connections high";
- severity = "warning";
- condition = "$B > 550 && $B <= 700";
- }
- )
- (
- replicaLag
- // {
- uid = "macrodb-replica-lag-crit";
- title = "macro-db-prod-read-replica lagging";
- severity = "critical";
- condition = "$B > 300";
- }
- )
- (
- replicaLag
- // {
- uid = "macrodb-replica-lag-warn";
- title = "macro-db-prod-read-replica lagging";
- severity = "warning";
- condition = "$B > 60 && $B <= 300";
- }
- )
- (
- readLatency
- // {
- uid = "macrodb-read-latency-crit";
- title = "macro-db-prod disk read latency high";
- severity = "critical";
- condition = "$B > 0.01";
- }
- )
- (
- readLatency
- // {
- uid = "macrodb-read-latency-warn";
- title = "macro-db-prod disk read latency high";
- severity = "warning";
- condition = "$B > 0.005 && $B <= 0.01";
- }
- )
- (
- freeStorage
- // {
- uid = "macrodb-free-storage-crit";
- title = "macro-db-prod free storage low";
- severity = "critical";
- condition = "$B < 1000000000000";
- }
- )
- (
- freeStorage
- // {
- uid = "macrodb-free-storage-warn";
- title = "macro-db-prod free storage low";
- severity = "warning";
- condition = "$B < 1500000000000 && $B >= 1000000000000";
- }
- )
- (
- redisMemory
- // {
- uid = "redis-memory-crit";
- title = "Redis memory high";
- severity = "critical";
- condition = "$B > 90";
- }
- )
- (
- redisMemory
- // {
- uid = "redis-memory-warn";
- title = "Redis memory high";
- severity = "warning";
- condition = "$B > 75 && $B <= 90";
- }
- )
- (
- redisEvictions
- // {
- uid = "redis-evictions-crit";
- title = "Redis evicting keys";
- severity = "critical";
- condition = "$B > 1000";
- }
- )
- (
- redisEvictions
- // {
- uid = "redis-evictions-warn";
- title = "Redis evicting keys";
- severity = "warning";
- condition = "$B > 0 && $B <= 1000";
- }
- )
- (
- backfillAge
- // {
- uid = "sqs-email-backfill-age-crit";
- title = "Email backfill queue falling behind";
- severity = "critical";
- condition = "$B > 10800";
- }
- )
- (
- backfillAge
- // {
- uid = "sqs-email-backfill-age-warn";
- title = "Email backfill queue falling behind";
- severity = "warning";
- condition = "$B > 3600 && $B <= 10800";
- }
- )
- (
- workQueueAge
- // {
- uid = "sqs-work-queue-age-crit";
- title = "Prod work queue falling behind";
- severity = "critical";
- condition = "$B > 7200";
- }
- )
- (
- workQueueAge
- // {
- uid = "sqs-work-queue-age-warn";
- title = "Prod work queue falling behind";
- severity = "warning";
- condition = "$B > 1800 && $B <= 7200";
- }
- )
- ];
+ prodRulePair =
+ alert:
+ let
+ critical = builtins.toJSON alert.critical;
+ warning = builtins.toJSON alert.warning;
+ in
+ if alert.firesWhen == "above" then
+ [
+ (prodRule alert {
+ severity = "critical";
+ condition = "$B > ${critical}";
+ })
+ (prodRule alert {
+ severity = "warning";
+ condition = "$B > ${warning} && $B <= ${critical}";
+ })
+ ]
+ else
+ [
+ (prodRule alert {
+ severity = "critical";
+ condition = "$B < ${critical}";
+ })
+ (prodRule alert {
+ severity = "warning";
+ condition = "$B < ${warning} && $B >= ${critical}";
+ })
+ ];
+ prodRules = lib.concatMap (row: lib.concatMap prodRulePair row.alerts) prodAlerts.rows;
dss = [
(lokiQuery ''sum(count_over_time({service_name="cloud-storage-service",deployment_environment="dev"} |~ "(?i)\\berror\\w*" [5m]))'')
];
diff --git a/infra/stacks/observability/nixos/goalert.nix b/infra/stacks/observability/nixos/goalert.nix
index 480fb341af..fea82f3a14 100644
--- a/infra/stacks/observability/nixos/goalert.nix
+++ b/infra/stacks/observability/nixos/goalert.nix
@@ -2,8 +2,7 @@
#
# Everything here runs in Ohio and talks only to loopback, so paging keeps
# working while production or its database is down. Pages reach the Macro
-# Alerts channel through the relay; phone paging (Twilio) and Google login are
-# not set up yet.
+# Alerts channel through the relay; phone paging (Twilio) is not set up yet.
{
config,
lib,
@@ -23,8 +22,20 @@ let
# start. `@name@` strings are replaced with the credential of that name.
goalertConfig = {
General.ApplicationName = "Macro On-Call";
- # One `admin` login, created by apply-config.py from goalert_admin_password.
+ # Google login, with Grafana's OAuth client. GoAlert cannot restrict OIDC
+ # logins to a domain; the client's Internal audience keeps them to Workspace
+ # accounts. Basic auth stays on for the break-glass `admin` login, created
+ # by apply-config.py from goalert_admin_password.
Auth.DisableBasic = false;
+ OIDC = {
+ Enable = true;
+ NewUsers = true;
+ OverrideName = "Google";
+ IssuerURL = "https://accounts.google.com";
+ ClientID = "@google_client_id@";
+ ClientSecret = "@google_client_secret@";
+ Scopes = "openid profile email";
+ };
Webhook = {
Enable = true;
AllowedURLs = [ relayUrl ];
@@ -93,6 +104,8 @@ in
LoadCredential = map credential [
"goalert_encryption_key"
"goalert_admin_password"
+ "google_client_id"
+ "google_client_secret"
];
ExecStartPre = [
"${pkgs.python3}/bin/python3 ${./goalert/apply-config.py} ${lib.getExe goalert} ${pkgs.writeText "goalert-config.json" (builtins.toJSON goalertConfig)} ${config.services.postgresql.package}/bin/psql"
diff --git a/infra/stacks/observability/nixos/prod-alerts-dashboard.nix b/infra/stacks/observability/nixos/prod-alerts-dashboard.nix
new file mode 100644
index 0000000000..314c9bf873
--- /dev/null
+++ b/infra/stacks/observability/nixos/prod-alerts-dashboard.nix
@@ -0,0 +1,143 @@
+{ lib, pkgs, ... }:
+let
+ prodAlerts = import ./prod-alerts.nix;
+ datasource = {
+ type = "cloudwatch";
+ uid = "cloudwatch";
+ };
+
+ # Threshold steps colour a value from the step it reaches upwards.
+ thresholdSteps =
+ alert:
+ if alert.firesWhen == "above" then
+ [
+ {
+ color = "green";
+ value = null;
+ }
+ {
+ color = "orange";
+ value = alert.warning;
+ }
+ {
+ color = "red";
+ value = alert.critical;
+ }
+ ]
+ else
+ [
+ {
+ color = "red";
+ value = null;
+ }
+ {
+ color = "orange";
+ value = alert.critical;
+ }
+ {
+ color = "green";
+ value = alert.warning;
+ }
+ ];
+
+ panel = alert: gridPos: {
+ id = alert.panelId;
+ inherit (alert) title;
+ inherit datasource gridPos;
+ type = "timeseries";
+ description = "Alerts on the ${alert.reducer} over ${toString (alert.window / 60)} minutes: warning (orange) and critical (red). Rules ${alert.uid}-warn and ${alert.uid}-crit.";
+ targets = [ (prodAlerts.cloudwatchQuery alert.metric // { inherit datasource; }) ];
+ fieldConfig.defaults = {
+ inherit (alert) unit;
+ thresholds = {
+ mode = "absolute";
+ steps = thresholdSteps alert;
+ };
+ custom.thresholdsStyle.mode = "dashed";
+ };
+ options.legend = {
+ displayMode = "list";
+ placement = "bottom";
+ showLegend = true;
+ };
+ };
+
+ # A row header, then two panels per line.
+ rowPanels =
+ y: row:
+ [
+ {
+ inherit (row) id title;
+ type = "row";
+ collapsed = false;
+ panels = [ ];
+ gridPos = {
+ x = 0;
+ inherit y;
+ w = 24;
+ h = 1;
+ };
+ }
+ ]
+ ++ lib.imap0 (
+ index: alert:
+ panel alert {
+ x = lib.mod index 2 * 12;
+ y = y + 1 + builtins.div index 2 * 8;
+ w = 12;
+ h = 8;
+ }
+ ) row.alerts;
+ rowHeight = row: 1 + (builtins.length row.alerts + 1) / 2 * 8;
+ panels =
+ (lib.foldl'
+ (layout: row: {
+ y = layout.y + rowHeight row;
+ panels = layout.panels ++ rowPanels layout.y row;
+ })
+ {
+ y = 0;
+ panels = [ ];
+ }
+ prodAlerts.rows
+ ).panels;
+
+ dashboard = {
+ uid = prodAlerts.dashboardUid;
+ title = "Prod alerts";
+ schemaVersion = 39;
+ version = 1;
+ editable = false;
+ tags = [
+ "macro"
+ "prod"
+ "alerts"
+ ];
+ timezone = "browser";
+ refresh = "1m";
+ time = {
+ from = "now-6h";
+ to = "now";
+ };
+ description = "One panel per prod alert rule in nixos/alerting.nix, with its warning and critical thresholds. Alert notifications link to the panel that fired. Reads CloudWatch in us-east-1.";
+ inherit panels;
+ };
+in
+{
+ services.grafana.provision.dashboards.settings.providers = [
+ {
+ name = "macro-prod-alerts";
+ orgId = 1;
+ folder = "Macro";
+ type = "file";
+ disableDeletion = true;
+ allowUiUpdates = false;
+ options.path = pkgs.linkFarm "macro-prod-alerts-dashboard" [
+ {
+ name = "prod-alerts.json";
+ path = pkgs.writeText "prod-alerts.json" (builtins.toJSON dashboard);
+ }
+ ];
+ }
+ ];
+}
diff --git a/infra/stacks/observability/nixos/prod-alerts.nix b/infra/stacks/observability/nixos/prod-alerts.nix
new file mode 100644
index 0000000000..68810ffe24
--- /dev/null
+++ b/infra/stacks/observability/nixos/prod-alerts.nix
@@ -0,0 +1,320 @@
+# Prod alert metrics, shared by the rules in alerting.nix and the panels in
+# prod-alerts-dashboard.nix so a rule and its graph cannot drift apart.
+#
+# Each entry becomes a critical and a warning rule (`<uid>-crit`, `<uid>-warn`)
+# and one dashboard panel (`panelId`). Each has a matching Datadog monitor in
+# infra/stacks/monitoring/monitors/{macrodb,redis,queues}.ts; keep thresholds
+# and windows in sync by hand. Grafana pages for the entries marked `pages`, so
+# their Datadog copies only notify.
+#
+# `firesWhen` is "above" or "below": critical fires past `critical`, warning
+# fires between `warning` and `critical`. `window` is in seconds; the rule
+# reduces it with `reducer` before comparing.
+let
+ # The CloudWatch query a rule evaluates and its panel draws.
+ cloudwatchQuery =
+ {
+ namespace,
+ metricName,
+ dimensions,
+ statistic,
+ }:
+ {
+ refId = "A";
+ id = "a";
+ region = "us-east-1";
+ inherit
+ namespace
+ metricName
+ dimensions
+ statistic
+ ;
+ period = "60";
+ matchExact = true;
+ metricQueryType = 0;
+ metricEditorMode = 0;
+ queryMode = "Metrics";
+ };
+
+ rds = metricName: instances: statistic: {
+ namespace = "AWS/RDS";
+ inherit metricName statistic;
+ dimensions.DBInstanceIdentifier = instances;
+ };
+ bothInstances = [
+ "macro-db-prod"
+ "macro-db-prod-read-replica"
+ ];
+ redisNodes = [
+ "macro-cache-prod-001"
+ "macro-cache-prod-002"
+ "email-service-redis-prod-c862c10"
+ "calendar-service-redis-prod-aee80d2"
+ "connection-gateway-redis-prod-f9d8cd0"
+ ];
+ redis = metricName: statistic: {
+ namespace = "AWS/ElastiCache";
+ inherit metricName statistic;
+ dimensions.CacheClusterId = redisNodes;
+ };
+ queueAge = queues: {
+ namespace = "AWS/SQS";
+ metricName = "ApproximateAgeOfOldestMessage";
+ statistic = "Maximum";
+ dimensions.QueueName = queues;
+ };
+ workQueues = [
+ "email-service-refresh-queue-prod"
+ "convert-service-queue-prod"
+ "ai-projection-queue-prod"
+ "email-service-gmail-webhook-queue-prod"
+ "notification-ingress-queue-prod"
+ ];
+in
+{
+ inherit cloudwatchQuery;
+ dashboardUid = "prod-alerts";
+
+ # Dashboard rows, in order. Panel ids are stable: notification links use them.
+ rows = [
+ {
+ id = 101;
+ title = "macrodb";
+ alerts = [
+ {
+ panelId = 1;
+ uid = "macrodb-primary-freeable-memory";
+ title = "macro-db-prod freeable memory low";
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "FreeableMemory" [ "macro-db-prod" ] "Average";
+ unit = "decbytes";
+ window = 900;
+ reducer = "mean";
+ firesWhen = "below";
+ warning = 2000000000;
+ critical = 1000000000;
+ description = "Freeable memory is {{ $values.B }} bytes. Normal is 7-18 GB; it fell to about 0.45 GB during the October overload. Each connection holds about 14 MB of private memory, so check connection count and swap first.";
+ runbook = "macrodb.md#memory";
+ }
+ {
+ panelId = 2;
+ uid = "macrodb-replica-freeable-memory";
+ title = "macro-db-prod-read-replica freeable memory low";
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "FreeableMemory" [ "macro-db-prod-read-replica" ] "Average";
+ unit = "decbytes";
+ window = 900;
+ reducer = "mean";
+ firesWhen = "below";
+ warning = 1500000000;
+ critical = 750000000;
+ description = "Replica freeable memory is {{ $values.B }} bytes. Normal minimum is about 3 GB. The replica serves GraphQL reads, so memory pressure shows up as slow lists.";
+ runbook = "macrodb.md#memory";
+ }
+ {
+ panelId = 3;
+ uid = "macrodb-swap";
+ title = "macro-db-prod swapping";
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "SwapUsage" bothInstances "Maximum";
+ unit = "decbytes";
+ window = 900;
+ reducer = "mean";
+ firesWhen = "above";
+ warning = 2000000000;
+ critical = 5000000000;
+ description = "{{ $labels.DBInstanceIdentifier }} is using {{ $values.B }} bytes of swap. Normal is under 0.6 GB. Swap above 5 GB preceded both October overloads by hours.";
+ runbook = "macrodb.md#memory";
+ }
+ {
+ panelId = 4;
+ uid = "macrodb-load";
+ title = "macro-db-prod database load above vCPU count";
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "DBLoadRelativeToNumVCPUs" bothInstances "Average";
+ unit = "short";
+ window = 900;
+ reducer = "mean";
+ firesWhen = "above";
+ warning = 2;
+ critical = 4;
+ description = "{{ $labels.DBInstanceIdentifier }} has {{ $values.B }}x as many active sessions as vCPUs. Normal is under 1.3; the October overload reached 8. Look at Performance Insights top waits (locks, IO) and the services holding the most connections.";
+ runbook = "macrodb.md#load";
+ }
+ {
+ panelId = 5;
+ uid = "macrodb-cpu";
+ title = "macro-db-prod CPU high";
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "CPUUtilization" bothInstances "Average";
+ unit = "percent";
+ window = 900;
+ reducer = "mean";
+ firesWhen = "above";
+ warning = 70;
+ critical = 85;
+ description = "{{ $labels.DBInstanceIdentifier }} CPU averaged {{ $values.B }}% over 15 minutes. Normal peaks are about 30%.";
+ runbook = "macrodb.md#load";
+ }
+ {
+ panelId = 6;
+ uid = "macrodb-primary-connections";
+ title = "macro-db-prod connections high";
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "DatabaseConnections" [ "macro-db-prod" ] "Maximum";
+ unit = "short";
+ window = 600;
+ reducer = "mean";
+ firesWhen = "above";
+ warning = 1700;
+ critical = 2200;
+ description = "macro-db-prod has {{ $values.B }} connections. Normal 15-minute peaks are about 1,100; max_connections is 3,439. Each connection costs about 14 MB, so this also drains freeable memory.";
+ runbook = "macrodb.md#connections";
+ }
+ {
+ panelId = 7;
+ uid = "macrodb-replica-connections";
+ title = "macro-db-prod-read-replica connections high";
+ service = "macro-db-prod";
+ metric = rds "DatabaseConnections" [ "macro-db-prod-read-replica" ] "Maximum";
+ unit = "short";
+ window = 600;
+ reducer = "mean";
+ firesWhen = "above";
+ warning = 550;
+ critical = 700;
+ description = "The read replica has {{ $values.B }} connections. Normal peaks are about 460.";
+ runbook = "macrodb.md#connections";
+ }
+ {
+ panelId = 8;
+ uid = "macrodb-replica-lag";
+ title = "macro-db-prod-read-replica lagging";
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "ReplicaLag" [ "macro-db-prod-read-replica" ] "Maximum";
+ unit = "s";
+ window = 600;
+ reducer = "mean";
+ firesWhen = "above";
+ warning = 60;
+ critical = 300;
+ description = "The read replica is {{ $values.B }} seconds behind the primary. Normal is under 6 seconds. GraphQL reads use the replica, so users see stale data and missing new items.";
+ runbook = "macrodb.md#replica-lag";
+ }
+ {
+ panelId = 9;
+ uid = "macrodb-read-latency";
+ title = "macro-db-prod disk read latency high";
+ service = "macro-db-prod";
+ metric = rds "ReadLatency" bothInstances "Average";
+ unit = "s";
+ window = 900;
+ reducer = "mean";
+ firesWhen = "above";
+ warning = 0.005;
+ critical = 0.01;
+ description = "{{ $labels.DBInstanceIdentifier }} disk reads take {{ $values.B }} seconds on average. Normal is about 0.6 ms.";
+ runbook = "macrodb.md#load";
+ }
+ {
+ panelId = 10;
+ uid = "macrodb-free-storage";
+ title = "macro-db-prod free storage low";
+ service = "macro-db-prod";
+ pages = true;
+ metric = rds "FreeStorageSpace" bothInstances "Minimum";
+ unit = "decbytes";
+ window = 3600;
+ reducer = "min";
+ firesWhen = "below";
+ warning = 1500000000000;
+ critical = 1000000000000;
+ description = "{{ $labels.DBInstanceIdentifier }} has {{ $values.B }} bytes of free storage out of 5.37 TB.";
+ runbook = "macrodb.md#storage";
+ }
+ ];
+ }
+ {
+ id = 102;
+ title = "Redis";
+ alerts = [
+ {
+ panelId = 11;
+ uid = "redis-memory";
+ title = "Redis memory high";
+ service = "elasticache";
+ pages = true;
+ metric = redis "DatabaseMemoryUsagePercentage" "Maximum";
+ unit = "percent";
+ window = 600;
+ reducer = "mean";
+ firesWhen = "above";
+ warning = 75;
+ critical = 90;
+ description = "{{ $labels.CacheClusterId }} is using {{ $values.B }}% of maxmemory. Keys without a TTL are never evicted under volatile-lru, so at 100% writes fail.";
+ runbook = "redis.md#memory";
+ }
+ {
+ panelId = 12;
+ uid = "redis-evictions";
+ title = "Redis evicting keys";
+ service = "elasticache";
+ metric = redis "Evictions" "Sum";
+ unit = "short";
+ window = 600;
+ # Thresholds apply to the ten-minute sum; the panel shows per-minute sums.
+ reducer = "sum";
+ firesWhen = "above";
+ warning = 0;
+ critical = 1000;
+ description = "{{ $labels.CacheClusterId }} evicted {{ $values.B }} keys in 10 minutes. Prod caches normally evict nothing.";
+ runbook = "redis.md#memory";
+ }
+ ];
+ }
+ {
+ id = 103;
+ title = "Queues";
+ alerts = [
+ {
+ panelId = 13;
+ uid = "sqs-email-backfill-age";
+ title = "Email backfill queue falling behind";
+ service = "email-service";
+ metric = queueAge [ "email-service-backfill-queue-prod" ];
+ unit = "s";
+ window = 900;
+ reducer = "mean";
+ firesWhen = "above";
+ warning = 3600;
+ critical = 10800;
+ description = "The oldest message in email-service-backfill-queue-prod is {{ $values.B }} seconds old. Normal peaks are about 50 minutes; new users wait this long for their mailbox to fill.";
+ runbook = "queues.md#backlog";
+ }
+ {
+ panelId = 14;
+ uid = "sqs-work-queue-age";
+ title = "Prod work queue falling behind";
+ service = "sqs";
+ pages = true;
+ metric = queueAge workQueues;
+ unit = "s";
+ window = 900;
+ reducer = "mean";
+ firesWhen = "above";
+ warning = 1800;
+ critical = 7200;
+ description = "The oldest message in {{ $labels.QueueName }} is {{ $values.B }} seconds old. These queues normally drain within minutes.";
+ runbook = "queues.md#backlog";
+ }
+ ];
+ }
+ ];
+}
diff --git a/infra/stacks/observability/nixos/services.nix b/infra/stacks/observability/nixos/services.nix
index e7b15641d9..c10cf01397 100644
--- a/infra/stacks/observability/nixos/services.nix
+++ b/infra/stacks/observability/nixos/services.nix
@@ -19,6 +19,7 @@ in
./ci-dashboard.nix
./database-dashboard.nix
./database-queries-dashboard.nix
+ ./prod-alerts-dashboard.nix
./alerting.nix
./goalert.nix
];