Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

fix(monitoring): size the RDS freeable memory floor for small instances (#9608)

teoe8065cffbfff
diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.md
index ff3205cc3c..c7b8d3a759 100644
--- a/infra/stacks/monitoring/AGENTS.md
+++ b/infra/stacks/monitoring/AGENTS.md
@@ -20,7 +20,8 @@ From `infra/`:
- `bun run check`
- Biome on changed files.
- An authenticated `pulumi preview` may change adopted monitors only in
- `message` and `renotifyInterval`; README.md lists the expected counts.
+ `message` and `renotifyInterval`, apart from the exceptions README.md lists
+ with the expected counts.
## Gotchas
diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.md
index 88f9679306..a95a3d96e0 100644
--- a/infra/stacks/monitoring/README.md
+++ b/infra/stacks/monitoring/README.md
@@ -108,7 +108,9 @@ The gate is that preview:
`renotifyInterval` only** — the routing move to GoAlert. The two AI-editing
monitors, already deployed from this stack, update their `message` the same
way. `dss-errors-prod` also changes its query and critical threshold
- (150 → 800).
+ (150 → 800), and `rds-low-freeable-memory` its query and thresholds
+ (critical 2 GB → 250 MB, warning 3 GB → 500 MB) so 2 GiB instances stop
+ alerting at their normal level.
- **Already-deployed monitors update their `message`** (GoAlert handles instead
of the channel), and the GraphQL soup volume, gateway connections and task
count monitors their `query` and thresholds.
@@ -240,7 +242,7 @@ otherwise the missing-telemetry alerts intentionally fire. Until it ships, a
downtime in `notification-retention.ts` mutes the prod one; delete the downtime
once the instrumentation is deployed. This monitoring stack
still requires a separate, explicitly approved manual deployment. Adopted
-monitors may change only in `message` and `renotifyInterval` (see the gate above).
+monitors may change only as the gate above lists.
Consumers are intentionally ungrouped and do not commit offsets, so native MSK
consumer-group lag is not meaningful. The new logs expose broker high watermark
diff --git a/infra/stacks/monitoring/monitors/rds.ts b/infra/stacks/monitoring/monitors/rds.ts
index 9d2b1b6956..9d97be4fcc 100644
--- a/infra/stacks/monitoring/monitors/rds.ts
+++ b/infra/stacks/monitoring/monitors/rds.ts
@@ -147,15 +147,18 @@ adopted('rds-low-freeable-memory', {
name: 'Low freeable memory for {{database_instance.name}}',
type: 'query alert',
query:
- 'avg(last_1h):avg:aws.rds.freeable_memory{environment:prod OR database_instance:macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com} by {database_instance} < 2000000000',
- // macrodb.ts and Grafana watch the same metric over a shorter window; this
- // one only notifies.
- message: `{{database_instance.name}} freeable memory is {{value}} bytes.
+ 'avg(last_1h):avg:aws.rds.freeable_memory{environment:prod OR database_instance:macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com} by {database_instance} < 250000000',
+ // One threshold covers instances from 2 GiB to 128 GiB, and CloudWatch has
+ // no total-memory metric to alert on a percentage, so the floor is set below
+ // the normal minimum of the smallest instance. macrodb.ts and Grafana watch
+ // macro-db-prod and its replica at their own baselines; this one only
+ // notifies.
+ message: `{{database_instance.name}} freeable memory is {{value}} bytes, averaged over an hour. A healthy 2 GiB instance keeps about 1 GB freeable, so this is memory pressure on any instance class. Check connection count and swap usage.
${notify({ pages: false })}`,
monitorThresholds: {
- critical: '2000000000',
- warning: '3000000000',
+ critical: '250000000',
+ warning: '500000000',
},
onMissingData: 'default',
notifyAudit: false,