Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
fix(monitoring): size the RDS freeable memory floor for small instances (#9608)
teoe8065cffbfff
diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.mdindex ff3205cc3c..c7b8d3a759 100644--- a/infra/stacks/monitoring/AGENTS.md+++ b/infra/stacks/monitoring/AGENTS.md@@ -20,7 +20,8 @@ From `infra/`:- `bun run check`- Biome on changed files.- An authenticated `pulumi preview` may change adopted monitors only in- `message` and `renotifyInterval`; README.md lists the expected counts.+ `message` and `renotifyInterval`, apart from the exceptions README.md lists+ with the expected counts.## Gotchasdiff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.mdindex 88f9679306..a95a3d96e0 100644--- a/infra/stacks/monitoring/README.md+++ b/infra/stacks/monitoring/README.md@@ -108,7 +108,9 @@ The gate is that preview:`renotifyInterval` only** — the routing move to GoAlert. The two AI-editingmonitors, already deployed from this stack, update their `message` the sameway. `dss-errors-prod` also changes its query and critical threshold- (150 → 800).+ (150 → 800), and `rds-low-freeable-memory` its query and thresholds+ (critical 2 GB → 250 MB, warning 3 GB → 500 MB) so 2 GiB instances stop+ alerting at their normal level.- **Already-deployed monitors update their `message`** (GoAlert handles insteadof the channel), and the GraphQL soup volume, gateway connections and taskcount monitors their `query` and thresholds.@@ -240,7 +242,7 @@ otherwise the missing-telemetry alerts intentionally fire. Until it ships, adowntime in `notification-retention.ts` mutes the prod one; delete the downtimeonce the instrumentation is deployed. This monitoring stackstill requires a separate, explicitly approved manual deployment. Adopted-monitors may change only in `message` and `renotifyInterval` (see the gate above).+monitors may change only as the gate above lists.Consumers are intentionally ungrouped and do not commit offsets, so native MSKconsumer-group lag is not meaningful. The new logs expose broker high watermarkdiff --git a/infra/stacks/monitoring/monitors/rds.ts b/infra/stacks/monitoring/monitors/rds.tsindex 9d2b1b6956..9d97be4fcc 100644--- a/infra/stacks/monitoring/monitors/rds.ts+++ b/infra/stacks/monitoring/monitors/rds.ts@@ -147,15 +147,18 @@ adopted('rds-low-freeable-memory', {name: 'Low freeable memory for {{database_instance.name}}',type: 'query alert',query:- 'avg(last_1h):avg:aws.rds.freeable_memory{environment:prod OR database_instance:macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com} by {database_instance} < 2000000000',- // macrodb.ts and Grafana watch the same metric over a shorter window; this- // one only notifies.- message: `{{database_instance.name}} freeable memory is {{value}} bytes.+ 'avg(last_1h):avg:aws.rds.freeable_memory{environment:prod OR database_instance:macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com} by {database_instance} < 250000000',+ // One threshold covers instances from 2 GiB to 128 GiB, and CloudWatch has+ // no total-memory metric to alert on a percentage, so the floor is set below+ // the normal minimum of the smallest instance. macrodb.ts and Grafana watch+ // macro-db-prod and its replica at their own baselines; this one only+ // notifies.+ message: `{{database_instance.name}} freeable memory is {{value}} bytes, averaged over an hour. A healthy 2 GiB instance keeps about 1 GB freeable, so this is memory pressure on any instance class. Check connection count and swap usage.${notify({ pages: false })}`,monitorThresholds: {- critical: '2000000000',- warning: '3000000000',+ critical: '250000000',+ warning: '500000000',},onMissingData: 'default',notifyAudit: false,