Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
MacroDB alerts
Runbook for the macro-db-prod and macro-db-prod-read-replica monitors in
monitors/macrodb.ts. Both instances are
db.m9g.2xlarge: 8 vCPU, about 30 GB of memory, max_connections 3,439.
Start in three places:
- Performance Insights for the instance (RDS console → the instance → Monitoring → Database load). Group by waits, then by SQL, to see what the active sessions are doing.
- Datadog Database Monitoring for
macro-db-prod(Query Metrics, Query Samples, and the blocking queries view). - Recent deploys and queue backlogs. Most overloads have started with a deploy, a backfill, or a queue draining all at once. Check the ECS service events and the SQS monitors in the same window.
Memory
Freeable memory and swap alerts. Every Postgres backend keeps several MB of private memory, about 14 MB per connection under prod load, so memory falls as connections rise. The OS page cache shrinks first; once it is gone the instance swaps and every query slows.
- Check
aws.rds.database_connectionsagainst the same window. If connections climbed, find which services added them (see Connections). - Check swap (
aws.rds.swap_usage). Sustained swap above a few GB means the instance is out of memory, not just low on cache. - Levers: lower the pool size of the service that grew; pause backfills; as a last resort fail over or resize the instance (a resize restarts it).
Load
Database load (average active sessions) relative to vCPUs, CPU, and disk read latency. Load above the vCPU count means sessions are queueing for CPU, locks or IO.
- In Performance Insights, the top wait event says which:
CPU,Lock:*/LWLock:*, orIO:*. - If lock waits dominate, go to Locks.
- If one query dominates CPU, find its caller from the query sample's application name and the APM span with the same statement.
- Levers: pause the backfill or worker driving the load; roll back the deploy that introduced the query; add the missing index in a follow-up.
Connections
Connection count on the primary or replica. Each ECS task opens its own pool, so connections scale with task count times pool size.
- Group
postgresql.connections(Database Monitoring) or Performance Insights sessions by application or user to see which service holds them. - Check whether a service scaled out (
aws.ecs.service.running) or a pool setting changed in a recent deploy. - Levers: lower that service's pool maximum or task count. RDS Proxy is not a quick fix: moving a service onto it has caused pool timeouts before.
Replica lag
The read replica is behind the primary. GraphQL Soup reads go to the replica, so users see stale lists and items they just created go missing.
- Check primary write volume and WAL (
aws.rds.transaction_logs_generation) and replica CPU and IO. Lag usually follows a write burst on the primary or a long query on the replica holding back replay. - Look for long-running queries on the replica in Database Monitoring.
- Levers: stop the write burst (often a backfill); cancel the long replica query.
Storage
Free storage space, now or forecast to run low within a week.
- Compare
aws.rds.free_storage_spacewithaws.rds.transaction_logs_disk_usageandaws.rds.replication_slot_disk_usage: WAL held by a replication slot grows without bound. - Otherwise find the tables that grew (
pg_total_relation_sizeon the largest relations) and what started writing to them. - Levers: drop an abandoned replication slot; stop the writer; increase allocated storage (online, but only once every six hours).
Locks
Connections blocked on locks, or a transaction open for a long time.
- In Database Monitoring's blocking queries view, find the blocking session and
its query. Advisory locks (
pg_advisory_xact_lock) and tuple locks from concurrent updates of the same rows have caused past overloads. - A long open transaction holds its locks and stops vacuum. The long-running
transaction alert fires once the oldest transaction has stayed old for five
minutes. Find it and its SQL in
pg_stat_activity(xact_start,state = 'idle in transaction') or Database Monitoring. - A single sample of
postgresql.activity.xact_start_ageas old as the newestnotification-servicetask, at about :09 past the hour, is not a transaction. notification-service'sLISTENconnection (noapplication_name) briefly runs a server-side transaction while it delivers theNOTIFYburst from the hourly notification retention sweep. That transaction'sxact_startis the connection's last statement, itsLISTENat task start. A freshbackend_xmin(thousands of transaction IDs old, not millions) shows it is not holding back vacuum. - Levers: cancel (
pg_cancel_backend) or terminate (pg_terminate_backend) the blocking session once you know what it is; stop the job that keeps starting new ones.