Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
fix(monitoring): reduce long-running transaction monitor noise (alerts 615, 621) (#9553)
teo4771ab7f0037
diff --git a/docs/runbooks/macrodb.md b/docs/runbooks/macrodb.mdindex 4fde31f698..4c818d1707 100644--- a/docs/runbooks/macrodb.md+++ b/docs/runbooks/macrodb.md@@ -88,8 +88,17 @@ Connections blocked on locks, or a transaction open for a long time.its query. Advisory locks (`pg_advisory_xact_lock`) and tuple locks fromconcurrent updates of the same rows have caused past overloads.- A long open transaction holds its locks and stops vacuum. The long-running- transaction alert includes the normalized SQL. Confirm it in- `pg_stat_activity` (`xact_start`, `state = 'idle in transaction'`).+ transaction alert fires once the oldest transaction has stayed old for five+ minutes. Find it and its SQL in `pg_stat_activity` (`xact_start`,+ `state = 'idle in transaction'`) or Database Monitoring.+- A single sample of `postgresql.activity.xact_start_age` as old as the+ newest `notification-service` task, at about :09 past the hour, is not a+ transaction. notification-service's `LISTEN` connection (no+ `application_name`) briefly runs a server-side transaction while it delivers+ the `NOTIFY` burst from the hourly notification retention sweep. That+ transaction's `xact_start` is the connection's last statement, its `LISTEN`+ at task start. A fresh `backend_xmin` (thousands of transaction IDs old,+ not millions) shows it is not holding back vacuum.- Levers: cancel (`pg_cancel_backend`) or terminate (`pg_terminate_backend`)the blocking session once you know what it is; stop the job that keepsstarting new ones.diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.mdindex d615267b19..ff3205cc3c 100644--- a/infra/stacks/monitoring/AGENTS.md+++ b/infra/stacks/monitoring/AGENTS.md@@ -35,9 +35,11 @@ From `infra/`:at oncall.macro-internal.com.- A prod monitor's graph on its file's dashboard repeats its metric andthresholds by hand; change both. `routing.test.ts` matches them by title.- The long-running transaction monitor is a Database Monitoring alert so the- notification can include the SQL; its graph still plots- `postgresql.activity.xact_start_age`.+ The long-running transaction monitor takes the lowest+ `postgresql.activity.xact_start_age` over 5 minutes, and its graph plots the+ same `moving_rollup(..., 300, 'min')`. A single-sample spike from+ notification-service's `LISTEN` connection fires hourly otherwise+ (docs/runbooks/macrodb.md#locks).- End every monitor message with `notify({ pages })`, which routes only toGoAlert, and pass `dependsOn: goalertWebhooks`. Never mention a channelwebhook, Slack handle or email address: GoAlert posts to the channel. Onlydiff --git a/infra/stacks/monitoring/monitors/macrodb.ts b/infra/stacks/monitoring/monitors/macrodb.tsindex d0d8fd3933..d08d38cdfe 100644--- a/infra/stacks/monitoring/monitors/macrodb.ts+++ b/infra/stacks/monitoring/monitors/macrodb.ts@@ -94,9 +94,9 @@ const dashboard = new datadog.Dashboard('macrodb-alerts', {thresholdGraph({title: 'macro-db-prod long-running transaction',query:- 'max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host}',- critical: '1800',- warning: '600',+ "moving_rollup(max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host}, 300, 'min')",+ critical: '1500',+ warning: '300',}),]),graphGroup('Storage', [@@ -276,22 +276,20 @@ prodMonitor('macrodb-blocked-connections', {runbook: 'docs/runbooks/macrodb.md#locks',});-// Database Monitoring query samples, not the host-level xact_start_age gauge:-// that metric has no SQL tag, so the notification could not name the statement.-// The dashboard graph still plots xact_start_age with the same age markers.+// The lowest age over 5 minutes, so one sample cannot trigger it: an idle+// LISTEN backend draining NOTIFYs reports its connect time as xact_start for+// a few seconds (docs/runbooks/macrodb.md#locks). An open transaction stays+// over 300s for 5 minutes once it is 10 minutes old, so warning and critical+// fire at about 10 and 30 minutes.prodMonitor('macrodb-long-transactions', {title: 'macro-db-prod long-running transaction',service: 'macro-db-prod',- type: 'database-monitoring alert',query:- 'database-monitoring("dbm_type:activity host:macro-db-prod* @duration:>1800s").rollup("count").by("host,@db.statement").last("10m") > 0',- monitorThresholds: { critical: '0' },+ 'min(last_5m):max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host} > 1500',+ monitorThresholds: { critical: '1500', warning: '300' },pages: false,- enableLogsSample: true,- alertDetail: 'oldest transaction >= 1800s',- recoveryDetail: 'no transaction over 1800s',description:- 'The oldest open transaction on {{host.name}}, "{{[@db.statement].name}}", is at least 1800 seconds old. Long transactions hold locks and stop vacuum.',+ 'The oldest open transaction on {{host.name}} has been over {{value}} seconds old for 5 minutes, so it has been open at least 5 minutes longer than that. Long transactions hold locks and stop vacuum; find the session and its SQL in Database Monitoring.',dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#locks',});diff --git a/infra/stacks/monitoring/routing.test.ts b/infra/stacks/monitoring/routing.test.tsindex 9b87f53ef6..0bce80a488 100644--- a/infra/stacks/monitoring/routing.test.ts+++ b/infra/stacks/monitoring/routing.test.ts@@ -182,12 +182,6 @@ test('every prod monitor is graphed on its dashboard with its thresholds', () =>);const { critical, warning } = monitor.inputs.monitorThresholds;const anomaly = monitor.inputs.query.includes('anomalies(');- // The long-running transaction monitor reads DBM query samples so the- // notification can include the SQL. Its graph still plots xact_start_age.- const graphThresholds =- monitor.name === 'macrodb-long-transactions'- ? { critical: '1800', warning: '600' }- : { critical, warning };expect({name: monitor.name,markers: graph?.timeseriesDefinition.markers.map(@@ -197,21 +191,7 @@ test('every prod monitor is graphed on its dashboard with its thresholds', () =>name: monitor.name,markers: anomaly? []- : [- `y = ${graphThresholds.critical}`,- ...(graphThresholds.warning- ? [`y = ${graphThresholds.warning}`]- : []),- ],+ : [`y = ${critical}`, ...(warning ? [`y = ${warning}`] : [])],});}});--test('the long-running transaction alert names the SQL', () => {- const monitor = monitors().find(- (r) => r.name === 'macrodb-long-transactions'- );- expect(monitor?.inputs.type).toBe('database-monitoring alert');- expect(monitor?.inputs.query).toContain('by("host,@db.statement")');- expect(monitor?.inputs.message).toContain('{{[@db.statement].name}}');-});