Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

fix(monitoring): reduce long-running transaction monitor noise (alerts 615, 621) (#9553)

teo4771ab7f0037
diff --git a/docs/runbooks/macrodb.md b/docs/runbooks/macrodb.md
index 4fde31f698..4c818d1707 100644
--- a/docs/runbooks/macrodb.md
+++ b/docs/runbooks/macrodb.md
@@ -88,8 +88,17 @@ Connections blocked on locks, or a transaction open for a long time.
its query. Advisory locks (`pg_advisory_xact_lock`) and tuple locks from
concurrent updates of the same rows have caused past overloads.
- A long open transaction holds its locks and stops vacuum. The long-running
- transaction alert includes the normalized SQL. Confirm it in
- `pg_stat_activity` (`xact_start`, `state = 'idle in transaction'`).
+ transaction alert fires once the oldest transaction has stayed old for five
+ minutes. Find it and its SQL in `pg_stat_activity` (`xact_start`,
+ `state = 'idle in transaction'`) or Database Monitoring.
+- A single sample of `postgresql.activity.xact_start_age` as old as the
+ newest `notification-service` task, at about :09 past the hour, is not a
+ transaction. notification-service's `LISTEN` connection (no
+ `application_name`) briefly runs a server-side transaction while it delivers
+ the `NOTIFY` burst from the hourly notification retention sweep. That
+ transaction's `xact_start` is the connection's last statement, its `LISTEN`
+ at task start. A fresh `backend_xmin` (thousands of transaction IDs old,
+ not millions) shows it is not holding back vacuum.
- Levers: cancel (`pg_cancel_backend`) or terminate (`pg_terminate_backend`)
the blocking session once you know what it is; stop the job that keeps
starting new ones.
diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.md
index d615267b19..ff3205cc3c 100644
--- a/infra/stacks/monitoring/AGENTS.md
+++ b/infra/stacks/monitoring/AGENTS.md
@@ -35,9 +35,11 @@ From `infra/`:
at oncall.macro-internal.com.
- A prod monitor's graph on its file's dashboard repeats its metric and
thresholds by hand; change both. `routing.test.ts` matches them by title.
- The long-running transaction monitor is a Database Monitoring alert so the
- notification can include the SQL; its graph still plots
- `postgresql.activity.xact_start_age`.
+ The long-running transaction monitor takes the lowest
+ `postgresql.activity.xact_start_age` over 5 minutes, and its graph plots the
+ same `moving_rollup(..., 300, 'min')`. A single-sample spike from
+ notification-service's `LISTEN` connection fires hourly otherwise
+ (docs/runbooks/macrodb.md#locks).
- End every monitor message with `notify({ pages })`, which routes only to
GoAlert, and pass `dependsOn: goalertWebhooks`. Never mention a channel
webhook, Slack handle or email address: GoAlert posts to the channel. Only
diff --git a/infra/stacks/monitoring/monitors/macrodb.ts b/infra/stacks/monitoring/monitors/macrodb.ts
index d0d8fd3933..d08d38cdfe 100644
--- a/infra/stacks/monitoring/monitors/macrodb.ts
+++ b/infra/stacks/monitoring/monitors/macrodb.ts
@@ -94,9 +94,9 @@ const dashboard = new datadog.Dashboard('macrodb-alerts', {
thresholdGraph({
title: 'macro-db-prod long-running transaction',
query:
- 'max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host}',
- critical: '1800',
- warning: '600',
+ "moving_rollup(max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host}, 300, 'min')",
+ critical: '1500',
+ warning: '300',
}),
]),
graphGroup('Storage', [
@@ -276,22 +276,20 @@ prodMonitor('macrodb-blocked-connections', {
runbook: 'docs/runbooks/macrodb.md#locks',
});
-// Database Monitoring query samples, not the host-level xact_start_age gauge:
-// that metric has no SQL tag, so the notification could not name the statement.
-// The dashboard graph still plots xact_start_age with the same age markers.
+// The lowest age over 5 minutes, so one sample cannot trigger it: an idle
+// LISTEN backend draining NOTIFYs reports its connect time as xact_start for
+// a few seconds (docs/runbooks/macrodb.md#locks). An open transaction stays
+// over 300s for 5 minutes once it is 10 minutes old, so warning and critical
+// fire at about 10 and 30 minutes.
prodMonitor('macrodb-long-transactions', {
title: 'macro-db-prod long-running transaction',
service: 'macro-db-prod',
- type: 'database-monitoring alert',
query:
- 'database-monitoring("dbm_type:activity host:macro-db-prod* @duration:>1800s").rollup("count").by("host,@db.statement").last("10m") > 0',
- monitorThresholds: { critical: '0' },
+ 'min(last_5m):max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host} > 1500',
+ monitorThresholds: { critical: '1500', warning: '300' },
pages: false,
- enableLogsSample: true,
- alertDetail: 'oldest transaction >= 1800s',
- recoveryDetail: 'no transaction over 1800s',
description:
- 'The oldest open transaction on {{host.name}}, "{{[@db.statement].name}}", is at least 1800 seconds old. Long transactions hold locks and stop vacuum.',
+ 'The oldest open transaction on {{host.name}} has been over {{value}} seconds old for 5 minutes, so it has been open at least 5 minutes longer than that. Long transactions hold locks and stop vacuum; find the session and its SQL in Database Monitoring.',
dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#locks',
});
diff --git a/infra/stacks/monitoring/routing.test.ts b/infra/stacks/monitoring/routing.test.ts
index 9b87f53ef6..0bce80a488 100644
--- a/infra/stacks/monitoring/routing.test.ts
+++ b/infra/stacks/monitoring/routing.test.ts
@@ -182,12 +182,6 @@ test('every prod monitor is graphed on its dashboard with its thresholds', () =>
);
const { critical, warning } = monitor.inputs.monitorThresholds;
const anomaly = monitor.inputs.query.includes('anomalies(');
- // The long-running transaction monitor reads DBM query samples so the
- // notification can include the SQL. Its graph still plots xact_start_age.
- const graphThresholds =
- monitor.name === 'macrodb-long-transactions'
- ? { critical: '1800', warning: '600' }
- : { critical, warning };
expect({
name: monitor.name,
markers: graph?.timeseriesDefinition.markers.map(
@@ -197,21 +191,7 @@ test('every prod monitor is graphed on its dashboard with its thresholds', () =>
name: monitor.name,
markers: anomaly
? []
- : [
- `y = ${graphThresholds.critical}`,
- ...(graphThresholds.warning
- ? [`y = ${graphThresholds.warning}`]
- : []),
- ],
+ : [`y = ${critical}`, ...(warning ? [`y = ${warning}`] : [])],
});
}
});
-
-test('the long-running transaction alert names the SQL', () => {
- const monitor = monitors().find(
- (r) => r.name === 'macrodb-long-transactions'
- );
- expect(monitor?.inputs.type).toBe('database-monitoring alert');
- expect(monitor?.inputs.query).toContain('by("host,@db.statement")');
- expect(monitor?.inputs.message).toContain('{{[@db.statement].name}}');
-});