Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

fix(monitoring): include the SQL in the long-running transaction alert (#9317)

Wolf Mermelsteind2a3b2a7f19c
diff --git a/docs/runbooks/macrodb.md b/docs/runbooks/macrodb.md
index 1ed4428e90..4fde31f698 100644
--- a/docs/runbooks/macrodb.md
+++ b/docs/runbooks/macrodb.md
@@ -87,7 +87,8 @@ Connections blocked on locks, or a transaction open for a long time.
- In Database Monitoring's blocking queries view, find the blocking session and
its query. Advisory locks (`pg_advisory_xact_lock`) and tuple locks from
concurrent updates of the same rows have caused past overloads.
-- A long open transaction holds its locks and stops vacuum. Find it in
+- A long open transaction holds its locks and stops vacuum. The long-running
+ transaction alert includes the normalized SQL. Confirm it in
`pg_stat_activity` (`xact_start`, `state = 'idle in transaction'`).
- Levers: cancel (`pg_cancel_backend`) or terminate (`pg_terminate_backend`)
the blocking session once you know what it is; stop the job that keeps
diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.md
index d9d5f1cca4..c78f3fef5c 100644
--- a/infra/stacks/monitoring/AGENTS.md
+++ b/infra/stacks/monitoring/AGENTS.md
@@ -33,6 +33,9 @@ From `infra/`:
at oncall.macro-internal.com.
- A prod monitor's graph on its file's dashboard repeats its metric and
thresholds by hand; change both. `routing.test.ts` matches them by title.
+ The long-running transaction monitor is a Database Monitoring alert so the
+ notification can include the SQL; its graph still plots
+ `postgresql.activity.xact_start_age`.
- End every monitor message with `notify({ pages })`, which routes only to
GoAlert, and pass `dependsOn: goalertWebhooks`. Never mention a channel
webhook, Slack handle or email address: GoAlert posts to the channel. Only
diff --git a/infra/stacks/monitoring/monitors/macrodb.ts b/infra/stacks/monitoring/monitors/macrodb.ts
index bcc9020726..d0d8fd3933 100644
--- a/infra/stacks/monitoring/monitors/macrodb.ts
+++ b/infra/stacks/monitoring/monitors/macrodb.ts
@@ -276,15 +276,22 @@ prodMonitor('macrodb-blocked-connections', {
runbook: 'docs/runbooks/macrodb.md#locks',
});
+// Database Monitoring query samples, not the host-level xact_start_age gauge:
+// that metric has no SQL tag, so the notification could not name the statement.
+// The dashboard graph still plots xact_start_age with the same age markers.
prodMonitor('macrodb-long-transactions', {
title: 'macro-db-prod long-running transaction',
service: 'macro-db-prod',
+ type: 'database-monitoring alert',
query:
- 'avg(last_10m):max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host} > 1800',
- monitorThresholds: { critical: '1800', warning: '600' },
+ 'database-monitoring("dbm_type:activity host:macro-db-prod* @duration:>1800s").rollup("count").by("host,@db.statement").last("10m") > 0',
+ monitorThresholds: { critical: '0' },
pages: false,
+ enableLogsSample: true,
+ alertDetail: 'oldest transaction >= 1800s',
+ recoveryDetail: 'no transaction over 1800s',
description:
- 'The oldest open transaction on {{host.name}} is {{value}} seconds old. Long transactions hold locks and stop vacuum.',
+ 'The oldest open transaction on {{host.name}}, "{{[@db.statement].name}}", is at least 1800 seconds old. Long transactions hold locks and stop vacuum.',
dashboard: dashboard.url,
runbook: 'docs/runbooks/macrodb.md#locks',
});
diff --git a/infra/stacks/monitoring/notify.ts b/infra/stacks/monitoring/notify.ts
index cfc335ef36..97c877f68c 100644
--- a/infra/stacks/monitoring/notify.ts
+++ b/infra/stacks/monitoring/notify.ts
@@ -60,19 +60,28 @@ export function alertMessage({
runbook,
dashboard,
pages,
+ alertDetail = '{{value}} (threshold {{threshold}})',
+ warningDetail = '{{value}} (warning threshold {{warn_threshold}})',
+ recoveryDetail = '{{value}}',
}: {
description: string;
runbook: string;
/** A dashboard's `url`, a path such as `/dashboard/abc-def-ghi/title`. */
dashboard?: pulumi.Input<string>;
pages: boolean;
+ /** Text after `CRITICAL:` in the state line. */
+ alertDetail?: string;
+ /** Text after `WARNING:` in the state line. */
+ warningDetail?: string;
+ /** Text after `RECOVERED:` in the state line. */
+ recoveryDetail?: string;
}): pulumi.Output<string> {
return pulumi.output(dashboard).apply((path) => {
const links = [
`Runbook: https://github.com/macro-inc/macro/blob/main/${runbook}`,
];
if (path !== undefined) links.push(`Dashboard: ${DATADOG_SITE}${path}`);
- return `{{#is_alert}}🔴 CRITICAL: {{value}} (threshold {{threshold}}){{/is_alert}}{{#is_warning}}🟠 WARNING: {{value}} (warning threshold {{warn_threshold}}){{/is_warning}}{{#is_recovery}}✅ RECOVERED: {{value}}{{/is_recovery}}
+ return `{{#is_alert}}🔴 CRITICAL: ${alertDetail}{{/is_alert}}{{#is_warning}}🟠 WARNING: ${warningDetail}{{/is_warning}}{{#is_recovery}}✅ RECOVERED: ${recoveryDetail}{{/is_recovery}}
${description}
diff --git a/infra/stacks/monitoring/prod-monitor.ts b/infra/stacks/monitoring/prod-monitor.ts
index 54e13aaac0..9120c6fce7 100644
--- a/infra/stacks/monitoring/prod-monitor.ts
+++ b/infra/stacks/monitoring/prod-monitor.ts
@@ -24,19 +24,30 @@ export function prodMonitor(
runbook: string;
/** `url` of the dashboard graphing this monitor, under a `thresholdGraph` titled `title`. */
dashboard?: pulumi.Input<string>;
+ /** Defaults to a metric `query alert`. */
+ type?: string;
+ /** Attach matching event samples. Needed for Database Monitoring attributes. */
+ enableLogsSample?: boolean;
+ /** Overrides for the state-line value text in `alertMessage`. */
+ alertDetail?: string;
+ warningDetail?: string;
+ recoveryDetail?: string;
}
): datadog.Monitor {
return new datadog.Monitor(
name,
{
name: `[PROD] ${args.title}`,
- type: 'query alert',
+ type: args.type ?? 'query alert',
query: args.query,
message: alertMessage({
description: args.description,
runbook: args.runbook,
dashboard: args.dashboard,
pages: args.pages,
+ alertDetail: args.alertDetail,
+ warningDetail: args.warningDetail,
+ recoveryDetail: args.recoveryDetail,
}),
tags: ['env:prod', `service:${args.service}`],
priority: args.pages ? '2' : '3',
@@ -47,6 +58,9 @@ export function prodMonitor(
onMissingData: 'default',
includeTags: true,
notifyAudit: false,
+ ...(args.enableLogsSample === undefined
+ ? {}
+ : { enableLogsSample: args.enableLogsSample }),
},
// The message mentions the GoAlert webhooks, which must exist first.
{ dependsOn: goalertWebhooks }
diff --git a/infra/stacks/monitoring/routing.test.ts b/infra/stacks/monitoring/routing.test.ts
index 06bac7fdc6..0d9ead019c 100644
--- a/infra/stacks/monitoring/routing.test.ts
+++ b/infra/stacks/monitoring/routing.test.ts
@@ -182,6 +182,12 @@ test('every prod monitor is graphed on its dashboard with its thresholds', () =>
);
const { critical, warning } = monitor.inputs.monitorThresholds;
const anomaly = monitor.inputs.query.includes('anomalies(');
+ // The long-running transaction monitor reads DBM query samples so the
+ // notification can include the SQL. Its graph still plots xact_start_age.
+ const graphThresholds =
+ monitor.name === 'macrodb-long-transactions'
+ ? { critical: '1800', warning: '600' }
+ : { critical, warning };
expect({
name: monitor.name,
markers: graph?.timeseriesDefinition.markers.map(
@@ -191,7 +197,21 @@ test('every prod monitor is graphed on its dashboard with its thresholds', () =>
name: monitor.name,
markers: anomaly
? []
- : [`y = ${critical}`, ...(warning ? [`y = ${warning}`] : [])],
+ : [
+ `y = ${graphThresholds.critical}`,
+ ...(graphThresholds.warning
+ ? [`y = ${graphThresholds.warning}`]
+ : []),
+ ],
});
}
});
+
+test('the long-running transaction alert names the SQL', () => {
+ const monitor = monitors().find(
+ (r) => r.name === 'macrodb-long-transactions'
+ );
+ expect(monitor?.inputs.type).toBe('database-monitoring alert');
+ expect(monitor?.inputs.query).toContain('by("host,@db.statement")');
+ expect(monitor?.inputs.message).toContain('{{[@db.statement].name}}');
+});
diff --git a/infra/stacks/observability/nixos/goalert/test_macro_alerts_relay.py b/infra/stacks/observability/nixos/goalert/test_macro_alerts_relay.py
index 715dbf9625..b6ec39a5d8 100644
--- a/infra/stacks/observability/nixos/goalert/test_macro_alerts_relay.py
+++ b/infra/stacks/observability/nixos/goalert/test_macro_alerts_relay.py
@@ -11,18 +11,18 @@ spec.loader.exec_module(relay)
DATADOG_DETAILS = """https://us5.datadoghq.com/event/event?id=8876996053944573810
-🔴 CRITICAL: 2631.578 (threshold 1800.0)
+🔴 CRITICAL: oldest transaction >= 1800s
-The oldest open transaction on macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com is 2631.578 seconds old. Long transactions hold locks and stop vacuum.
+The oldest open transaction on macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com, "SELECT * FROM events WHERE id = ?", is at least 1800 seconds old. Long transactions hold locks and stop vacuum.
Runbook: https://github.com/macro-inc/macro/blob/main/docs/runbooks/macrodb.md#locks
Dashboard: https://us5.datadoghq.com/dashboard/42x-f55-h6z/macrodb-alerts
@webhook-goalert-notify
-`avg(last_10m):max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host} > 1800`
+`database-monitoring("dbm_type:activity host:macro-db-prod* @duration:>1800s").rollup("count").by("host,@db.statement").last("10m") > 0`
-Metric value: 2631.578
+Metric value: 1
Metric Graph: https://us5.datadoghq.com/monitors/23404621?group=host · Monitor Status: https://us5.datadoghq.com/monitors/23404621"""
@@ -57,8 +57,8 @@ class AlertPostTest(unittest.TestCase):
})
self.assertEqual(post, """---
🔔 · **macro-db-prod long-running transaction — macro-db-prod**
-🔴 critical: 2631.578 (threshold 1800.0)
-The oldest open transaction on macro-db-prod is 2631.578 seconds old. Long transactions hold locks and stop vacuum.
+🔴 critical: oldest transaction >= 1800s
+The oldest open transaction on macro-db-prod, "SELECT * FROM events WHERE id = ?", is at least 1800 seconds old. Long transactions hold locks and stop vacuum.
[Acknowledge](https://oncall.macro-internal.com/alerts/130) · [Graph](https://us5.datadoghq.com/dashboard/42x-f55-h6z/macrodb-alerts) · [Runbook](https://github.com/macro-inc/macro/blob/main/docs/runbooks/macrodb.md#locks)""")
def test_grafana_page(self):
diff --git a/services/bots/goalert-bot/test/format.test.ts b/services/bots/goalert-bot/test/format.test.ts
index df15881c6d..f6aaf387fb 100644
--- a/services/bots/goalert-bot/test/format.test.ts
+++ b/services/bots/goalert-bot/test/format.test.ts
@@ -10,18 +10,18 @@ import {
// forwards them in `Details`.
const DATADOG_DETAILS = `https://us5.datadoghq.com/event/event?id=8876996053944573810
-🔴 CRITICAL: 2631.578 (threshold 1800.0)
+🔴 CRITICAL: oldest transaction >= 1800s
-The oldest open transaction on macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com is 2631.578 seconds old. Long transactions hold locks and stop vacuum.
+The oldest open transaction on macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com, "SELECT * FROM events WHERE id = ?", is at least 1800 seconds old. Long transactions hold locks and stop vacuum.
Runbook: https://github.com/macro-inc/macro/blob/main/docs/runbooks/macrodb.md#locks
Dashboard: https://us5.datadoghq.com/dashboard/42x-f55-h6z/macrodb-alerts
@webhook-goalert-notify
-\`avg(last_10m):max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host} > 1800\`
+\`database-monitoring("dbm_type:activity host:macro-db-prod* @duration:>1800s").rollup("count").by("host,@db.statement").last("10m") > 0\`
-Metric value: 2631.578
+Metric value: 1
Metric Graph: https://us5.datadoghq.com/monitors/23404621?group=host · Monitor Status: https://us5.datadoghq.com/monitors/23404621`;
@@ -57,8 +57,8 @@ describe('compact alert posts', () => {
});
expect(renderPost(post)).toEqual({
content: `🔔 · **macro-db-prod long-running transaction — macro-db-prod**
-🔴 critical: 2631.578 (threshold 1800.0)
-The oldest open transaction on macro-db-prod is 2631.578 seconds old. Long transactions hold locks and stop vacuum.
+🔴 critical: oldest transaction >= 1800s
+The oldest open transaction on macro-db-prod, "SELECT * FROM events WHERE id = ?", is at least 1800 seconds old. Long transactions hold locks and stop vacuum.
[Acknowledge](https://oncall.macro-internal.com/alerts/130) · [Graph](https://us5.datadoghq.com/dashboard/42x-f55-h6z/macrodb-alerts) · [Runbook](https://github.com/macro-inc/macro/blob/main/docs/runbooks/macrodb.md#locks)`,
mentions: [],
});