Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
fix(monitoring): include the SQL in the long-running transaction alert (#9317)
Wolf Mermelsteind2a3b2a7f19c
diff --git a/docs/runbooks/macrodb.md b/docs/runbooks/macrodb.mdindex 1ed4428e90..4fde31f698 100644--- a/docs/runbooks/macrodb.md+++ b/docs/runbooks/macrodb.md@@ -87,7 +87,8 @@ Connections blocked on locks, or a transaction open for a long time.- In Database Monitoring's blocking queries view, find the blocking session andits query. Advisory locks (`pg_advisory_xact_lock`) and tuple locks fromconcurrent updates of the same rows have caused past overloads.-- A long open transaction holds its locks and stops vacuum. Find it in+- A long open transaction holds its locks and stops vacuum. The long-running+ transaction alert includes the normalized SQL. Confirm it in`pg_stat_activity` (`xact_start`, `state = 'idle in transaction'`).- Levers: cancel (`pg_cancel_backend`) or terminate (`pg_terminate_backend`)the blocking session once you know what it is; stop the job that keepsdiff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.mdindex d9d5f1cca4..c78f3fef5c 100644--- a/infra/stacks/monitoring/AGENTS.md+++ b/infra/stacks/monitoring/AGENTS.md@@ -33,6 +33,9 @@ From `infra/`:at oncall.macro-internal.com.- A prod monitor's graph on its file's dashboard repeats its metric andthresholds by hand; change both. `routing.test.ts` matches them by title.+ The long-running transaction monitor is a Database Monitoring alert so the+ notification can include the SQL; its graph still plots+ `postgresql.activity.xact_start_age`.- End every monitor message with `notify({ pages })`, which routes only toGoAlert, and pass `dependsOn: goalertWebhooks`. Never mention a channelwebhook, Slack handle or email address: GoAlert posts to the channel. Onlydiff --git a/infra/stacks/monitoring/monitors/macrodb.ts b/infra/stacks/monitoring/monitors/macrodb.tsindex bcc9020726..d0d8fd3933 100644--- a/infra/stacks/monitoring/monitors/macrodb.ts+++ b/infra/stacks/monitoring/monitors/macrodb.ts@@ -276,15 +276,22 @@ prodMonitor('macrodb-blocked-connections', {runbook: 'docs/runbooks/macrodb.md#locks',});+// Database Monitoring query samples, not the host-level xact_start_age gauge:+// that metric has no SQL tag, so the notification could not name the statement.+// The dashboard graph still plots xact_start_age with the same age markers.prodMonitor('macrodb-long-transactions', {title: 'macro-db-prod long-running transaction',service: 'macro-db-prod',+ type: 'database-monitoring alert',query:- 'avg(last_10m):max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host} > 1800',- monitorThresholds: { critical: '1800', warning: '600' },+ 'database-monitoring("dbm_type:activity host:macro-db-prod* @duration:>1800s").rollup("count").by("host,@db.statement").last("10m") > 0',+ monitorThresholds: { critical: '0' },pages: false,+ enableLogsSample: true,+ alertDetail: 'oldest transaction >= 1800s',+ recoveryDetail: 'no transaction over 1800s',description:- 'The oldest open transaction on {{host.name}} is {{value}} seconds old. Long transactions hold locks and stop vacuum.',+ 'The oldest open transaction on {{host.name}}, "{{[@db.statement].name}}", is at least 1800 seconds old. Long transactions hold locks and stop vacuum.',dashboard: dashboard.url,runbook: 'docs/runbooks/macrodb.md#locks',});diff --git a/infra/stacks/monitoring/notify.ts b/infra/stacks/monitoring/notify.tsindex cfc335ef36..97c877f68c 100644--- a/infra/stacks/monitoring/notify.ts+++ b/infra/stacks/monitoring/notify.ts@@ -60,19 +60,28 @@ export function alertMessage({runbook,dashboard,pages,+ alertDetail = '{{value}} (threshold {{threshold}})',+ warningDetail = '{{value}} (warning threshold {{warn_threshold}})',+ recoveryDetail = '{{value}}',}: {description: string;runbook: string;/** A dashboard's `url`, a path such as `/dashboard/abc-def-ghi/title`. */dashboard?: pulumi.Input<string>;pages: boolean;+ /** Text after `CRITICAL:` in the state line. */+ alertDetail?: string;+ /** Text after `WARNING:` in the state line. */+ warningDetail?: string;+ /** Text after `RECOVERED:` in the state line. */+ recoveryDetail?: string;}): pulumi.Output<string> {return pulumi.output(dashboard).apply((path) => {const links = [`Runbook: https://github.com/macro-inc/macro/blob/main/${runbook}`,];if (path !== undefined) links.push(`Dashboard: ${DATADOG_SITE}${path}`);- return `{{#is_alert}}🔴 CRITICAL: {{value}} (threshold {{threshold}}){{/is_alert}}{{#is_warning}}🟠 WARNING: {{value}} (warning threshold {{warn_threshold}}){{/is_warning}}{{#is_recovery}}✅ RECOVERED: {{value}}{{/is_recovery}}+ return `{{#is_alert}}🔴 CRITICAL: ${alertDetail}{{/is_alert}}{{#is_warning}}🟠 WARNING: ${warningDetail}{{/is_warning}}{{#is_recovery}}✅ RECOVERED: ${recoveryDetail}{{/is_recovery}}${description}diff --git a/infra/stacks/monitoring/prod-monitor.ts b/infra/stacks/monitoring/prod-monitor.tsindex 54e13aaac0..9120c6fce7 100644--- a/infra/stacks/monitoring/prod-monitor.ts+++ b/infra/stacks/monitoring/prod-monitor.ts@@ -24,19 +24,30 @@ export function prodMonitor(runbook: string;/** `url` of the dashboard graphing this monitor, under a `thresholdGraph` titled `title`. */dashboard?: pulumi.Input<string>;+ /** Defaults to a metric `query alert`. */+ type?: string;+ /** Attach matching event samples. Needed for Database Monitoring attributes. */+ enableLogsSample?: boolean;+ /** Overrides for the state-line value text in `alertMessage`. */+ alertDetail?: string;+ warningDetail?: string;+ recoveryDetail?: string;}): datadog.Monitor {return new datadog.Monitor(name,{name: `[PROD] ${args.title}`,- type: 'query alert',+ type: args.type ?? 'query alert',query: args.query,message: alertMessage({description: args.description,runbook: args.runbook,dashboard: args.dashboard,pages: args.pages,+ alertDetail: args.alertDetail,+ warningDetail: args.warningDetail,+ recoveryDetail: args.recoveryDetail,}),tags: ['env:prod', `service:${args.service}`],priority: args.pages ? '2' : '3',@@ -47,6 +58,9 @@ export function prodMonitor(onMissingData: 'default',includeTags: true,notifyAudit: false,+ ...(args.enableLogsSample === undefined+ ? {}+ : { enableLogsSample: args.enableLogsSample }),},// The message mentions the GoAlert webhooks, which must exist first.{ dependsOn: goalertWebhooks }diff --git a/infra/stacks/monitoring/routing.test.ts b/infra/stacks/monitoring/routing.test.tsindex 06bac7fdc6..0d9ead019c 100644--- a/infra/stacks/monitoring/routing.test.ts+++ b/infra/stacks/monitoring/routing.test.ts@@ -182,6 +182,12 @@ test('every prod monitor is graphed on its dashboard with its thresholds', () =>);const { critical, warning } = monitor.inputs.monitorThresholds;const anomaly = monitor.inputs.query.includes('anomalies(');+ // The long-running transaction monitor reads DBM query samples so the+ // notification can include the SQL. Its graph still plots xact_start_age.+ const graphThresholds =+ monitor.name === 'macrodb-long-transactions'+ ? { critical: '1800', warning: '600' }+ : { critical, warning };expect({name: monitor.name,markers: graph?.timeseriesDefinition.markers.map(@@ -191,7 +197,21 @@ test('every prod monitor is graphed on its dashboard with its thresholds', () =>name: monitor.name,markers: anomaly? []- : [`y = ${critical}`, ...(warning ? [`y = ${warning}`] : [])],+ : [+ `y = ${graphThresholds.critical}`,+ ...(graphThresholds.warning+ ? [`y = ${graphThresholds.warning}`]+ : []),+ ],});}});++test('the long-running transaction alert names the SQL', () => {+ const monitor = monitors().find(+ (r) => r.name === 'macrodb-long-transactions'+ );+ expect(monitor?.inputs.type).toBe('database-monitoring alert');+ expect(monitor?.inputs.query).toContain('by("host,@db.statement")');+ expect(monitor?.inputs.message).toContain('{{[@db.statement].name}}');+});diff --git a/infra/stacks/observability/nixos/goalert/test_macro_alerts_relay.py b/infra/stacks/observability/nixos/goalert/test_macro_alerts_relay.pyindex 715dbf9625..b6ec39a5d8 100644--- a/infra/stacks/observability/nixos/goalert/test_macro_alerts_relay.py+++ b/infra/stacks/observability/nixos/goalert/test_macro_alerts_relay.py@@ -11,18 +11,18 @@ spec.loader.exec_module(relay)DATADOG_DETAILS = """https://us5.datadoghq.com/event/event?id=8876996053944573810-🔴 CRITICAL: 2631.578 (threshold 1800.0)+🔴 CRITICAL: oldest transaction >= 1800s-The oldest open transaction on macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com is 2631.578 seconds old. Long transactions hold locks and stop vacuum.+The oldest open transaction on macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com, "SELECT * FROM events WHERE id = ?", is at least 1800 seconds old. Long transactions hold locks and stop vacuum.Runbook: https://github.com/macro-inc/macro/blob/main/docs/runbooks/macrodb.md#locksDashboard: https://us5.datadoghq.com/dashboard/42x-f55-h6z/macrodb-alerts@webhook-goalert-notify-`avg(last_10m):max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host} > 1800`+`database-monitoring("dbm_type:activity host:macro-db-prod* @duration:>1800s").rollup("count").by("host,@db.statement").last("10m") > 0`-Metric value: 2631.578+Metric value: 1Metric Graph: https://us5.datadoghq.com/monitors/23404621?group=host · Monitor Status: https://us5.datadoghq.com/monitors/23404621"""@@ -57,8 +57,8 @@ class AlertPostTest(unittest.TestCase):})self.assertEqual(post, """---🔔 · **macro-db-prod long-running transaction — macro-db-prod**-🔴 critical: 2631.578 (threshold 1800.0)-The oldest open transaction on macro-db-prod is 2631.578 seconds old. Long transactions hold locks and stop vacuum.+🔴 critical: oldest transaction >= 1800s+The oldest open transaction on macro-db-prod, "SELECT * FROM events WHERE id = ?", is at least 1800 seconds old. Long transactions hold locks and stop vacuum.[Acknowledge](https://oncall.macro-internal.com/alerts/130) · [Graph](https://us5.datadoghq.com/dashboard/42x-f55-h6z/macrodb-alerts) · [Runbook](https://github.com/macro-inc/macro/blob/main/docs/runbooks/macrodb.md#locks)""")def test_grafana_page(self):diff --git a/services/bots/goalert-bot/test/format.test.ts b/services/bots/goalert-bot/test/format.test.tsindex df15881c6d..f6aaf387fb 100644--- a/services/bots/goalert-bot/test/format.test.ts+++ b/services/bots/goalert-bot/test/format.test.ts@@ -10,18 +10,18 @@ import {// forwards them in `Details`.const DATADOG_DETAILS = `https://us5.datadoghq.com/event/event?id=8876996053944573810-🔴 CRITICAL: 2631.578 (threshold 1800.0)+🔴 CRITICAL: oldest transaction >= 1800s-The oldest open transaction on macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com is 2631.578 seconds old. Long transactions hold locks and stop vacuum.+The oldest open transaction on macro-db-prod.ctkwyjgndnfr.us-east-1.rds.amazonaws.com, "SELECT * FROM events WHERE id = ?", is at least 1800 seconds old. Long transactions hold locks and stop vacuum.Runbook: https://github.com/macro-inc/macro/blob/main/docs/runbooks/macrodb.md#locksDashboard: https://us5.datadoghq.com/dashboard/42x-f55-h6z/macrodb-alerts@webhook-goalert-notify-\`avg(last_10m):max:postgresql.activity.xact_start_age{host:macro-db-prod*} by {host} > 1800\`+\`database-monitoring("dbm_type:activity host:macro-db-prod* @duration:>1800s").rollup("count").by("host,@db.statement").last("10m") > 0\`-Metric value: 2631.578+Metric value: 1Metric Graph: https://us5.datadoghq.com/monitors/23404621?group=host · Monitor Status: https://us5.datadoghq.com/monitors/23404621`;@@ -57,8 +57,8 @@ describe('compact alert posts', () => {});expect(renderPost(post)).toEqual({content: `🔔 · **macro-db-prod long-running transaction — macro-db-prod**-🔴 critical: 2631.578 (threshold 1800.0)-The oldest open transaction on macro-db-prod is 2631.578 seconds old. Long transactions hold locks and stop vacuum.+🔴 critical: oldest transaction >= 1800s+The oldest open transaction on macro-db-prod, "SELECT * FROM events WHERE id = ?", is at least 1800 seconds old. Long transactions hold locks and stop vacuum.[Acknowledge](https://oncall.macro-internal.com/alerts/130) · [Graph](https://us5.datadoghq.com/dashboard/42x-f55-h6z/macrodb-alerts) · [Runbook](https://github.com/macro-inc/macro/blob/main/docs/runbooks/macrodb.md#locks)`,mentions: [],});