Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
feat(monitoring): alert when native call pushes fail at SNS (#8933)
Peter Chinmanad9bc3abeb19
diff --git a/crates/notification/src/domain/service/voip.rs b/crates/notification/src/domain/service/voip.rsindex b9bfeaae49..24a8dd2866 100644--- a/crates/notification/src/domain/service/voip.rs+++ b/crates/notification/src/domain/service/voip.rs@@ -113,6 +113,7 @@ wherematch mobile.send_voip_push(arn, &user_id, payload).await {Ok(_) => Some((user_id, ios)),Err(e) => {+ // infra/stacks/monitoring/monitors/calls.ts alerts on this message.tracing::error!(error=?e,"voip push: SNS delivery failed"diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.mdindex bab17a84d0..e83f993df0 100644--- a/infra/stacks/monitoring/AGENTS.md+++ b/infra/stacks/monitoring/AGENTS.md@@ -25,3 +25,6 @@ From `infra/`:- `monitors/whatsapp-session.ts` reads log events by `telemetry_event`; itsheader is the field contract with `services/whatsapp_session_service`.Renaming a field there silently turns its monitor into missing data.+- `monitors/calls.ts` matches the text of the `voip push: SNS delivery failed`+ log in `crates/notification/src/domain/service/voip.rs` and the+ `PushEndpointUnavailable` message. Rewording either silently breaks it.diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.mdindex 3a65e55b6a..14677af3c4 100644--- a/infra/stacks/monitoring/README.md+++ b/infra/stacks/monitoring/README.md@@ -122,6 +122,15 @@ Neither new alert detects successful responses that apply no edits, or failuresduring periods without editing requests. The end-to-end canary remains afollow-up.+## Native call push coverage++`monitors/calls.ts` adds one log monitor that alerts on any rejected iOS or+Android call push, excluding stale device registrations. A `pulumi preview`+after it is added should show that one create and nothing else.++It sees only publishes SNS rejects. A push that SNS accepts and the device then+drops leaves no backend trace.+## GraphQL soup rollout coverage`monitors/graphql-soup.ts` declares 15 new monitors for the `enable-graphql-soup`diff --git a/infra/stacks/monitoring/index.ts b/infra/stacks/monitoring/index.tsindex f02f2df0b0..9be3ec78f4 100644--- a/infra/stacks/monitoring/index.ts+++ b/infra/stacks/monitoring/index.ts@@ -19,6 +19,7 @@ import './synthetics';// Declared here first.import './monitors/ai-editing';+import './monitors/calls';import './monitors/graphql-soup';import './monitors/whatsapp-session';import { notificationRetentionDashboard } from './monitors/notification-retention';diff --git a/infra/stacks/monitoring/monitors/calls.ts b/infra/stacks/monitoring/monitors/calls.tsnew file mode 100644index 0000000000..91089c3414--- /dev/null+++ b/infra/stacks/monitoring/monitors/calls.ts@@ -0,0 +1,44 @@+// Native call pushes (crates/notification/src/domain/service/voip.rs).+//+// New monitor, not adopted from the UI, so it is not `protect`ed.+//+// cloud-storage-service (document_storage_service, the only service that+// builds the VoIP push sender) publishes iOS PushKit and Android FCM call+// pushes itself and logs "voip push: SNS delivery failed" for each rejected publish.+// The callee then gets an ordinary alert instead of a ringing phone, and+// nothing else errors. Every Android call push failed this way until #8880+// granted the task role `sns:Publish` on the FCM platform application: 11+// failures in the 30 days to 2026-10-27, far below the DSS error-log monitor's+// 150 per 5 minutes. Call volume is that low, so a single failure alerts.+//+// Pushes to stale device registrations are expected after an uninstall. SNS+// rejects them as disabled or missing endpoints, and `@error` carries the+// `PushEndpointUnavailable` context text, so they are excluded.+import * as datadog from '@pulumi/datadog';++export const nativeCallPushFailing = new datadog.Monitor(+ 'native-call-push-failing',+ {+ name: '[PROD] Native call pushes failing',+ type: 'log alert',+ query:+ 'logs("service:cloud-storage-service env:prod \\"voip push: SNS delivery failed\\" -@error:*disabled?or?no?longer?exists*").index("*").rollup("count").last("1h") >= 1',+ message: `SNS rejected a native call push, so the callee's phone did not ring.++The endpoint ARN in \`@error\` names the platform (\`GCM\` is Android,+\`APNS_VOIP\` is iOS). \`AuthorizationError\` means the cloud-storage-service+task role lacks \`sns:Publish\` on that platform application: add it to+\`snsPlatformArns\` in infra/stacks/cloud-storage-service/index.ts.++- [Failure logs](https://us5.datadoghq.com/logs?query=service%3Acloud-storage-service%20env%3Aprod%20%22voip%20push%3A%20SNS%20delivery%20failed%22)++@webhook-macro-alert-hook`,+ tags: ['env:prod', 'service:cloud-storage-service'],+ monitorThresholds: { critical: '1' },+ onMissingData: 'default',+ includeTags: false,+ notifyAudit: false,+ enableLogsSample: true,+ groupbySimpleMonitor: false,+ }+);