Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(monitoring): alert when native call pushes fail at SNS (#8933)

Peter Chinmanad9bc3abeb19
diff --git a/crates/notification/src/domain/service/voip.rs b/crates/notification/src/domain/service/voip.rs
index b9bfeaae49..24a8dd2866 100644
--- a/crates/notification/src/domain/service/voip.rs
+++ b/crates/notification/src/domain/service/voip.rs
@@ -113,6 +113,7 @@ where
match mobile.send_voip_push(arn, &user_id, payload).await {
Ok(_) => Some((user_id, ios)),
Err(e) => {
+ // infra/stacks/monitoring/monitors/calls.ts alerts on this message.
tracing::error!(
error=?e,
"voip push: SNS delivery failed"
diff --git a/infra/stacks/monitoring/AGENTS.md b/infra/stacks/monitoring/AGENTS.md
index bab17a84d0..e83f993df0 100644
--- a/infra/stacks/monitoring/AGENTS.md
+++ b/infra/stacks/monitoring/AGENTS.md
@@ -25,3 +25,6 @@ From `infra/`:
- `monitors/whatsapp-session.ts` reads log events by `telemetry_event`; its
header is the field contract with `services/whatsapp_session_service`.
Renaming a field there silently turns its monitor into missing data.
+- `monitors/calls.ts` matches the text of the `voip push: SNS delivery failed`
+ log in `crates/notification/src/domain/service/voip.rs` and the
+ `PushEndpointUnavailable` message. Rewording either silently breaks it.
diff --git a/infra/stacks/monitoring/README.md b/infra/stacks/monitoring/README.md
index 3a65e55b6a..14677af3c4 100644
--- a/infra/stacks/monitoring/README.md
+++ b/infra/stacks/monitoring/README.md
@@ -122,6 +122,15 @@ Neither new alert detects successful responses that apply no edits, or failures
during periods without editing requests. The end-to-end canary remains a
follow-up.
+## Native call push coverage
+
+`monitors/calls.ts` adds one log monitor that alerts on any rejected iOS or
+Android call push, excluding stale device registrations. A `pulumi preview`
+after it is added should show that one create and nothing else.
+
+It sees only publishes SNS rejects. A push that SNS accepts and the device then
+drops leaves no backend trace.
+
## GraphQL soup rollout coverage
`monitors/graphql-soup.ts` declares 15 new monitors for the `enable-graphql-soup`
diff --git a/infra/stacks/monitoring/index.ts b/infra/stacks/monitoring/index.ts
index f02f2df0b0..9be3ec78f4 100644
--- a/infra/stacks/monitoring/index.ts
+++ b/infra/stacks/monitoring/index.ts
@@ -19,6 +19,7 @@ import './synthetics';
// Declared here first.
import './monitors/ai-editing';
+import './monitors/calls';
import './monitors/graphql-soup';
import './monitors/whatsapp-session';
import { notificationRetentionDashboard } from './monitors/notification-retention';
diff --git a/infra/stacks/monitoring/monitors/calls.ts b/infra/stacks/monitoring/monitors/calls.ts
new file mode 100644
index 0000000000..91089c3414
--- /dev/null
+++ b/infra/stacks/monitoring/monitors/calls.ts
@@ -0,0 +1,44 @@
+// Native call pushes (crates/notification/src/domain/service/voip.rs).
+//
+// New monitor, not adopted from the UI, so it is not `protect`ed.
+//
+// cloud-storage-service (document_storage_service, the only service that
+// builds the VoIP push sender) publishes iOS PushKit and Android FCM call
+// pushes itself and logs "voip push: SNS delivery failed" for each rejected publish.
+// The callee then gets an ordinary alert instead of a ringing phone, and
+// nothing else errors. Every Android call push failed this way until #8880
+// granted the task role `sns:Publish` on the FCM platform application: 11
+// failures in the 30 days to 2026-10-27, far below the DSS error-log monitor's
+// 150 per 5 minutes. Call volume is that low, so a single failure alerts.
+//
+// Pushes to stale device registrations are expected after an uninstall. SNS
+// rejects them as disabled or missing endpoints, and `@error` carries the
+// `PushEndpointUnavailable` context text, so they are excluded.
+import * as datadog from '@pulumi/datadog';
+
+export const nativeCallPushFailing = new datadog.Monitor(
+ 'native-call-push-failing',
+ {
+ name: '[PROD] Native call pushes failing',
+ type: 'log alert',
+ query:
+ 'logs("service:cloud-storage-service env:prod \\"voip push: SNS delivery failed\\" -@error:*disabled?or?no?longer?exists*").index("*").rollup("count").last("1h") >= 1',
+ message: `SNS rejected a native call push, so the callee's phone did not ring.
+
+The endpoint ARN in \`@error\` names the platform (\`GCM\` is Android,
+\`APNS_VOIP\` is iOS). \`AuthorizationError\` means the cloud-storage-service
+task role lacks \`sns:Publish\` on that platform application: add it to
+\`snsPlatformArns\` in infra/stacks/cloud-storage-service/index.ts.
+
+- [Failure logs](https://us5.datadoghq.com/logs?query=service%3Acloud-storage-service%20env%3Aprod%20%22voip%20push%3A%20SNS%20delivery%20failed%22)
+
+@webhook-macro-alert-hook`,
+ tags: ['env:prod', 'service:cloud-storage-service'],
+ monitorThresholds: { critical: '1' },
+ onMissingData: 'default',
+ includeTags: false,
+ notifyAudit: false,
+ enableLogsSample: true,
+ groupbySimpleMonitor: false,
+ }
+);