Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
monitoring
What it is
Org-global Datadog monitors, synthetics, and rollout dashboards. This stack is
manual and has only a prod stack, even for monitors covering both environments.
Prod threshold and anomaly monitors are in monitors/{macrodb,redis,queues,regressions,outlook-sync}.ts
(prodMonitor() in prod-monitor.ts), each file with a dashboard graphing its
monitors that their messages link to; every monitor notifies through notify.ts.
dashboards/web-performance.ts holds the web app performance dashboard with the
APM retention filter and span-based metrics it reads; it has no monitors.
Checks
From infra/:
bun test --isolate stacks/monitoring(routing and notification retention tests).--isolatematters: the Pulumi mock tests each import modules that register resources, and a shared module cache hides them from the second file.bun run check- Biome on changed files.
- An authenticated
pulumi previewmay change adopted monitors only inmessageandrenotifyInterval, apart from the exceptions README.md lists with the expected counts.
Gotchas
- The CloudWatch-based monitors in
macrodb.ts,redis.tsandqueues.tshave hand-written Grafana copies ininfra/stacks/observability/nixos/prod-alerts.nix. Change thresholds and windows in both. Grafana pages for those, so their Datadog copies usepages: false. - Read
README.mdbefore deployment: imported resources must be adopted first to avoid duplicate alerts. Never deploy without explicit approval. - Deploying needs the
goalertIntegrationKeyandgoalertNotifyIntegrationKeysecrets in stack config, and GoAlert (infra/stacks/observability) reachable at oncall.macro-internal.com. - A prod monitor's graph on its file's dashboard repeats its metric and
thresholds by hand; change both.
routing.test.tsmatches them by title. The long-running transaction monitor takes the lowestpostgresql.activity.xact_start_ageover 5 minutes, and its graph plots the samemoving_rollup(..., 300, 'min'). A single-sample spike from notification-service'sLISTENconnection fires hourly otherwise (docs/runbooks/macrodb.md#locks). - End every monitor message with
notify({ pages }), which routes only to GoAlert, and passdependsOn: goalertWebhooks. Never mention a channel webhook, Slack handle or email address: GoAlert posts to the channel. Onlygoalert-healthposts to@webhook-macro-alert-hookdirectly.routing.test.tsenforces it. - Notification retention adds nine monitors, one dashboard and a downtime muting the prod missing-telemetry alarm; delete the downtime once DSS health instrumentation is deployed. Missing telemetry is deliberately an alert, not zero lag.
- Do not use native committed-offset lag for the independent notification consumers. Their per-instance log measurements preserve broadcast semantics.
notificationRetentionDashboardUrlis a stack output, not an app route.dashboards/web-performance.tsreads span attributes set byapps/web/src/observability(the span model indocs/agent-guide/observability.md). Renaming one there silently empties its widgets. Span-based metric names and computes cannot be changed after creation; add a new metric instead.monitors/whatsapp-session.tsreads log events bytelemetry_event; its header is the field contract withservices/whatsapp_session_service. Renaming a field there silently turns its monitor into missing data.monitors/harness-presence.tsreadsharness_presence_disagreementlogged bycrates/harnesses/src/inbound/axum_router.rs; its header is the field contract.monitors/outlook-sync.tsreads the health log, sync spans and Graph and token span fields email-service emits; its header is the field contract. It finds the Graph and token spans by@outlook.throttle_originand@outlook.token_cache, not by name: Datadog renames an HTTP client span tohttp.client.requestwith its method as the resource.monitors/calls.tsmatches the text of thevoip push: SNS delivery failedlog incrates/notification/src/domain/service/voip.rsand thePushEndpointUnavailablemessage. Rewording either silently breaks it.