Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
observability
What it is
The Ohio (us-east-2) Grafana host: Grafana, Loki, Tempo, Prometheus, Alloy and
the GoAlert on-call pager on one NixOS EC2 instance, separate from production in
us-east-1. README.md is the full operations guide.
Start here
nixos/host.nix: boot order, retained volume, stable uids, service hardening.nixos/services.nix,nixos/goalert.nix: the systemd services and credentials.index.ts: VPC, ALB, certificate, DNS, alarms and EC2 user data.
Recipes
- Change a Grafana prod alert rule: edit its entry in
nixos/prod-alerts.nix(rules and theProd alertsdashboard panel both read it), and its Datadog copy ininfra/stacks/monitoring/monitors/by hand. A new entry takes the next freepanelId; never renumber one, since sent notifications link to it. - Change a CI percentile, queue or CI hours query: edit
nixos/ci-report.json; theMacro CIdashboard's Loki panels andscripts/ci-report.tsboth read it. - Add a runtime setting:
settings.ts→render.ts(bump the schema version) →nixos/read-user-data.py(keys, pattern, version) → the consuming.nixfile. - Add a secret:
assets/refresh-secrets.py(validation) →LoadCredentialin the unit → README secret list. Add the unit tosecretConsumersinhost.nix. - Upgrade GoAlert: bump
versionand the hashes innixos/goalert/package.nix.
Checks
From this directory:
nix build --dry-run .#nixosConfigurations.observability.config.system.build.toplevelnix fmt -- <changed .nix files>;python3 -m py_compileon changed Python.python3 -m unittest test_macro_alerts_relayfromnixos/goalert/when the relay's channel post format changes.
From infra/: bun run check, then Biome on changed .ts.
Gotchas
- Changes ship only as a new AMI plus an approved deploy; never
pulumi upor publish an image without explicit permission. - The host reads secrets from the stack's own Secrets Manager secret in
us-east-2, so it never depends on production to boot. Doppler
observability/prdholds the values; copy changes into the secret. - A service listed in
host.nixservicesgetsProtectSystem=strict, metadata blocking and the bootstrap dependency; one that needs the instance role must be in theIPAddressDenyexception list or run as its own unit. - GoAlert's application config lives in its database;
goalert/apply-config.pyoverwrites it on every start fromgoalert.nix.set-configaccepts invalid values silently, so check the journal forvalidate config. - Grafana keeps a provisioned contact point removed from
alerting.nix; list its uid indeleteContactPointsto delete it. - Grafana provisioning expands
$VARin contact point settings, but not in rule queries or annotations, so$Band$valueswork there and secrets must come in through the Grafana unit's environment.