Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(infra): add native NixOS Grafana observability pilot in Ohio (#7783)

Wolf Mermelsteind9ca08c352cf
diff --git a/infra/README.md b/infra/README.md
index 6ddd5a0c4f..52bb3a5e07 100644
--- a/infra/README.md
+++ b/infra/README.md
@@ -5,7 +5,7 @@ This monorepo contains the code for all Macro Cloud IaC.
## IaC
Infrastructure is written with [pulumi](https://www.pulumi.com/docs/) with typescript and bun.
-We use AWS for all infrastructure so ask to be added to the aws org and setup [pulumi with aws](https://www.pulumi.com/registry/packages/aws/installation-configuration/). All of our infra is on `us-east-1`.
+We use AWS for all infrastructure so ask to be added to the aws org and setup [pulumi with aws](https://www.pulumi.com/registry/packages/aws/installation-configuration/). Production infrastructure is in `us-east-1`; the [observability pilot](stacks/observability/README.md) runs independently in `us-east-2`.
The stacks directory represents the infrastructure for each service and pulumi should be run from inside a stack directory. From `stacks/my-service/` run `pulumi up --stack [dev | prod]` to deploy.
diff --git a/infra/bun.lock b/infra/bun.lock
index 475c98989a..550909f839 100644
--- a/infra/bun.lock
+++ b/infra/bun.lock
@@ -187,6 +187,10 @@
"name": "notification-service-stack",
"version": "0.0.0",
},
+ "stacks/observability": {
+ "name": "observability-stack",
+ "version": "0.0.0",
+ },
"stacks/opensearch": {
"name": "opensearch-stack",
"version": "0.0.0",
@@ -753,6 +757,8 @@
"npm-run-path": ["[email protected]", "", { "dependencies": { "path-key": "^3.0.0" } }, "sha512-S48WzZW777zhNIrn7gxOlISNAqi9ZC/uQFnRdbeIHhZhCA6UqpkOT8T1G7BvfdgP4Er8gF4sUbaS0i7QvIfCWw=="],
+ "observability-stack": ["observability-stack@workspace:stacks/observability"],
+
"onetime": ["[email protected]", "", { "dependencies": { "mimic-fn": "^2.1.0" } }, "sha512-kbpaSSGJTWdAY5KPVeMOKXSrPtr8C8C7wodJbcsd51jRnmD+GZu8Y0VoU6Dm5Z4vWr0Ig/1NKuWRKf7j5aaYSg=="],
"opensearch-stack": ["opensearch-stack@workspace:stacks/opensearch"],
diff --git a/infra/stacks/observability/Pulumi.yaml b/infra/stacks/observability/Pulumi.yaml
new file mode 100644
index 0000000000..2c09586032
--- /dev/null
+++ b/infra/stacks/observability/Pulumi.yaml
@@ -0,0 +1,3 @@
+name: observability
+runtime: nodejs
+description: Single-host Grafana pilot alongside Datadog
diff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.md
new file mode 100644
index 0000000000..5a5cff6c2e
--- /dev/null
+++ b/infra/stacks/observability/README.md
@@ -0,0 +1,359 @@
+# Observability pilot
+
+This stack starts Grafana alongside Datadog on one private EC2 instance. It
+runs Grafana, Loki, Tempo, Prometheus, Alloy and nginx as native systemd services
+on a prebuilt NixOS image. There is no Docker or Compose dependency. Nix owns
+packages, configuration, service users, resource limits and startup dependencies:
+
+- `nixos/services.nix`: native application services and runtime credentials.
+- `nixos/grafana.nix` and `nixos/datasources.nix`: Grafana access policy and datasources.
+- `nixos/backend-config.nix`: Loki, Tempo and Prometheus settings.
+- `nixos/ingest.alloy`: unprivileged OTLP receiver and export pipeline.
+- `nixos/host-telemetry.nix` and `nixos/host.alloy`: host metrics and journal logs.
+- `nixos/cloudwatch.nix`: independent disk and memory health metrics.
+- `nixos/proxy.nix`: native nginx virtual hosts and query-only backend gateway.
+- `nixos/host.nix`: boot dependencies, retained storage and stable service identities.
+
+NixOS modules generate application configuration from Nix attributes. Alloy's
+pipelines live in native `.alloy` files read by Nix. Pulumi sends only
+validated runtime values such as hostnames, bucket names and approved identities.
+Datadog instrumentation, collection and alerts remain unchanged. Nothing sends
+application telemetry here until a subsequent dual-export change is deployed.
+
+## Architecture and storage
+
+```text
+Team browser -- HTTPS --> ALB --> nginx --> Grafana -- Google Workspace OAuth
+ --> query proxy --> backends
+OTEL exporter -- HTTPS + bearer token --> ALB --> nginx --> Alloy
+ --> Loki --> S3 logs
+ --> Tempo --> S3 traces
+ --> Prometheus --> EBS
+```
+
+- One `m7i.xlarge` (4 vCPU / 16 GiB) in a new VPC's private `us-east-2a`
+ subnet in Ohio, separate from production in `us-east-1`. This is a provisional
+ pilot size, not a capacity commitment.
+- Independent regional dependencies: two public ALB subnets, one NAT gateway,
+ a regional ACM certificate, EBS/snapshots, S3, Secrets Manager and alarm topic.
+ An S3 gateway endpoint keeps bucket traffic off the NAT gateway. No production
+ VPC, peering, NAT or regional certificate is reused. Route53 and IAM are global.
+- Separate encrypted 300 GiB gp3 volume: Grafana SQLite and plugins,
+ Prometheus TSDB, Alloy metrics WAL, Loki WAL/index/cache, Tempo WAL/blocks.
+ The root disk is disposable. Volume and S3 buckets are protected and retained.
+- Two private, encrypted S3 buckets, scoped instance-role access, HTTPS required.
+ Loki/Tempo compactors own retention; no lifecycle rule can expire live blocks.
+- Defaults: 30 days of logs, 7 days of traces, metrics up to 30 days or 100 GB,
+ whichever is reached first. WAL/cache usage is additional, not covered by the
+ metrics limit. Review ingestion volume and disk growth before broad rollout.
+- Daily EBS snapshots retained for seven days; these are crash-consistent, not
+ an application-consistent or cross-region backup. Snapshot recovery can lose
+ up to a day of local state. S3 does not protect telemetry still buffered locally.
+- No RDS, shared filesystem, Kafka or cluster. Grafana's SQLite stores users,
+ dashboards and settings, not log/trace/metric history.
+- All service and OS versions are pinned by this stack's `flake.lock` on NixOS
+ 26.05. Grafana plugin auto-install/update is disabled. Reviewed lock updates
+ and replacement AMIs deliver upgrades; there is no installation or build at boot.
+
+Previously received telemetry remains accessible if the production region fails.
+Data that has not left production can still be lost, and a single Ohio host is
+not highly available. Ohio failure is not covered by these regional snapshots
+or buckets. Subsequent dual export must use independent queues and HTTPS across
+regions, and account for inter-region transfer costs and latency. A production
+region outage should not block Grafana login or secret retrieval in Ohio.
+
+## Host and service collection
+
+A native Alloy service collects CPU, memory, filesystem and network metrics
+using its node exporter, including systemd unit state, tasks, restart counts and
+start times. It reads selected application/systemd service logs and kernel logs
+from the journal. Metrics go to Prometheus; logs go to Loki. Container discovery
+and cAdvisor are unnecessary because the stack runs directly on systemd. Other
+production hosts and application OTEL dual export are subsequent work.
+
+The collector runs as a dedicated unprivileged user with journal access, separate
+from the unprivileged OTLP ingress service. Its UI binds only to loopback, EC2
+metadata access is blocked, and its memory limit is 1 GiB. State is retained on
+EBS under `alloy`; the metrics WAL retains at most one hour of unsent samples.
+Log delivery uses bounded retries and does not guarantee lossless delivery
+through a long outage. Nginx access logs use a restricted format without query
+strings and flow through journald into Loki, as do service errors. Host Alloy
+uses the host mount namespace so filesystem metrics reflect the actual mounts;
+Unix permissions restrict writes to its own retained directory.
+
+CloudWatch Agent runs as a separate unprivileged NixOS service, independent of
+the data mount and the application stack. It sends only root/data-disk utilization and memory
+utilization to the `Macro/Observability` namespace using the instance role.
+Alarms identify the current EC2 instance: either ext4 disk above 80% or memory
+above 90% over two five-minute periods notifies the configured SNS topic.
+Missing data breaches the alarm, including when the data volume is not mounted.
+CloudWatch collection runs every minute. No custom disk polling script remains.
+
+Both native agent versions are pinned by `flake.lock`; their NixOS modules own
+startup and restart behavior. Verify dashboards and alarms with real EC2 data
+before enabling broad production ingestion.
+
+## Team authentication and security
+
+Production URLs are `https://grafana.macro-internal.com` and `https://otlp.macro-internal.com`;
+dev uses `grafana-dev.macro-internal.com` and `otlp-dev.macro-internal.com`.
+
+Grafana uses a dedicated **Google Workspace OAuth client**. Only explicitly
+approved lowercase `@macro.com` emails in `allowedEmails` may sign in; membership
+in the domain alone is insufficient. Named `adminEmails` get Grafana server
+administrator access; other approved users are Viewers. Admins must also appear
+in the allowlist. Unknown identities map to an invalid role and are denied with
+`role_attribute_strict`. Google hosted-domain validation, ID-token signature
+validation, PKCE and refresh-token checking are enabled. Workspace administrators
+must enforce MFA for the approved accounts. No password login, default admin,
+anonymous access or public signup is enabled.
+
+Sessions have a one-hour inactivity limit and eight-hour maximum. Removing an
+email from configuration blocks the next login; it does **not** immediately
+invalidate an existing session. For offboarding, first disable the user and
+revoke their sessions in Grafana using another admin, then update the allowlist.
+Keep at least two admins. This uses OSS role mapping, not Enterprise team sync.
+All approved users can query the pilot's telemetry; there is no per-service data
+isolation. Apply redaction in the producer pipeline before copying sensitive data.
+
+Only the ALB is internet-facing, on TLS 1.2/1.3 port 443. The host has no public IP,
+SSH key or SSH ingress. Its only ingress is port 8080 from the ALB security group.
+ALB-to-host traffic is HTTP inside the VPC. Administrative access uses AWS SSM and
+the operator's IAM identity. Grafana, Loki, Tempo, Prometheus and both Alloy
+services bind to loopback. The default ALB action is 404. No HTTP listener is opened.
+
+Grafana data sources use a separate internal nginx listener on port 8081 that
+allows only named query endpoints and their required HTTP methods. This matters
+because Viewers can call Grafana's data-source proxy: direct backend URLs would
+also expose maintenance endpoints such as Tempo `/shutdown` and Loki `/flush`
+(including mutating GET requests). Port 8081 binds only to loopback, and
+Alloy's ingestion path is separate. New data-source features may require reviewed
+additions to this query allowlist.
+
+Ingestion is a separate hostname permitting only POST to `/v1/logs`, `/v1/traces`
+and `/v1/metrics`. Alloy validates a bearer token independently of browser login.
+The token cannot query telemetry through that endpoint. Requests have an 8 MiB
+limit and a shared 100 requests/second limit with a 200-request burst. Only use
+this token in trusted server-side collectors, never browser or mobile clients.
+The initial token is shared; provision per-producer credentials before expanding
+beyond trusted internal services.
+
+Secrets Manager holds Google client credentials, the Grafana encryption key and
+the ingestion token. EC2 retrieves them at startup into `/run` with restricted
+file permissions. Systemd `LoadCredential` gives Grafana only its three credentials
+and ingress Alloy only the OTLP token. Secret values never enter Pulumi state,
+user-data, environment variables or configuration committed here. Grafana's nonsecret host
+and approved-user role expression are supplied through environment variables;
+the generated INI refers to them through Grafana's environment provider. Secrets
+remain file references in the Nix store, never secret values. The secret must use the
+AWS-managed Secrets Manager encryption key; a customer-managed key needs an
+explicit scoped KMS policy addition. IMDSv2 is required with hop limit 1. Only Loki, Tempo, the bootstrap/secret helper,
+CloudWatch and SSM need the instance role; application units without an AWS
+requirement block metadata access. The host is still one administrative trust
+boundary; split roles/hosts when stronger isolation is required.
+
+## First deployment
+
+The stack is deliberately absent from `.github/services-config.json`, so this PR
+does not enroll it in automatic deployments. Use the repository's Pulumi backend
+and AWS account `569036502058`, region `us-east-2`. Deployment requires the
+existing public `macro-internal.com` Route53 zone; this stack creates its own VPC/NAT and
+DNS-validated regional ACM certificate. Region validation rejects `us-east-1`.
+
+1. Create a dedicated Google OAuth Web application under the company's Google
+ organization with an **Internal** consent audience. Register exactly
+ `https://grafana-dev.macro-internal.com/login/google` for dev or
+ `https://grafana.macro-internal.com/login/google` for prod. Prefer separate clients and
+ secrets per environment. Confirm Workspace MFA enforcement.
+2. Through the approved secret-management process, create a Secrets Manager JSON
+ secret in **us-east-2** containing `google_client_id`, `google_client_secret`,
+ `grafana_secret_key`, and `otlp_token`. Generate independent cryptographically
+ random values of at least 32 characters for the last two. Preserve
+ `grafana_secret_key` through recovery; changing it can make stored credentials
+ unreadable. Never paste secret values into shell commands, this file or PRs.
+3. Select approved users and admins. Select an existing **us-east-2** SNS topic
+ with a confirmed, monitored subscription for infrastructure alarms. Its
+ delivery destination must remain accessible during a production outage.
+ Configuration rejects secrets and topics in another region; the host reads
+ the local secret directly, without fetching credentials from production.
+4. Build, publish and smoke-boot the NixOS image as described below. Set its
+ reviewed Ohio AMI ID; there is deliberately no mutable "latest image" lookup.
+5. Set the nonsecret configuration below, substituting actual identifiers. No
+ example account or placeholder is authorized automatically.
+
+```bash
+\cd infra
+bun install --frozen-lockfile
+\cd stacks/observability
+pulumi stack select macro-inc/dev --create
+pulumi config set aws:region us-east-2
+pulumi config set amiId '<reviewed NixOS AMI ID in Ohio>'
+pulumi config set secretArn '<existing Secrets Manager ARN>'
+pulumi config set alarmTopicArn '<existing monitored SNS topic ARN>'
+pulumi config set --path 'allowedEmails[0]' '<[email protected]>'
+pulumi config set --path 'allowedEmails[1]' '<[email protected]>'
+pulumi config set --path 'adminEmails[0]' '<[email protected]>'
+pulumi preview --diff
+```
+
+Deploy with `pulumi up` after
+reviewing the resource plan. Allow up to 15 minutes for bootstrap.
+Pulumi resource creation does not prove bootstrap or OAuth has succeeded.
+Do not change the region of a stack that already owns resources: that requires
+an explicit migration and recovery plan for its protected storage.
+
+Before enabling application traffic, validate all of these against AWS:
+
+- Approved Viewer and Admin can log in; an unapproved Workspace account and a
+ personal Google account cannot. Viewer cannot administer Grafana.
+- All three data sources pass their health checks. A small OTLP fixture appears
+ in logs, traces and metrics with the expected `service.name`.
+- Missing/wrong tokens fail; valid token cannot read backend APIs. No backend
+ ports or SSH are reachable directly.
+- S3 objects appear after flushing; IAM uses the expected two buckets only.
+- Reboot preserves data and starts services; stop/replace the host and verify
+ the existing volume is reattached. Restore a snapshot to an isolated host and
+ verify SQLite/Prometheus/WAL recovery before relying on the backup.
+- Host/systemd metrics and service logs appear in Grafana. CloudWatch's
+ instance, target-health, root/data-disk and memory alarms reach the selected
+ SNS subscription. Verify disk metrics use `InstanceId`, `path` and `fstype`
+ dimensions, memory uses `InstanceId`, and missing data alerts. Confirm a daily snapshot is created. Inspect disk/queue
+ growth before increasing ingestion. These alarms do not cover every data-path
+ failure; keep Datadog primary and add end-to-end ingestion checks next.
+
+## Operations and recovery
+
+### NixOS image build and publication
+
+From `infra/stacks/observability`, on an x86_64 Linux Nix builder:
+
+```bash
+nix build . --cores 2
+python3 nixos/publish-image.py result '<private Ohio image-artifact bucket>'
+```
+
+The build produces a UEFI VHD with the complete host configuration. KVM speeds up
+image creation; software emulation is supported but slower. The publishing command
+is an explicit AWS write: it uploads the image, imports an encrypted snapshot and
+registers a private AMI in `us-east-2`. It never launches or modifies an instance.
+It requires an existing private S3 artifact bucket in Ohio and an operator with
+the [VM Import permissions and service role](https://docs.aws.amazon.com/vm-import/latest/userguide/required-permissions.html)
+plus image registration/tagging permissions. Use `--role` for a dedicated import
+role; scope its S3 access to the artifact bucket. This role is separate from the
+runtime EC2 role. Publishing credentials never enter the image.
+
+Record the printed image SHA256, import task, snapshot and AMI ID with the release.
+If publication is interrupted, inspect those identifiers before retrying; do not
+blindly create duplicate import tasks. Retain the previous AMI and snapshot until
+the replacement has passed acceptance checks. Remove obsolete image artifacts,
+AMIs and their root snapshots through the normal infrastructure cleanup process;
+they are separate from the protected telemetry volume and its daily snapshots.
+
+Review the AMI and boot it in the pilot's private subnet before enabling ingestion.
+Check SSM access, the OS version, exact data volume mount, service health, reboot,
+secret-outage recovery and replacement/reattachment. Pin `amiId` only after review.
+The AWS image import and actual EC2 boot checks require deployment access and are
+separate from local configuration and startup validation.
+
+For host changes or security updates, update `nixos/host.nix` or the pinned input
+with `nix flake update nixpkgs`, build/test/publish a new image, then change `amiId`
+and review `pulumi preview`. Do not run an ad-hoc `nixos-rebuild switch` on the
+host: Pulumi's pinned image is the source of truth. Rolling back an AMI does not
+roll back Grafana schema migrations or telemetry state; use a tested snapshot
+recovery plan when the old application version cannot read current data.
+
+### Runtime
+
+Use SSM Session Manager with the `instanceId` output. On the host:
+
+```bash
+sudo systemctl status observability-bootstrap observability-secrets grafana loki tempo prometheus alloy alloy-ingest nginx
+sudo journalctl -u observability-bootstrap -u observability-secrets --since boot
+sudo journalctl -u grafana -u loki -u tempo -u prometheus -u alloy -u alloy-ingest -u nginx --since '30 minutes ago'
+```
+
+Nonsecret runtime values arrive as versioned, compressed JSON in EC2 user-data.
+`read-user-data.py` retrieves them through IMDSv2 and validates the exact schema,
+region, hostnames, bucket/volume/secret identifiers and approved identity lists.
+It writes a systemd environment file, nginx hostname maps, the secret's identifier
+and the expected volume ID under `/opt/observability`. It does not execute
+user-data or read secret values. The standard NixOS user-data evaluator is disabled.
+Schema version 3 rejects older payloads, unknown settings and caller-supplied files.
+
+`observability-bootstrap` mounts that exact EBS volume and creates each service's
+data directory using stable, named NixOS users. `prepare-volume.sh` formats only
+a disk with no filesystem or signatures; existing ext4 data is preserved.
+Application units verify the mount before every startup, preventing fallback to
+the root disk. Bootstrap retries every 30 seconds after metadata or attachment
+failure, and its systemd `Upholds` relationships start dependents after recovery.
+
+`observability-secrets` fetches the credential bundle into root-only files under
+`/run`. Grafana and ingress Alloy wait for successful retrieval and receive private
+credential copies from systemd. Secret-service outages retry every 30 seconds;
+Loki, Tempo, Prometheus and host collection can run independently of that service.
+Systemd restarts crashed processes individually with memory and filesystem limits.
+
+Configuration/image changes require a new image and reviewed `amiId`; runtime
+value changes do not need an image rebuild. Both kinds of changes replace the
+instance and cause downtime. The old instance is deleted before replacement, its
+volume is detached, then the new host waits for that retained volume. This pilot
+has not been deployed with the earlier container layout; an existing container
+installation would need an explicit ownership/path migration before using this image.
+
+To rotate the OAuth secret or ingestion token, update Secrets Manager, then run
+`sudo systemctl restart observability-secrets`. This stops Grafana and ingress
+Alloy while credentials refresh, then restarts both with new private copies.
+Restarting an individual application uses the last fetched bundle. For token
+rotation coordinate producers and their retry queues; there is no dual-token
+grace window. Do not casually rotate Grafana's encryption key.
+
+For instance failure in the same AZ, replace the instance through Pulumi and
+reuse its retained volume. Do not run two hosts against the same data directory.
+For corrupt/lost data, select a known snapshot, set `dataSnapshotId`, and preview
+the resulting volume/instance replacements. A deliberate recovery requires
+removing Pulumi protection from the old volume before replacement; its
+`retainOnDelete` setting preserves it for investigation. Verify the new volume's
+ID and mounted data before accepting traffic. Cross-AZ recovery requires choosing
+a private subnet in that AZ and restoring the snapshot there; the initial code
+deliberately fixes the AZ. Never reformat an existing volume to resolve a mount
+failure. Buckets are independent and should be reused during recovery.
+
+This is a single point of failure. Alloy's logs/traces batch buffers and export
+queues are in memory; successful OTLP acceptance is not an end-to-end durability
+guarantee. Producer retries cover rejected requests, not accepted data lost in a
+crash. Metrics remote-write has a local WAL. EBS-backed backend WALs protect data
+already delivered to Loki/Tempo. Establish failure/loss tolerance while Datadog
+still receives the authoritative copy.
+
+## Validation and subsequent passes
+
+From `infra/`, run `bunx biome check stacks/observability` and `bun run check`.
+Build the complete NixOS image from this stack directory with `nix build . --cores 2`.
+Before publishing an image, validate native service startup, Google authorization redirects,
+anonymous/token denial, three-signal ingestion/readback, S3 flushing, process
+restart recovery and systemd credential/storage permissions in an isolated local
+fixture. Real Google login, AWS IAM, EC2 attachment/boot, ALB/TLS and snapshot
+restore remain the deployment acceptance checks above.
+
+Follow-up PRs:
+
+1. Duplicate OTEL at the existing collector to both Datadog and this stack with
+ independent bounded queues, retries and matching sampling. Check actual
+ metrics temporality and attribute mapping before enabling all services.
+2. Inventory non-OTEL sources: Datadog agents, infrastructure metrics, CloudWatch,
+ RDS/Postgres query monitoring and logs, browser RUM, synthetics, profiling and
+ security features. They are not covered by duplicating OTLP. Add DB collectors
+ with dedicated read-only monitoring identities and evaluate query-level
+ parity before replacing Datadog DBM.
+3. Port dashboards/alerts and run representative incidents in both tools. Add
+ pipeline/queue alerts, ingestion canaries, host/root-disk monitoring and a
+ tested restore procedure. Measure volume, cost, query latency and data gaps.
+4. Decide whether to split services or add replicas. Loki/Tempo can reuse S3;
+ metrics need a deliberate remote-store/migration plan. Move Grafana metadata
+ to managed Postgres/RDS before multiple Grafana instances. Cut off Datadog only
+ after coverage and operational acceptance are demonstrated.
+
+References: [Google OAuth](https://grafana.com/docs/grafana/latest/setup-grafana/configure-access/configure-authentication/google/),
+[Alloy bearer authentication](https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.auth.bearer/),
+[Loki retention](https://grafana.com/docs/loki/latest/operations/storage/retention/),
+[Tempo supported versions](https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/recommended-versions/).
diff --git a/infra/stacks/observability/assets/prepare-volume.sh b/infra/stacks/observability/assets/prepare-volume.sh
new file mode 100644
index 0000000000..54de6df53a
--- /dev/null
+++ b/infra/stacks/observability/assets/prepare-volume.sh
@@ -0,0 +1,13 @@
+#!/bin/bash
+set -euo pipefail
+device=$1
+filesystem=$(blkid -s TYPE -o value "$device" || true)
+if [ -z "$filesystem" ]; then
+ # Fail closed if inspection fails, even when it produces no output.
+ signatures=$(wipefs --no-act --noheadings --output TYPE "$device")
+ test -z "$signatures"
+ mkfs.ext4 "$device"
+elif [ "$filesystem" != ext4 ]; then
+ echo 'Refusing to format an existing data volume' >&2
+ exit 1
+fi
diff --git a/infra/stacks/observability/assets/refresh-secrets.py b/infra/stacks/observability/assets/refresh-secrets.py
new file mode 100644
index 0000000000..404640c16b
--- /dev/null
+++ b/infra/stacks/observability/assets/refresh-secrets.py
@@ -0,0 +1,36 @@
+#!/usr/bin/env python3
+"""Fetch credentials without putting their contents in user-data or logs."""
+import json
+import os
+from pathlib import Path
+import subprocess
+
+config = json.loads(Path('/opt/observability/bootstrap.json').read_text())
+result = subprocess.run(
+ ['aws', 'secretsmanager', 'get-secret-value', '--region', config['region'],
+ '--secret-id', config['secretArn'], '--output', 'json'],
+ check=True, capture_output=True, text=True,
+)
+values = json.loads(json.loads(result.stdout)['SecretString'])
+keys = ['google_client_id', 'google_client_secret', 'grafana_secret_key', 'otlp_token']
+# Validate the entire bundle before writing any file. Token is at least 32 bytes.
+for key in keys:
+ value = values.get(key)
+ if not isinstance(value, str) or not value or value.strip() != value:
+ raise SystemExit(f'Missing or invalid secret field: {key}')
+ if '\n' in value or '\r' in value:
+ raise SystemExit(f'Secret must be a single line: {key}')
+for key in ['grafana_secret_key', 'otlp_token']:
+ if len(values[key]) < 32:
+ raise SystemExit(f'Secret too short: {key}')
+directory = Path('/run/macro-observability')
+directory.mkdir(mode=0o700, exist_ok=True)
+os.chown(directory, 0, 0)
+for key in keys:
+ path = directory / key
+ # Dependent services stop during refresh and receive private systemd credentials.
+ path.unlink(missing_ok=True)
+ fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o400)
+ with os.fdopen(fd, 'w') as output:
+ output.write(values[key])
+ os.fchown(output.fileno(), 0, 0)
diff --git a/infra/stacks/observability/flake.lock b/infra/stacks/observability/flake.lock
new file mode 100644
index 0000000000..ba6568a3d0
--- /dev/null
+++ b/infra/stacks/observability/flake.lock
@@ -0,0 +1,27 @@
+{
+ "nodes": {
+ "nixpkgs": {
+ "locked": {
+ "lastModified": 1791426274,
+ "narHash": "sha256-MWhktfFtFE8Jl7bf9Pkp7TR5MSdYCjCswSo0Gkb4ywc=",
+ "owner": "NixOS",
+ "repo": "nixpkgs",
+ "rev": "7c8764b7c7b09b34f632464276218ef9090eaa11",
+ "type": "github"
+ },
+ "original": {
+ "owner": "NixOS",
+ "ref": "nixos-26.05",
+ "repo": "nixpkgs",
+ "type": "github"
+ }
+ },
+ "root": {
+ "inputs": {
+ "nixpkgs": "nixpkgs"
+ }
+ }
+ },
+ "root": "root",
+ "version": 7
+}
diff --git a/infra/stacks/observability/flake.nix b/infra/stacks/observability/flake.nix
new file mode 100644
index 0000000000..073cfffa19
--- /dev/null
+++ b/infra/stacks/observability/flake.nix
@@ -0,0 +1,25 @@
+{
+ description = "Macro observability EC2 image";
+ inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";
+
+ outputs =
+ { nixpkgs, ... }:
+ let
+ system = "x86_64-linux";
+ host = nixpkgs.lib.nixosSystem {
+ inherit system;
+ modules = [
+ "${nixpkgs}/nixos/maintainers/scripts/ec2/amazon-image.nix"
+ ./nixos/host.nix
+ ];
+ };
+ in
+ {
+ nixosConfigurations.observability = host;
+ packages.${system} = {
+ default = host.config.system.build.amazonImage;
+
+ };
+ formatter.${system} = nixpkgs.legacyPackages.${system}.nixfmt;
+ };
+}
diff --git a/infra/stacks/observability/index.ts b/infra/stacks/observability/index.ts
new file mode 100644
index 0000000000..4d99c5719a
--- /dev/null
+++ b/infra/stacks/observability/index.ts
@@ -0,0 +1,430 @@
+import * as aws from '@pulumi/aws';
+import * as pulumi from '@pulumi/pulumi';
+import { createNetwork } from './network';
+import { renderUserData } from './render';
+import {
+ validateRegion,
+ validateRegionalArn,
+ validateSettings,
+} from './settings';
+
+const config = new pulumi.Config();
+const stack = pulumi.getStack();
+if (!['dev', 'prod'].includes(stack)) {
+ throw new Error('Use the dev or prod stack to select a unique hostname');
+}
+const region = aws.config.requireRegion();
+validateRegion(region);
+const baseDomain = 'macro-internal.com';
+const suffix = stack === 'prod' ? '' : '-dev';
+const grafanaHost = `grafana${suffix}.${baseDomain}`;
+const otlpHost = `otlp${suffix}.${baseDomain}`;
+const tags = { project: 'observability', environment: stack };
+const durable = { protect: true, retainOnDelete: true };
+const secretArn = config.require('secretArn');
+const allowedEmails = config.requireObject<string[]>('allowedEmails');
+const adminEmails = config.requireObject<string[]>('adminEmails');
+const alarmTopicArn = config.require('alarmTopicArn');
+// Built from this stack's pinned NixOS flake and published in Ohio.
+const amiId = config.require('amiId');
+if (!/^ami-[a-f0-9]+$/.test(amiId)) {
+ throw new Error('amiId must identify the reviewed observability NixOS image');
+}
+// Validate operator input before registering any infrastructure resources.
+validateSettings({
+ region,
+ grafanaHost,
+ otlpHost,
+ allowedEmails,
+ adminEmails,
+ secretArn,
+ volumeId: 'vol-0',
+ logsBucket: 'validate-logs',
+ tracesBucket: 'validate-traces',
+});
+validateRegionalArn(alarmTopicArn, 'sns', region);
+const vpc = createNetwork(region, tags);
+const subnet = vpc.privateSubnet;
+const subnetId = subnet.id;
+
+// Route53 is global; issuance and use of the certificate are both in Ohio.
+const zone = aws.route53.getZoneOutput({
+ name: baseDomain,
+ privateZone: false,
+});
+const certificate = new aws.acm.Certificate('observability', {
+ domainName: grafanaHost,
+ subjectAlternativeNames: [otlpHost],
+ validationMethod: 'DNS',
+ tags,
+});
+const validationRecords = [grafanaHost, otlpHost].map((hostname) => {
+ const option = certificate.domainValidationOptions.apply((options) => {
+ const match = options.find((entry) => entry.domainName === hostname);
+ if (!match) throw new Error(`Missing ACM validation for ${hostname}`);
+ return match;
+ });
+ return new aws.route53.Record(`observability-certificate-${hostname}`, {
+ zoneId: zone.zoneId,
+ name: option.resourceRecordName,
+ type: option.resourceRecordType,
+ records: [option.resourceRecordValue],
+ ttl: 300,
+ });
+});
+const validatedCertificate = new aws.acm.CertificateValidation(
+ 'observability',
+ {
+ certificateArn: certificate.arn,
+ validationRecordFqdns: validationRecords.map((record) => record.fqdn),
+ }
+);
+
+function telemetryBucket(name: string) {
+ const bucket = new aws.s3.BucketV2(
+ name,
+ { forceDestroy: false, tags },
+ durable
+ );
+ new aws.s3.BucketPublicAccessBlock(`${name}-private`, {
+ bucket: bucket.id,
+ blockPublicAcls: true,
+ blockPublicPolicy: true,
+ ignorePublicAcls: true,
+ restrictPublicBuckets: true,
+ });
+ new aws.s3.BucketServerSideEncryptionConfigurationV2(`${name}-encryption`, {
+ bucket: bucket.id,
+ rules: [{ applyServerSideEncryptionByDefault: { sseAlgorithm: 'AES256' } }],
+ });
+ new aws.s3.BucketPolicy(`${name}-tls`, {
+ bucket: bucket.id,
+ policy: bucket.arn.apply((arn) =>
+ JSON.stringify({
+ Version: '2012-10-17',
+ Statement: [
+ {
+ Effect: 'Deny',
+ Principal: '*',
+ Action: 's3:*',
+ Resource: [arn, `${arn}/*`],
+ Condition: { Bool: { 'aws:SecureTransport': 'false' } },
+ },
+ ],
+ })
+ ),
+ });
+ // Loki/Tempo own retention. Do not expire active blocks with S3 lifecycle.
+ new aws.s3.BucketLifecycleConfigurationV2(`${name}-multipart`, {
+ bucket: bucket.id,
+ rules: [
+ {
+ id: 'abort-incomplete-uploads',
+ status: 'Enabled',
+ filter: { prefix: '' },
+ abortIncompleteMultipartUpload: { daysAfterInitiation: 1 },
+ },
+ ],
+ });
+ return bucket;
+}
+const logs = telemetryBucket('observability-logs');
+const traces = telemetryBucket('observability-traces');
+const data = new aws.ebs.Volume(
+ 'observability-data',
+ {
+ availabilityZone: subnet.availabilityZone,
+ type: 'gp3',
+ size: 300,
+ encrypted: true,
+ snapshotId: config.get('dataSnapshotId'),
+ tags: { ...tags, ObservabilityBackup: stack },
+ },
+ durable
+);
+
+const role = new aws.iam.Role('observability-host', {
+ assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
+ Service: 'ec2.amazonaws.com',
+ }),
+ tags,
+});
+const ssm = new aws.iam.RolePolicyAttachment('observability-ssm', {
+ role: role.name,
+ policyArn: 'arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore',
+});
+const policy = new aws.iam.RolePolicy('observability-storage', {
+ role: role.id,
+ policy: pulumi.all([logs.arn, traces.arn]).apply((arns) =>
+ JSON.stringify({
+ Version: '2012-10-17',
+ Statement: [
+ {
+ Effect: 'Allow',
+ Action: [
+ 's3:ListBucket',
+ 's3:GetBucketLocation',
+ 's3:ListBucketMultipartUploads',
+ ],
+ Resource: arns,
+ },
+ {
+ Effect: 'Allow',
+ Action: [
+ 's3:GetObject',
+ 's3:PutObject',
+ 's3:DeleteObject',
+ 's3:AbortMultipartUpload',
+ 's3:ListMultipartUploadParts',
+ ],
+ Resource: arns.map((arn) => `${arn}/*`),
+ },
+ {
+ Effect: 'Allow',
+ Action: ['secretsmanager:GetSecretValue'],
+ Resource: secretArn,
+ },
+ {
+ Effect: 'Allow',
+ Action: ['cloudwatch:PutMetricData'],
+ Resource: '*',
+ Condition: {
+ StringEquals: { 'cloudwatch:namespace': 'Macro/Observability' },
+ },
+ },
+ ],
+ })
+ ),
+});
+const profile = new aws.iam.InstanceProfile('observability-host', {
+ role: role.name,
+});
+const albSg = new aws.ec2.SecurityGroup('observability-alb', {
+ vpcId: vpc.vpcId,
+ description: 'Public HTTPS only',
+ ingress: [
+ { protocol: 'tcp', fromPort: 443, toPort: 443, cidrBlocks: ['0.0.0.0/0'] },
+ ],
+ tags,
+});
+const hostSg = new aws.ec2.SecurityGroup('observability-host', {
+ vpcId: vpc.vpcId,
+ description: 'Only the ALB can reach the proxy; administration via SSM',
+ ingress: [
+ {
+ protocol: 'tcp',
+ fromPort: 8080,
+ toPort: 8080,
+ securityGroups: [albSg.id],
+ },
+ ],
+ egress: [
+ { protocol: '-1', fromPort: 0, toPort: 0, cidrBlocks: ['0.0.0.0/0'] },
+ ],
+ tags,
+});
+new aws.ec2.SecurityGroupRule('observability-alb-to-host', {
+ securityGroupId: albSg.id,
+ type: 'egress',
+ protocol: 'tcp',
+ fromPort: 8080,
+ toPort: 8080,
+ sourceSecurityGroupId: hostSg.id,
+});
+const instance = new aws.ec2.Instance(
+ 'observability',
+ {
+ ami: amiId,
+ instanceType: 'm7i.xlarge',
+ subnetId,
+ associatePublicIpAddress: false,
+ vpcSecurityGroupIds: [hostSg.id],
+ iamInstanceProfile: profile.name,
+ metadataOptions: {
+ httpEndpoint: 'enabled',
+ httpTokens: 'required',
+ httpPutResponseHopLimit: 1,
+ },
+ rootBlockDevice: {
+ volumeType: 'gp3',
+ volumeSize: 30,
+ encrypted: true,
+ deleteOnTermination: true,
+ },
+ userDataReplaceOnChange: true,
+ userDataBase64: pulumi
+ .all([data.id, logs.bucket, traces.bucket])
+ .apply(([volumeId, logsBucket, tracesBucket]) =>
+ renderUserData({
+ region,
+ grafanaHost,
+ otlpHost,
+ allowedEmails,
+ adminEmails,
+ secretArn,
+ volumeId,
+ logsBucket,
+ tracesBucket,
+ })
+ ),
+ tags,
+ },
+ { deleteBeforeReplace: true, dependsOn: [policy, ssm, ...vpc.privateReady] }
+);
+new aws.ec2.VolumeAttachment('observability-data', {
+ deviceName: '/dev/sdf',
+ volumeId: data.id,
+ instanceId: instance.id,
+ stopInstanceBeforeDetaching: true,
+ forceDetach: false,
+});
+
+const alb = new aws.lb.LoadBalancer(
+ 'observability',
+ {
+ loadBalancerType: 'application',
+ internal: false,
+ subnets: vpc.publicSubnetIds,
+ securityGroups: [albSg.id],
+ dropInvalidHeaderFields: true,
+ desyncMitigationMode: 'strictest',
+ tags,
+ },
+ { dependsOn: vpc.publicReady }
+);
+const target = new aws.lb.TargetGroup('observability', {
+ port: 8080,
+ protocol: 'HTTP',
+ targetType: 'instance',
+ vpcId: vpc.vpcId,
+ healthCheck: { path: '/healthz', matcher: '200', interval: 30 },
+ deregistrationDelay: 30,
+ tags,
+});
+new aws.lb.TargetGroupAttachment('observability', {
+ targetGroupArn: target.arn,
+ targetId: instance.id,
+ port: 8080,
+});
+const listener = new aws.lb.Listener('observability-https', {
+ loadBalancerArn: alb.arn,
+ port: 443,
+ protocol: 'HTTPS',
+ certificateArn: validatedCertificate.certificateArn,
+ sslPolicy: 'ELBSecurityPolicy-TLS13-1-2-2021-06',
+ defaultActions: [
+ {
+ type: 'fixed-response',
+ fixedResponse: { contentType: 'text/plain', statusCode: '404' },
+ },
+ ],
+});
+new aws.lb.ListenerRule('observability-hosts', {
+ listenerArn: listener.arn,
+ priority: 1,
+ conditions: [{ hostHeader: { values: [grafanaHost, otlpHost] } }],
+ actions: [{ type: 'forward', targetGroupArn: target.arn }],
+});
+for (const hostname of [grafanaHost, otlpHost]) {
+ new aws.route53.Record(hostname, {
+ zoneId: zone.zoneId,
+ name: hostname,
+ type: 'A',
+ aliases: [
+ { name: alb.dnsName, zoneId: alb.zoneId, evaluateTargetHealth: true },
+ ],
+ });
+}
+
+const backupRole = new aws.iam.Role('observability-snapshots', {
+ assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
+ Service: 'dlm.amazonaws.com',
+ }),
+ tags,
+});
+const backupPolicy = new aws.iam.RolePolicyAttachment(
+ 'observability-snapshots',
+ {
+ role: backupRole.name,
+ policyArn:
+ 'arn:aws:iam::aws:policy/service-role/AWSDataLifecycleManagerServiceRole',
+ }
+);
+new aws.dlm.LifecyclePolicy(
+ 'observability-snapshots',
+ {
+ description: 'Daily crash-consistent data volume snapshots retained 7 days',
+ executionRoleArn: backupRole.arn,
+ state: 'ENABLED',
+ policyDetails: {
+ resourceTypes: ['VOLUME'],
+ targetTags: { ObservabilityBackup: stack },
+ schedules: [
+ {
+ name: 'daily',
+ copyTags: true,
+ createRule: { interval: 24, intervalUnit: 'HOURS', times: '04:00' },
+ retainRule: { count: 7 },
+ },
+ ],
+ },
+ tags,
+ },
+ { dependsOn: [backupPolicy] }
+);
+const alarm = {
+ comparisonOperator: 'GreaterThanThreshold',
+ evaluationPeriods: 2,
+ period: 60,
+ statistic: 'Maximum',
+ alarmActions: [alarmTopicArn],
+ okActions: [alarmTopicArn],
+ tags,
+};
+new aws.cloudwatch.MetricAlarm('observability-instance', {
+ ...alarm,
+ namespace: 'AWS/EC2',
+ metricName: 'StatusCheckFailed',
+ threshold: 0,
+ dimensions: { InstanceId: instance.id },
+ treatMissingData: 'breaching',
+});
+new aws.cloudwatch.MetricAlarm('observability-ui', {
+ ...alarm,
+ namespace: 'AWS/ApplicationELB',
+ metricName: 'UnHealthyHostCount',
+ threshold: 0,
+ dimensions: { LoadBalancer: alb.arnSuffix, TargetGroup: target.arnSuffix },
+ treatMissingData: 'breaching',
+});
+for (const [name, path] of [
+ ['observability-disk', '/srv/observability'],
+ ['observability-root-disk', '/'],
+]) {
+ new aws.cloudwatch.MetricAlarm(name, {
+ ...alarm,
+ namespace: 'Macro/Observability',
+ metricName: 'disk_used_percent',
+ threshold: 80,
+ dimensions: { InstanceId: instance.id, path, fstype: 'ext4' },
+ period: 300,
+ treatMissingData: 'breaching',
+ });
+}
+new aws.cloudwatch.MetricAlarm('observability-memory', {
+ ...alarm,
+ namespace: 'Macro/Observability',
+ metricName: 'mem_used_percent',
+ threshold: 90,
+ dimensions: { InstanceId: instance.id },
+ period: 300,
+ treatMissingData: 'breaching',
+});
+
+export const grafanaUrl = `https://${grafanaHost}`;
+export const otlpHttpEndpoint = `https://${otlpHost}`;
+export const instanceId = instance.id;
+export const dataVolumeId = data.id;
+export const logsBucket = logs.bucket;
+export const tracesBucket = traces.bucket;
+export const observabilityRegion = region;
diff --git a/infra/stacks/observability/network.ts b/infra/stacks/observability/network.ts
new file mode 100644
index 0000000000..b991894037
--- /dev/null
+++ b/infra/stacks/observability/network.ts
@@ -0,0 +1,92 @@
+import * as aws from '@pulumi/aws';
+
+// Independent of prod's VPC, NAT, and regional endpoints. One private host
+// subnet and one NAT are intentional for the single-instance pilot.
+export function createNetwork(region: string, tags: Record<string, string>) {
+ const vpc = new aws.ec2.Vpc('observability', {
+ cidrBlock: '10.88.0.0/16',
+ enableDnsSupport: true,
+ enableDnsHostnames: true,
+ tags,
+ });
+ const internet = new aws.ec2.InternetGateway('observability', {
+ vpcId: vpc.id,
+ tags,
+ });
+ const publicRoutes = new aws.ec2.RouteTable('observability-public', {
+ vpcId: vpc.id,
+ tags,
+ });
+ const internetRoute = new aws.ec2.Route('observability-internet', {
+ routeTableId: publicRoutes.id,
+ destinationCidrBlock: '0.0.0.0/0',
+ gatewayId: internet.id,
+ });
+ const publicSubnets = ['a', 'b'].map(
+ (zone, index) =>
+ new aws.ec2.Subnet(`observability-public-${zone}`, {
+ vpcId: vpc.id,
+ availabilityZone: `${region}${zone}`,
+ cidrBlock: `10.88.${index}.0/24`,
+ mapPublicIpOnLaunch: false,
+ tags,
+ })
+ );
+ const publicAssociations = publicSubnets.map(
+ (subnet, index) =>
+ new aws.ec2.RouteTableAssociation(`observability-public-${index}`, {
+ subnetId: subnet.id,
+ routeTableId: publicRoutes.id,
+ })
+ );
+ const natAddress = new aws.ec2.Eip('observability-nat', {
+ domain: 'vpc',
+ tags,
+ });
+ const nat = new aws.ec2.NatGateway(
+ 'observability',
+ {
+ subnetId: publicSubnets[0].id,
+ allocationId: natAddress.id,
+ tags,
+ },
+ { dependsOn: [internetRoute, ...publicAssociations] }
+ );
+ const privateSubnet = new aws.ec2.Subnet('observability-private', {
+ vpcId: vpc.id,
+ availabilityZone: `${region}a`,
+ cidrBlock: '10.88.10.0/24',
+ mapPublicIpOnLaunch: false,
+ tags,
+ });
+ const privateRoutes = new aws.ec2.RouteTable('observability-private', {
+ vpcId: vpc.id,
+ tags,
+ });
+ const natRoute = new aws.ec2.Route('observability-egress', {
+ routeTableId: privateRoutes.id,
+ destinationCidrBlock: '0.0.0.0/0',
+ natGatewayId: nat.id,
+ });
+ const privateAssociation = new aws.ec2.RouteTableAssociation(
+ 'observability-private',
+ {
+ subnetId: privateSubnet.id,
+ routeTableId: privateRoutes.id,
+ }
+ );
+ const s3 = new aws.ec2.VpcEndpoint('observability-s3', {
+ vpcId: vpc.id,
+ serviceName: `com.amazonaws.${region}.s3`,
+ vpcEndpointType: 'Gateway',
+ routeTableIds: [privateRoutes.id],
+ tags,
+ });
+ return {
+ vpcId: vpc.id,
+ privateSubnet,
+ publicSubnetIds: publicSubnets.map((subnet) => subnet.id),
+ publicReady: [internetRoute, ...publicAssociations],
+ privateReady: [natRoute, privateAssociation, s3],
+ };
+}
diff --git a/infra/stacks/observability/nixos/backend-config.nix b/infra/stacks/observability/nixos/backend-config.nix
new file mode 100644
index 0000000000..c31221baac
--- /dev/null
+++ b/infra/stacks/observability/nixos/backend-config.nix
@@ -0,0 +1,100 @@
+{
+ loki = {
+ auth_enabled = false;
+ server = {
+ http_listen_address = "127.0.0.1";
+ http_listen_port = 3100;
+ grpc_listen_address = "127.0.0.1";
+ grpc_listen_port = 9095;
+ };
+ common = {
+ instance_addr = "127.0.0.1";
+ path_prefix = "/srv/observability/loki";
+ replication_factor = 1;
+ ring.kvstore.store = "inmemory";
+ };
+ schema_config.configs = [
+ {
+ from = "2024-01-01";
+ store = "tsdb";
+ object_store = "s3";
+ schema = "v13";
+ index = {
+ prefix = "index_";
+ period = "24h";
+ };
+ }
+ ];
+ storage_config = {
+ aws = {
+ region = "\${AWS_REGION}";
+ bucketnames = "\${LOGS_BUCKET}";
+ };
+ tsdb_shipper = {
+ active_index_directory = "/srv/observability/loki/index";
+ cache_location = "/srv/observability/loki/index-cache";
+ };
+ };
+ ingester.wal = {
+ enabled = true;
+ dir = "/srv/observability/loki/wal";
+ };
+ compactor = {
+ working_directory = "/srv/observability/loki/compactor";
+ retention_enabled = true;
+ delete_request_store = "s3";
+ };
+ limits_config = {
+ retention_period = "720h";
+ allow_structured_metadata = true;
+ ingestion_rate_mb = 8;
+ ingestion_burst_size_mb = 16;
+ max_query_parallelism = 4;
+ };
+ querier.max_concurrent = 4;
+ analytics.reporting_enabled = false;
+ };
+ tempo = {
+ server = {
+ http_listen_address = "127.0.0.1";
+ http_listen_port = 3200;
+ grpc_listen_address = "127.0.0.1";
+ grpc_listen_port = 9096;
+ };
+ memberlist.bind_addr = [ "127.0.0.1" ];
+ distributor.receivers.otlp.protocols.grpc.endpoint = "127.0.0.1:4317";
+ ingester = {
+ max_block_duration = "5m";
+ lifecycler.address = "127.0.0.1";
+ };
+ compactor.compaction.block_retention = "168h";
+ storage.trace = {
+ backend = "s3";
+ wal.path = "/srv/observability/tempo/wal";
+ s3 = {
+ bucket = "\${TRACES_BUCKET}";
+ region = "\${AWS_REGION}";
+ endpoint = "s3.\${AWS_REGION}.amazonaws.com";
+ };
+ };
+ usage_report.reporting_enabled = false;
+ };
+ prometheus = {
+ global.scrape_interval = "30s";
+ scrape_configs = [
+ {
+ job_name = "observability";
+ static_configs = [
+ {
+ targets = [
+ "127.0.0.1:9090"
+ "127.0.0.1:3100"
+ "127.0.0.1:3200"
+ "127.0.0.1:12345"
+ ];
+ }
+ ];
+ }
+ ];
+ };
+}
diff --git a/infra/stacks/observability/nixos/cloudwatch.nix b/infra/stacks/observability/nixos/cloudwatch.nix
new file mode 100644
index 0000000000..9493ea29c1
--- /dev/null
+++ b/infra/stacks/observability/nixos/cloudwatch.nix
@@ -0,0 +1,25 @@
+{
+ agent = {
+ region = "us-east-2";
+ metrics_collection_interval = 60;
+ omit_hostname = true;
+ usage_data = false;
+ logfile = "";
+ };
+ metrics = {
+ namespace = "Macro/Observability";
+ append_dimensions.InstanceId = "\${aws:InstanceId}";
+ metrics_collected = {
+ disk = {
+ measurement = [ "used_percent" ];
+ resources = [
+ "/"
+ "/srv/observability"
+ ];
+ drop_device = true;
+ };
+ mem.measurement = [ "used_percent" ];
+ };
+ force_flush_interval = 60;
+ };
+}
diff --git a/infra/stacks/observability/nixos/datasources.nix b/infra/stacks/observability/nixos/datasources.nix
new file mode 100644
index 0000000000..f9c39326be
--- /dev/null
+++ b/infra/stacks/observability/nixos/datasources.nix
@@ -0,0 +1,30 @@
+{
+ apiVersion = 1;
+ datasources = [
+ {
+ name = "Prometheus";
+ uid = "prometheus";
+ type = "prometheus";
+ access = "proxy";
+ url = "http://127.0.0.1:8081/prometheus";
+ editable = false;
+ isDefault = true;
+ }
+ {
+ name = "Loki";
+ uid = "loki";
+ type = "loki";
+ access = "proxy";
+ url = "http://127.0.0.1:8081/loki";
+ editable = false;
+ }
+ {
+ name = "Tempo";
+ uid = "tempo";
+ type = "tempo";
+ access = "proxy";
+ url = "http://127.0.0.1:8081/tempo";
+ editable = false;
+ }
+ ];
+}
diff --git a/infra/stacks/observability/nixos/grafana.nix b/infra/stacks/observability/nixos/grafana.nix
new file mode 100644
index 0000000000..ff61f97816
--- /dev/null
+++ b/infra/stacks/observability/nixos/grafana.nix
@@ -0,0 +1,61 @@
+{
+ "server" = {
+ http_addr = "127.0.0.1";
+ domain = "$__env{GRAFANA_HOST}";
+ root_url = "$__env{GRAFANA_ROOT_URL}";
+ enforce_domain = true;
+ };
+ "security" = {
+ disable_initial_admin_creation = true;
+ secret_key = "$__file{/run/credentials/grafana.service/grafana_secret_key}";
+ cookie_secure = true;
+ cookie_samesite = "lax";
+ strict_transport_security = true;
+ content_security_policy = true;
+ };
+ "users" = {
+ allow_sign_up = false;
+ allow_org_create = false;
+ viewers_can_edit = false;
+ };
+ "auth" = {
+ disable_login_form = true;
+ login_maximum_inactive_lifetime_duration = "1h";
+ login_maximum_lifetime_duration = "8h";
+ };
+ "auth.basic" = {
+ enabled = false;
+ };
+ "auth.anonymous" = {
+ enabled = false;
+ };
+ "auth.google" = {
+ enabled = true;
+ allow_sign_up = true;
+ client_id = "$__file{/run/credentials/grafana.service/google_client_id}";
+ client_secret = "$__file{/run/credentials/grafana.service/google_client_secret}";
+ scopes = "openid email profile";
+ auth_url = "https://accounts.google.com/o/oauth2/v2/auth";
+ token_url = "https://oauth2.googleapis.com/token";
+ api_url = "https://openidconnect.googleapis.com/v1/userinfo";
+ allowed_domains = "macro.com";
+ hosted_domain = "macro.com";
+ validate_hd = true;
+ use_pkce = true;
+ use_refresh_token = true;
+ validate_id_token = true;
+ jwk_set_url = "https://www.googleapis.com/oauth2/v3/certs";
+ role_attribute_path = "$__env{GRAFANA_ROLE_EXPRESSION}";
+ role_attribute_strict = true;
+ allow_assign_grafana_admin = true;
+ skip_org_role_sync = false;
+ };
+ "analytics" = {
+ reporting_enabled = false;
+ check_for_updates = false;
+ };
+ "plugins" = {
+ preinstall_disabled = true;
+ preinstall_auto_update = false;
+ };
+}
diff --git a/infra/stacks/observability/nixos/host-telemetry.nix b/infra/stacks/observability/nixos/host-telemetry.nix
new file mode 100644
index 0000000000..4a6cf0c046
--- /dev/null
+++ b/infra/stacks/observability/nixos/host-telemetry.nix
@@ -0,0 +1,50 @@
+{ lib, pkgs, ... }:
+{
+ services.alloy = {
+ enable = true;
+ configPath = pkgs.writeText "host.alloy" (builtins.readFile ./host.alloy);
+ extraFlags = [
+ "--server.http.listen-addr=127.0.0.1:12346"
+ "--storage.path=/srv/observability/alloy"
+ "--disable-reporting"
+ ];
+ };
+ # Journal access is separate from the externally reachable OTLP receiver.
+ users.users.alloy.extraGroups = [ "systemd-journal" ];
+ systemd.services.alloy.serviceConfig = {
+ DynamicUser = lib.mkForce false;
+ User = "alloy";
+ Group = "alloy";
+ # No private mount namespace: filesystem metrics must describe the host,
+ # not sandbox read-only bind mounts. Unix permissions restrict this user.
+ StateDirectory = lib.mkForce "";
+ WorkingDirectory = lib.mkForce "/srv/observability/alloy";
+ MemoryMax = "1G";
+ };
+
+ users.groups.cloudwatch-agent = { };
+ users.users.cloudwatch-agent = {
+ isSystemUser = true;
+ group = "cloudwatch-agent";
+ };
+ services.amazon-cloudwatch-agent = {
+ enable = true;
+ user = "cloudwatch-agent";
+ mode = "ec2";
+ configuration = import ./cloudwatch.nix;
+ };
+ # Independent of Alloy, the data mount and the Grafana stack. Missing
+ # data-disk samples must reach the alarm's missing-data policy, not a fallback.
+ systemd.services.amazon-cloudwatch-agent = {
+ after = [ "network-online.target" ];
+ wants = [ "network-online.target" ];
+ unitConfig.StartLimitIntervalSec = 0;
+ serviceConfig = {
+ NoNewPrivileges = true;
+ ProtectSystem = "strict";
+ ProtectHome = true;
+ PrivateTmp = true;
+ MemoryMax = "512M";
+ };
+ };
+}
diff --git a/infra/stacks/observability/nixos/host.alloy b/infra/stacks/observability/nixos/host.alloy
new file mode 100644
index 0000000000..7a41cb78a1
--- /dev/null
+++ b/infra/stacks/observability/nixos/host.alloy
@@ -0,0 +1,68 @@
+prometheus.exporter.unix "host" {
+ disable_collectors = ["textfile"]
+ enable_collectors = ["systemd"]
+ systemd {
+ unit_include = "(grafana|loki|tempo|prometheus|nginx|alloy|alloy-ingest|observability-.*|amazon-cloudwatch-agent)\\.service"
+ task_metrics = true
+ enable_restarts = true
+ start_time = true
+ }
+}
+
+prometheus.scrape "host" {
+ targets = prometheus.exporter.unix.host.targets
+ job_name = "observability/host"
+ scrape_interval = "30s"
+ forward_to = [prometheus.remote_write.local.receiver]
+}
+
+prometheus.exporter.self "host_alloy" {}
+prometheus.scrape "host_alloy" {
+ targets = prometheus.exporter.self.host_alloy.targets
+ job_name = "observability/host-alloy"
+ scrape_interval = "30s"
+ forward_to = [prometheus.remote_write.local.receiver]
+}
+
+prometheus.remote_write "local" {
+ endpoint { url = "http://127.0.0.1:9090/api/v1/write" }
+ wal {
+ min_keepalive_time = "5m"
+ max_keepalive_time = "1h"
+ }
+}
+
+loki.relabel "journal" {
+ forward_to = []
+ rule {
+ source_labels = ["__journal__systemd_unit"]
+ regex = "(grafana|loki|tempo|prometheus|nginx|alloy|alloy-ingest|observability-.*|amazon-cloudwatch-agent|systemd-.*)\\.service"
+ action = "keep"
+ }
+ rule {
+ source_labels = ["__journal__systemd_unit"]
+ target_label = "unit"
+ }
+ rule {
+ source_labels = ["__journal_priority_keyword"]
+ target_label = "level"
+ }
+}
+
+loki.source.journal "host" {
+ max_age = "1h"
+ labels = {job = "observability/journal", instance = constants.hostname}
+ relabel_rules = loki.relabel.journal.rules
+ forward_to = [loki.write.local.receiver]
+}
+
+loki.source.journal "kernel" {
+ matches = "_TRANSPORT=kernel"
+ max_age = "1h"
+ labels = {job = "observability/kernel", instance = constants.hostname}
+ forward_to = [loki.write.local.receiver]
+}
+
+loki.write "local" {
+ endpoint { url = "http://127.0.0.1:3100/loki/api/v1/push" }
+}
diff --git a/infra/stacks/observability/nixos/host.nix b/infra/stacks/observability/nixos/host.nix
new file mode 100644
index 0000000000..f5715ac459
--- /dev/null
+++ b/infra/stacks/observability/nixos/host.nix
@@ -0,0 +1,144 @@
+{ lib, pkgs, ... }:
+let
+ runtimePath = with pkgs; [
+ awscli2
+ bash
+ coreutils
+ e2fsprogs
+ python3
+ util-linux
+ ];
+ services = [
+ "grafana"
+ "loki"
+ "tempo"
+ "prometheus"
+ "alloy"
+ "alloy-ingest"
+ "nginx"
+ ];
+ secretConsumers = [
+ "grafana.service"
+ "alloy-ingest.service"
+ ];
+ dataUsers = {
+ grafana = 20001;
+ loki = 20002;
+ tempo = 20003;
+ prometheus = 20004;
+ alloy = 20005;
+ alloy-ingest = 20006;
+ };
+in
+{
+ imports = [
+ ./services.nix
+ ./host-telemetry.nix
+ ./proxy.nix
+ ];
+ system.stateVersion = "26.05";
+ image.baseName = "macro-observability";
+ virtualisation.diskSize = 12 * 1024;
+ ec2.efi = true;
+ networking.hostName = "observability";
+ networking.firewall.allowedTCPPorts = [ 8080 ];
+ services.openssh.enable = lib.mkForce false;
+ services.amazon-ssm-agent.enable = true;
+ # EC2 user data contains validated JSON settings, never executable Nix.
+ virtualisation.amazon-init.enable = false;
+ nix.settings.experimental-features = [
+ "nix-command"
+ "flakes"
+ ];
+ environment.systemPackages = runtimePath;
+
+ # Stable identities preserve ownership when a replacement AMI reuses EBS.
+ users.users = lib.mapAttrs (name: uid: {
+ uid = lib.mkForce uid;
+ isSystemUser = true;
+ group = name;
+ createHome = lib.mkForce false;
+ }) dataUsers;
+ users.groups = lib.mapAttrs (_: gid: { gid = lib.mkForce gid; }) dataUsers;
+
+ systemd.services =
+ lib.genAttrs services (name: {
+ requires = [ "observability-bootstrap.service" ];
+ after = [ "observability-bootstrap.service" ];
+ partOf = [ "observability-bootstrap.service" ];
+ startLimitIntervalSec = lib.mkForce 0;
+ serviceConfig = {
+ ExecStartPre = lib.mkBefore [ "${pkgs.util-linux}/bin/mountpoint -q /srv/observability" ];
+ Restart = lib.mkForce "always";
+ RestartSec = lib.mkForce 30;
+ # Stay activating during automatic restarts so Upholds cannot bypass
+ # the backoff by starting a briefly failed/inactive unit.
+ RestartMode = "direct";
+ NoNewPrivileges = true;
+ ProtectSystem = lib.mkForce (if name == "alloy" then false else "strict");
+ ProtectHome = name != "alloy";
+ PrivateTmp = name != "alloy";
+ UMask = lib.mkForce "0077";
+ TimeoutStopSec = 90;
+ # Only Loki/Tempo need the EC2 role for S3 access.
+ IPAddressDeny = lib.optionals (
+ !builtins.elem name [
+ "loki"
+ "tempo"
+ ]
+ ) [ "169.254.169.254/32" ];
+ };
+ })
+ // {
+ fetch-ec2-metadata.enable = false;
+ observability-bootstrap = {
+ description = "Validate EC2 settings and mount retained observability storage";
+ wantedBy = [ "multi-user.target" ];
+ wants = [ "network-online.target" ];
+ after = [ "network-online.target" ];
+ path = runtimePath;
+ startLimitIntervalSec = 0;
+ # Only the secret service upholds its consumers. Starting them here would
+ # repeatedly pull a failed secret service out of its restart backoff.
+ unitConfig.Upholds = [
+ "observability-secrets.service"
+ ]
+ ++ lib.subtractLists secretConsumers (map (name: "${name}.service") services);
+ script = ''
+ ${pkgs.python3}/bin/python3 ${./read-user-data.py}
+ ${pkgs.bash}/bin/bash ${./mount-data.sh}
+ ${lib.concatMapStringsSep "\n" (
+ name: "install -d -m 0700 -o ${name} -g ${name} /srv/observability/${name}"
+ ) (builtins.attrNames dataUsers)}
+ '';
+ serviceConfig = {
+ Type = "oneshot";
+ RemainAfterExit = true;
+ Restart = "on-failure";
+ RestartMode = "direct";
+ RestartSec = 30;
+ TimeoutStartSec = 900;
+ };
+ };
+ observability-secrets = {
+ description = "Fetch runtime credentials for Grafana and OTLP ingress";
+ requires = [ "observability-bootstrap.service" ];
+ after = [ "observability-bootstrap.service" ];
+ partOf = [ "observability-bootstrap.service" ];
+ path = [ pkgs.awscli2 ];
+ startLimitIntervalSec = 0;
+ unitConfig.Upholds = secretConsumers;
+ serviceConfig = {
+ Type = "oneshot";
+ RemainAfterExit = true;
+ ExecStart = "${pkgs.python3}/bin/python3 ${../assets/refresh-secrets.py}";
+ Restart = "on-failure";
+ RestartMode = "direct";
+ RestartSec = 30;
+ TimeoutStartSec = 120;
+ UMask = lib.mkForce "0077";
+ };
+ };
+ };
+ environment.etc."observability/prepare-volume.sh".source = ../assets/prepare-volume.sh;
+}
diff --git a/infra/stacks/observability/nixos/ingest.alloy b/infra/stacks/observability/nixos/ingest.alloy
new file mode 100644
index 0000000000..24a112ecb2
--- /dev/null
+++ b/infra/stacks/observability/nixos/ingest.alloy
@@ -0,0 +1,64 @@
+local.file "otlp_token" {
+ filename = sys.env("CREDENTIALS_DIRECTORY") + "/otlp_token"
+ is_secret = true
+}
+
+otelcol.auth.bearer "ingest" {
+ token = local.file.otlp_token.content
+}
+
+otelcol.receiver.otlp "ingest" {
+ http {
+ endpoint = "127.0.0.1:4318"
+ auth = otelcol.auth.bearer.ingest.handler
+ max_request_body_size = "8MiB"
+ }
+ output {
+ logs = [otelcol.processor.memory_limiter.ingest.input]
+ traces = [otelcol.processor.memory_limiter.ingest.input]
+ metrics = [otelcol.processor.memory_limiter.ingest.input]
+ }
+}
+
+otelcol.processor.memory_limiter "ingest" {
+ check_interval = "1s"
+ limit = "768MiB"
+ spike_limit = "128MiB"
+ output {
+ logs = [otelcol.processor.batch.ingest.input]
+ traces = [otelcol.processor.batch.ingest.input]
+ metrics = [otelcol.processor.batch.ingest.input]
+ }
+}
+
+otelcol.processor.batch "ingest" {
+ timeout = "1s"
+ send_batch_size = 512
+ send_batch_max_size = 1024
+ output {
+ logs = [otelcol.exporter.otlphttp.loki.input]
+ traces = [otelcol.exporter.otlp.tempo.input]
+ metrics = [otelcol.exporter.prometheus.metrics.input]
+ }
+}
+
+otelcol.exporter.otlphttp "loki" {
+ client { endpoint = "http://127.0.0.1:3100/otlp" }
+ sending_queue { queue_size = 256 }
+}
+
+otelcol.exporter.otlp "tempo" {
+ client {
+ endpoint = "127.0.0.1:4317"
+ tls { insecure = true }
+ }
+ sending_queue { queue_size = 256 }
+}
+
+otelcol.exporter.prometheus "metrics" {
+ forward_to = [prometheus.remote_write.local.receiver]
+}
+
+prometheus.remote_write "local" {
+ endpoint { url = "http://127.0.0.1:9090/api/v1/write" }
+}
diff --git a/infra/stacks/observability/nixos/mount-data.sh b/infra/stacks/observability/nixos/mount-data.sh
new file mode 100644
index 0000000000..1edc005966
--- /dev/null
+++ b/infra/stacks/observability/nixos/mount-data.sh
@@ -0,0 +1,13 @@
+set -euo pipefail
+volume_id=$(cat /opt/observability/volume-id)
+device="/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_${volume_id//-/}"
+for attempt in $(seq 1 120); do
+ if [ -b "$device" ]; then break; fi
+ sleep 5
+done
+test -b "$device"
+bash /etc/observability/prepare-volume.sh "$device"
+uuid=$(blkid -s UUID -o value "$device")
+mkdir -p /srv/observability
+mountpoint -q /srv/observability || mount -t ext4 "$device" /srv/observability
+test "$(findmnt --noheadings --output UUID --target /srv/observability)" = "$uuid"
diff --git a/infra/stacks/observability/nixos/proxy.nix b/infra/stacks/observability/nixos/proxy.nix
new file mode 100644
index 0000000000..bda092d357
--- /dev/null
+++ b/infra/stacks/observability/nixos/proxy.nix
@@ -0,0 +1,98 @@
+{ ... }:
+let
+ deny = {
+ extraConfig = "return 404;";
+ };
+ websocketHeaders = ''
+ proxy_http_version 1.1;
+ proxy_set_header Upgrade $http_upgrade;
+ proxy_set_header Connection $connection_upgrade;
+ '';
+ queryLocation = backend: methods: {
+ proxyPass = "http://${backend}";
+ extraConfig = ''
+ limit_except ${methods} { deny all; }
+ rewrite ^/[^/]+(/.*)$ $1 break;
+ '';
+ };
+in
+{
+ services.nginx = {
+ enable = true;
+ # Hostnames arrive through validated EC2 metadata. The native unit runs
+ # nginx -t after bootstrap has written this runtime include.
+ validateConfigFile = false;
+ clientMaxBodySize = "8m";
+ commonHttpConfig = ''
+ include /opt/observability/hosts.conf;
+ log_format safe '$remote_addr $host $request_method $uri $status';
+ access_log syslog:server=unix:/dev/log,facility=local7,tag=nginx_access,severity=info safe;
+ limit_req_zone $server_name zone=ingest:1m rate=100r/s;
+ map $http_upgrade $connection_upgrade { default upgrade; ''' close; }
+ '';
+ virtualHosts.public = {
+ listen = [
+ {
+ addr = "0.0.0.0";
+ port = 8080;
+ }
+ ];
+ default = true;
+ serverName = "_";
+ extraConfig = "add_header Strict-Transport-Security 'max-age=31536000' always;";
+ locations = {
+ "= /healthz" = {
+ proxyPass = "http://127.0.0.1:3000/api/health";
+ extraConfig = "access_log off; proxy_set_header Host $grafana_host;";
+ };
+ "/" = {
+ proxyPass = "http://127.0.0.1:3000";
+ extraConfig = ''
+ if ($observability_host != grafana) { return 404; }
+ proxy_set_header Host $host;
+ proxy_set_header X-Forwarded-Proto https;
+ proxy_set_header X-Forwarded-For $remote_addr;
+ ${websocketHeaders}
+ '';
+ };
+ "~ ^/v1/(traces|logs|metrics)$" = {
+ proxyPass = "http://127.0.0.1:4318";
+ extraConfig = ''
+ if ($observability_host != otlp) { return 404; }
+ limit_except POST { deny all; }
+ limit_req zone=ingest burst=200 nodelay;
+ limit_req_status 429;
+ proxy_read_timeout 30s;
+ '';
+ };
+ };
+ };
+ # Grafana Viewers can choose arbitrary datasource proxy paths. Allow query
+ # endpoints only, including protection against maintenance APIs using GET.
+ virtualHosts.queries = {
+ listen = [
+ {
+ addr = "127.0.0.1";
+ port = 8081;
+ }
+ ];
+ serverName = "_";
+ locations = {
+ "~ ^/prometheus/api/v1/(query|query_range|query_exemplars|series|labels|label/[^/]+/values)$" =
+ queryLocation "127.0.0.1:9090" "GET POST";
+ "~ ^/prometheus/api/v1/(metadata|status/buildinfo|status/config|rules|alerts|targets|targets/metadata)$" =
+ queryLocation "127.0.0.1:9090" "GET";
+ "~ ^/loki/loki/api/v1/(query|query_range)$" = queryLocation "127.0.0.1:3100" "GET POST";
+ "~ ^/loki/loki/api/v1/(labels|label/[^/]+/values|series|index/stats|index/volume|index/volume_range|patterns|tail|status/buildinfo|format_query|detected_fields|detected_labels|detected_field/[^/]+/values)$" =
+ let
+ location = queryLocation "127.0.0.1:3100" "GET";
+ in
+ location // { extraConfig = location.extraConfig + websocketHeaders; };
+ "~ ^/tempo/api/(echo|search|search/tags|search/tag/[^/]+/values|traces/[a-fA-F0-9]+|v2/traces/[a-fA-F0-9]+|v2/search/tags|v2/search/tag/[^/]+/values|metrics/query|metrics/query_range|status/buildinfo)$" =
+ queryLocation "127.0.0.1:3200" "GET";
+ "/" = deny;
+ };
+ };
+ };
+ systemd.services.nginx.serviceConfig.MemoryMax = "256M";
+}
diff --git a/infra/stacks/observability/nixos/publish-image.py b/infra/stacks/observability/nixos/publish-image.py
new file mode 100644
index 0000000000..f62484b66c
--- /dev/null
+++ b/infra/stacks/observability/nixos/publish-image.py
@@ -0,0 +1,76 @@
+#!/usr/bin/env python3
+"""Publish a built image in Ohio. Explicit operator action; never run by Pulumi."""
+import argparse
+import hashlib
+import json
+from pathlib import Path
+import subprocess
+import time
+
+parser = argparse.ArgumentParser(description=__doc__)
+parser.add_argument('image_directory', type=Path)
+parser.add_argument('bucket', help='Existing private Ohio image-artifact bucket')
+parser.add_argument('--role', default='vmimport', help='Existing VM Import service role')
+args = parser.parse_args()
+
+def aws(*arguments):
+ result = subprocess.run(
+ ['aws', '--region', 'us-east-2', '--output', 'json', *arguments],
+ check=True, capture_output=True, text=True,
+ )
+ return json.loads(result.stdout) if result.stdout.strip() else {}
+
+identity = aws('sts', 'get-caller-identity')
+if identity['Account'] != '569036502058':
+ raise SystemExit('Use the Macro AWS account (569036502058)')
+if aws('s3api', 'get-bucket-location', '--bucket', args.bucket)['LocationConstraint'] != 'us-east-2':
+ raise SystemExit('Image bucket must be in Ohio')
+public_access = aws('s3api', 'get-public-access-block', '--bucket', args.bucket)['PublicAccessBlockConfiguration']
+if not all(public_access.get(key) for key in [
+ 'BlockPublicAcls', 'IgnorePublicAcls', 'BlockPublicPolicy', 'RestrictPublicBuckets',
+]):
+ raise SystemExit('Image bucket must have all S3 public-access blocks enabled')
+info = json.loads((args.image_directory / 'nix-support/image-info.json').read_text())
+if info['system'] != 'x86_64-linux' or info['boot_mode'] != 'uefi':
+ raise SystemExit('Expected an x86_64 UEFI image from this flake')
+image = args.image_directory / Path(info['file']).name
+if image.suffix != '.vhd':
+ raise SystemExit('Expected the VHD image format')
+with image.open('rb') as source:
+ digest = hashlib.file_digest(source, 'sha256').hexdigest()
+key = f'observability/{digest}.vhd'
+subprocess.run(['aws', '--region', 'us-east-2', 's3', 'cp', str(image),
+ f's3://{args.bucket}/{key}', '--sse', 'AES256'], check=True)
+task = aws('ec2', 'import-snapshot', '--role-name', args.role, '--encrypted',
+ '--description', f'Macro observability {digest}',
+ '--disk-container', json.dumps({
+ 'Format': 'VHD', 'UserBucket': {'S3Bucket': args.bucket, 'S3Key': key},
+ }))['ImportTaskId']
+print(f'Import task: {task}; artifact: s3://{args.bucket}/{key}', flush=True)
+deadline = time.monotonic() + 3600
+while time.monotonic() < deadline:
+ detail = aws('ec2', 'describe-import-snapshot-tasks',
+ '--import-task-ids', task)['ImportSnapshotTasks'][0]['SnapshotTaskDetail']
+ if detail['Status'] == 'completed':
+ snapshot = detail['SnapshotId']
+ break
+ if detail['Status'] not in ['active', 'pending']:
+ raise SystemExit(f'Import failed: {detail}')
+ time.sleep(20)
+else:
+ raise SystemExit(f'Import still pending: inspect {task} before retrying publication')
+print(f'Imported snapshot: {snapshot}', flush=True)
+ami = aws('ec2', 'register-image', '--name', f'macro-observability-{digest[:20]}',
+ '--architecture', 'x86_64', '--virtualization-type', 'hvm',
+ '--boot-mode', 'uefi', '--ena-support', '--imds-support', 'v2.0',
+ '--root-device-name', '/dev/xvda',
+ '--block-device-mappings', json.dumps([{
+ 'DeviceName': '/dev/xvda',
+ 'Ebs': {'SnapshotId': snapshot, 'VolumeType': 'gp3',
+ 'VolumeSize': 30, 'DeleteOnTermination': True},
+ }]))['ImageId']
+print(f'AMI: {ami}; SHA256: {digest}', flush=True)
+aws('ec2', 'create-tags', '--resources', ami, snapshot, '--tags',
+ 'Key=project,Value=observability', f'Key=ImageSHA256,Value={digest}')
+aws('ec2', 'wait', 'image-available', '--image-ids', ami)
+print(f'Review and smoke-boot {ami}, then set Pulumi amiId. No instance was launched.')
diff --git a/infra/stacks/observability/nixos/read-user-data.py b/infra/stacks/observability/nixos/read-user-data.py
new file mode 100644
index 0000000000..2c20f6001c
--- /dev/null
+++ b/infra/stacks/observability/nixos/read-user-data.py
@@ -0,0 +1,99 @@
+"""Validate nonsecret EC2 settings for native NixOS services."""
+import gzip
+import json
+from pathlib import Path
+import re
+import urllib.request
+
+CONFIG_ROOT = Path('/opt/observability')
+SETTING_KEYS = {
+ 'region', 'grafanaHost', 'otlpHost', 'allowedEmails', 'adminEmails',
+ 'secretArn', 'volumeId', 'logsBucket', 'tracesBucket',
+}
+
+
+def validate_payload(payload):
+ if not isinstance(payload, dict) or set(payload) != {'version', 'settings'} or payload['version'] != 3:
+ raise ValueError('Unsupported observability user-data schema')
+ settings = payload['settings']
+ if not isinstance(settings, dict) or set(settings) != SETTING_KEYS:
+ raise ValueError('Invalid observability runtime setting keys')
+ patterns = {
+ 'region': r'us-east-2',
+ 'grafanaHost': r'[a-z0-9-]+\.macro-internal\.com',
+ 'otlpHost': r'[a-z0-9-]+\.macro-internal\.com',
+ 'secretArn': r'arn:aws:secretsmanager:us-east-2:\d{12}:secret:[a-zA-Z0-9/_+=.@-]+',
+ 'volumeId': r'vol-[a-f0-9]+',
+ 'logsBucket': r'[a-z0-9][a-z0-9-]{1,61}[a-z0-9]',
+ 'tracesBucket': r'[a-z0-9][a-z0-9-]{1,61}[a-z0-9]',
+ }
+ for key, pattern in patterns.items():
+ if not isinstance(settings[key], str) or not re.fullmatch(pattern, settings[key]):
+ raise ValueError(f'Invalid runtime setting: {key}')
+ if settings['grafanaHost'] == settings['otlpHost']:
+ raise ValueError('UI and ingestion require separate hostnames')
+ for key in ['allowedEmails', 'adminEmails']:
+ emails = settings[key]
+ if not isinstance(emails, list) or not emails or any(
+ not isinstance(email, str) or not re.fullmatch(r'[a-z0-9._+-]+@macro\.com', email)
+ for email in emails
+ ):
+ raise ValueError(f'Invalid approved identity list: {key}')
+ if not set(settings['adminEmails']).issubset(settings['allowedEmails']):
+ raise ValueError('Every admin must also be approved')
+ return settings
+
+
+def write_runtime(payload, output_root):
+ settings = validate_payload(payload)
+ admin_list = json.dumps(settings['adminEmails'], separators=(',', ':'))
+ allowed_list = json.dumps(settings['allowedEmails'], separators=(',', ':'))
+ environment = {
+ 'AWS_REGION': settings['region'],
+ 'GRAFANA_HOST': settings['grafanaHost'],
+ 'GRAFANA_ROOT_URL': 'https://' + settings['grafanaHost'] + '/',
+ 'LOGS_BUCKET': settings['logsBucket'],
+ 'TRACES_BUCKET': settings['tracesBucket'],
+ 'GRAFANA_ROLE_EXPRESSION': f"contains(`{admin_list}`, email) && 'GrafanaAdmin' || contains(`{allowed_list}`, email) && 'Viewer' || 'Denied'",
+ }
+ # systemd EnvironmentFile accepts double-quoted, escaped values. Inputs are
+ # validated ASCII; no shell ever evaluates these values or the role expression.
+ files = {
+ 'runtime.env': ''.join(f'{key}={json.dumps(value)}\n' for key, value in environment.items()),
+ 'hosts.conf': (
+ 'map $host $observability_host {\n default denied;\n'
+ f" {settings['grafanaHost']} grafana;\n {settings['otlpHost']} otlp;\n"
+ '}\nmap "" $grafana_host {\n'
+ f" default {settings['grafanaHost']};\n" + '}\n'
+ ),
+ 'bootstrap.json': json.dumps({'region': settings['region'], 'secretArn': settings['secretArn']}),
+ 'volume-id': settings['volumeId'],
+ }
+ output_root.mkdir(mode=0o755, parents=True, exist_ok=True)
+ for name, content in files.items():
+ (output_root / name).write_text(content)
+
+
+def main():
+ endpoint = 'http://169.254.169.254/latest/'
+ # Metadata access must never use a configured HTTP proxy.
+ http = urllib.request.build_opener(urllib.request.ProxyHandler({}))
+ token_request = urllib.request.Request(
+ endpoint + 'api/token', method='PUT',
+ headers={'X-aws-ec2-metadata-token-ttl-seconds': '60'},
+ )
+ with http.open(token_request, timeout=10) as response:
+ token = response.read().decode()
+ request = urllib.request.Request(
+ endpoint + 'user-data', headers={'X-aws-ec2-metadata-token': token},
+ )
+ with http.open(request, timeout=10) as response:
+ raw = response.read(16385)
+ if len(raw) > 16384:
+ raise ValueError('User data exceeds EC2 limit')
+ payload = json.loads(gzip.decompress(raw))
+ write_runtime(payload, CONFIG_ROOT)
+
+
+if __name__ == '__main__':
+ main()
diff --git a/infra/stacks/observability/nixos/services.nix b/infra/stacks/observability/nixos/services.nix
new file mode 100644
index 0000000000..affa0e357a
--- /dev/null
+++ b/infra/stacks/observability/nixos/services.nix
@@ -0,0 +1,97 @@
+{
+ config,
+ lib,
+ pkgs,
+ ...
+}:
+let
+ backends = import ./backend-config.nix;
+ credential = name: "${name}:/run/macro-observability/${name}";
+ needsSecrets = {
+ requires = [ "observability-secrets.service" ];
+ after = [ "observability-secrets.service" ];
+ partOf = [ "observability-secrets.service" ];
+ };
+in
+{
+ services.grafana = {
+ enable = true;
+ dataDir = "/srv/observability/grafana";
+ settings = import ./grafana.nix;
+ provision = {
+ enable = true;
+ datasources.settings = import ./datasources.nix;
+ };
+ };
+ services.loki = {
+ enable = true;
+ dataDir = "/srv/observability/loki";
+ configuration = backends.loki;
+ extraFlags = [ "-config.expand-env=true" ];
+ };
+ services.tempo = {
+ enable = true;
+ settings = backends.tempo;
+ extraFlags = [ "-config.expand-env=true" ];
+ };
+ services.prometheus = {
+ enable = true;
+ listenAddress = "127.0.0.1";
+ retentionTime = "30d";
+ globalConfig = backends.prometheus.global;
+ scrapeConfigs = backends.prometheus.scrape_configs;
+ extraFlags = [
+ "--storage.tsdb.retention.size=100GB"
+ "--web.enable-remote-write-receiver"
+ ];
+ };
+ systemd.services = {
+ grafana = needsSecrets // {
+ serviceConfig = {
+ EnvironmentFile = "/opt/observability/runtime.env";
+ LoadCredential = map credential [
+ "google_client_id"
+ "google_client_secret"
+ "grafana_secret_key"
+ ];
+ ReadWritePaths = [ config.services.grafana.dataDir ];
+ MemoryMax = "1G";
+ };
+ };
+ loki.serviceConfig = {
+ EnvironmentFile = "/opt/observability/runtime.env";
+ ReadWritePaths = [ config.services.loki.dataDir ];
+ MemoryMax = "4G";
+ };
+ tempo.serviceConfig = {
+ EnvironmentFile = "/opt/observability/runtime.env";
+ DynamicUser = lib.mkForce false;
+ User = "tempo";
+ Group = "tempo";
+ WorkingDirectory = lib.mkForce "/srv/observability/tempo";
+ StateDirectory = lib.mkForce "";
+ ReadWritePaths = [ "/srv/observability/tempo" ];
+ MemoryMax = "3G";
+ };
+ prometheus.serviceConfig = {
+ # The upstream module manages /var/lib/prometheus2; keep its whole state
+ # directory on retained storage without overriding its generated command.
+ BindPaths = [ "/srv/observability/prometheus:/var/lib/${config.services.prometheus.stateDir}" ];
+ MemoryMax = "3G";
+ };
+ alloy-ingest = needsSecrets // {
+ description = "Unprivileged OTLP ingress";
+ wantedBy = [ "multi-user.target" ];
+ serviceConfig = {
+ User = "alloy-ingest";
+ Group = "alloy-ingest";
+ LoadCredential = [ (credential "otlp_token") ];
+ ExecStart = "${lib.getExe pkgs.grafana-alloy} run ${pkgs.writeText "ingest.alloy" (builtins.readFile ./ingest.alloy)} --storage.path=/srv/observability/alloy-ingest --server.http.listen-addr=127.0.0.1:12345 --disable-reporting";
+ ReadWritePaths = [ "/srv/observability/alloy-ingest" ];
+ MemoryMax = "1536M";
+ CapabilityBoundingSet = [ "" ];
+ PrivateDevices = true;
+ };
+ };
+ };
+}
diff --git a/infra/stacks/observability/package.json b/infra/stacks/observability/package.json
new file mode 100644
index 0000000000..d5ad5d0633
--- /dev/null
+++ b/infra/stacks/observability/package.json
@@ -0,0 +1,7 @@
+{
+ "name": "observability-stack",
+ "version": "0.0.0",
+ "private": true,
+ "license": "MIT",
+ "main": "index.ts"
+}
diff --git a/infra/stacks/observability/render.ts b/infra/stacks/observability/render.ts
new file mode 100644
index 0000000000..1ec03801e8
--- /dev/null
+++ b/infra/stacks/observability/render.ts
@@ -0,0 +1,12 @@
+import { gzipSync } from 'node:zlib';
+import { type Settings, validateSettings } from './settings';
+
+// Nix owns service configuration. User data carries only validated runtime values.
+export function renderUserData(settings: Settings): string {
+ validateSettings(settings);
+ const compressed = gzipSync(JSON.stringify({ version: 3, settings }));
+ if (compressed.length > 16 * 1024) {
+ throw new Error('Configuration exceeds the EC2 user-data limit');
+ }
+ return compressed.toString('base64');
+}
diff --git a/infra/stacks/observability/settings.ts b/infra/stacks/observability/settings.ts
new file mode 100644
index 0000000000..bb106e9ce2
--- /dev/null
+++ b/infra/stacks/observability/settings.ts
@@ -0,0 +1,88 @@
+export interface Settings {
+ region: string;
+ grafanaHost: string;
+ otlpHost: string;
+ allowedEmails: string[];
+ adminEmails: string[];
+ secretArn: string;
+ volumeId: string;
+ logsBucket: string;
+ tracesBucket: string;
+}
+
+export function validateRegion(region: string): void {
+ if (region !== 'us-east-2') {
+ throw new Error(
+ 'Observability must run in us-east-2, outside prod us-east-1'
+ );
+ }
+}
+
+export function validateRegionalArn(
+ arn: string,
+ service: 'sns' | 'secretsmanager',
+ region: string
+): void {
+ const parts = arn.split(':');
+ if (
+ parts[0] !== 'arn' ||
+ parts[1] !== 'aws' ||
+ parts[2] !== service ||
+ parts[3] !== region ||
+ !/^\d{12}$/.test(parts[4]) ||
+ (service === 'sns' &&
+ (parts.length !== 6 || !/^[a-zA-Z0-9_-]+$/.test(parts[5])))
+ ) {
+ throw new Error(`${service} ARN must reference ${region}`);
+ }
+}
+
+// Validate runtime values before they reach the NixOS configuration renderer.
+export function validateSettings(settings: Settings): void {
+ for (const host of [settings.grafanaHost, settings.otlpHost]) {
+ if (!/^[a-z0-9-]+\.macro-internal\.com$/.test(host)) {
+ throw new Error(
+ 'Observability hostnames must be subdomains of macro-internal.com'
+ );
+ }
+ }
+ if (settings.grafanaHost === settings.otlpHost) {
+ throw new Error('UI and ingestion require separate hostnames');
+ }
+ if (!settings.allowedEmails.length || !settings.adminEmails.length) {
+ throw new Error('Configure approved users and at least one admin');
+ }
+ for (const email of [...settings.allowedEmails, ...settings.adminEmails]) {
+ if (!/^[a-z0-9._+-]+@macro\.com$/.test(email)) {
+ throw new Error(
+ 'Access lists require lowercase macro.com email addresses'
+ );
+ }
+ }
+ if (
+ settings.adminEmails.some(
+ (email) => !settings.allowedEmails.includes(email)
+ )
+ ) {
+ throw new Error('Every admin must also be in allowedEmails');
+ }
+ if (
+ !/^arn:aws:secretsmanager:[a-z0-9-]+:\d{12}:secret:[a-zA-Z0-9/_+=.@-]+$/.test(
+ settings.secretArn
+ )
+ ) {
+ throw new Error(
+ 'secretArn must reference an existing Secrets Manager secret'
+ );
+ }
+ validateRegion(settings.region);
+ validateRegionalArn(settings.secretArn, 'secretsmanager', settings.region);
+ if (!/^vol-[a-f0-9]+$/.test(settings.volumeId)) {
+ throw new Error('Invalid data volume ID');
+ }
+ for (const bucket of [settings.logsBucket, settings.tracesBucket]) {
+ if (!/^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$/.test(bucket)) {
+ throw new Error('Invalid telemetry bucket name');
+ }
+ }
+}
diff --git a/infra/stacks/observability/tsconfig.json b/infra/stacks/observability/tsconfig.json
new file mode 100644
index 0000000000..4082f16a5d
--- /dev/null
+++ b/infra/stacks/observability/tsconfig.json
@@ -0,0 +1,3 @@
+{
+ "extends": "../../tsconfig.json"
+}