Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
feat(infra): add native NixOS Grafana observability pilot in Ohio (#7783)
Wolf Mermelsteind9ca08c352cf
diff --git a/infra/README.md b/infra/README.mdindex 6ddd5a0c4f..52bb3a5e07 100644--- a/infra/README.md+++ b/infra/README.md@@ -5,7 +5,7 @@ This monorepo contains the code for all Macro Cloud IaC.## IaCInfrastructure is written with [pulumi](https://www.pulumi.com/docs/) with typescript and bun.-We use AWS for all infrastructure so ask to be added to the aws org and setup [pulumi with aws](https://www.pulumi.com/registry/packages/aws/installation-configuration/). All of our infra is on `us-east-1`.+We use AWS for all infrastructure so ask to be added to the aws org and setup [pulumi with aws](https://www.pulumi.com/registry/packages/aws/installation-configuration/). Production infrastructure is in `us-east-1`; the [observability pilot](stacks/observability/README.md) runs independently in `us-east-2`.The stacks directory represents the infrastructure for each service and pulumi should be run from inside a stack directory. From `stacks/my-service/` run `pulumi up --stack [dev | prod]` to deploy.diff --git a/infra/bun.lock b/infra/bun.lockindex 475c98989a..550909f839 100644--- a/infra/bun.lock+++ b/infra/bun.lock@@ -187,6 +187,10 @@"name": "notification-service-stack","version": "0.0.0",},+ "stacks/observability": {+ "name": "observability-stack",+ "version": "0.0.0",+ },"stacks/opensearch": {"name": "opensearch-stack","version": "0.0.0",@@ -753,6 +757,8 @@"npm-run-path": ["[email protected]", "", { "dependencies": { "path-key": "^3.0.0" } }, "sha512-S48WzZW777zhNIrn7gxOlISNAqi9ZC/uQFnRdbeIHhZhCA6UqpkOT8T1G7BvfdgP4Er8gF4sUbaS0i7QvIfCWw=="],+ "observability-stack": ["observability-stack@workspace:stacks/observability"],+"onetime": ["[email protected]", "", { "dependencies": { "mimic-fn": "^2.1.0" } }, "sha512-kbpaSSGJTWdAY5KPVeMOKXSrPtr8C8C7wodJbcsd51jRnmD+GZu8Y0VoU6Dm5Z4vWr0Ig/1NKuWRKf7j5aaYSg=="],"opensearch-stack": ["opensearch-stack@workspace:stacks/opensearch"],diff --git a/infra/stacks/observability/Pulumi.yaml b/infra/stacks/observability/Pulumi.yamlnew file mode 100644index 0000000000..2c09586032--- /dev/null+++ b/infra/stacks/observability/Pulumi.yaml@@ -0,0 +1,3 @@+name: observability+runtime: nodejs+description: Single-host Grafana pilot alongside Datadogdiff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.mdnew file mode 100644index 0000000000..5a5cff6c2e--- /dev/null+++ b/infra/stacks/observability/README.md@@ -0,0 +1,359 @@+# Observability pilot++This stack starts Grafana alongside Datadog on one private EC2 instance. It+runs Grafana, Loki, Tempo, Prometheus, Alloy and nginx as native systemd services+on a prebuilt NixOS image. There is no Docker or Compose dependency. Nix owns+packages, configuration, service users, resource limits and startup dependencies:++- `nixos/services.nix`: native application services and runtime credentials.+- `nixos/grafana.nix` and `nixos/datasources.nix`: Grafana access policy and datasources.+- `nixos/backend-config.nix`: Loki, Tempo and Prometheus settings.+- `nixos/ingest.alloy`: unprivileged OTLP receiver and export pipeline.+- `nixos/host-telemetry.nix` and `nixos/host.alloy`: host metrics and journal logs.+- `nixos/cloudwatch.nix`: independent disk and memory health metrics.+- `nixos/proxy.nix`: native nginx virtual hosts and query-only backend gateway.+- `nixos/host.nix`: boot dependencies, retained storage and stable service identities.++NixOS modules generate application configuration from Nix attributes. Alloy's+pipelines live in native `.alloy` files read by Nix. Pulumi sends only+validated runtime values such as hostnames, bucket names and approved identities.+Datadog instrumentation, collection and alerts remain unchanged. Nothing sends+application telemetry here until a subsequent dual-export change is deployed.++## Architecture and storage++```text+Team browser -- HTTPS --> ALB --> nginx --> Grafana -- Google Workspace OAuth+ --> query proxy --> backends+OTEL exporter -- HTTPS + bearer token --> ALB --> nginx --> Alloy+ --> Loki --> S3 logs+ --> Tempo --> S3 traces+ --> Prometheus --> EBS+```++- One `m7i.xlarge` (4 vCPU / 16 GiB) in a new VPC's private `us-east-2a`+ subnet in Ohio, separate from production in `us-east-1`. This is a provisional+ pilot size, not a capacity commitment.+- Independent regional dependencies: two public ALB subnets, one NAT gateway,+ a regional ACM certificate, EBS/snapshots, S3, Secrets Manager and alarm topic.+ An S3 gateway endpoint keeps bucket traffic off the NAT gateway. No production+ VPC, peering, NAT or regional certificate is reused. Route53 and IAM are global.+- Separate encrypted 300 GiB gp3 volume: Grafana SQLite and plugins,+ Prometheus TSDB, Alloy metrics WAL, Loki WAL/index/cache, Tempo WAL/blocks.+ The root disk is disposable. Volume and S3 buckets are protected and retained.+- Two private, encrypted S3 buckets, scoped instance-role access, HTTPS required.+ Loki/Tempo compactors own retention; no lifecycle rule can expire live blocks.+- Defaults: 30 days of logs, 7 days of traces, metrics up to 30 days or 100 GB,+ whichever is reached first. WAL/cache usage is additional, not covered by the+ metrics limit. Review ingestion volume and disk growth before broad rollout.+- Daily EBS snapshots retained for seven days; these are crash-consistent, not+ an application-consistent or cross-region backup. Snapshot recovery can lose+ up to a day of local state. S3 does not protect telemetry still buffered locally.+- No RDS, shared filesystem, Kafka or cluster. Grafana's SQLite stores users,+ dashboards and settings, not log/trace/metric history.+- All service and OS versions are pinned by this stack's `flake.lock` on NixOS+ 26.05. Grafana plugin auto-install/update is disabled. Reviewed lock updates+ and replacement AMIs deliver upgrades; there is no installation or build at boot.++Previously received telemetry remains accessible if the production region fails.+Data that has not left production can still be lost, and a single Ohio host is+not highly available. Ohio failure is not covered by these regional snapshots+or buckets. Subsequent dual export must use independent queues and HTTPS across+regions, and account for inter-region transfer costs and latency. A production+region outage should not block Grafana login or secret retrieval in Ohio.++## Host and service collection++A native Alloy service collects CPU, memory, filesystem and network metrics+using its node exporter, including systemd unit state, tasks, restart counts and+start times. It reads selected application/systemd service logs and kernel logs+from the journal. Metrics go to Prometheus; logs go to Loki. Container discovery+and cAdvisor are unnecessary because the stack runs directly on systemd. Other+production hosts and application OTEL dual export are subsequent work.++The collector runs as a dedicated unprivileged user with journal access, separate+from the unprivileged OTLP ingress service. Its UI binds only to loopback, EC2+metadata access is blocked, and its memory limit is 1 GiB. State is retained on+EBS under `alloy`; the metrics WAL retains at most one hour of unsent samples.+Log delivery uses bounded retries and does not guarantee lossless delivery+through a long outage. Nginx access logs use a restricted format without query+strings and flow through journald into Loki, as do service errors. Host Alloy+uses the host mount namespace so filesystem metrics reflect the actual mounts;+Unix permissions restrict writes to its own retained directory.++CloudWatch Agent runs as a separate unprivileged NixOS service, independent of+the data mount and the application stack. It sends only root/data-disk utilization and memory+utilization to the `Macro/Observability` namespace using the instance role.+Alarms identify the current EC2 instance: either ext4 disk above 80% or memory+above 90% over two five-minute periods notifies the configured SNS topic.+Missing data breaches the alarm, including when the data volume is not mounted.+CloudWatch collection runs every minute. No custom disk polling script remains.++Both native agent versions are pinned by `flake.lock`; their NixOS modules own+startup and restart behavior. Verify dashboards and alarms with real EC2 data+before enabling broad production ingestion.++## Team authentication and security++Production URLs are `https://grafana.macro-internal.com` and `https://otlp.macro-internal.com`;+dev uses `grafana-dev.macro-internal.com` and `otlp-dev.macro-internal.com`.++Grafana uses a dedicated **Google Workspace OAuth client**. Only explicitly+approved lowercase `@macro.com` emails in `allowedEmails` may sign in; membership+in the domain alone is insufficient. Named `adminEmails` get Grafana server+administrator access; other approved users are Viewers. Admins must also appear+in the allowlist. Unknown identities map to an invalid role and are denied with+`role_attribute_strict`. Google hosted-domain validation, ID-token signature+validation, PKCE and refresh-token checking are enabled. Workspace administrators+must enforce MFA for the approved accounts. No password login, default admin,+anonymous access or public signup is enabled.++Sessions have a one-hour inactivity limit and eight-hour maximum. Removing an+email from configuration blocks the next login; it does **not** immediately+invalidate an existing session. For offboarding, first disable the user and+revoke their sessions in Grafana using another admin, then update the allowlist.+Keep at least two admins. This uses OSS role mapping, not Enterprise team sync.+All approved users can query the pilot's telemetry; there is no per-service data+isolation. Apply redaction in the producer pipeline before copying sensitive data.++Only the ALB is internet-facing, on TLS 1.2/1.3 port 443. The host has no public IP,+SSH key or SSH ingress. Its only ingress is port 8080 from the ALB security group.+ALB-to-host traffic is HTTP inside the VPC. Administrative access uses AWS SSM and+the operator's IAM identity. Grafana, Loki, Tempo, Prometheus and both Alloy+services bind to loopback. The default ALB action is 404. No HTTP listener is opened.++Grafana data sources use a separate internal nginx listener on port 8081 that+allows only named query endpoints and their required HTTP methods. This matters+because Viewers can call Grafana's data-source proxy: direct backend URLs would+also expose maintenance endpoints such as Tempo `/shutdown` and Loki `/flush`+(including mutating GET requests). Port 8081 binds only to loopback, and+Alloy's ingestion path is separate. New data-source features may require reviewed+additions to this query allowlist.++Ingestion is a separate hostname permitting only POST to `/v1/logs`, `/v1/traces`+and `/v1/metrics`. Alloy validates a bearer token independently of browser login.+The token cannot query telemetry through that endpoint. Requests have an 8 MiB+limit and a shared 100 requests/second limit with a 200-request burst. Only use+this token in trusted server-side collectors, never browser or mobile clients.+The initial token is shared; provision per-producer credentials before expanding+beyond trusted internal services.++Secrets Manager holds Google client credentials, the Grafana encryption key and+the ingestion token. EC2 retrieves them at startup into `/run` with restricted+file permissions. Systemd `LoadCredential` gives Grafana only its three credentials+and ingress Alloy only the OTLP token. Secret values never enter Pulumi state,+user-data, environment variables or configuration committed here. Grafana's nonsecret host+and approved-user role expression are supplied through environment variables;+the generated INI refers to them through Grafana's environment provider. Secrets+remain file references in the Nix store, never secret values. The secret must use the+AWS-managed Secrets Manager encryption key; a customer-managed key needs an+explicit scoped KMS policy addition. IMDSv2 is required with hop limit 1. Only Loki, Tempo, the bootstrap/secret helper,+CloudWatch and SSM need the instance role; application units without an AWS+requirement block metadata access. The host is still one administrative trust+boundary; split roles/hosts when stronger isolation is required.++## First deployment++The stack is deliberately absent from `.github/services-config.json`, so this PR+does not enroll it in automatic deployments. Use the repository's Pulumi backend+and AWS account `569036502058`, region `us-east-2`. Deployment requires the+existing public `macro-internal.com` Route53 zone; this stack creates its own VPC/NAT and+DNS-validated regional ACM certificate. Region validation rejects `us-east-1`.++1. Create a dedicated Google OAuth Web application under the company's Google+ organization with an **Internal** consent audience. Register exactly+ `https://grafana-dev.macro-internal.com/login/google` for dev or+ `https://grafana.macro-internal.com/login/google` for prod. Prefer separate clients and+ secrets per environment. Confirm Workspace MFA enforcement.+2. Through the approved secret-management process, create a Secrets Manager JSON+ secret in **us-east-2** containing `google_client_id`, `google_client_secret`,+ `grafana_secret_key`, and `otlp_token`. Generate independent cryptographically+ random values of at least 32 characters for the last two. Preserve+ `grafana_secret_key` through recovery; changing it can make stored credentials+ unreadable. Never paste secret values into shell commands, this file or PRs.+3. Select approved users and admins. Select an existing **us-east-2** SNS topic+ with a confirmed, monitored subscription for infrastructure alarms. Its+ delivery destination must remain accessible during a production outage.+ Configuration rejects secrets and topics in another region; the host reads+ the local secret directly, without fetching credentials from production.+4. Build, publish and smoke-boot the NixOS image as described below. Set its+ reviewed Ohio AMI ID; there is deliberately no mutable "latest image" lookup.+5. Set the nonsecret configuration below, substituting actual identifiers. No+ example account or placeholder is authorized automatically.++```bash+\cd infra+bun install --frozen-lockfile+\cd stacks/observability+pulumi stack select macro-inc/dev --create+pulumi config set aws:region us-east-2+pulumi config set amiId '<reviewed NixOS AMI ID in Ohio>'+pulumi config set secretArn '<existing Secrets Manager ARN>'+pulumi config set alarmTopicArn '<existing monitored SNS topic ARN>'+pulumi config set --path 'allowedEmails[0]' '<[email protected]>'+pulumi config set --path 'allowedEmails[1]' '<[email protected]>'+pulumi config set --path 'adminEmails[0]' '<[email protected]>'+pulumi preview --diff+```++Deploy with `pulumi up` after+reviewing the resource plan. Allow up to 15 minutes for bootstrap.+Pulumi resource creation does not prove bootstrap or OAuth has succeeded.+Do not change the region of a stack that already owns resources: that requires+an explicit migration and recovery plan for its protected storage.++Before enabling application traffic, validate all of these against AWS:++- Approved Viewer and Admin can log in; an unapproved Workspace account and a+ personal Google account cannot. Viewer cannot administer Grafana.+- All three data sources pass their health checks. A small OTLP fixture appears+ in logs, traces and metrics with the expected `service.name`.+- Missing/wrong tokens fail; valid token cannot read backend APIs. No backend+ ports or SSH are reachable directly.+- S3 objects appear after flushing; IAM uses the expected two buckets only.+- Reboot preserves data and starts services; stop/replace the host and verify+ the existing volume is reattached. Restore a snapshot to an isolated host and+ verify SQLite/Prometheus/WAL recovery before relying on the backup.+- Host/systemd metrics and service logs appear in Grafana. CloudWatch's+ instance, target-health, root/data-disk and memory alarms reach the selected+ SNS subscription. Verify disk metrics use `InstanceId`, `path` and `fstype`+ dimensions, memory uses `InstanceId`, and missing data alerts. Confirm a daily snapshot is created. Inspect disk/queue+ growth before increasing ingestion. These alarms do not cover every data-path+ failure; keep Datadog primary and add end-to-end ingestion checks next.++## Operations and recovery++### NixOS image build and publication++From `infra/stacks/observability`, on an x86_64 Linux Nix builder:++```bash+nix build . --cores 2+python3 nixos/publish-image.py result '<private Ohio image-artifact bucket>'+```++The build produces a UEFI VHD with the complete host configuration. KVM speeds up+image creation; software emulation is supported but slower. The publishing command+is an explicit AWS write: it uploads the image, imports an encrypted snapshot and+registers a private AMI in `us-east-2`. It never launches or modifies an instance.+It requires an existing private S3 artifact bucket in Ohio and an operator with+the [VM Import permissions and service role](https://docs.aws.amazon.com/vm-import/latest/userguide/required-permissions.html)+plus image registration/tagging permissions. Use `--role` for a dedicated import+role; scope its S3 access to the artifact bucket. This role is separate from the+runtime EC2 role. Publishing credentials never enter the image.++Record the printed image SHA256, import task, snapshot and AMI ID with the release.+If publication is interrupted, inspect those identifiers before retrying; do not+blindly create duplicate import tasks. Retain the previous AMI and snapshot until+the replacement has passed acceptance checks. Remove obsolete image artifacts,+AMIs and their root snapshots through the normal infrastructure cleanup process;+they are separate from the protected telemetry volume and its daily snapshots.++Review the AMI and boot it in the pilot's private subnet before enabling ingestion.+Check SSM access, the OS version, exact data volume mount, service health, reboot,+secret-outage recovery and replacement/reattachment. Pin `amiId` only after review.+The AWS image import and actual EC2 boot checks require deployment access and are+separate from local configuration and startup validation.++For host changes or security updates, update `nixos/host.nix` or the pinned input+with `nix flake update nixpkgs`, build/test/publish a new image, then change `amiId`+and review `pulumi preview`. Do not run an ad-hoc `nixos-rebuild switch` on the+host: Pulumi's pinned image is the source of truth. Rolling back an AMI does not+roll back Grafana schema migrations or telemetry state; use a tested snapshot+recovery plan when the old application version cannot read current data.++### Runtime++Use SSM Session Manager with the `instanceId` output. On the host:++```bash+sudo systemctl status observability-bootstrap observability-secrets grafana loki tempo prometheus alloy alloy-ingest nginx+sudo journalctl -u observability-bootstrap -u observability-secrets --since boot+sudo journalctl -u grafana -u loki -u tempo -u prometheus -u alloy -u alloy-ingest -u nginx --since '30 minutes ago'+```++Nonsecret runtime values arrive as versioned, compressed JSON in EC2 user-data.+`read-user-data.py` retrieves them through IMDSv2 and validates the exact schema,+region, hostnames, bucket/volume/secret identifiers and approved identity lists.+It writes a systemd environment file, nginx hostname maps, the secret's identifier+and the expected volume ID under `/opt/observability`. It does not execute+user-data or read secret values. The standard NixOS user-data evaluator is disabled.+Schema version 3 rejects older payloads, unknown settings and caller-supplied files.++`observability-bootstrap` mounts that exact EBS volume and creates each service's+data directory using stable, named NixOS users. `prepare-volume.sh` formats only+a disk with no filesystem or signatures; existing ext4 data is preserved.+Application units verify the mount before every startup, preventing fallback to+the root disk. Bootstrap retries every 30 seconds after metadata or attachment+failure, and its systemd `Upholds` relationships start dependents after recovery.++`observability-secrets` fetches the credential bundle into root-only files under+`/run`. Grafana and ingress Alloy wait for successful retrieval and receive private+credential copies from systemd. Secret-service outages retry every 30 seconds;+Loki, Tempo, Prometheus and host collection can run independently of that service.+Systemd restarts crashed processes individually with memory and filesystem limits.++Configuration/image changes require a new image and reviewed `amiId`; runtime+value changes do not need an image rebuild. Both kinds of changes replace the+instance and cause downtime. The old instance is deleted before replacement, its+volume is detached, then the new host waits for that retained volume. This pilot+has not been deployed with the earlier container layout; an existing container+installation would need an explicit ownership/path migration before using this image.++To rotate the OAuth secret or ingestion token, update Secrets Manager, then run+`sudo systemctl restart observability-secrets`. This stops Grafana and ingress+Alloy while credentials refresh, then restarts both with new private copies.+Restarting an individual application uses the last fetched bundle. For token+rotation coordinate producers and their retry queues; there is no dual-token+grace window. Do not casually rotate Grafana's encryption key.++For instance failure in the same AZ, replace the instance through Pulumi and+reuse its retained volume. Do not run two hosts against the same data directory.+For corrupt/lost data, select a known snapshot, set `dataSnapshotId`, and preview+the resulting volume/instance replacements. A deliberate recovery requires+removing Pulumi protection from the old volume before replacement; its+`retainOnDelete` setting preserves it for investigation. Verify the new volume's+ID and mounted data before accepting traffic. Cross-AZ recovery requires choosing+a private subnet in that AZ and restoring the snapshot there; the initial code+deliberately fixes the AZ. Never reformat an existing volume to resolve a mount+failure. Buckets are independent and should be reused during recovery.++This is a single point of failure. Alloy's logs/traces batch buffers and export+queues are in memory; successful OTLP acceptance is not an end-to-end durability+guarantee. Producer retries cover rejected requests, not accepted data lost in a+crash. Metrics remote-write has a local WAL. EBS-backed backend WALs protect data+already delivered to Loki/Tempo. Establish failure/loss tolerance while Datadog+still receives the authoritative copy.++## Validation and subsequent passes++From `infra/`, run `bunx biome check stacks/observability` and `bun run check`.+Build the complete NixOS image from this stack directory with `nix build . --cores 2`.+Before publishing an image, validate native service startup, Google authorization redirects,+anonymous/token denial, three-signal ingestion/readback, S3 flushing, process+restart recovery and systemd credential/storage permissions in an isolated local+fixture. Real Google login, AWS IAM, EC2 attachment/boot, ALB/TLS and snapshot+restore remain the deployment acceptance checks above.++Follow-up PRs:++1. Duplicate OTEL at the existing collector to both Datadog and this stack with+ independent bounded queues, retries and matching sampling. Check actual+ metrics temporality and attribute mapping before enabling all services.+2. Inventory non-OTEL sources: Datadog agents, infrastructure metrics, CloudWatch,+ RDS/Postgres query monitoring and logs, browser RUM, synthetics, profiling and+ security features. They are not covered by duplicating OTLP. Add DB collectors+ with dedicated read-only monitoring identities and evaluate query-level+ parity before replacing Datadog DBM.+3. Port dashboards/alerts and run representative incidents in both tools. Add+ pipeline/queue alerts, ingestion canaries, host/root-disk monitoring and a+ tested restore procedure. Measure volume, cost, query latency and data gaps.+4. Decide whether to split services or add replicas. Loki/Tempo can reuse S3;+ metrics need a deliberate remote-store/migration plan. Move Grafana metadata+ to managed Postgres/RDS before multiple Grafana instances. Cut off Datadog only+ after coverage and operational acceptance are demonstrated.++References: [Google OAuth](https://grafana.com/docs/grafana/latest/setup-grafana/configure-access/configure-authentication/google/),+[Alloy bearer authentication](https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.auth.bearer/),+[Loki retention](https://grafana.com/docs/loki/latest/operations/storage/retention/),+[Tempo supported versions](https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/recommended-versions/).diff --git a/infra/stacks/observability/assets/prepare-volume.sh b/infra/stacks/observability/assets/prepare-volume.shnew file mode 100644index 0000000000..54de6df53a--- /dev/null+++ b/infra/stacks/observability/assets/prepare-volume.sh@@ -0,0 +1,13 @@+#!/bin/bash+set -euo pipefail+device=$1+filesystem=$(blkid -s TYPE -o value "$device" || true)+if [ -z "$filesystem" ]; then+ # Fail closed if inspection fails, even when it produces no output.+ signatures=$(wipefs --no-act --noheadings --output TYPE "$device")+ test -z "$signatures"+ mkfs.ext4 "$device"+elif [ "$filesystem" != ext4 ]; then+ echo 'Refusing to format an existing data volume' >&2+ exit 1+fidiff --git a/infra/stacks/observability/assets/refresh-secrets.py b/infra/stacks/observability/assets/refresh-secrets.pynew file mode 100644index 0000000000..404640c16b--- /dev/null+++ b/infra/stacks/observability/assets/refresh-secrets.py@@ -0,0 +1,36 @@+#!/usr/bin/env python3+"""Fetch credentials without putting their contents in user-data or logs."""+import json+import os+from pathlib import Path+import subprocess++config = json.loads(Path('/opt/observability/bootstrap.json').read_text())+result = subprocess.run(+ ['aws', 'secretsmanager', 'get-secret-value', '--region', config['region'],+ '--secret-id', config['secretArn'], '--output', 'json'],+ check=True, capture_output=True, text=True,+)+values = json.loads(json.loads(result.stdout)['SecretString'])+keys = ['google_client_id', 'google_client_secret', 'grafana_secret_key', 'otlp_token']+# Validate the entire bundle before writing any file. Token is at least 32 bytes.+for key in keys:+ value = values.get(key)+ if not isinstance(value, str) or not value or value.strip() != value:+ raise SystemExit(f'Missing or invalid secret field: {key}')+ if '\n' in value or '\r' in value:+ raise SystemExit(f'Secret must be a single line: {key}')+for key in ['grafana_secret_key', 'otlp_token']:+ if len(values[key]) < 32:+ raise SystemExit(f'Secret too short: {key}')+directory = Path('/run/macro-observability')+directory.mkdir(mode=0o700, exist_ok=True)+os.chown(directory, 0, 0)+for key in keys:+ path = directory / key+ # Dependent services stop during refresh and receive private systemd credentials.+ path.unlink(missing_ok=True)+ fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o400)+ with os.fdopen(fd, 'w') as output:+ output.write(values[key])+ os.fchown(output.fileno(), 0, 0)diff --git a/infra/stacks/observability/flake.lock b/infra/stacks/observability/flake.locknew file mode 100644index 0000000000..ba6568a3d0--- /dev/null+++ b/infra/stacks/observability/flake.lock@@ -0,0 +1,27 @@+{+ "nodes": {+ "nixpkgs": {+ "locked": {+ "lastModified": 1791426274,+ "narHash": "sha256-MWhktfFtFE8Jl7bf9Pkp7TR5MSdYCjCswSo0Gkb4ywc=",+ "owner": "NixOS",+ "repo": "nixpkgs",+ "rev": "7c8764b7c7b09b34f632464276218ef9090eaa11",+ "type": "github"+ },+ "original": {+ "owner": "NixOS",+ "ref": "nixos-26.05",+ "repo": "nixpkgs",+ "type": "github"+ }+ },+ "root": {+ "inputs": {+ "nixpkgs": "nixpkgs"+ }+ }+ },+ "root": "root",+ "version": 7+}diff --git a/infra/stacks/observability/flake.nix b/infra/stacks/observability/flake.nixnew file mode 100644index 0000000000..073cfffa19--- /dev/null+++ b/infra/stacks/observability/flake.nix@@ -0,0 +1,25 @@+{+ description = "Macro observability EC2 image";+ inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";++ outputs =+ { nixpkgs, ... }:+ let+ system = "x86_64-linux";+ host = nixpkgs.lib.nixosSystem {+ inherit system;+ modules = [+ "${nixpkgs}/nixos/maintainers/scripts/ec2/amazon-image.nix"+ ./nixos/host.nix+ ];+ };+ in+ {+ nixosConfigurations.observability = host;+ packages.${system} = {+ default = host.config.system.build.amazonImage;++ };+ formatter.${system} = nixpkgs.legacyPackages.${system}.nixfmt;+ };+}diff --git a/infra/stacks/observability/index.ts b/infra/stacks/observability/index.tsnew file mode 100644index 0000000000..4d99c5719a--- /dev/null+++ b/infra/stacks/observability/index.ts@@ -0,0 +1,430 @@+import * as aws from '@pulumi/aws';+import * as pulumi from '@pulumi/pulumi';+import { createNetwork } from './network';+import { renderUserData } from './render';+import {+ validateRegion,+ validateRegionalArn,+ validateSettings,+} from './settings';++const config = new pulumi.Config();+const stack = pulumi.getStack();+if (!['dev', 'prod'].includes(stack)) {+ throw new Error('Use the dev or prod stack to select a unique hostname');+}+const region = aws.config.requireRegion();+validateRegion(region);+const baseDomain = 'macro-internal.com';+const suffix = stack === 'prod' ? '' : '-dev';+const grafanaHost = `grafana${suffix}.${baseDomain}`;+const otlpHost = `otlp${suffix}.${baseDomain}`;+const tags = { project: 'observability', environment: stack };+const durable = { protect: true, retainOnDelete: true };+const secretArn = config.require('secretArn');+const allowedEmails = config.requireObject<string[]>('allowedEmails');+const adminEmails = config.requireObject<string[]>('adminEmails');+const alarmTopicArn = config.require('alarmTopicArn');+// Built from this stack's pinned NixOS flake and published in Ohio.+const amiId = config.require('amiId');+if (!/^ami-[a-f0-9]+$/.test(amiId)) {+ throw new Error('amiId must identify the reviewed observability NixOS image');+}+// Validate operator input before registering any infrastructure resources.+validateSettings({+ region,+ grafanaHost,+ otlpHost,+ allowedEmails,+ adminEmails,+ secretArn,+ volumeId: 'vol-0',+ logsBucket: 'validate-logs',+ tracesBucket: 'validate-traces',+});+validateRegionalArn(alarmTopicArn, 'sns', region);+const vpc = createNetwork(region, tags);+const subnet = vpc.privateSubnet;+const subnetId = subnet.id;++// Route53 is global; issuance and use of the certificate are both in Ohio.+const zone = aws.route53.getZoneOutput({+ name: baseDomain,+ privateZone: false,+});+const certificate = new aws.acm.Certificate('observability', {+ domainName: grafanaHost,+ subjectAlternativeNames: [otlpHost],+ validationMethod: 'DNS',+ tags,+});+const validationRecords = [grafanaHost, otlpHost].map((hostname) => {+ const option = certificate.domainValidationOptions.apply((options) => {+ const match = options.find((entry) => entry.domainName === hostname);+ if (!match) throw new Error(`Missing ACM validation for ${hostname}`);+ return match;+ });+ return new aws.route53.Record(`observability-certificate-${hostname}`, {+ zoneId: zone.zoneId,+ name: option.resourceRecordName,+ type: option.resourceRecordType,+ records: [option.resourceRecordValue],+ ttl: 300,+ });+});+const validatedCertificate = new aws.acm.CertificateValidation(+ 'observability',+ {+ certificateArn: certificate.arn,+ validationRecordFqdns: validationRecords.map((record) => record.fqdn),+ }+);++function telemetryBucket(name: string) {+ const bucket = new aws.s3.BucketV2(+ name,+ { forceDestroy: false, tags },+ durable+ );+ new aws.s3.BucketPublicAccessBlock(`${name}-private`, {+ bucket: bucket.id,+ blockPublicAcls: true,+ blockPublicPolicy: true,+ ignorePublicAcls: true,+ restrictPublicBuckets: true,+ });+ new aws.s3.BucketServerSideEncryptionConfigurationV2(`${name}-encryption`, {+ bucket: bucket.id,+ rules: [{ applyServerSideEncryptionByDefault: { sseAlgorithm: 'AES256' } }],+ });+ new aws.s3.BucketPolicy(`${name}-tls`, {+ bucket: bucket.id,+ policy: bucket.arn.apply((arn) =>+ JSON.stringify({+ Version: '2012-10-17',+ Statement: [+ {+ Effect: 'Deny',+ Principal: '*',+ Action: 's3:*',+ Resource: [arn, `${arn}/*`],+ Condition: { Bool: { 'aws:SecureTransport': 'false' } },+ },+ ],+ })+ ),+ });+ // Loki/Tempo own retention. Do not expire active blocks with S3 lifecycle.+ new aws.s3.BucketLifecycleConfigurationV2(`${name}-multipart`, {+ bucket: bucket.id,+ rules: [+ {+ id: 'abort-incomplete-uploads',+ status: 'Enabled',+ filter: { prefix: '' },+ abortIncompleteMultipartUpload: { daysAfterInitiation: 1 },+ },+ ],+ });+ return bucket;+}+const logs = telemetryBucket('observability-logs');+const traces = telemetryBucket('observability-traces');+const data = new aws.ebs.Volume(+ 'observability-data',+ {+ availabilityZone: subnet.availabilityZone,+ type: 'gp3',+ size: 300,+ encrypted: true,+ snapshotId: config.get('dataSnapshotId'),+ tags: { ...tags, ObservabilityBackup: stack },+ },+ durable+);++const role = new aws.iam.Role('observability-host', {+ assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({+ Service: 'ec2.amazonaws.com',+ }),+ tags,+});+const ssm = new aws.iam.RolePolicyAttachment('observability-ssm', {+ role: role.name,+ policyArn: 'arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore',+});+const policy = new aws.iam.RolePolicy('observability-storage', {+ role: role.id,+ policy: pulumi.all([logs.arn, traces.arn]).apply((arns) =>+ JSON.stringify({+ Version: '2012-10-17',+ Statement: [+ {+ Effect: 'Allow',+ Action: [+ 's3:ListBucket',+ 's3:GetBucketLocation',+ 's3:ListBucketMultipartUploads',+ ],+ Resource: arns,+ },+ {+ Effect: 'Allow',+ Action: [+ 's3:GetObject',+ 's3:PutObject',+ 's3:DeleteObject',+ 's3:AbortMultipartUpload',+ 's3:ListMultipartUploadParts',+ ],+ Resource: arns.map((arn) => `${arn}/*`),+ },+ {+ Effect: 'Allow',+ Action: ['secretsmanager:GetSecretValue'],+ Resource: secretArn,+ },+ {+ Effect: 'Allow',+ Action: ['cloudwatch:PutMetricData'],+ Resource: '*',+ Condition: {+ StringEquals: { 'cloudwatch:namespace': 'Macro/Observability' },+ },+ },+ ],+ })+ ),+});+const profile = new aws.iam.InstanceProfile('observability-host', {+ role: role.name,+});+const albSg = new aws.ec2.SecurityGroup('observability-alb', {+ vpcId: vpc.vpcId,+ description: 'Public HTTPS only',+ ingress: [+ { protocol: 'tcp', fromPort: 443, toPort: 443, cidrBlocks: ['0.0.0.0/0'] },+ ],+ tags,+});+const hostSg = new aws.ec2.SecurityGroup('observability-host', {+ vpcId: vpc.vpcId,+ description: 'Only the ALB can reach the proxy; administration via SSM',+ ingress: [+ {+ protocol: 'tcp',+ fromPort: 8080,+ toPort: 8080,+ securityGroups: [albSg.id],+ },+ ],+ egress: [+ { protocol: '-1', fromPort: 0, toPort: 0, cidrBlocks: ['0.0.0.0/0'] },+ ],+ tags,+});+new aws.ec2.SecurityGroupRule('observability-alb-to-host', {+ securityGroupId: albSg.id,+ type: 'egress',+ protocol: 'tcp',+ fromPort: 8080,+ toPort: 8080,+ sourceSecurityGroupId: hostSg.id,+});+const instance = new aws.ec2.Instance(+ 'observability',+ {+ ami: amiId,+ instanceType: 'm7i.xlarge',+ subnetId,+ associatePublicIpAddress: false,+ vpcSecurityGroupIds: [hostSg.id],+ iamInstanceProfile: profile.name,+ metadataOptions: {+ httpEndpoint: 'enabled',+ httpTokens: 'required',+ httpPutResponseHopLimit: 1,+ },+ rootBlockDevice: {+ volumeType: 'gp3',+ volumeSize: 30,+ encrypted: true,+ deleteOnTermination: true,+ },+ userDataReplaceOnChange: true,+ userDataBase64: pulumi+ .all([data.id, logs.bucket, traces.bucket])+ .apply(([volumeId, logsBucket, tracesBucket]) =>+ renderUserData({+ region,+ grafanaHost,+ otlpHost,+ allowedEmails,+ adminEmails,+ secretArn,+ volumeId,+ logsBucket,+ tracesBucket,+ })+ ),+ tags,+ },+ { deleteBeforeReplace: true, dependsOn: [policy, ssm, ...vpc.privateReady] }+);+new aws.ec2.VolumeAttachment('observability-data', {+ deviceName: '/dev/sdf',+ volumeId: data.id,+ instanceId: instance.id,+ stopInstanceBeforeDetaching: true,+ forceDetach: false,+});++const alb = new aws.lb.LoadBalancer(+ 'observability',+ {+ loadBalancerType: 'application',+ internal: false,+ subnets: vpc.publicSubnetIds,+ securityGroups: [albSg.id],+ dropInvalidHeaderFields: true,+ desyncMitigationMode: 'strictest',+ tags,+ },+ { dependsOn: vpc.publicReady }+);+const target = new aws.lb.TargetGroup('observability', {+ port: 8080,+ protocol: 'HTTP',+ targetType: 'instance',+ vpcId: vpc.vpcId,+ healthCheck: { path: '/healthz', matcher: '200', interval: 30 },+ deregistrationDelay: 30,+ tags,+});+new aws.lb.TargetGroupAttachment('observability', {+ targetGroupArn: target.arn,+ targetId: instance.id,+ port: 8080,+});+const listener = new aws.lb.Listener('observability-https', {+ loadBalancerArn: alb.arn,+ port: 443,+ protocol: 'HTTPS',+ certificateArn: validatedCertificate.certificateArn,+ sslPolicy: 'ELBSecurityPolicy-TLS13-1-2-2021-06',+ defaultActions: [+ {+ type: 'fixed-response',+ fixedResponse: { contentType: 'text/plain', statusCode: '404' },+ },+ ],+});+new aws.lb.ListenerRule('observability-hosts', {+ listenerArn: listener.arn,+ priority: 1,+ conditions: [{ hostHeader: { values: [grafanaHost, otlpHost] } }],+ actions: [{ type: 'forward', targetGroupArn: target.arn }],+});+for (const hostname of [grafanaHost, otlpHost]) {+ new aws.route53.Record(hostname, {+ zoneId: zone.zoneId,+ name: hostname,+ type: 'A',+ aliases: [+ { name: alb.dnsName, zoneId: alb.zoneId, evaluateTargetHealth: true },+ ],+ });+}++const backupRole = new aws.iam.Role('observability-snapshots', {+ assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({+ Service: 'dlm.amazonaws.com',+ }),+ tags,+});+const backupPolicy = new aws.iam.RolePolicyAttachment(+ 'observability-snapshots',+ {+ role: backupRole.name,+ policyArn:+ 'arn:aws:iam::aws:policy/service-role/AWSDataLifecycleManagerServiceRole',+ }+);+new aws.dlm.LifecyclePolicy(+ 'observability-snapshots',+ {+ description: 'Daily crash-consistent data volume snapshots retained 7 days',+ executionRoleArn: backupRole.arn,+ state: 'ENABLED',+ policyDetails: {+ resourceTypes: ['VOLUME'],+ targetTags: { ObservabilityBackup: stack },+ schedules: [+ {+ name: 'daily',+ copyTags: true,+ createRule: { interval: 24, intervalUnit: 'HOURS', times: '04:00' },+ retainRule: { count: 7 },+ },+ ],+ },+ tags,+ },+ { dependsOn: [backupPolicy] }+);+const alarm = {+ comparisonOperator: 'GreaterThanThreshold',+ evaluationPeriods: 2,+ period: 60,+ statistic: 'Maximum',+ alarmActions: [alarmTopicArn],+ okActions: [alarmTopicArn],+ tags,+};+new aws.cloudwatch.MetricAlarm('observability-instance', {+ ...alarm,+ namespace: 'AWS/EC2',+ metricName: 'StatusCheckFailed',+ threshold: 0,+ dimensions: { InstanceId: instance.id },+ treatMissingData: 'breaching',+});+new aws.cloudwatch.MetricAlarm('observability-ui', {+ ...alarm,+ namespace: 'AWS/ApplicationELB',+ metricName: 'UnHealthyHostCount',+ threshold: 0,+ dimensions: { LoadBalancer: alb.arnSuffix, TargetGroup: target.arnSuffix },+ treatMissingData: 'breaching',+});+for (const [name, path] of [+ ['observability-disk', '/srv/observability'],+ ['observability-root-disk', '/'],+]) {+ new aws.cloudwatch.MetricAlarm(name, {+ ...alarm,+ namespace: 'Macro/Observability',+ metricName: 'disk_used_percent',+ threshold: 80,+ dimensions: { InstanceId: instance.id, path, fstype: 'ext4' },+ period: 300,+ treatMissingData: 'breaching',+ });+}+new aws.cloudwatch.MetricAlarm('observability-memory', {+ ...alarm,+ namespace: 'Macro/Observability',+ metricName: 'mem_used_percent',+ threshold: 90,+ dimensions: { InstanceId: instance.id },+ period: 300,+ treatMissingData: 'breaching',+});++export const grafanaUrl = `https://${grafanaHost}`;+export const otlpHttpEndpoint = `https://${otlpHost}`;+export const instanceId = instance.id;+export const dataVolumeId = data.id;+export const logsBucket = logs.bucket;+export const tracesBucket = traces.bucket;+export const observabilityRegion = region;diff --git a/infra/stacks/observability/network.ts b/infra/stacks/observability/network.tsnew file mode 100644index 0000000000..b991894037--- /dev/null+++ b/infra/stacks/observability/network.ts@@ -0,0 +1,92 @@+import * as aws from '@pulumi/aws';++// Independent of prod's VPC, NAT, and regional endpoints. One private host+// subnet and one NAT are intentional for the single-instance pilot.+export function createNetwork(region: string, tags: Record<string, string>) {+ const vpc = new aws.ec2.Vpc('observability', {+ cidrBlock: '10.88.0.0/16',+ enableDnsSupport: true,+ enableDnsHostnames: true,+ tags,+ });+ const internet = new aws.ec2.InternetGateway('observability', {+ vpcId: vpc.id,+ tags,+ });+ const publicRoutes = new aws.ec2.RouteTable('observability-public', {+ vpcId: vpc.id,+ tags,+ });+ const internetRoute = new aws.ec2.Route('observability-internet', {+ routeTableId: publicRoutes.id,+ destinationCidrBlock: '0.0.0.0/0',+ gatewayId: internet.id,+ });+ const publicSubnets = ['a', 'b'].map(+ (zone, index) =>+ new aws.ec2.Subnet(`observability-public-${zone}`, {+ vpcId: vpc.id,+ availabilityZone: `${region}${zone}`,+ cidrBlock: `10.88.${index}.0/24`,+ mapPublicIpOnLaunch: false,+ tags,+ })+ );+ const publicAssociations = publicSubnets.map(+ (subnet, index) =>+ new aws.ec2.RouteTableAssociation(`observability-public-${index}`, {+ subnetId: subnet.id,+ routeTableId: publicRoutes.id,+ })+ );+ const natAddress = new aws.ec2.Eip('observability-nat', {+ domain: 'vpc',+ tags,+ });+ const nat = new aws.ec2.NatGateway(+ 'observability',+ {+ subnetId: publicSubnets[0].id,+ allocationId: natAddress.id,+ tags,+ },+ { dependsOn: [internetRoute, ...publicAssociations] }+ );+ const privateSubnet = new aws.ec2.Subnet('observability-private', {+ vpcId: vpc.id,+ availabilityZone: `${region}a`,+ cidrBlock: '10.88.10.0/24',+ mapPublicIpOnLaunch: false,+ tags,+ });+ const privateRoutes = new aws.ec2.RouteTable('observability-private', {+ vpcId: vpc.id,+ tags,+ });+ const natRoute = new aws.ec2.Route('observability-egress', {+ routeTableId: privateRoutes.id,+ destinationCidrBlock: '0.0.0.0/0',+ natGatewayId: nat.id,+ });+ const privateAssociation = new aws.ec2.RouteTableAssociation(+ 'observability-private',+ {+ subnetId: privateSubnet.id,+ routeTableId: privateRoutes.id,+ }+ );+ const s3 = new aws.ec2.VpcEndpoint('observability-s3', {+ vpcId: vpc.id,+ serviceName: `com.amazonaws.${region}.s3`,+ vpcEndpointType: 'Gateway',+ routeTableIds: [privateRoutes.id],+ tags,+ });+ return {+ vpcId: vpc.id,+ privateSubnet,+ publicSubnetIds: publicSubnets.map((subnet) => subnet.id),+ publicReady: [internetRoute, ...publicAssociations],+ privateReady: [natRoute, privateAssociation, s3],+ };+}diff --git a/infra/stacks/observability/nixos/backend-config.nix b/infra/stacks/observability/nixos/backend-config.nixnew file mode 100644index 0000000000..c31221baac--- /dev/null+++ b/infra/stacks/observability/nixos/backend-config.nix@@ -0,0 +1,100 @@+{+ loki = {+ auth_enabled = false;+ server = {+ http_listen_address = "127.0.0.1";+ http_listen_port = 3100;+ grpc_listen_address = "127.0.0.1";+ grpc_listen_port = 9095;+ };+ common = {+ instance_addr = "127.0.0.1";+ path_prefix = "/srv/observability/loki";+ replication_factor = 1;+ ring.kvstore.store = "inmemory";+ };+ schema_config.configs = [+ {+ from = "2024-01-01";+ store = "tsdb";+ object_store = "s3";+ schema = "v13";+ index = {+ prefix = "index_";+ period = "24h";+ };+ }+ ];+ storage_config = {+ aws = {+ region = "\${AWS_REGION}";+ bucketnames = "\${LOGS_BUCKET}";+ };+ tsdb_shipper = {+ active_index_directory = "/srv/observability/loki/index";+ cache_location = "/srv/observability/loki/index-cache";+ };+ };+ ingester.wal = {+ enabled = true;+ dir = "/srv/observability/loki/wal";+ };+ compactor = {+ working_directory = "/srv/observability/loki/compactor";+ retention_enabled = true;+ delete_request_store = "s3";+ };+ limits_config = {+ retention_period = "720h";+ allow_structured_metadata = true;+ ingestion_rate_mb = 8;+ ingestion_burst_size_mb = 16;+ max_query_parallelism = 4;+ };+ querier.max_concurrent = 4;+ analytics.reporting_enabled = false;+ };+ tempo = {+ server = {+ http_listen_address = "127.0.0.1";+ http_listen_port = 3200;+ grpc_listen_address = "127.0.0.1";+ grpc_listen_port = 9096;+ };+ memberlist.bind_addr = [ "127.0.0.1" ];+ distributor.receivers.otlp.protocols.grpc.endpoint = "127.0.0.1:4317";+ ingester = {+ max_block_duration = "5m";+ lifecycler.address = "127.0.0.1";+ };+ compactor.compaction.block_retention = "168h";+ storage.trace = {+ backend = "s3";+ wal.path = "/srv/observability/tempo/wal";+ s3 = {+ bucket = "\${TRACES_BUCKET}";+ region = "\${AWS_REGION}";+ endpoint = "s3.\${AWS_REGION}.amazonaws.com";+ };+ };+ usage_report.reporting_enabled = false;+ };+ prometheus = {+ global.scrape_interval = "30s";+ scrape_configs = [+ {+ job_name = "observability";+ static_configs = [+ {+ targets = [+ "127.0.0.1:9090"+ "127.0.0.1:3100"+ "127.0.0.1:3200"+ "127.0.0.1:12345"+ ];+ }+ ];+ }+ ];+ };+}diff --git a/infra/stacks/observability/nixos/cloudwatch.nix b/infra/stacks/observability/nixos/cloudwatch.nixnew file mode 100644index 0000000000..9493ea29c1--- /dev/null+++ b/infra/stacks/observability/nixos/cloudwatch.nix@@ -0,0 +1,25 @@+{+ agent = {+ region = "us-east-2";+ metrics_collection_interval = 60;+ omit_hostname = true;+ usage_data = false;+ logfile = "";+ };+ metrics = {+ namespace = "Macro/Observability";+ append_dimensions.InstanceId = "\${aws:InstanceId}";+ metrics_collected = {+ disk = {+ measurement = [ "used_percent" ];+ resources = [+ "/"+ "/srv/observability"+ ];+ drop_device = true;+ };+ mem.measurement = [ "used_percent" ];+ };+ force_flush_interval = 60;+ };+}diff --git a/infra/stacks/observability/nixos/datasources.nix b/infra/stacks/observability/nixos/datasources.nixnew file mode 100644index 0000000000..f9c39326be--- /dev/null+++ b/infra/stacks/observability/nixos/datasources.nix@@ -0,0 +1,30 @@+{+ apiVersion = 1;+ datasources = [+ {+ name = "Prometheus";+ uid = "prometheus";+ type = "prometheus";+ access = "proxy";+ url = "http://127.0.0.1:8081/prometheus";+ editable = false;+ isDefault = true;+ }+ {+ name = "Loki";+ uid = "loki";+ type = "loki";+ access = "proxy";+ url = "http://127.0.0.1:8081/loki";+ editable = false;+ }+ {+ name = "Tempo";+ uid = "tempo";+ type = "tempo";+ access = "proxy";+ url = "http://127.0.0.1:8081/tempo";+ editable = false;+ }+ ];+}diff --git a/infra/stacks/observability/nixos/grafana.nix b/infra/stacks/observability/nixos/grafana.nixnew file mode 100644index 0000000000..ff61f97816--- /dev/null+++ b/infra/stacks/observability/nixos/grafana.nix@@ -0,0 +1,61 @@+{+ "server" = {+ http_addr = "127.0.0.1";+ domain = "$__env{GRAFANA_HOST}";+ root_url = "$__env{GRAFANA_ROOT_URL}";+ enforce_domain = true;+ };+ "security" = {+ disable_initial_admin_creation = true;+ secret_key = "$__file{/run/credentials/grafana.service/grafana_secret_key}";+ cookie_secure = true;+ cookie_samesite = "lax";+ strict_transport_security = true;+ content_security_policy = true;+ };+ "users" = {+ allow_sign_up = false;+ allow_org_create = false;+ viewers_can_edit = false;+ };+ "auth" = {+ disable_login_form = true;+ login_maximum_inactive_lifetime_duration = "1h";+ login_maximum_lifetime_duration = "8h";+ };+ "auth.basic" = {+ enabled = false;+ };+ "auth.anonymous" = {+ enabled = false;+ };+ "auth.google" = {+ enabled = true;+ allow_sign_up = true;+ client_id = "$__file{/run/credentials/grafana.service/google_client_id}";+ client_secret = "$__file{/run/credentials/grafana.service/google_client_secret}";+ scopes = "openid email profile";+ auth_url = "https://accounts.google.com/o/oauth2/v2/auth";+ token_url = "https://oauth2.googleapis.com/token";+ api_url = "https://openidconnect.googleapis.com/v1/userinfo";+ allowed_domains = "macro.com";+ hosted_domain = "macro.com";+ validate_hd = true;+ use_pkce = true;+ use_refresh_token = true;+ validate_id_token = true;+ jwk_set_url = "https://www.googleapis.com/oauth2/v3/certs";+ role_attribute_path = "$__env{GRAFANA_ROLE_EXPRESSION}";+ role_attribute_strict = true;+ allow_assign_grafana_admin = true;+ skip_org_role_sync = false;+ };+ "analytics" = {+ reporting_enabled = false;+ check_for_updates = false;+ };+ "plugins" = {+ preinstall_disabled = true;+ preinstall_auto_update = false;+ };+}diff --git a/infra/stacks/observability/nixos/host-telemetry.nix b/infra/stacks/observability/nixos/host-telemetry.nixnew file mode 100644index 0000000000..4a6cf0c046--- /dev/null+++ b/infra/stacks/observability/nixos/host-telemetry.nix@@ -0,0 +1,50 @@+{ lib, pkgs, ... }:+{+ services.alloy = {+ enable = true;+ configPath = pkgs.writeText "host.alloy" (builtins.readFile ./host.alloy);+ extraFlags = [+ "--server.http.listen-addr=127.0.0.1:12346"+ "--storage.path=/srv/observability/alloy"+ "--disable-reporting"+ ];+ };+ # Journal access is separate from the externally reachable OTLP receiver.+ users.users.alloy.extraGroups = [ "systemd-journal" ];+ systemd.services.alloy.serviceConfig = {+ DynamicUser = lib.mkForce false;+ User = "alloy";+ Group = "alloy";+ # No private mount namespace: filesystem metrics must describe the host,+ # not sandbox read-only bind mounts. Unix permissions restrict this user.+ StateDirectory = lib.mkForce "";+ WorkingDirectory = lib.mkForce "/srv/observability/alloy";+ MemoryMax = "1G";+ };++ users.groups.cloudwatch-agent = { };+ users.users.cloudwatch-agent = {+ isSystemUser = true;+ group = "cloudwatch-agent";+ };+ services.amazon-cloudwatch-agent = {+ enable = true;+ user = "cloudwatch-agent";+ mode = "ec2";+ configuration = import ./cloudwatch.nix;+ };+ # Independent of Alloy, the data mount and the Grafana stack. Missing+ # data-disk samples must reach the alarm's missing-data policy, not a fallback.+ systemd.services.amazon-cloudwatch-agent = {+ after = [ "network-online.target" ];+ wants = [ "network-online.target" ];+ unitConfig.StartLimitIntervalSec = 0;+ serviceConfig = {+ NoNewPrivileges = true;+ ProtectSystem = "strict";+ ProtectHome = true;+ PrivateTmp = true;+ MemoryMax = "512M";+ };+ };+}diff --git a/infra/stacks/observability/nixos/host.alloy b/infra/stacks/observability/nixos/host.alloynew file mode 100644index 0000000000..7a41cb78a1--- /dev/null+++ b/infra/stacks/observability/nixos/host.alloy@@ -0,0 +1,68 @@+prometheus.exporter.unix "host" {+ disable_collectors = ["textfile"]+ enable_collectors = ["systemd"]+ systemd {+ unit_include = "(grafana|loki|tempo|prometheus|nginx|alloy|alloy-ingest|observability-.*|amazon-cloudwatch-agent)\\.service"+ task_metrics = true+ enable_restarts = true+ start_time = true+ }+}++prometheus.scrape "host" {+ targets = prometheus.exporter.unix.host.targets+ job_name = "observability/host"+ scrape_interval = "30s"+ forward_to = [prometheus.remote_write.local.receiver]+}++prometheus.exporter.self "host_alloy" {}+prometheus.scrape "host_alloy" {+ targets = prometheus.exporter.self.host_alloy.targets+ job_name = "observability/host-alloy"+ scrape_interval = "30s"+ forward_to = [prometheus.remote_write.local.receiver]+}++prometheus.remote_write "local" {+ endpoint { url = "http://127.0.0.1:9090/api/v1/write" }+ wal {+ min_keepalive_time = "5m"+ max_keepalive_time = "1h"+ }+}++loki.relabel "journal" {+ forward_to = []+ rule {+ source_labels = ["__journal__systemd_unit"]+ regex = "(grafana|loki|tempo|prometheus|nginx|alloy|alloy-ingest|observability-.*|amazon-cloudwatch-agent|systemd-.*)\\.service"+ action = "keep"+ }+ rule {+ source_labels = ["__journal__systemd_unit"]+ target_label = "unit"+ }+ rule {+ source_labels = ["__journal_priority_keyword"]+ target_label = "level"+ }+}++loki.source.journal "host" {+ max_age = "1h"+ labels = {job = "observability/journal", instance = constants.hostname}+ relabel_rules = loki.relabel.journal.rules+ forward_to = [loki.write.local.receiver]+}++loki.source.journal "kernel" {+ matches = "_TRANSPORT=kernel"+ max_age = "1h"+ labels = {job = "observability/kernel", instance = constants.hostname}+ forward_to = [loki.write.local.receiver]+}++loki.write "local" {+ endpoint { url = "http://127.0.0.1:3100/loki/api/v1/push" }+}diff --git a/infra/stacks/observability/nixos/host.nix b/infra/stacks/observability/nixos/host.nixnew file mode 100644index 0000000000..f5715ac459--- /dev/null+++ b/infra/stacks/observability/nixos/host.nix@@ -0,0 +1,144 @@+{ lib, pkgs, ... }:+let+ runtimePath = with pkgs; [+ awscli2+ bash+ coreutils+ e2fsprogs+ python3+ util-linux+ ];+ services = [+ "grafana"+ "loki"+ "tempo"+ "prometheus"+ "alloy"+ "alloy-ingest"+ "nginx"+ ];+ secretConsumers = [+ "grafana.service"+ "alloy-ingest.service"+ ];+ dataUsers = {+ grafana = 20001;+ loki = 20002;+ tempo = 20003;+ prometheus = 20004;+ alloy = 20005;+ alloy-ingest = 20006;+ };+in+{+ imports = [+ ./services.nix+ ./host-telemetry.nix+ ./proxy.nix+ ];+ system.stateVersion = "26.05";+ image.baseName = "macro-observability";+ virtualisation.diskSize = 12 * 1024;+ ec2.efi = true;+ networking.hostName = "observability";+ networking.firewall.allowedTCPPorts = [ 8080 ];+ services.openssh.enable = lib.mkForce false;+ services.amazon-ssm-agent.enable = true;+ # EC2 user data contains validated JSON settings, never executable Nix.+ virtualisation.amazon-init.enable = false;+ nix.settings.experimental-features = [+ "nix-command"+ "flakes"+ ];+ environment.systemPackages = runtimePath;++ # Stable identities preserve ownership when a replacement AMI reuses EBS.+ users.users = lib.mapAttrs (name: uid: {+ uid = lib.mkForce uid;+ isSystemUser = true;+ group = name;+ createHome = lib.mkForce false;+ }) dataUsers;+ users.groups = lib.mapAttrs (_: gid: { gid = lib.mkForce gid; }) dataUsers;++ systemd.services =+ lib.genAttrs services (name: {+ requires = [ "observability-bootstrap.service" ];+ after = [ "observability-bootstrap.service" ];+ partOf = [ "observability-bootstrap.service" ];+ startLimitIntervalSec = lib.mkForce 0;+ serviceConfig = {+ ExecStartPre = lib.mkBefore [ "${pkgs.util-linux}/bin/mountpoint -q /srv/observability" ];+ Restart = lib.mkForce "always";+ RestartSec = lib.mkForce 30;+ # Stay activating during automatic restarts so Upholds cannot bypass+ # the backoff by starting a briefly failed/inactive unit.+ RestartMode = "direct";+ NoNewPrivileges = true;+ ProtectSystem = lib.mkForce (if name == "alloy" then false else "strict");+ ProtectHome = name != "alloy";+ PrivateTmp = name != "alloy";+ UMask = lib.mkForce "0077";+ TimeoutStopSec = 90;+ # Only Loki/Tempo need the EC2 role for S3 access.+ IPAddressDeny = lib.optionals (+ !builtins.elem name [+ "loki"+ "tempo"+ ]+ ) [ "169.254.169.254/32" ];+ };+ })+ // {+ fetch-ec2-metadata.enable = false;+ observability-bootstrap = {+ description = "Validate EC2 settings and mount retained observability storage";+ wantedBy = [ "multi-user.target" ];+ wants = [ "network-online.target" ];+ after = [ "network-online.target" ];+ path = runtimePath;+ startLimitIntervalSec = 0;+ # Only the secret service upholds its consumers. Starting them here would+ # repeatedly pull a failed secret service out of its restart backoff.+ unitConfig.Upholds = [+ "observability-secrets.service"+ ]+ ++ lib.subtractLists secretConsumers (map (name: "${name}.service") services);+ script = ''+ ${pkgs.python3}/bin/python3 ${./read-user-data.py}+ ${pkgs.bash}/bin/bash ${./mount-data.sh}+ ${lib.concatMapStringsSep "\n" (+ name: "install -d -m 0700 -o ${name} -g ${name} /srv/observability/${name}"+ ) (builtins.attrNames dataUsers)}+ '';+ serviceConfig = {+ Type = "oneshot";+ RemainAfterExit = true;+ Restart = "on-failure";+ RestartMode = "direct";+ RestartSec = 30;+ TimeoutStartSec = 900;+ };+ };+ observability-secrets = {+ description = "Fetch runtime credentials for Grafana and OTLP ingress";+ requires = [ "observability-bootstrap.service" ];+ after = [ "observability-bootstrap.service" ];+ partOf = [ "observability-bootstrap.service" ];+ path = [ pkgs.awscli2 ];+ startLimitIntervalSec = 0;+ unitConfig.Upholds = secretConsumers;+ serviceConfig = {+ Type = "oneshot";+ RemainAfterExit = true;+ ExecStart = "${pkgs.python3}/bin/python3 ${../assets/refresh-secrets.py}";+ Restart = "on-failure";+ RestartMode = "direct";+ RestartSec = 30;+ TimeoutStartSec = 120;+ UMask = lib.mkForce "0077";+ };+ };+ };+ environment.etc."observability/prepare-volume.sh".source = ../assets/prepare-volume.sh;+}diff --git a/infra/stacks/observability/nixos/ingest.alloy b/infra/stacks/observability/nixos/ingest.alloynew file mode 100644index 0000000000..24a112ecb2--- /dev/null+++ b/infra/stacks/observability/nixos/ingest.alloy@@ -0,0 +1,64 @@+local.file "otlp_token" {+ filename = sys.env("CREDENTIALS_DIRECTORY") + "/otlp_token"+ is_secret = true+}++otelcol.auth.bearer "ingest" {+ token = local.file.otlp_token.content+}++otelcol.receiver.otlp "ingest" {+ http {+ endpoint = "127.0.0.1:4318"+ auth = otelcol.auth.bearer.ingest.handler+ max_request_body_size = "8MiB"+ }+ output {+ logs = [otelcol.processor.memory_limiter.ingest.input]+ traces = [otelcol.processor.memory_limiter.ingest.input]+ metrics = [otelcol.processor.memory_limiter.ingest.input]+ }+}++otelcol.processor.memory_limiter "ingest" {+ check_interval = "1s"+ limit = "768MiB"+ spike_limit = "128MiB"+ output {+ logs = [otelcol.processor.batch.ingest.input]+ traces = [otelcol.processor.batch.ingest.input]+ metrics = [otelcol.processor.batch.ingest.input]+ }+}++otelcol.processor.batch "ingest" {+ timeout = "1s"+ send_batch_size = 512+ send_batch_max_size = 1024+ output {+ logs = [otelcol.exporter.otlphttp.loki.input]+ traces = [otelcol.exporter.otlp.tempo.input]+ metrics = [otelcol.exporter.prometheus.metrics.input]+ }+}++otelcol.exporter.otlphttp "loki" {+ client { endpoint = "http://127.0.0.1:3100/otlp" }+ sending_queue { queue_size = 256 }+}++otelcol.exporter.otlp "tempo" {+ client {+ endpoint = "127.0.0.1:4317"+ tls { insecure = true }+ }+ sending_queue { queue_size = 256 }+}++otelcol.exporter.prometheus "metrics" {+ forward_to = [prometheus.remote_write.local.receiver]+}++prometheus.remote_write "local" {+ endpoint { url = "http://127.0.0.1:9090/api/v1/write" }+}diff --git a/infra/stacks/observability/nixos/mount-data.sh b/infra/stacks/observability/nixos/mount-data.shnew file mode 100644index 0000000000..1edc005966--- /dev/null+++ b/infra/stacks/observability/nixos/mount-data.sh@@ -0,0 +1,13 @@+set -euo pipefail+volume_id=$(cat /opt/observability/volume-id)+device="/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_${volume_id//-/}"+for attempt in $(seq 1 120); do+ if [ -b "$device" ]; then break; fi+ sleep 5+done+test -b "$device"+bash /etc/observability/prepare-volume.sh "$device"+uuid=$(blkid -s UUID -o value "$device")+mkdir -p /srv/observability+mountpoint -q /srv/observability || mount -t ext4 "$device" /srv/observability+test "$(findmnt --noheadings --output UUID --target /srv/observability)" = "$uuid"diff --git a/infra/stacks/observability/nixos/proxy.nix b/infra/stacks/observability/nixos/proxy.nixnew file mode 100644index 0000000000..bda092d357--- /dev/null+++ b/infra/stacks/observability/nixos/proxy.nix@@ -0,0 +1,98 @@+{ ... }:+let+ deny = {+ extraConfig = "return 404;";+ };+ websocketHeaders = ''+ proxy_http_version 1.1;+ proxy_set_header Upgrade $http_upgrade;+ proxy_set_header Connection $connection_upgrade;+ '';+ queryLocation = backend: methods: {+ proxyPass = "http://${backend}";+ extraConfig = ''+ limit_except ${methods} { deny all; }+ rewrite ^/[^/]+(/.*)$ $1 break;+ '';+ };+in+{+ services.nginx = {+ enable = true;+ # Hostnames arrive through validated EC2 metadata. The native unit runs+ # nginx -t after bootstrap has written this runtime include.+ validateConfigFile = false;+ clientMaxBodySize = "8m";+ commonHttpConfig = ''+ include /opt/observability/hosts.conf;+ log_format safe '$remote_addr $host $request_method $uri $status';+ access_log syslog:server=unix:/dev/log,facility=local7,tag=nginx_access,severity=info safe;+ limit_req_zone $server_name zone=ingest:1m rate=100r/s;+ map $http_upgrade $connection_upgrade { default upgrade; ''' close; }+ '';+ virtualHosts.public = {+ listen = [+ {+ addr = "0.0.0.0";+ port = 8080;+ }+ ];+ default = true;+ serverName = "_";+ extraConfig = "add_header Strict-Transport-Security 'max-age=31536000' always;";+ locations = {+ "= /healthz" = {+ proxyPass = "http://127.0.0.1:3000/api/health";+ extraConfig = "access_log off; proxy_set_header Host $grafana_host;";+ };+ "/" = {+ proxyPass = "http://127.0.0.1:3000";+ extraConfig = ''+ if ($observability_host != grafana) { return 404; }+ proxy_set_header Host $host;+ proxy_set_header X-Forwarded-Proto https;+ proxy_set_header X-Forwarded-For $remote_addr;+ ${websocketHeaders}+ '';+ };+ "~ ^/v1/(traces|logs|metrics)$" = {+ proxyPass = "http://127.0.0.1:4318";+ extraConfig = ''+ if ($observability_host != otlp) { return 404; }+ limit_except POST { deny all; }+ limit_req zone=ingest burst=200 nodelay;+ limit_req_status 429;+ proxy_read_timeout 30s;+ '';+ };+ };+ };+ # Grafana Viewers can choose arbitrary datasource proxy paths. Allow query+ # endpoints only, including protection against maintenance APIs using GET.+ virtualHosts.queries = {+ listen = [+ {+ addr = "127.0.0.1";+ port = 8081;+ }+ ];+ serverName = "_";+ locations = {+ "~ ^/prometheus/api/v1/(query|query_range|query_exemplars|series|labels|label/[^/]+/values)$" =+ queryLocation "127.0.0.1:9090" "GET POST";+ "~ ^/prometheus/api/v1/(metadata|status/buildinfo|status/config|rules|alerts|targets|targets/metadata)$" =+ queryLocation "127.0.0.1:9090" "GET";+ "~ ^/loki/loki/api/v1/(query|query_range)$" = queryLocation "127.0.0.1:3100" "GET POST";+ "~ ^/loki/loki/api/v1/(labels|label/[^/]+/values|series|index/stats|index/volume|index/volume_range|patterns|tail|status/buildinfo|format_query|detected_fields|detected_labels|detected_field/[^/]+/values)$" =+ let+ location = queryLocation "127.0.0.1:3100" "GET";+ in+ location // { extraConfig = location.extraConfig + websocketHeaders; };+ "~ ^/tempo/api/(echo|search|search/tags|search/tag/[^/]+/values|traces/[a-fA-F0-9]+|v2/traces/[a-fA-F0-9]+|v2/search/tags|v2/search/tag/[^/]+/values|metrics/query|metrics/query_range|status/buildinfo)$" =+ queryLocation "127.0.0.1:3200" "GET";+ "/" = deny;+ };+ };+ };+ systemd.services.nginx.serviceConfig.MemoryMax = "256M";+}diff --git a/infra/stacks/observability/nixos/publish-image.py b/infra/stacks/observability/nixos/publish-image.pynew file mode 100644index 0000000000..f62484b66c--- /dev/null+++ b/infra/stacks/observability/nixos/publish-image.py@@ -0,0 +1,76 @@+#!/usr/bin/env python3+"""Publish a built image in Ohio. Explicit operator action; never run by Pulumi."""+import argparse+import hashlib+import json+from pathlib import Path+import subprocess+import time++parser = argparse.ArgumentParser(description=__doc__)+parser.add_argument('image_directory', type=Path)+parser.add_argument('bucket', help='Existing private Ohio image-artifact bucket')+parser.add_argument('--role', default='vmimport', help='Existing VM Import service role')+args = parser.parse_args()++def aws(*arguments):+ result = subprocess.run(+ ['aws', '--region', 'us-east-2', '--output', 'json', *arguments],+ check=True, capture_output=True, text=True,+ )+ return json.loads(result.stdout) if result.stdout.strip() else {}++identity = aws('sts', 'get-caller-identity')+if identity['Account'] != '569036502058':+ raise SystemExit('Use the Macro AWS account (569036502058)')+if aws('s3api', 'get-bucket-location', '--bucket', args.bucket)['LocationConstraint'] != 'us-east-2':+ raise SystemExit('Image bucket must be in Ohio')+public_access = aws('s3api', 'get-public-access-block', '--bucket', args.bucket)['PublicAccessBlockConfiguration']+if not all(public_access.get(key) for key in [+ 'BlockPublicAcls', 'IgnorePublicAcls', 'BlockPublicPolicy', 'RestrictPublicBuckets',+]):+ raise SystemExit('Image bucket must have all S3 public-access blocks enabled')+info = json.loads((args.image_directory / 'nix-support/image-info.json').read_text())+if info['system'] != 'x86_64-linux' or info['boot_mode'] != 'uefi':+ raise SystemExit('Expected an x86_64 UEFI image from this flake')+image = args.image_directory / Path(info['file']).name+if image.suffix != '.vhd':+ raise SystemExit('Expected the VHD image format')+with image.open('rb') as source:+ digest = hashlib.file_digest(source, 'sha256').hexdigest()+key = f'observability/{digest}.vhd'+subprocess.run(['aws', '--region', 'us-east-2', 's3', 'cp', str(image),+ f's3://{args.bucket}/{key}', '--sse', 'AES256'], check=True)+task = aws('ec2', 'import-snapshot', '--role-name', args.role, '--encrypted',+ '--description', f'Macro observability {digest}',+ '--disk-container', json.dumps({+ 'Format': 'VHD', 'UserBucket': {'S3Bucket': args.bucket, 'S3Key': key},+ }))['ImportTaskId']+print(f'Import task: {task}; artifact: s3://{args.bucket}/{key}', flush=True)+deadline = time.monotonic() + 3600+while time.monotonic() < deadline:+ detail = aws('ec2', 'describe-import-snapshot-tasks',+ '--import-task-ids', task)['ImportSnapshotTasks'][0]['SnapshotTaskDetail']+ if detail['Status'] == 'completed':+ snapshot = detail['SnapshotId']+ break+ if detail['Status'] not in ['active', 'pending']:+ raise SystemExit(f'Import failed: {detail}')+ time.sleep(20)+else:+ raise SystemExit(f'Import still pending: inspect {task} before retrying publication')+print(f'Imported snapshot: {snapshot}', flush=True)+ami = aws('ec2', 'register-image', '--name', f'macro-observability-{digest[:20]}',+ '--architecture', 'x86_64', '--virtualization-type', 'hvm',+ '--boot-mode', 'uefi', '--ena-support', '--imds-support', 'v2.0',+ '--root-device-name', '/dev/xvda',+ '--block-device-mappings', json.dumps([{+ 'DeviceName': '/dev/xvda',+ 'Ebs': {'SnapshotId': snapshot, 'VolumeType': 'gp3',+ 'VolumeSize': 30, 'DeleteOnTermination': True},+ }]))['ImageId']+print(f'AMI: {ami}; SHA256: {digest}', flush=True)+aws('ec2', 'create-tags', '--resources', ami, snapshot, '--tags',+ 'Key=project,Value=observability', f'Key=ImageSHA256,Value={digest}')+aws('ec2', 'wait', 'image-available', '--image-ids', ami)+print(f'Review and smoke-boot {ami}, then set Pulumi amiId. No instance was launched.')diff --git a/infra/stacks/observability/nixos/read-user-data.py b/infra/stacks/observability/nixos/read-user-data.pynew file mode 100644index 0000000000..2c20f6001c--- /dev/null+++ b/infra/stacks/observability/nixos/read-user-data.py@@ -0,0 +1,99 @@+"""Validate nonsecret EC2 settings for native NixOS services."""+import gzip+import json+from pathlib import Path+import re+import urllib.request++CONFIG_ROOT = Path('/opt/observability')+SETTING_KEYS = {+ 'region', 'grafanaHost', 'otlpHost', 'allowedEmails', 'adminEmails',+ 'secretArn', 'volumeId', 'logsBucket', 'tracesBucket',+}+++def validate_payload(payload):+ if not isinstance(payload, dict) or set(payload) != {'version', 'settings'} or payload['version'] != 3:+ raise ValueError('Unsupported observability user-data schema')+ settings = payload['settings']+ if not isinstance(settings, dict) or set(settings) != SETTING_KEYS:+ raise ValueError('Invalid observability runtime setting keys')+ patterns = {+ 'region': r'us-east-2',+ 'grafanaHost': r'[a-z0-9-]+\.macro-internal\.com',+ 'otlpHost': r'[a-z0-9-]+\.macro-internal\.com',+ 'secretArn': r'arn:aws:secretsmanager:us-east-2:\d{12}:secret:[a-zA-Z0-9/_+=.@-]+',+ 'volumeId': r'vol-[a-f0-9]+',+ 'logsBucket': r'[a-z0-9][a-z0-9-]{1,61}[a-z0-9]',+ 'tracesBucket': r'[a-z0-9][a-z0-9-]{1,61}[a-z0-9]',+ }+ for key, pattern in patterns.items():+ if not isinstance(settings[key], str) or not re.fullmatch(pattern, settings[key]):+ raise ValueError(f'Invalid runtime setting: {key}')+ if settings['grafanaHost'] == settings['otlpHost']:+ raise ValueError('UI and ingestion require separate hostnames')+ for key in ['allowedEmails', 'adminEmails']:+ emails = settings[key]+ if not isinstance(emails, list) or not emails or any(+ not isinstance(email, str) or not re.fullmatch(r'[a-z0-9._+-]+@macro\.com', email)+ for email in emails+ ):+ raise ValueError(f'Invalid approved identity list: {key}')+ if not set(settings['adminEmails']).issubset(settings['allowedEmails']):+ raise ValueError('Every admin must also be approved')+ return settings+++def write_runtime(payload, output_root):+ settings = validate_payload(payload)+ admin_list = json.dumps(settings['adminEmails'], separators=(',', ':'))+ allowed_list = json.dumps(settings['allowedEmails'], separators=(',', ':'))+ environment = {+ 'AWS_REGION': settings['region'],+ 'GRAFANA_HOST': settings['grafanaHost'],+ 'GRAFANA_ROOT_URL': 'https://' + settings['grafanaHost'] + '/',+ 'LOGS_BUCKET': settings['logsBucket'],+ 'TRACES_BUCKET': settings['tracesBucket'],+ 'GRAFANA_ROLE_EXPRESSION': f"contains(`{admin_list}`, email) && 'GrafanaAdmin' || contains(`{allowed_list}`, email) && 'Viewer' || 'Denied'",+ }+ # systemd EnvironmentFile accepts double-quoted, escaped values. Inputs are+ # validated ASCII; no shell ever evaluates these values or the role expression.+ files = {+ 'runtime.env': ''.join(f'{key}={json.dumps(value)}\n' for key, value in environment.items()),+ 'hosts.conf': (+ 'map $host $observability_host {\n default denied;\n'+ f" {settings['grafanaHost']} grafana;\n {settings['otlpHost']} otlp;\n"+ '}\nmap "" $grafana_host {\n'+ f" default {settings['grafanaHost']};\n" + '}\n'+ ),+ 'bootstrap.json': json.dumps({'region': settings['region'], 'secretArn': settings['secretArn']}),+ 'volume-id': settings['volumeId'],+ }+ output_root.mkdir(mode=0o755, parents=True, exist_ok=True)+ for name, content in files.items():+ (output_root / name).write_text(content)+++def main():+ endpoint = 'http://169.254.169.254/latest/'+ # Metadata access must never use a configured HTTP proxy.+ http = urllib.request.build_opener(urllib.request.ProxyHandler({}))+ token_request = urllib.request.Request(+ endpoint + 'api/token', method='PUT',+ headers={'X-aws-ec2-metadata-token-ttl-seconds': '60'},+ )+ with http.open(token_request, timeout=10) as response:+ token = response.read().decode()+ request = urllib.request.Request(+ endpoint + 'user-data', headers={'X-aws-ec2-metadata-token': token},+ )+ with http.open(request, timeout=10) as response:+ raw = response.read(16385)+ if len(raw) > 16384:+ raise ValueError('User data exceeds EC2 limit')+ payload = json.loads(gzip.decompress(raw))+ write_runtime(payload, CONFIG_ROOT)+++if __name__ == '__main__':+ main()diff --git a/infra/stacks/observability/nixos/services.nix b/infra/stacks/observability/nixos/services.nixnew file mode 100644index 0000000000..affa0e357a--- /dev/null+++ b/infra/stacks/observability/nixos/services.nix@@ -0,0 +1,97 @@+{+ config,+ lib,+ pkgs,+ ...+}:+let+ backends = import ./backend-config.nix;+ credential = name: "${name}:/run/macro-observability/${name}";+ needsSecrets = {+ requires = [ "observability-secrets.service" ];+ after = [ "observability-secrets.service" ];+ partOf = [ "observability-secrets.service" ];+ };+in+{+ services.grafana = {+ enable = true;+ dataDir = "/srv/observability/grafana";+ settings = import ./grafana.nix;+ provision = {+ enable = true;+ datasources.settings = import ./datasources.nix;+ };+ };+ services.loki = {+ enable = true;+ dataDir = "/srv/observability/loki";+ configuration = backends.loki;+ extraFlags = [ "-config.expand-env=true" ];+ };+ services.tempo = {+ enable = true;+ settings = backends.tempo;+ extraFlags = [ "-config.expand-env=true" ];+ };+ services.prometheus = {+ enable = true;+ listenAddress = "127.0.0.1";+ retentionTime = "30d";+ globalConfig = backends.prometheus.global;+ scrapeConfigs = backends.prometheus.scrape_configs;+ extraFlags = [+ "--storage.tsdb.retention.size=100GB"+ "--web.enable-remote-write-receiver"+ ];+ };+ systemd.services = {+ grafana = needsSecrets // {+ serviceConfig = {+ EnvironmentFile = "/opt/observability/runtime.env";+ LoadCredential = map credential [+ "google_client_id"+ "google_client_secret"+ "grafana_secret_key"+ ];+ ReadWritePaths = [ config.services.grafana.dataDir ];+ MemoryMax = "1G";+ };+ };+ loki.serviceConfig = {+ EnvironmentFile = "/opt/observability/runtime.env";+ ReadWritePaths = [ config.services.loki.dataDir ];+ MemoryMax = "4G";+ };+ tempo.serviceConfig = {+ EnvironmentFile = "/opt/observability/runtime.env";+ DynamicUser = lib.mkForce false;+ User = "tempo";+ Group = "tempo";+ WorkingDirectory = lib.mkForce "/srv/observability/tempo";+ StateDirectory = lib.mkForce "";+ ReadWritePaths = [ "/srv/observability/tempo" ];+ MemoryMax = "3G";+ };+ prometheus.serviceConfig = {+ # The upstream module manages /var/lib/prometheus2; keep its whole state+ # directory on retained storage without overriding its generated command.+ BindPaths = [ "/srv/observability/prometheus:/var/lib/${config.services.prometheus.stateDir}" ];+ MemoryMax = "3G";+ };+ alloy-ingest = needsSecrets // {+ description = "Unprivileged OTLP ingress";+ wantedBy = [ "multi-user.target" ];+ serviceConfig = {+ User = "alloy-ingest";+ Group = "alloy-ingest";+ LoadCredential = [ (credential "otlp_token") ];+ ExecStart = "${lib.getExe pkgs.grafana-alloy} run ${pkgs.writeText "ingest.alloy" (builtins.readFile ./ingest.alloy)} --storage.path=/srv/observability/alloy-ingest --server.http.listen-addr=127.0.0.1:12345 --disable-reporting";+ ReadWritePaths = [ "/srv/observability/alloy-ingest" ];+ MemoryMax = "1536M";+ CapabilityBoundingSet = [ "" ];+ PrivateDevices = true;+ };+ };+ };+}diff --git a/infra/stacks/observability/package.json b/infra/stacks/observability/package.jsonnew file mode 100644index 0000000000..d5ad5d0633--- /dev/null+++ b/infra/stacks/observability/package.json@@ -0,0 +1,7 @@+{+ "name": "observability-stack",+ "version": "0.0.0",+ "private": true,+ "license": "MIT",+ "main": "index.ts"+}diff --git a/infra/stacks/observability/render.ts b/infra/stacks/observability/render.tsnew file mode 100644index 0000000000..1ec03801e8--- /dev/null+++ b/infra/stacks/observability/render.ts@@ -0,0 +1,12 @@+import { gzipSync } from 'node:zlib';+import { type Settings, validateSettings } from './settings';++// Nix owns service configuration. User data carries only validated runtime values.+export function renderUserData(settings: Settings): string {+ validateSettings(settings);+ const compressed = gzipSync(JSON.stringify({ version: 3, settings }));+ if (compressed.length > 16 * 1024) {+ throw new Error('Configuration exceeds the EC2 user-data limit');+ }+ return compressed.toString('base64');+}diff --git a/infra/stacks/observability/settings.ts b/infra/stacks/observability/settings.tsnew file mode 100644index 0000000000..bb106e9ce2--- /dev/null+++ b/infra/stacks/observability/settings.ts@@ -0,0 +1,88 @@+export interface Settings {+ region: string;+ grafanaHost: string;+ otlpHost: string;+ allowedEmails: string[];+ adminEmails: string[];+ secretArn: string;+ volumeId: string;+ logsBucket: string;+ tracesBucket: string;+}++export function validateRegion(region: string): void {+ if (region !== 'us-east-2') {+ throw new Error(+ 'Observability must run in us-east-2, outside prod us-east-1'+ );+ }+}++export function validateRegionalArn(+ arn: string,+ service: 'sns' | 'secretsmanager',+ region: string+): void {+ const parts = arn.split(':');+ if (+ parts[0] !== 'arn' ||+ parts[1] !== 'aws' ||+ parts[2] !== service ||+ parts[3] !== region ||+ !/^\d{12}$/.test(parts[4]) ||+ (service === 'sns' &&+ (parts.length !== 6 || !/^[a-zA-Z0-9_-]+$/.test(parts[5])))+ ) {+ throw new Error(`${service} ARN must reference ${region}`);+ }+}++// Validate runtime values before they reach the NixOS configuration renderer.+export function validateSettings(settings: Settings): void {+ for (const host of [settings.grafanaHost, settings.otlpHost]) {+ if (!/^[a-z0-9-]+\.macro-internal\.com$/.test(host)) {+ throw new Error(+ 'Observability hostnames must be subdomains of macro-internal.com'+ );+ }+ }+ if (settings.grafanaHost === settings.otlpHost) {+ throw new Error('UI and ingestion require separate hostnames');+ }+ if (!settings.allowedEmails.length || !settings.adminEmails.length) {+ throw new Error('Configure approved users and at least one admin');+ }+ for (const email of [...settings.allowedEmails, ...settings.adminEmails]) {+ if (!/^[a-z0-9._+-]+@macro\.com$/.test(email)) {+ throw new Error(+ 'Access lists require lowercase macro.com email addresses'+ );+ }+ }+ if (+ settings.adminEmails.some(+ (email) => !settings.allowedEmails.includes(email)+ )+ ) {+ throw new Error('Every admin must also be in allowedEmails');+ }+ if (+ !/^arn:aws:secretsmanager:[a-z0-9-]+:\d{12}:secret:[a-zA-Z0-9/_+=.@-]+$/.test(+ settings.secretArn+ )+ ) {+ throw new Error(+ 'secretArn must reference an existing Secrets Manager secret'+ );+ }+ validateRegion(settings.region);+ validateRegionalArn(settings.secretArn, 'secretsmanager', settings.region);+ if (!/^vol-[a-f0-9]+$/.test(settings.volumeId)) {+ throw new Error('Invalid data volume ID');+ }+ for (const bucket of [settings.logsBucket, settings.tracesBucket]) {+ if (!/^[a-z0-9][a-z0-9-]{1,61}[a-z0-9]$/.test(bucket)) {+ throw new Error('Invalid telemetry bucket name');+ }+ }+}diff --git a/infra/stacks/observability/tsconfig.json b/infra/stacks/observability/tsconfig.jsonnew file mode 100644index 0000000000..4082f16a5d--- /dev/null+++ b/infra/stacks/observability/tsconfig.json@@ -0,0 +1,3 @@+{+ "extends": "../../tsconfig.json"+}