Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
feat(infra): grant Grafana admin access to Macro Workspace (#7854)
Wolf Mermelstein488f93102959
diff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.mdindex 5a5cff6c2e..320248233e 100644--- a/infra/stacks/observability/README.md+++ b/infra/stacks/observability/README.md@@ -16,7 +16,7 @@ packages, configuration, service users, resource limits and startup dependenciesNixOS modules generate application configuration from Nix attributes. Alloy'spipelines live in native `.alloy` files read by Nix. Pulumi sends only-validated runtime values such as hostnames, bucket names and approved identities.+validated runtime values such as hostnames and bucket names.Datadog instrumentation, collection and alerts remain unchanged. Nothing sendsapplication telemetry here until a subsequent dual-export change is deployed.@@ -98,23 +98,23 @@ before enabling broad production ingestion.Production URLs are `https://grafana.macro-internal.com` and `https://otlp.macro-internal.com`;dev uses `grafana-dev.macro-internal.com` and `otlp-dev.macro-internal.com`.-Grafana uses a dedicated **Google Workspace OAuth client**. Only explicitly-approved lowercase `@macro.com` emails in `allowedEmails` may sign in; membership-in the domain alone is insufficient. Named `adminEmails` get Grafana server-administrator access; other approved users are Viewers. Admins must also appear-in the allowlist. Unknown identities map to an invalid role and are denied with-`role_attribute_strict`. Google hosted-domain validation, ID-token signature-validation, PKCE and refresh-token checking are enabled. Workspace administrators-must enforce MFA for the approved accounts. No password login, default admin,-anonymous access or public signup is enabled.--Sessions have a one-hour inactivity limit and eight-hour maximum. Removing an-email from configuration blocks the next login; it does **not** immediately-invalidate an existing session. For offboarding, first disable the user and-revoke their sessions in Grafana using another admin, then update the allowlist.-Keep at least two admins. This uses OSS role mapping, not Enterprise team sync.-All approved users can query the pilot's telemetry; there is no per-service data-isolation. Apply redaction in the producer pipeline before copying sensitive data.+Grafana uses a **Google Workspace OAuth client**. Every member of the `macro.com`+Workspace may sign in and receives Grafana server administrator and organization+administrator access. Google hosted-domain validation and the `macro.com` domain+restriction reject accounts outside that Workspace, including personal Google+accounts. ID-token signature validation, PKCE and refresh-token checking are enabled.+Workspace administrators must enforce MFA. No password login, default admin or+anonymous access is enabled; account creation happens only after Google login.+The role policy is declared directly in `nixos/grafana.nix`; there are no per-user+Pulumi access lists.++Sessions have a one-hour inactivity limit and eight-hour maximum. Suspending a+Workspace account does **not** immediately invalidate an existing Grafana session.+For offboarding, disable the Workspace account and use another Grafana admin to+disable the Grafana user and revoke their sessions. This uses OSS role mapping,+not Enterprise team sync. All team members can administer Grafana and query the+pilot's telemetry; there is no per-service data isolation. Apply redaction in the+producer pipeline before copying sensitive data.Only the ALB is internet-facing, on TLS 1.2/1.3 port 443. The host has no public IP,SSH key or SSH ingress. Its only ingress is port 8080 from the ALB security group.@@ -124,7 +124,7 @@ services bind to loopback. The default ALB action is 404. No HTTP listener is opGrafana data sources use a separate internal nginx listener on port 8081 thatallows only named query endpoints and their required HTTP methods. This matters-because Viewers can call Grafana's data-source proxy: direct backend URLs would+because users can call Grafana's data-source proxy: direct backend URLs wouldalso expose maintenance endpoints such as Tempo `/shutdown` and Loki `/flush`(including mutating GET requests). Port 8081 binds only to loopback, andAlloy's ingestion path is separate. New data-source features may require reviewed@@ -142,9 +142,9 @@ Secrets Manager holds Google client credentials, the Grafana encryption key andthe ingestion token. EC2 retrieves them at startup into `/run` with restrictedfile permissions. Systemd `LoadCredential` gives Grafana only its three credentialsand ingress Alloy only the OTLP token. Secret values never enter Pulumi state,-user-data, environment variables or configuration committed here. Grafana's nonsecret host-and approved-user role expression are supplied through environment variables;-the generated INI refers to them through Grafana's environment provider. Secrets+user-data, environment variables or configuration committed here. Grafana's nonsecret hostname+and public URL are supplied through environment variables; the generated INI+refers to them through Grafana's environment provider. Secretsremain file references in the Nix store, never secret values. The secret must use theAWS-managed Secrets Manager encryption key; a customer-managed key needs anexplicit scoped KMS policy addition. IMDSv2 is required with hop limit 1. Only Loki, Tempo, the bootstrap/secret helper,@@ -160,26 +160,28 @@ and AWS account `569036502058`, region `us-east-2`. Deployment requires theexisting public `macro-internal.com` Route53 zone; this stack creates its own VPC/NAT andDNS-validated regional ACM certificate. Region validation rejects `us-east-1`.-1. Create a dedicated Google OAuth Web application under the company's Google- organization with an **Internal** consent audience. Register exactly+1. Use a Google OAuth Web client managed by the company. For a new dedicated+ client, choose an **Internal** consent audience. When reusing the application+ client, preserve its existing audience and redirect URIs; Grafana independently+ enforces the `macro.com` Workspace restriction. Add the exact redirect URI`https://grafana-dev.macro-internal.com/login/google` for dev or- `https://grafana.macro-internal.com/login/google` for prod. Prefer separate clients and- secrets per environment. Confirm Workspace MFA enforcement.+ `https://grafana.macro-internal.com/login/google` for prod. Prefer separate clients+ and secrets per environment. Confirm Workspace MFA enforcement.2. Through the approved secret-management process, create a Secrets Manager JSONsecret in **us-east-2** containing `google_client_id`, `google_client_secret`,`grafana_secret_key`, and `otlp_token`. Generate independent cryptographicallyrandom values of at least 32 characters for the last two. Preserve`grafana_secret_key` through recovery; changing it can make stored credentialsunreadable. Never paste secret values into shell commands, this file or PRs.-3. Select approved users and admins. Select an existing **us-east-2** SNS topic+3. Select an existing **us-east-2** SNS topicwith a confirmed, monitored subscription for infrastructure alarms. Itsdelivery destination must remain accessible during a production outage.Configuration rejects secrets and topics in another region; the host readsthe local secret directly, without fetching credentials from production.4. Build, publish and smoke-boot the NixOS image as described below. Set itsreviewed Ohio AMI ID; there is deliberately no mutable "latest image" lookup.-5. Set the nonsecret configuration below, substituting actual identifiers. No- example account or placeholder is authorized automatically.+5. Set the nonsecret configuration below, substituting actual identifiers.+ Placeholder credentials are not usable for deployment.```bash\cd infra@@ -190,9 +192,6 @@ pulumi config set aws:region us-east-2pulumi config set amiId '<reviewed NixOS AMI ID in Ohio>'pulumi config set secretArn '<existing Secrets Manager ARN>'pulumi config set alarmTopicArn '<existing monitored SNS topic ARN>'-pulumi config set --path 'allowedEmails[0]' '<[email protected]>'-pulumi config set --path 'allowedEmails[1]' '<[email protected]>'-pulumi config set --path 'adminEmails[0]' '<[email protected]>'pulumi preview --diff```@@ -204,8 +203,8 @@ an explicit migration and recovery plan for its protected storage.Before enabling application traffic, validate all of these against AWS:-- Approved Viewer and Admin can log in; an unapproved Workspace account and a- personal Google account cannot. Viewer cannot administer Grafana.+- A `macro.com` Workspace member can log in with server and organization admin+ access. A different Workspace domain and personal Google accounts cannot log in.- All three data sources pass their health checks. A small OTLP fixture appearsin logs, traces and metrics with the expected `service.name`.- Missing/wrong tokens fail; valid token cannot read backend APIs. No backend@@ -274,11 +273,11 @@ sudo journalctl -u grafana -u loki -u tempo -u prometheus -u alloy -u alloy-ingeNonsecret runtime values arrive as versioned, compressed JSON in EC2 user-data.`read-user-data.py` retrieves them through IMDSv2 and validates the exact schema,-region, hostnames, bucket/volume/secret identifiers and approved identity lists.+region, hostnames and bucket/volume/secret identifiers.It writes a systemd environment file, nginx hostname maps, the secret's identifierand the expected volume ID under `/opt/observability`. It does not executeuser-data or read secret values. The standard NixOS user-data evaluator is disabled.-Schema version 3 rejects older payloads, unknown settings and caller-supplied files.+Schema version 4 rejects older payloads, unknown settings and caller-supplied files.`observability-bootstrap` mounts that exact EBS volume and creates each service'sdata directory using stable, named NixOS users. `prepare-volume.sh` formats onlydiff --git a/infra/stacks/observability/index.ts b/infra/stacks/observability/index.tsindex 4d99c5719a..3897c547e1 100644--- a/infra/stacks/observability/index.ts+++ b/infra/stacks/observability/index.ts@@ -22,8 +22,6 @@ const otlpHost = `otlp${suffix}.${baseDomain}`;const tags = { project: 'observability', environment: stack };const durable = { protect: true, retainOnDelete: true };const secretArn = config.require('secretArn');-const allowedEmails = config.requireObject<string[]>('allowedEmails');-const adminEmails = config.requireObject<string[]>('adminEmails');const alarmTopicArn = config.require('alarmTopicArn');// Built from this stack's pinned NixOS flake and published in Ohio.const amiId = config.require('amiId');@@ -35,8 +33,6 @@ validateSettings({region,grafanaHost,otlpHost,- allowedEmails,- adminEmails,secretArn,volumeId: 'vol-0',logsBucket: 'validate-logs',@@ -259,8 +255,6 @@ const instance = new aws.ec2.Instance(region,grafanaHost,otlpHost,- allowedEmails,- adminEmails,secretArn,volumeId,logsBucket,diff --git a/infra/stacks/observability/nixos/grafana.nix b/infra/stacks/observability/nixos/grafana.nixindex ff61f97816..88715066b0 100644--- a/infra/stacks/observability/nixos/grafana.nix+++ b/infra/stacks/observability/nixos/grafana.nix@@ -45,7 +45,7 @@use_refresh_token = true;validate_id_token = true;jwk_set_url = "https://www.googleapis.com/oauth2/v3/certs";- role_attribute_path = "$__env{GRAFANA_ROLE_EXPRESSION}";+ role_attribute_path = "\"'GrafanaAdmin'\"";role_attribute_strict = true;allow_assign_grafana_admin = true;skip_org_role_sync = false;diff --git a/infra/stacks/observability/nixos/read-user-data.py b/infra/stacks/observability/nixos/read-user-data.pyindex 2c20f6001c..9ae2205c82 100644--- a/infra/stacks/observability/nixos/read-user-data.py+++ b/infra/stacks/observability/nixos/read-user-data.py@@ -7,13 +7,13 @@ import urllib.requestCONFIG_ROOT = Path('/opt/observability')SETTING_KEYS = {- 'region', 'grafanaHost', 'otlpHost', 'allowedEmails', 'adminEmails',+ 'region', 'grafanaHost', 'otlpHost','secretArn', 'volumeId', 'logsBucket', 'tracesBucket',}def validate_payload(payload):- if not isinstance(payload, dict) or set(payload) != {'version', 'settings'} or payload['version'] != 3:+ if not isinstance(payload, dict) or set(payload) != {'version', 'settings'} or payload['version'] != 4:raise ValueError('Unsupported observability user-data schema')settings = payload['settings']if not isinstance(settings, dict) or set(settings) != SETTING_KEYS:@@ -32,32 +32,20 @@ def validate_payload(payload):raise ValueError(f'Invalid runtime setting: {key}')if settings['grafanaHost'] == settings['otlpHost']:raise ValueError('UI and ingestion require separate hostnames')- for key in ['allowedEmails', 'adminEmails']:- emails = settings[key]- if not isinstance(emails, list) or not emails or any(- not isinstance(email, str) or not re.fullmatch(r'[a-z0-9._+-]+@macro\.com', email)- for email in emails- ):- raise ValueError(f'Invalid approved identity list: {key}')- if not set(settings['adminEmails']).issubset(settings['allowedEmails']):- raise ValueError('Every admin must also be approved')return settingsdef write_runtime(payload, output_root):settings = validate_payload(payload)- admin_list = json.dumps(settings['adminEmails'], separators=(',', ':'))- allowed_list = json.dumps(settings['allowedEmails'], separators=(',', ':'))environment = {'AWS_REGION': settings['region'],'GRAFANA_HOST': settings['grafanaHost'],'GRAFANA_ROOT_URL': 'https://' + settings['grafanaHost'] + '/','LOGS_BUCKET': settings['logsBucket'],'TRACES_BUCKET': settings['tracesBucket'],- 'GRAFANA_ROLE_EXPRESSION': f"contains(`{admin_list}`, email) && 'GrafanaAdmin' || contains(`{allowed_list}`, email) && 'Viewer' || 'Denied'",}# systemd EnvironmentFile accepts double-quoted, escaped values. Inputs are- # validated ASCII; no shell ever evaluates these values or the role expression.+ # validated ASCII; no shell ever evaluates these values.files = {'runtime.env': ''.join(f'{key}={json.dumps(value)}\n' for key, value in environment.items()),'hosts.conf': (diff --git a/infra/stacks/observability/render.ts b/infra/stacks/observability/render.tsindex 1ec03801e8..95058ae212 100644--- a/infra/stacks/observability/render.ts+++ b/infra/stacks/observability/render.ts@@ -4,7 +4,7 @@ import { type Settings, validateSettings } from './settings';// Nix owns service configuration. User data carries only validated runtime values.export function renderUserData(settings: Settings): string {validateSettings(settings);- const compressed = gzipSync(JSON.stringify({ version: 3, settings }));+ const compressed = gzipSync(JSON.stringify({ version: 4, settings }));if (compressed.length > 16 * 1024) {throw new Error('Configuration exceeds the EC2 user-data limit');}diff --git a/infra/stacks/observability/settings.ts b/infra/stacks/observability/settings.tsindex bb106e9ce2..b5b879d3c6 100644--- a/infra/stacks/observability/settings.ts+++ b/infra/stacks/observability/settings.ts@@ -2,8 +2,6 @@ export interface Settings {region: string;grafanaHost: string;otlpHost: string;- allowedEmails: string[];- adminEmails: string[];secretArn: string;volumeId: string;logsBucket: string;@@ -49,23 +47,6 @@ export function validateSettings(settings: Settings): void {if (settings.grafanaHost === settings.otlpHost) {throw new Error('UI and ingestion require separate hostnames');}- if (!settings.allowedEmails.length || !settings.adminEmails.length) {- throw new Error('Configure approved users and at least one admin');- }- for (const email of [...settings.allowedEmails, ...settings.adminEmails]) {- if (!/^[a-z0-9._+-]+@macro\.com$/.test(email)) {- throw new Error(- 'Access lists require lowercase macro.com email addresses'- );- }- }- if (- settings.adminEmails.some(- (email) => !settings.allowedEmails.includes(email)- )- ) {- throw new Error('Every admin must also be in allowedEmails');- }if (!/^arn:aws:secretsmanager:[a-z0-9-]+:\d{12}:secret:[a-zA-Z0-9/_+=.@-]+$/.test(settings.secretArn