Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
Provision Grafana dashboards and staged dev alert rules with Nix (#7868)
Wolf Mermelsteinc3c3e0efd059
diff --git a/infra/stacks/observability/Pulumi.prod.yaml b/infra/stacks/observability/Pulumi.prod.yamlindex ab79e16f03..8be74a8c10 100644--- a/infra/stacks/observability/Pulumi.prod.yaml+++ b/infra/stacks/observability/Pulumi.prod.yaml@@ -1,5 +1,5 @@config:aws:region: us-east-2observability:alarmTopicArn: arn:aws:sns:us-east-2:569036502058:observability-prod-alerts- observability:amiId: ami-081ab5dfe46213a68+ observability:amiId: ami-07c508886bdbdba10observability:secretArn: arn:aws:secretsmanager:us-east-2:569036502058:secret:observability/prod-TwQpBJdiff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.mdindex 2ebb350f5e..b53d101149 100644--- a/infra/stacks/observability/README.md+++ b/infra/stacks/observability/README.md@@ -277,7 +277,7 @@ region, hostnames and bucket/volume/secret identifiers.It writes a systemd environment file, nginx hostname maps, the secret's identifierand the expected volume ID under `/opt/observability`. It does not executeuser-data or read secret values. The standard NixOS user-data evaluator is disabled.-Schema version 4 rejects older payloads, unknown settings and caller-supplied files.+Schema version 5 rejects older payloads, unknown settings and caller-supplied files.`observability-bootstrap` mounts that exact EBS volume and creates each service'sdata directory using stable, named NixOS users. `prepare-volume.sh` formats only@@ -324,6 +324,40 @@ crash. Metrics remote-write has a local WAL. EBS-backed backend WALs protect datalready delivered to Loki/Tempo. Establish failure/loss tolerance while Datadogstill receives the authoritative copy.+## Dashboards and alert migration++Nix provisions the `Macro` dashboard folder, a dev telemetry dashboard, and a+CloudWatch database/Lambda dashboard. CloudWatch reads native metrics in Virginia;+it does not replicate their history to Ohio, so these panels depend on Virginia+remaining available. These are not a replacement for Datadog database query monitoring, RUM, or anomaly+models. Grafana uses the EC2 role for regional metric reads and publishing to the+configured alarm SNS topic. Other unprivileged HTTP/ingest services still cannot+access instance metadata.++`nixos/alerting.nix` starts with the three dev monitors declared in the monitoring+stack: DSS errors (1013923), channel-invite spam (4876245), and document text+extractor failure ratio (1291772). Warning and critical rules are separate and+mutually exclusive. Their windows and thresholds come from those definitions;+the Lambda ratio sums four hours ending fifteen minutes ago. No-data is OK for+these event-count/error-ratio rules, while datasource execution errors remain+errors. Notification grouping and repeat timing are Grafana settings, not exact+copies of Datadog notification behavior.++All migrated rules initially remain **paused**. Confirm source coverage and query+results before enabling each rule. The channel-invite monitor's old `processing+message` event was not found in current notification source, so it particularly+needs reconciliation against live logs. Paused rules do not imply healthy coverage.+Production telemetry monitors remain in Datadog. Complete the live dashboard and+monitor inventory before claiming parity; repository definitions may have drifted.++The `Macro team email` contact point publishes through SNS using the host role,+without SMTP credentials. The configured topic already has [email protected] and[email protected] subscriptions; each recipient must confirm their SNS subscription.+Test the contact point and verify receipt before relying on notifications.++This release adds `alarmTopicArn` to user-data schema 5. Deploy it together with+the new reviewed AMI; a schema-4 image cannot boot schema-5 user data.+## Dev dual exportEvery dev ECS service using the shared Datadog sidecars automatically also runsdiff --git a/infra/stacks/observability/index.ts b/infra/stacks/observability/index.tsindex 3897c547e1..6ed1c0d3bd 100644--- a/infra/stacks/observability/index.ts+++ b/infra/stacks/observability/index.ts@@ -34,6 +34,7 @@ validateSettings({grafanaHost,otlpHost,secretArn,+ alarmTopicArn,volumeId: 'vol-0',logsBucket: 'validate-logs',tracesBucket: 'validate-traces',@@ -180,6 +181,23 @@ const policy = new aws.iam.RolePolicy('observability-storage', {Action: ['secretsmanager:GetSecretValue'],Resource: secretArn,},+ {+ Effect: 'Allow',+ Action: [+ 'cloudwatch:ListMetrics',+ 'cloudwatch:GetMetricData',+ 'cloudwatch:GetMetricStatistics',+ ],+ Resource: '*',+ Condition: {+ StringEquals: { 'aws:RequestedRegion': ['us-east-1', 'us-east-2'] },+ },+ },+ {+ Effect: 'Allow',+ Action: ['sns:Publish'],+ Resource: alarmTopicArn,+ },{Effect: 'Allow',Action: ['cloudwatch:PutMetricData'],@@ -256,6 +274,7 @@ const instance = new aws.ec2.Instance(grafanaHost,otlpHost,secretArn,+ alarmTopicArn,volumeId,logsBucket,tracesBucket,diff --git a/infra/stacks/observability/nixos/alerting.nix b/infra/stacks/observability/nixos/alerting.nixnew file mode 100644index 0000000000..dacf000b2e--- /dev/null+++ b/infra/stacks/observability/nixos/alerting.nix@@ -0,0 +1,188 @@+{ ... }:+let+ lokiQuery = expr: {+ refId = "A";+ datasourceUid = "loki";+ relativeTimeRange = {+ from = 3600;+ to = 0;+ };+ model = {+ inherit expr;+ refId = "A";+ queryType = "instant";+ intervalMs = 1000;+ maxDataPoints = 43200;+ };+ };+ rule = uid: title: monitorId: query: severity: condition: {+ inherit uid title;+ condition = "C";+ for = "0s";+ noDataState = "OK";+ execErrState = "Error";+ # Enable after live source queries and notification delivery are verified.+ isPaused = true;+ labels = {+ environment = "dev";+ inherit severity;+ migration = "datadog";+ };+ annotations = {+ summary = title;+ datadog_url = "https://us5.datadoghq.com/monitors/${monitorId}";+ description = "Migrated threshold and window; Datadog remains active during validation.";+ };+ data = query ++ [+ {+ refId = "B";+ datasourceUid = "__expr__";+ relativeTimeRange = {+ from = 0;+ to = 0;+ };+ model = {+ refId = "B";+ type = "reduce";+ expression = "A";+ reducer = "last";+ };+ }+ {+ refId = "C";+ datasourceUid = "__expr__";+ relativeTimeRange = {+ from = 0;+ to = 0;+ };+ model = {+ refId = "C";+ type = "math";+ expression = condition;+ };+ }+ ];+ };+ dss = [+ (lokiQuery ''sum(count_over_time({service_name="cloud-storage-service",deployment_environment="dev"} |~ "(?i)\\berror\\w*" [5m]))'')+ ];+ invites = [+ (lokiQuery ''sum by (sender_id) (count_over_time({service_name="notification-service",deployment_environment="dev"} | json | notification_event_type="channel_invite" | message="processing message" | sender_id!="" | __error__="" [1h]))'')+ ];+ lambdaMetric = refId: id: metricName: {+ inherit refId;+ datasourceUid = "cloudwatch";+ relativeTimeRange = {+ from = 15300;+ to = 900;+ };+ model = {+ inherit refId id metricName;+ region = "us-east-1";+ namespace = "AWS/Lambda";+ dimensions.FunctionName = [ "document-text-extractor-dev" ];+ statistic = "Sum";+ period = "300";+ matchExact = true;+ metricQueryType = 0;+ metricEditorMode = 0;+ queryMode = "Metrics";+ };+ };+ extractor = [+ (lambdaMetric "Errors" "errors" "Errors")+ (lambdaMetric "Invocations" "invocations" "Invocations")+ {+ refId = "A";+ datasourceUid = "cloudwatch";+ relativeTimeRange = {+ from = 15300;+ to = 900;+ };+ model = {+ refId = "A";+ id = "error_ratio";+ statistic = "Sum";+ region = "us-east-1";+ expression = "IF(TIME_SERIES(SUM(invocations)) > 0, 100 * TIME_SERIES(SUM(errors)) / TIME_SERIES(SUM(invocations)), 0)";+ period = "300";+ metricQueryType = 0;+ metricEditorMode = 1;+ queryMode = "Metrics";+ };+ }+ ];+in+{+ services.grafana.provision.alerting = {+ contactPoints.settings = {+ apiVersion = 1;+ contactPoints = [+ {+ orgId = 1;+ name = "Macro team email";+ receivers = [+ {+ uid = "macro-team-sns";+ type = "sns";+ disableResolveMessage = false;+ settings = {+ sigv4.region = "us-east-2";+ topic_arn = "$GRAFANA_ALERT_TOPIC_ARN";+ };+ }+ ];+ }+ ];+ };+ policies.settings = {+ apiVersion = 1;+ policies = [+ {+ orgId = 1;+ receiver = "Macro team email";+ group_by = [+ "grafana_folder"+ "alertname"+ "severity"+ ];+ group_wait = "30s";+ group_interval = "5m";+ repeat_interval = "4h";+ }+ ];+ };+ rules.settings = {+ apiVersion = 1;+ groups = [+ {+ orgId = 1;+ name = "Datadog dev migration";+ folder = "Macro";+ interval = "1m";+ rules = [+ (rule "dd-1291772-critical" "[DEV] Document text extractor failing" "1291772" extractor "critical"+ "$B > 80"+ )+ (rule "dd-1291772-warning" "[DEV] Document text extractor warning" "1291772" extractor "warning"+ "$B > 50 && $B <= 80"+ )+ (rule "dd-1013923-critical" "[DEV] Increased number of errors for DSS" "1013923" dss "critical"+ "$B > 200"+ )+ (rule "dd-1013923-warning" "[DEV] DSS error count warning" "1013923" dss "warning"+ "$B > 100 && $B <= 200"+ )+ (rule "dd-4876245-critical" "[DEV] Potential email spam on channel invites" "4876245" invites+ "critical"+ "$B > 10"+ )+ (rule "dd-4876245-warning" "[DEV] Channel invite count warning" "4876245" invites "warning"+ "$B > 5 && $B <= 10"+ )+ ];+ }+ ];+ };+ };+}diff --git a/infra/stacks/observability/nixos/dashboards.nix b/infra/stacks/observability/nixos/dashboards.nixnew file mode 100644index 0000000000..7cf078d0e9--- /dev/null+++ b/infra/stacks/observability/nixos/dashboards.nix@@ -0,0 +1,161 @@+{ lib, pkgs, ... }:+let+ cloudwatch = {+ type = "cloudwatch";+ uid = "cloudwatch";+ };+ prometheus = {+ type = "prometheus";+ uid = "prometheus";+ };+ loki = {+ type = "loki";+ uid = "loki";+ };+ cwTarget = namespace: metric: dimensions: statistic: {+ refId = "A";+ region = "us-east-1";+ inherit namespace dimensions statistic;+ metricName = metric;+ metricQueryType = 0;+ metricEditorMode = 0;+ queryMode = "Metrics";+ matchExact = true;+ period = "300";+ };+ panel = id: title: datasource: targets: unit: {+ inherit+ id+ title+ datasource+ targets+ ;+ type = "timeseries";+ gridPos = {+ x = if lib.mod (id - 1) 2 == 0 then 0 else 12;+ y = ((id - 1) / 2) * 8;+ w = 12;+ h = 8;+ };+ fieldConfig.defaults = { inherit unit; };+ };+ dashboard = uid: title: panels: {+ inherit uid title panels;+ schemaVersion = 39;+ version = 1;+ editable = false;+ tags = [+ "macro"+ "migration"+ ];+ timezone = "browser";+ refresh = "1m";+ time = {+ from = "now-6h";+ to = "now";+ };+ };+ applications = dashboard "macro-dev-telemetry" "Macro dev telemetry" [+ (panel 1 "Logs per second by service" loki [+ {+ refId = "A";+ expr = ''sum by (service_name) (rate({deployment_environment="dev"}[1m]))'';+ }+ ] "ops")+ (panel 2 "Error logs in five minutes" loki [+ {+ refId = "A";+ expr = ''sum by (service_name) (count_over_time({deployment_environment="dev"} | json parsed_level="level" | drop __error__, __error_details__ | parsed_level=~"(?i)error|fatal|critical" or severity_text=~"(?i)error|fatal|critical" [5m]))'';+ }+ ] "short")+ (panel 3 "Task CPU usage" prometheus [+ {+ refId = "A";+ expr = ''sum by (job) (ecs_task_cpu_usage_vcpu_vCPU{deployment_environment="dev"})'';+ }+ ] "short")+ (panel 4 "Task memory usage" prometheus [+ {+ refId = "A";+ expr = ''sum by (job) (ecs_task_memory_usage_Bytes{deployment_environment="dev"})'';+ }+ ] "bytes")+ (+ (panel 5 "Recent dev logs" loki [+ {+ refId = "A";+ expr = ''{deployment_environment="dev"}'';+ }+ ] "short")+ // {+ type = "logs";+ gridPos = {+ x = 0;+ y = 16;+ w = 24;+ h = 14;+ };+ }+ )+ ];+ database =+ (dashboard "macro-database" "Macro database and Lambda health" [+ (panel 1 "Database CPU" cloudwatch [+ (cwTarget "AWS/RDS" "CPUUtilization" { DBInstanceIdentifier = "$database"; } "Average")+ ] "percent")+ (panel 2 "Database connections" cloudwatch [+ (cwTarget "AWS/RDS" "DatabaseConnections" { DBInstanceIdentifier = "$database"; } "Average")+ ] "short")+ (panel 3 "Freeable memory" cloudwatch [+ (cwTarget "AWS/RDS" "FreeableMemory" { DBInstanceIdentifier = "$database"; } "Average")+ ] "bytes")+ (panel 4 "Free storage" cloudwatch [+ (cwTarget "AWS/RDS" "FreeStorageSpace" { DBInstanceIdentifier = "$database"; } "Average")+ ] "bytes")+ (panel 5 "Text extractor errors" cloudwatch [+ (cwTarget "AWS/Lambda" "Errors" { FunctionName = [ "document-text-extractor-dev" ]; } "Sum")+ ] "short")+ (panel 6 "Text extractor invocations" cloudwatch [+ (cwTarget "AWS/Lambda" "Invocations" { FunctionName = [ "document-text-extractor-dev" ]; } "Sum")+ ] "short")+ ])+ // {+ description = "Native AWS metrics. Query-level database monitoring and Datadog anomaly detection are separate migration work.";+ templating.list = [+ {+ name = "database";+ label = "Database";+ type = "custom";+ query = "macro-db-dev,macro-db-dev-read-replica,macro-db-prod,macro-db-prod-read-replica,fusionauth-db-dev,fusionauthdb-prod";+ current = {+ text = "macro-db-dev";+ value = "macro-db-dev";+ };+ options = [ ];+ }+ ];+ };+ files = pkgs.linkFarm "macro-grafana-dashboards" [+ {+ name = "dev.json";+ path = pkgs.writeText "dev.json" (builtins.toJSON applications);+ }+ {+ name = "database.json";+ path = pkgs.writeText "database.json" (builtins.toJSON database);+ }+ ];+in+{+ services.grafana.provision.dashboards.settings.providers = [+ {+ name = "macro";+ orgId = 1;+ folder = "Macro";+ type = "file";+ disableDeletion = true;+ allowUiUpdates = false;+ options.path = files;+ }+ ];+}diff --git a/infra/stacks/observability/nixos/datasources.nix b/infra/stacks/observability/nixos/datasources.nixindex f9c39326be..d766dabc2a 100644--- a/infra/stacks/observability/nixos/datasources.nix+++ b/infra/stacks/observability/nixos/datasources.nix@@ -1,6 +1,17 @@{apiVersion = 1;datasources = [+ {+ name = "CloudWatch";+ uid = "cloudwatch";+ type = "cloudwatch";+ access = "proxy";+ editable = false;+ jsonData = {+ authType = "default";+ defaultRegion = "us-east-1";+ };+ }{name = "Prometheus";uid = "prometheus";diff --git a/infra/stacks/observability/nixos/host.nix b/infra/stacks/observability/nixos/host.nixindex f5715ac459..bf05e75039 100644--- a/infra/stacks/observability/nixos/host.nix+++ b/infra/stacks/observability/nixos/host.nix@@ -80,17 +80,19 @@ inPrivateTmp = name != "alloy";UMask = lib.mkForce "0077";TimeoutStopSec = 90;- # Only Loki/Tempo need the EC2 role for S3 access.+ # Grafana uses the role for CloudWatch reads and SNS notifications.IPAddressDeny = lib.optionals (!builtins.elem name ["loki""tempo"+ "grafana"]) [ "169.254.169.254/32" ];};})// {fetch-ec2-metadata.enable = false;+ print-host-key.enable = false;observability-bootstrap = {description = "Validate EC2 settings and mount retained observability storage";wantedBy = [ "multi-user.target" ];diff --git a/infra/stacks/observability/nixos/read-user-data.py b/infra/stacks/observability/nixos/read-user-data.pyindex 9ae2205c82..a6cd081a08 100644--- a/infra/stacks/observability/nixos/read-user-data.py+++ b/infra/stacks/observability/nixos/read-user-data.py@@ -8,12 +8,12 @@ import urllib.requestCONFIG_ROOT = Path('/opt/observability')SETTING_KEYS = {'region', 'grafanaHost', 'otlpHost',- 'secretArn', 'volumeId', 'logsBucket', 'tracesBucket',+ 'secretArn', 'alarmTopicArn', 'volumeId', 'logsBucket', 'tracesBucket',}def validate_payload(payload):- if not isinstance(payload, dict) or set(payload) != {'version', 'settings'} or payload['version'] != 4:+ if not isinstance(payload, dict) or set(payload) != {'version', 'settings'} or payload['version'] != 5:raise ValueError('Unsupported observability user-data schema')settings = payload['settings']if not isinstance(settings, dict) or set(settings) != SETTING_KEYS:@@ -22,6 +22,7 @@ def validate_payload(payload):'region': r'us-east-2','grafanaHost': r'[a-z0-9-]+\.macro-internal\.com','otlpHost': r'[a-z0-9-]+\.macro-internal\.com',+ 'alarmTopicArn': r'arn:aws:sns:us-east-2:\d{12}:[a-zA-Z0-9_-]+','secretArn': r'arn:aws:secretsmanager:us-east-2:\d{12}:secret:[a-zA-Z0-9/_+=.@-]+','volumeId': r'vol-[a-f0-9]+','logsBucket': r'[a-z0-9][a-z0-9-]{1,61}[a-z0-9]',@@ -39,6 +40,7 @@ def write_runtime(payload, output_root):settings = validate_payload(payload)environment = {'AWS_REGION': settings['region'],+ 'GRAFANA_ALERT_TOPIC_ARN': settings['alarmTopicArn'],'GRAFANA_HOST': settings['grafanaHost'],'GRAFANA_ROOT_URL': 'https://' + settings['grafanaHost'] + '/','LOGS_BUCKET': settings['logsBucket'],diff --git a/infra/stacks/observability/nixos/services.nix b/infra/stacks/observability/nixos/services.nixindex affa0e357a..489b5624ac 100644--- a/infra/stacks/observability/nixos/services.nix+++ b/infra/stacks/observability/nixos/services.nix@@ -14,6 +14,10 @@ let};in{+ imports = [+ ./dashboards.nix+ ./alerting.nix+ ];services.grafana = {enable = true;dataDir = "/srv/observability/grafana";diff --git a/infra/stacks/observability/render.ts b/infra/stacks/observability/render.tsindex 95058ae212..0d1a10d3ef 100644--- a/infra/stacks/observability/render.ts+++ b/infra/stacks/observability/render.ts@@ -4,7 +4,7 @@ import { type Settings, validateSettings } from './settings';// Nix owns service configuration. User data carries only validated runtime values.export function renderUserData(settings: Settings): string {validateSettings(settings);- const compressed = gzipSync(JSON.stringify({ version: 4, settings }));+ const compressed = gzipSync(JSON.stringify({ version: 5, settings }));if (compressed.length > 16 * 1024) {throw new Error('Configuration exceeds the EC2 user-data limit');}diff --git a/infra/stacks/observability/settings.ts b/infra/stacks/observability/settings.tsindex b5b879d3c6..9f65d5ba3e 100644--- a/infra/stacks/observability/settings.ts+++ b/infra/stacks/observability/settings.ts@@ -3,6 +3,7 @@ export interface Settings {grafanaHost: string;otlpHost: string;secretArn: string;+ alarmTopicArn: string;volumeId: string;logsBucket: string;tracesBucket: string;@@ -58,6 +59,7 @@ export function validateSettings(settings: Settings): void {}validateRegion(settings.region);validateRegionalArn(settings.secretArn, 'secretsmanager', settings.region);+ validateRegionalArn(settings.alarmTopicArn, 'sns', settings.region);if (!/^vol-[a-f0-9]+$/.test(settings.volumeId)) {throw new Error('Invalid data volume ID');}