Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

Provision Grafana dashboards and staged dev alert rules with Nix (#7868)

Wolf Mermelsteinc3c3e0efd059
diff --git a/infra/stacks/observability/Pulumi.prod.yaml b/infra/stacks/observability/Pulumi.prod.yaml
index ab79e16f03..8be74a8c10 100644
--- a/infra/stacks/observability/Pulumi.prod.yaml
+++ b/infra/stacks/observability/Pulumi.prod.yaml
@@ -1,5 +1,5 @@
config:
aws:region: us-east-2
observability:alarmTopicArn: arn:aws:sns:us-east-2:569036502058:observability-prod-alerts
- observability:amiId: ami-081ab5dfe46213a68
+ observability:amiId: ami-07c508886bdbdba10
observability:secretArn: arn:aws:secretsmanager:us-east-2:569036502058:secret:observability/prod-TwQpBJ
diff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.md
index 2ebb350f5e..b53d101149 100644
--- a/infra/stacks/observability/README.md
+++ b/infra/stacks/observability/README.md
@@ -277,7 +277,7 @@ region, hostnames and bucket/volume/secret identifiers.
It writes a systemd environment file, nginx hostname maps, the secret's identifier
and the expected volume ID under `/opt/observability`. It does not execute
user-data or read secret values. The standard NixOS user-data evaluator is disabled.
-Schema version 4 rejects older payloads, unknown settings and caller-supplied files.
+Schema version 5 rejects older payloads, unknown settings and caller-supplied files.
`observability-bootstrap` mounts that exact EBS volume and creates each service's
data directory using stable, named NixOS users. `prepare-volume.sh` formats only
@@ -324,6 +324,40 @@ crash. Metrics remote-write has a local WAL. EBS-backed backend WALs protect dat
already delivered to Loki/Tempo. Establish failure/loss tolerance while Datadog
still receives the authoritative copy.
+## Dashboards and alert migration
+
+Nix provisions the `Macro` dashboard folder, a dev telemetry dashboard, and a
+CloudWatch database/Lambda dashboard. CloudWatch reads native metrics in Virginia;
+it does not replicate their history to Ohio, so these panels depend on Virginia
+remaining available. These are not a replacement for Datadog database query monitoring, RUM, or anomaly
+models. Grafana uses the EC2 role for regional metric reads and publishing to the
+configured alarm SNS topic. Other unprivileged HTTP/ingest services still cannot
+access instance metadata.
+
+`nixos/alerting.nix` starts with the three dev monitors declared in the monitoring
+stack: DSS errors (1013923), channel-invite spam (4876245), and document text
+extractor failure ratio (1291772). Warning and critical rules are separate and
+mutually exclusive. Their windows and thresholds come from those definitions;
+the Lambda ratio sums four hours ending fifteen minutes ago. No-data is OK for
+these event-count/error-ratio rules, while datasource execution errors remain
+errors. Notification grouping and repeat timing are Grafana settings, not exact
+copies of Datadog notification behavior.
+
+All migrated rules initially remain **paused**. Confirm source coverage and query
+results before enabling each rule. The channel-invite monitor's old `processing
+message` event was not found in current notification source, so it particularly
+needs reconciliation against live logs. Paused rules do not imply healthy coverage.
+Production telemetry monitors remain in Datadog. Complete the live dashboard and
+monitor inventory before claiming parity; repository definitions may have drifted.
+
+The `Macro team email` contact point publishes through SNS using the host role,
+without SMTP credentials. The configured topic already has [email protected] and
[email protected] subscriptions; each recipient must confirm their SNS subscription.
+Test the contact point and verify receipt before relying on notifications.
+
+This release adds `alarmTopicArn` to user-data schema 5. Deploy it together with
+the new reviewed AMI; a schema-4 image cannot boot schema-5 user data.
+
## Dev dual export
Every dev ECS service using the shared Datadog sidecars automatically also runs
diff --git a/infra/stacks/observability/index.ts b/infra/stacks/observability/index.ts
index 3897c547e1..6ed1c0d3bd 100644
--- a/infra/stacks/observability/index.ts
+++ b/infra/stacks/observability/index.ts
@@ -34,6 +34,7 @@ validateSettings({
grafanaHost,
otlpHost,
secretArn,
+ alarmTopicArn,
volumeId: 'vol-0',
logsBucket: 'validate-logs',
tracesBucket: 'validate-traces',
@@ -180,6 +181,23 @@ const policy = new aws.iam.RolePolicy('observability-storage', {
Action: ['secretsmanager:GetSecretValue'],
Resource: secretArn,
},
+ {
+ Effect: 'Allow',
+ Action: [
+ 'cloudwatch:ListMetrics',
+ 'cloudwatch:GetMetricData',
+ 'cloudwatch:GetMetricStatistics',
+ ],
+ Resource: '*',
+ Condition: {
+ StringEquals: { 'aws:RequestedRegion': ['us-east-1', 'us-east-2'] },
+ },
+ },
+ {
+ Effect: 'Allow',
+ Action: ['sns:Publish'],
+ Resource: alarmTopicArn,
+ },
{
Effect: 'Allow',
Action: ['cloudwatch:PutMetricData'],
@@ -256,6 +274,7 @@ const instance = new aws.ec2.Instance(
grafanaHost,
otlpHost,
secretArn,
+ alarmTopicArn,
volumeId,
logsBucket,
tracesBucket,
diff --git a/infra/stacks/observability/nixos/alerting.nix b/infra/stacks/observability/nixos/alerting.nix
new file mode 100644
index 0000000000..dacf000b2e
--- /dev/null
+++ b/infra/stacks/observability/nixos/alerting.nix
@@ -0,0 +1,188 @@
+{ ... }:
+let
+ lokiQuery = expr: {
+ refId = "A";
+ datasourceUid = "loki";
+ relativeTimeRange = {
+ from = 3600;
+ to = 0;
+ };
+ model = {
+ inherit expr;
+ refId = "A";
+ queryType = "instant";
+ intervalMs = 1000;
+ maxDataPoints = 43200;
+ };
+ };
+ rule = uid: title: monitorId: query: severity: condition: {
+ inherit uid title;
+ condition = "C";
+ for = "0s";
+ noDataState = "OK";
+ execErrState = "Error";
+ # Enable after live source queries and notification delivery are verified.
+ isPaused = true;
+ labels = {
+ environment = "dev";
+ inherit severity;
+ migration = "datadog";
+ };
+ annotations = {
+ summary = title;
+ datadog_url = "https://us5.datadoghq.com/monitors/${monitorId}";
+ description = "Migrated threshold and window; Datadog remains active during validation.";
+ };
+ data = query ++ [
+ {
+ refId = "B";
+ datasourceUid = "__expr__";
+ relativeTimeRange = {
+ from = 0;
+ to = 0;
+ };
+ model = {
+ refId = "B";
+ type = "reduce";
+ expression = "A";
+ reducer = "last";
+ };
+ }
+ {
+ refId = "C";
+ datasourceUid = "__expr__";
+ relativeTimeRange = {
+ from = 0;
+ to = 0;
+ };
+ model = {
+ refId = "C";
+ type = "math";
+ expression = condition;
+ };
+ }
+ ];
+ };
+ dss = [
+ (lokiQuery ''sum(count_over_time({service_name="cloud-storage-service",deployment_environment="dev"} |~ "(?i)\\berror\\w*" [5m]))'')
+ ];
+ invites = [
+ (lokiQuery ''sum by (sender_id) (count_over_time({service_name="notification-service",deployment_environment="dev"} | json | notification_event_type="channel_invite" | message="processing message" | sender_id!="" | __error__="" [1h]))'')
+ ];
+ lambdaMetric = refId: id: metricName: {
+ inherit refId;
+ datasourceUid = "cloudwatch";
+ relativeTimeRange = {
+ from = 15300;
+ to = 900;
+ };
+ model = {
+ inherit refId id metricName;
+ region = "us-east-1";
+ namespace = "AWS/Lambda";
+ dimensions.FunctionName = [ "document-text-extractor-dev" ];
+ statistic = "Sum";
+ period = "300";
+ matchExact = true;
+ metricQueryType = 0;
+ metricEditorMode = 0;
+ queryMode = "Metrics";
+ };
+ };
+ extractor = [
+ (lambdaMetric "Errors" "errors" "Errors")
+ (lambdaMetric "Invocations" "invocations" "Invocations")
+ {
+ refId = "A";
+ datasourceUid = "cloudwatch";
+ relativeTimeRange = {
+ from = 15300;
+ to = 900;
+ };
+ model = {
+ refId = "A";
+ id = "error_ratio";
+ statistic = "Sum";
+ region = "us-east-1";
+ expression = "IF(TIME_SERIES(SUM(invocations)) > 0, 100 * TIME_SERIES(SUM(errors)) / TIME_SERIES(SUM(invocations)), 0)";
+ period = "300";
+ metricQueryType = 0;
+ metricEditorMode = 1;
+ queryMode = "Metrics";
+ };
+ }
+ ];
+in
+{
+ services.grafana.provision.alerting = {
+ contactPoints.settings = {
+ apiVersion = 1;
+ contactPoints = [
+ {
+ orgId = 1;
+ name = "Macro team email";
+ receivers = [
+ {
+ uid = "macro-team-sns";
+ type = "sns";
+ disableResolveMessage = false;
+ settings = {
+ sigv4.region = "us-east-2";
+ topic_arn = "$GRAFANA_ALERT_TOPIC_ARN";
+ };
+ }
+ ];
+ }
+ ];
+ };
+ policies.settings = {
+ apiVersion = 1;
+ policies = [
+ {
+ orgId = 1;
+ receiver = "Macro team email";
+ group_by = [
+ "grafana_folder"
+ "alertname"
+ "severity"
+ ];
+ group_wait = "30s";
+ group_interval = "5m";
+ repeat_interval = "4h";
+ }
+ ];
+ };
+ rules.settings = {
+ apiVersion = 1;
+ groups = [
+ {
+ orgId = 1;
+ name = "Datadog dev migration";
+ folder = "Macro";
+ interval = "1m";
+ rules = [
+ (rule "dd-1291772-critical" "[DEV] Document text extractor failing" "1291772" extractor "critical"
+ "$B > 80"
+ )
+ (rule "dd-1291772-warning" "[DEV] Document text extractor warning" "1291772" extractor "warning"
+ "$B > 50 && $B <= 80"
+ )
+ (rule "dd-1013923-critical" "[DEV] Increased number of errors for DSS" "1013923" dss "critical"
+ "$B > 200"
+ )
+ (rule "dd-1013923-warning" "[DEV] DSS error count warning" "1013923" dss "warning"
+ "$B > 100 && $B <= 200"
+ )
+ (rule "dd-4876245-critical" "[DEV] Potential email spam on channel invites" "4876245" invites
+ "critical"
+ "$B > 10"
+ )
+ (rule "dd-4876245-warning" "[DEV] Channel invite count warning" "4876245" invites "warning"
+ "$B > 5 && $B <= 10"
+ )
+ ];
+ }
+ ];
+ };
+ };
+}
diff --git a/infra/stacks/observability/nixos/dashboards.nix b/infra/stacks/observability/nixos/dashboards.nix
new file mode 100644
index 0000000000..7cf078d0e9
--- /dev/null
+++ b/infra/stacks/observability/nixos/dashboards.nix
@@ -0,0 +1,161 @@
+{ lib, pkgs, ... }:
+let
+ cloudwatch = {
+ type = "cloudwatch";
+ uid = "cloudwatch";
+ };
+ prometheus = {
+ type = "prometheus";
+ uid = "prometheus";
+ };
+ loki = {
+ type = "loki";
+ uid = "loki";
+ };
+ cwTarget = namespace: metric: dimensions: statistic: {
+ refId = "A";
+ region = "us-east-1";
+ inherit namespace dimensions statistic;
+ metricName = metric;
+ metricQueryType = 0;
+ metricEditorMode = 0;
+ queryMode = "Metrics";
+ matchExact = true;
+ period = "300";
+ };
+ panel = id: title: datasource: targets: unit: {
+ inherit
+ id
+ title
+ datasource
+ targets
+ ;
+ type = "timeseries";
+ gridPos = {
+ x = if lib.mod (id - 1) 2 == 0 then 0 else 12;
+ y = ((id - 1) / 2) * 8;
+ w = 12;
+ h = 8;
+ };
+ fieldConfig.defaults = { inherit unit; };
+ };
+ dashboard = uid: title: panels: {
+ inherit uid title panels;
+ schemaVersion = 39;
+ version = 1;
+ editable = false;
+ tags = [
+ "macro"
+ "migration"
+ ];
+ timezone = "browser";
+ refresh = "1m";
+ time = {
+ from = "now-6h";
+ to = "now";
+ };
+ };
+ applications = dashboard "macro-dev-telemetry" "Macro dev telemetry" [
+ (panel 1 "Logs per second by service" loki [
+ {
+ refId = "A";
+ expr = ''sum by (service_name) (rate({deployment_environment="dev"}[1m]))'';
+ }
+ ] "ops")
+ (panel 2 "Error logs in five minutes" loki [
+ {
+ refId = "A";
+ expr = ''sum by (service_name) (count_over_time({deployment_environment="dev"} | json parsed_level="level" | drop __error__, __error_details__ | parsed_level=~"(?i)error|fatal|critical" or severity_text=~"(?i)error|fatal|critical" [5m]))'';
+ }
+ ] "short")
+ (panel 3 "Task CPU usage" prometheus [
+ {
+ refId = "A";
+ expr = ''sum by (job) (ecs_task_cpu_usage_vcpu_vCPU{deployment_environment="dev"})'';
+ }
+ ] "short")
+ (panel 4 "Task memory usage" prometheus [
+ {
+ refId = "A";
+ expr = ''sum by (job) (ecs_task_memory_usage_Bytes{deployment_environment="dev"})'';
+ }
+ ] "bytes")
+ (
+ (panel 5 "Recent dev logs" loki [
+ {
+ refId = "A";
+ expr = ''{deployment_environment="dev"}'';
+ }
+ ] "short")
+ // {
+ type = "logs";
+ gridPos = {
+ x = 0;
+ y = 16;
+ w = 24;
+ h = 14;
+ };
+ }
+ )
+ ];
+ database =
+ (dashboard "macro-database" "Macro database and Lambda health" [
+ (panel 1 "Database CPU" cloudwatch [
+ (cwTarget "AWS/RDS" "CPUUtilization" { DBInstanceIdentifier = "$database"; } "Average")
+ ] "percent")
+ (panel 2 "Database connections" cloudwatch [
+ (cwTarget "AWS/RDS" "DatabaseConnections" { DBInstanceIdentifier = "$database"; } "Average")
+ ] "short")
+ (panel 3 "Freeable memory" cloudwatch [
+ (cwTarget "AWS/RDS" "FreeableMemory" { DBInstanceIdentifier = "$database"; } "Average")
+ ] "bytes")
+ (panel 4 "Free storage" cloudwatch [
+ (cwTarget "AWS/RDS" "FreeStorageSpace" { DBInstanceIdentifier = "$database"; } "Average")
+ ] "bytes")
+ (panel 5 "Text extractor errors" cloudwatch [
+ (cwTarget "AWS/Lambda" "Errors" { FunctionName = [ "document-text-extractor-dev" ]; } "Sum")
+ ] "short")
+ (panel 6 "Text extractor invocations" cloudwatch [
+ (cwTarget "AWS/Lambda" "Invocations" { FunctionName = [ "document-text-extractor-dev" ]; } "Sum")
+ ] "short")
+ ])
+ // {
+ description = "Native AWS metrics. Query-level database monitoring and Datadog anomaly detection are separate migration work.";
+ templating.list = [
+ {
+ name = "database";
+ label = "Database";
+ type = "custom";
+ query = "macro-db-dev,macro-db-dev-read-replica,macro-db-prod,macro-db-prod-read-replica,fusionauth-db-dev,fusionauthdb-prod";
+ current = {
+ text = "macro-db-dev";
+ value = "macro-db-dev";
+ };
+ options = [ ];
+ }
+ ];
+ };
+ files = pkgs.linkFarm "macro-grafana-dashboards" [
+ {
+ name = "dev.json";
+ path = pkgs.writeText "dev.json" (builtins.toJSON applications);
+ }
+ {
+ name = "database.json";
+ path = pkgs.writeText "database.json" (builtins.toJSON database);
+ }
+ ];
+in
+{
+ services.grafana.provision.dashboards.settings.providers = [
+ {
+ name = "macro";
+ orgId = 1;
+ folder = "Macro";
+ type = "file";
+ disableDeletion = true;
+ allowUiUpdates = false;
+ options.path = files;
+ }
+ ];
+}
diff --git a/infra/stacks/observability/nixos/datasources.nix b/infra/stacks/observability/nixos/datasources.nix
index f9c39326be..d766dabc2a 100644
--- a/infra/stacks/observability/nixos/datasources.nix
+++ b/infra/stacks/observability/nixos/datasources.nix
@@ -1,6 +1,17 @@
{
apiVersion = 1;
datasources = [
+ {
+ name = "CloudWatch";
+ uid = "cloudwatch";
+ type = "cloudwatch";
+ access = "proxy";
+ editable = false;
+ jsonData = {
+ authType = "default";
+ defaultRegion = "us-east-1";
+ };
+ }
{
name = "Prometheus";
uid = "prometheus";
diff --git a/infra/stacks/observability/nixos/host.nix b/infra/stacks/observability/nixos/host.nix
index f5715ac459..bf05e75039 100644
--- a/infra/stacks/observability/nixos/host.nix
+++ b/infra/stacks/observability/nixos/host.nix
@@ -80,17 +80,19 @@ in
PrivateTmp = name != "alloy";
UMask = lib.mkForce "0077";
TimeoutStopSec = 90;
- # Only Loki/Tempo need the EC2 role for S3 access.
+ # Grafana uses the role for CloudWatch reads and SNS notifications.
IPAddressDeny = lib.optionals (
!builtins.elem name [
"loki"
"tempo"
+ "grafana"
]
) [ "169.254.169.254/32" ];
};
})
// {
fetch-ec2-metadata.enable = false;
+ print-host-key.enable = false;
observability-bootstrap = {
description = "Validate EC2 settings and mount retained observability storage";
wantedBy = [ "multi-user.target" ];
diff --git a/infra/stacks/observability/nixos/read-user-data.py b/infra/stacks/observability/nixos/read-user-data.py
index 9ae2205c82..a6cd081a08 100644
--- a/infra/stacks/observability/nixos/read-user-data.py
+++ b/infra/stacks/observability/nixos/read-user-data.py
@@ -8,12 +8,12 @@ import urllib.request
CONFIG_ROOT = Path('/opt/observability')
SETTING_KEYS = {
'region', 'grafanaHost', 'otlpHost',
- 'secretArn', 'volumeId', 'logsBucket', 'tracesBucket',
+ 'secretArn', 'alarmTopicArn', 'volumeId', 'logsBucket', 'tracesBucket',
}
def validate_payload(payload):
- if not isinstance(payload, dict) or set(payload) != {'version', 'settings'} or payload['version'] != 4:
+ if not isinstance(payload, dict) or set(payload) != {'version', 'settings'} or payload['version'] != 5:
raise ValueError('Unsupported observability user-data schema')
settings = payload['settings']
if not isinstance(settings, dict) or set(settings) != SETTING_KEYS:
@@ -22,6 +22,7 @@ def validate_payload(payload):
'region': r'us-east-2',
'grafanaHost': r'[a-z0-9-]+\.macro-internal\.com',
'otlpHost': r'[a-z0-9-]+\.macro-internal\.com',
+ 'alarmTopicArn': r'arn:aws:sns:us-east-2:\d{12}:[a-zA-Z0-9_-]+',
'secretArn': r'arn:aws:secretsmanager:us-east-2:\d{12}:secret:[a-zA-Z0-9/_+=.@-]+',
'volumeId': r'vol-[a-f0-9]+',
'logsBucket': r'[a-z0-9][a-z0-9-]{1,61}[a-z0-9]',
@@ -39,6 +40,7 @@ def write_runtime(payload, output_root):
settings = validate_payload(payload)
environment = {
'AWS_REGION': settings['region'],
+ 'GRAFANA_ALERT_TOPIC_ARN': settings['alarmTopicArn'],
'GRAFANA_HOST': settings['grafanaHost'],
'GRAFANA_ROOT_URL': 'https://' + settings['grafanaHost'] + '/',
'LOGS_BUCKET': settings['logsBucket'],
diff --git a/infra/stacks/observability/nixos/services.nix b/infra/stacks/observability/nixos/services.nix
index affa0e357a..489b5624ac 100644
--- a/infra/stacks/observability/nixos/services.nix
+++ b/infra/stacks/observability/nixos/services.nix
@@ -14,6 +14,10 @@ let
};
in
{
+ imports = [
+ ./dashboards.nix
+ ./alerting.nix
+ ];
services.grafana = {
enable = true;
dataDir = "/srv/observability/grafana";
diff --git a/infra/stacks/observability/render.ts b/infra/stacks/observability/render.ts
index 95058ae212..0d1a10d3ef 100644
--- a/infra/stacks/observability/render.ts
+++ b/infra/stacks/observability/render.ts
@@ -4,7 +4,7 @@ import { type Settings, validateSettings } from './settings';
// Nix owns service configuration. User data carries only validated runtime values.
export function renderUserData(settings: Settings): string {
validateSettings(settings);
- const compressed = gzipSync(JSON.stringify({ version: 4, settings }));
+ const compressed = gzipSync(JSON.stringify({ version: 5, settings }));
if (compressed.length > 16 * 1024) {
throw new Error('Configuration exceeds the EC2 user-data limit');
}
diff --git a/infra/stacks/observability/settings.ts b/infra/stacks/observability/settings.ts
index b5b879d3c6..9f65d5ba3e 100644
--- a/infra/stacks/observability/settings.ts
+++ b/infra/stacks/observability/settings.ts
@@ -3,6 +3,7 @@ export interface Settings {
grafanaHost: string;
otlpHost: string;
secretArn: string;
+ alarmTopicArn: string;
volumeId: string;
logsBucket: string;
tracesBucket: string;
@@ -58,6 +59,7 @@ export function validateSettings(settings: Settings): void {
}
validateRegion(settings.region);
validateRegionalArn(settings.secretArn, 'secretsmanager', settings.region);
+ validateRegionalArn(settings.alarmTopicArn, 'sns', settings.region);
if (!/^vol-[a-f0-9]+$/.test(settings.volumeId)) {
throw new Error('Invalid data volume ID');
}