Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
fix(observability): harden postgres collector startup and samples (#8124)
Wolf Mermelstein07b8b723c170
diff --git a/infra/stacks/database-observability/Pulumi.dev.yaml b/infra/stacks/database-observability/Pulumi.dev.yamlindex d704ece2d4..0783f55ae5 100644--- a/infra/stacks/database-observability/Pulumi.dev.yaml+++ b/infra/stacks/database-observability/Pulumi.dev.yaml@@ -3,4 +3,4 @@ config:database-observability:subnetId: subnet-0306dce66c915072adatabase-observability:ingestSecretArn: arn:aws:secretsmanager:us-east-1:569036502058:secret:observability/dev-ingest-7UhJbLdatabase-observability:secretArn: arn:aws:secretsmanager:us-east-1:569036502058:secret:observability/dev-postgres-KHe9RE- database-observability:amiId: ami-0fc11f7dc85b8d104+ database-observability:amiId: ami-0f1627b24df2d9329diff --git a/infra/stacks/database-observability/README.md b/infra/stacks/database-observability/README.mdindex 78e805ac1c..e30c781f95 100644--- a/infra/stacks/database-observability/README.md+++ b/infra/stacks/database-observability/README.md@@ -35,15 +35,23 @@ separately before expanding it.SELECT on application tables.`pg_monitor` can read activity/query information, so treat this credential assensitive.+ Timeouts and read-only defaults come from the role; the connection URI keeps+ TLS verification and connection timeout settings without libpq `options`.3. On the same dev writer / macrodb connection, run`psql --file query-monitoring.sql`. It checks `pg_stat_statements` is alreadypreloaded and query IDs are enabled, installs the standard extension if needed,+ updates its SQL views/functions to the installed engine's default version,and raises the monitoring role's connection limit from five to eight. Theextension and role changes replicate; do not run this on the reader. It doesnot reset query statistics or change RDS settings. If the preload check fails,review the existing RDS parameter group and plan a reboot separately; thisstack never restarts the database. Confirm the extension is visible on bothendpoints before starting Alloy.+ Review custom extension-view consumers before upgrading: old I/O column names+ and object-specific grants can change. RDS can retain an older extension version+ after an engine upgrade; dev PostgreSQL 18 required version 1.12 for the+ exporter's shared/local I/O columns. The SQL update uses bounded lock/statement+ timeouts and does not reset statistics.4. Save that password in Doppler `observability/dev` as`GRAFANA_POSTGRES_PASSWORD` and an AWS-managed-key Secrets Manager secret inVirginia with JSON shape `{"password":"..."}`. Use the existing raw-string@@ -129,6 +137,10 @@ writer and replica must be collected separately.Alloy samples `pg_stat_activity` every 15 seconds, emitting query samples andwait events as observed queries end or change state. Short queries can be missed.+Sample/wait records use collection time as their Loki timestamp; original timing+fields remain in the body. Old idle sessions can retain query-start times beyond+Loki's acceptance window, so forwarding those timestamps could reject a batch+containing fresh observations.`disable_query_redaction = false` omits raw SQL from activity samples; SQL textcomes from `pg_stat_statements`. Monitoring/system users and system databasesare excluded. The pipeline drops non-DML SQL associations because utilitydiff --git a/infra/stacks/database-observability/query-monitoring.sql b/infra/stacks/database-observability/query-monitoring.sqlindex a59877624d..01bb45a9ef 100644--- a/infra/stacks/database-observability/query-monitoring.sql+++ b/infra/stacks/database-observability/query-monitoring.sql@@ -14,7 +14,14 @@ BEGINRAISE EXCEPTION 'compute_query_id is off; review the RDS configuration separately';END IF;END $$;+-- RDS engine upgrades can leave older extension views installed. PostgreSQL 17++-- exporters need the shared/local I/O columns supplied by the current views.+BEGIN;+SET LOCAL lock_timeout = '1s';+SET LOCAL statement_timeout = '15s';CREATE EXTENSION IF NOT EXISTS pg_stat_statements;+ALTER EXTENSION pg_stat_statements UPDATE;+COMMIT;-- Allow bounded concurrent health, statement and activity collection.ALTER ROLE grafana_monitor CONNECTION LIMIT 8;-- Confirm statistics visibility without reading application rows.diff --git a/infra/stacks/observability/nixos/database-collector/bootstrap.py b/infra/stacks/observability/nixos/database-collector/bootstrap.pyindex 04ce4f1fc0..b73435ff52 100644--- a/infra/stacks/observability/nixos/database-collector/bootstrap.py+++ b/infra/stacks/observability/nixos/database-collector/bootstrap.py@@ -62,8 +62,9 @@ def main():options = urllib.parse.urlencode({'sslmode': 'verify-full', 'sslrootcert': '/etc/ssl/certs/rds-us-east-1.pem','connect_timeout': '5', 'application_name': 'grafana_database_collector',- 'options': '-c statement_timeout=5000 -c lock_timeout=1000 -c default_transaction_read_only=on',})+ # The monitoring role enforces timeouts/read-only defaults. Alloy's DSN+ # normalization cannot round-trip a libpq options value containing spaces.values = {'otlp_token': token}for target in ['writer', 'replica']:values[target + '_dsn'] = (diff --git a/infra/stacks/observability/nixos/database-collector/collector.alloy b/infra/stacks/observability/nixos/database-collector/collector.alloyindex d17e537f43..2c16f67d83 100644--- a/infra/stacks/observability/nixos/database-collector/collector.alloy+++ b/infra/stacks/observability/nixos/database-collector/collector.alloy@@ -228,6 +228,22 @@ loki.relabel "replica" {loki.process "database_queries" {forward_to = [loki.write.ohio.receiver]+ // Activity records are observations. Idle sessions can retain a query-start+ // timestamp older than Loki accepts; preserve durations in the log body.+ stage.match {+ selector = "{job=\"integrations/postgres-queries\",op=~\"query_sample|wait_event\"}"++ stage.template {+ source = "observed_at"+ template = "{{ now | unixEpoch }}"+ }++ stage.timestamp {+ source = "observed_at"+ format = "Unix"+ action_on_failure = "skip"+ }+ }// pg_stat_statements normalizes DML, but utility statements can retain literals.// Keep only DML associations and activity/wait samples; SQL is never a stream label.stage.match {