Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

fix(observability): harden postgres collector startup and samples (#8124)

Wolf Mermelstein07b8b723c170
diff --git a/infra/stacks/database-observability/Pulumi.dev.yaml b/infra/stacks/database-observability/Pulumi.dev.yaml
index d704ece2d4..0783f55ae5 100644
--- a/infra/stacks/database-observability/Pulumi.dev.yaml
+++ b/infra/stacks/database-observability/Pulumi.dev.yaml
@@ -3,4 +3,4 @@ config:
database-observability:subnetId: subnet-0306dce66c915072a
database-observability:ingestSecretArn: arn:aws:secretsmanager:us-east-1:569036502058:secret:observability/dev-ingest-7UhJbL
database-observability:secretArn: arn:aws:secretsmanager:us-east-1:569036502058:secret:observability/dev-postgres-KHe9RE
- database-observability:amiId: ami-0fc11f7dc85b8d104
+ database-observability:amiId: ami-0f1627b24df2d9329
diff --git a/infra/stacks/database-observability/README.md b/infra/stacks/database-observability/README.md
index 78e805ac1c..e30c781f95 100644
--- a/infra/stacks/database-observability/README.md
+++ b/infra/stacks/database-observability/README.md
@@ -35,15 +35,23 @@ separately before expanding it.
SELECT on application tables.
`pg_monitor` can read activity/query information, so treat this credential as
sensitive.
+ Timeouts and read-only defaults come from the role; the connection URI keeps
+ TLS verification and connection timeout settings without libpq `options`.
3. On the same dev writer / macrodb connection, run
`psql --file query-monitoring.sql`. It checks `pg_stat_statements` is already
preloaded and query IDs are enabled, installs the standard extension if needed,
+ updates its SQL views/functions to the installed engine's default version,
and raises the monitoring role's connection limit from five to eight. The
extension and role changes replicate; do not run this on the reader. It does
not reset query statistics or change RDS settings. If the preload check fails,
review the existing RDS parameter group and plan a reboot separately; this
stack never restarts the database. Confirm the extension is visible on both
endpoints before starting Alloy.
+ Review custom extension-view consumers before upgrading: old I/O column names
+ and object-specific grants can change. RDS can retain an older extension version
+ after an engine upgrade; dev PostgreSQL 18 required version 1.12 for the
+ exporter's shared/local I/O columns. The SQL update uses bounded lock/statement
+ timeouts and does not reset statistics.
4. Save that password in Doppler `observability/dev` as
`GRAFANA_POSTGRES_PASSWORD` and an AWS-managed-key Secrets Manager secret in
Virginia with JSON shape `{"password":"..."}`. Use the existing raw-string
@@ -129,6 +137,10 @@ writer and replica must be collected separately.
Alloy samples `pg_stat_activity` every 15 seconds, emitting query samples and
wait events as observed queries end or change state. Short queries can be missed.
+Sample/wait records use collection time as their Loki timestamp; original timing
+fields remain in the body. Old idle sessions can retain query-start times beyond
+Loki's acceptance window, so forwarding those timestamps could reject a batch
+containing fresh observations.
`disable_query_redaction = false` omits raw SQL from activity samples; SQL text
comes from `pg_stat_statements`. Monitoring/system users and system databases
are excluded. The pipeline drops non-DML SQL associations because utility
diff --git a/infra/stacks/database-observability/query-monitoring.sql b/infra/stacks/database-observability/query-monitoring.sql
index a59877624d..01bb45a9ef 100644
--- a/infra/stacks/database-observability/query-monitoring.sql
+++ b/infra/stacks/database-observability/query-monitoring.sql
@@ -14,7 +14,14 @@ BEGIN
RAISE EXCEPTION 'compute_query_id is off; review the RDS configuration separately';
END IF;
END $$;
+-- RDS engine upgrades can leave older extension views installed. PostgreSQL 17+
+-- exporters need the shared/local I/O columns supplied by the current views.
+BEGIN;
+SET LOCAL lock_timeout = '1s';
+SET LOCAL statement_timeout = '15s';
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
+ALTER EXTENSION pg_stat_statements UPDATE;
+COMMIT;
-- Allow bounded concurrent health, statement and activity collection.
ALTER ROLE grafana_monitor CONNECTION LIMIT 8;
-- Confirm statistics visibility without reading application rows.
diff --git a/infra/stacks/observability/nixos/database-collector/bootstrap.py b/infra/stacks/observability/nixos/database-collector/bootstrap.py
index 04ce4f1fc0..b73435ff52 100644
--- a/infra/stacks/observability/nixos/database-collector/bootstrap.py
+++ b/infra/stacks/observability/nixos/database-collector/bootstrap.py
@@ -62,8 +62,9 @@ def main():
options = urllib.parse.urlencode({
'sslmode': 'verify-full', 'sslrootcert': '/etc/ssl/certs/rds-us-east-1.pem',
'connect_timeout': '5', 'application_name': 'grafana_database_collector',
- 'options': '-c statement_timeout=5000 -c lock_timeout=1000 -c default_transaction_read_only=on',
})
+ # The monitoring role enforces timeouts/read-only defaults. Alloy's DSN
+ # normalization cannot round-trip a libpq options value containing spaces.
values = {'otlp_token': token}
for target in ['writer', 'replica']:
values[target + '_dsn'] = (
diff --git a/infra/stacks/observability/nixos/database-collector/collector.alloy b/infra/stacks/observability/nixos/database-collector/collector.alloy
index d17e537f43..2c16f67d83 100644
--- a/infra/stacks/observability/nixos/database-collector/collector.alloy
+++ b/infra/stacks/observability/nixos/database-collector/collector.alloy
@@ -228,6 +228,22 @@ loki.relabel "replica" {
loki.process "database_queries" {
forward_to = [loki.write.ohio.receiver]
+ // Activity records are observations. Idle sessions can retain a query-start
+ // timestamp older than Loki accepts; preserve durations in the log body.
+ stage.match {
+ selector = "{job=\"integrations/postgres-queries\",op=~\"query_sample|wait_event\"}"
+
+ stage.template {
+ source = "observed_at"
+ template = "{{ now | unixEpoch }}"
+ }
+
+ stage.timestamp {
+ source = "observed_at"
+ format = "Unix"
+ action_on_failure = "skip"
+ }
+ }
// pg_stat_statements normalizes DML, but utility statements can retain literals.
// Keep only DML associations and activity/wait samples; SQL is never a stream label.
stage.match {