Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(observability): hook dev postgres queries into grafana (#8111)

Wolf Mermelstein6aad76cf2269
diff --git a/infra/stacks/database-observability/README.md b/infra/stacks/database-observability/README.md
index 81dd0ac488..0e09076ae8 100644
--- a/infra/stacks/database-observability/README.md
+++ b/infra/stacks/database-observability/README.md
@@ -1,25 +1,28 @@
-# PostgreSQL metrics pilot
+# PostgreSQL monitoring pilot
This stack adds one native NixOS/Alloy collector in a private Virginia subnet,
with no inbound ports or public IP. It monitors the dev writer and replica over
-verified PostgreSQL TLS and forwards metrics to the existing Ohio Prometheus.
+verified PostgreSQL TLS and forwards metrics to Ohio Prometheus and query
+details to Ohio Loki.
Datadog, application tasks, RDS parameter groups, and production are unchanged.
-This is the database health baseline. It does not collect SQL text, query
-fingerprints, samples or execution plans, change existing alerts, or provide
-Grafana Cloud's Database Observability UI. PostgreSQL 14 and 18 compatibility
-must be verified before expanding the dev-only stack to production.
+The collector covers database health plus query-level statistics, normalized SQL,
+and sampled activity/waits. Grafana gets provisioned health and query dashboards,
+not Grafana Cloud's Database Observability UI. Execution plans, schema discovery
+and alert changes are excluded. The stack remains dev-only; validate production
+separately before expanding it.
## Deployment order
1. Build and publish a new **Ohio observability** AMI from
`infra/stacks/observability`; deploy it first. This adds
- `https://otlp.macro-internal.com/api/v1/write`, an exact POST-only endpoint
+ `https://otlp.macro-internal.com/api/v1/write` and
+ `/loki/api/v1/push`, exact POST-only endpoints
protected by HTTP Basic (`metrics`, existing OTLP token) over ALB TLS. It
exposes no query/administration routes. Existing OTLP bearer access is
unchanged. The password is hashed at boot and passed privately to nginx.
- Grafana gets the provisioned **Macro PostgreSQL** dashboard at
- `/d/macro-postgres`.
+ Grafana gets **Macro PostgreSQL** at `/d/macro-postgres` and
+ **Macro PostgreSQL queries** at `/d/macro-postgres-queries`.
2. As an authorized database administrator, connect **to dev writer / macrodb**
with TLS and run `psql --file monitoring-role.sql`. Choose a random password
of at least 32 characters at its secure prompt. The role/grants replicate to
@@ -27,20 +30,29 @@ must be verified before expanding the dev-only stack to production.
application-table SELECT; it sets connection and query timeouts and defaults
to read-only transactions. It doesn't install extensions or restart RDS.
`pg_monitor` can read activity/query information, so treat this credential as
- sensitive even though this collector does not export query text.
-3. Save that password in Doppler `observability/dev` as
+ sensitive.
+3. On the same dev writer / macrodb connection, run
+ `psql --file query-monitoring.sql`. It checks `pg_stat_statements` is already
+ preloaded and query IDs are enabled, installs the standard extension if needed,
+ and raises the monitoring role's connection limit from five to eight. The
+ extension and role changes replicate; do not run this on the reader. It does
+ not reset query statistics or change RDS settings. If the preload check fails,
+ review the existing RDS parameter group and plan a reboot separately; this
+ stack never restarts the database. Confirm the extension is visible on both
+ endpoints before starting Alloy.
+4. Save that password in Doppler `observability/dev` as
`GRAFANA_POSTGRES_PASSWORD` and an AWS-managed-key Secrets Manager secret in
Virginia with JSON shape `{"password":"..."}`. Use the existing raw-string
dev-ingest secret for `ingestSecretArn`. Doppler is the operator source;
the host retrieves only the two explicit AWS secrets using its instance
role. There is no automatic Doppler-to-Secrets-Manager synchronization.
Never put values in Pulumi config, command-line arguments or user data.
-4. Build the pinned collector image:
+5. Build the pinned collector image:
`nix build ./infra/stacks/observability#database-collector --out-link result-database-collector`.
Publish with
`python3 infra/stacks/observability/nixos/publish-image.py result-database-collector VIRGINIA_PRIVATE_IMAGE_BUCKET --region us-east-1 --name-prefix macro-database-collector`.
The image is fixed; EC2 user data carries validated nonsecret settings only.
-5. Initialize the separate Pulumi project here with a `dev` stack. Configure
+6. Initialize the separate Pulumi project here with a `dev` stack. Configure
`aws:region=us-east-1`, `amiId`, `subnetId`, `secretArn`, and `ingestSecretArn`.
The subnet must be private, in the database VPC, with outbound NAT/HTTPS
access to AWS APIs and Ohio. The AMI must be the reviewed collector image
@@ -64,9 +76,10 @@ endpoint must report `pg_up=1`, `pg_exporter_last_scrape_error=0`, and scrape
transaction, connection, lock and vacuum metrics during representative dev
traffic. Missing metrics are not replaced with reassuring zeros. The freshness
panel measures the latest sample's age within Prometheus's five-minute lookback;
-after that it shows no data, not an increasing outage timer. Verify no passwords,
-connection-string labels, SQL statements or parameter values appear
-in ingested data or logs. The exporter runs once per minute with five-second
+after that it shows no data, not an increasing outage timer. Verify no passwords
+or connection-string labels appear in ingested data or logs. SQL text is
+deliberately present in Loki, not in Prometheus labels. The exporter runs once
+per minute with five-second
SQL timeouts; a failing collector does not imply the database is down.
Exercise a temporary receiver outage in an isolated smoke environment, restore
@@ -75,6 +88,64 @@ samples. No alert notifications are enabled in this PR; collection-health
alert routing and independent host/disk alarms are follow-up rollout work.
This pilot must not be the sole database monitor.
+## Query investigation and limits
+
+Open **Macro PostgreSQL queries**, choose an instance, then click a query ID in
+the query overview. Normalized SQL appears directly beside execution time,
+workload share, call count, average duration and rows. Sort any metric column,
+filter the SQL column, select a database user, or raise the row limit to 200.
+The SQL filter searches the ranked rows currently loaded; use the SQL log panel
+and query-ID selector for other collected statements. Summary cards and query
+charts respect the selected instance, user and query ID. Identical query IDs
+across selected users are aggregated.
+
+The overview joins Prometheus counters to SQL associations observed within five
+minutes of the selected period's end. A missing SQL association does not hide an
+expensive query's counters. SQL text may be unavailable; use the SQL log panel
+to search a wider time range. Statements without interval
+execution time do not appear in the ranking. Each row links to the same dashboard
+with its query ID and the current user/time filters preserved. The wait chart
+counts emitted observations per minute; it does not represent time-weighted
+active sessions. The blocking panel shows sampled blocked PIDs and their blockers.
+
+The dashboard connects these counters to sampled sessions, waits and I/O timing.
+The health dashboard links to it, and the query dashboard links back to collection
+health. Empty panels mean missing data or no observed activity, not a healthy DB.
+
+`pg_stat_statements` metrics and SQL associations are collected once per minute,
+limited to 200 statements per endpoint ranked by cumulative execution time, with
+the upstream collector's low-cost percentile cutoff. This can omit new or cheap
+queries; resetting statistics, evictions and first observations also affect
+interval calculations. Database execution time is not CPU time, and aggregate
+means are not p95/p99. Block I/O time requires `track_io_timing`; this PR does not
+enable it. The extension covers statements executed on each endpoint, so the
+writer and replica must be collected separately.
+
+Alloy samples `pg_stat_activity` every 15 seconds, emitting query samples and
+wait events as observed queries end or change state. Short queries can be missed.
+`disable_query_redaction = false` omits raw SQL from activity samples; SQL text
+comes from `pg_stat_statements`. Monitoring/system users and system databases
+are excluded. The pipeline drops non-DML SQL associations because utility
+statements may retain literals. Normalization is not anonymization: identifiers,
+application names, client addresses and occasionally literals can be sensitive.
+Treat Grafana/Loki access and their backups accordingly. Do not put personal data
+or credentials in SQL comments or identifiers.
+
+Before rollout, confirm both endpoints produce
+`pg_stat_statements_calls_total` and `pg_stat_statements_seconds_total`, and Loki
+has `job="integrations/postgres-queries"` streams with `op="query_association"`.
+Run a harmless representative dev query, locate its normalized SQL and query ID,
+then observe a long-running query and wait event. Confirm no raw SQL appears in
+`op="query_sample"`, and that `grafana_monitor` cannot SELECT application rows.
+An hourly `op="health_status"` stream includes upstream requirements for disabled
+features; use the actual statement/activity records to verify enabled collection.
+A healthy metrics exporter alone does not prove query details work.
+
+Automatic `EXPLAIN` and schema collection are disabled. The pinned Alloy plan
+collector prepares statements and needs application-object permissions beyond
+`pg_monitor`; those grants need a separate review. Existing Datadog and
+`auto_explain` configuration stay unchanged.
+
## Storage, rotation and rollback
The encrypted 20 GiB gp3 data volume is retained/protected across host
@@ -82,7 +153,10 @@ replacement; stable ownership and volume-ID verification prevent silent
fallback to root storage. Prometheus remote-write WAL is retained up to 24 hours,
not an unlimited queue or a guarantee against a whole Virginia failure. Monitor
volume capacity, measure actual telemetry volume and tune the retention before
-production. Metrics delivered to Ohio use its existing Prometheus retention.
+production. Metrics delivered to Ohio use its existing Prometheus retention; query logs use
+Loki's existing 30-day retention. Loki's disk WAL buffers delivery but has no
+configured 24-hour cutoff like Prometheus: a sustained outage can fill the
+collector volume. Keep disk monitoring and Datadog in place.
For credential rotation, update the role password and AWS secret, then restart
`database-collector-bootstrap.service` over SSM. Alloy stops with bootstrap and
@@ -93,5 +167,30 @@ requires updating its Ohio source and all existing OTLP clients.
Rollback by stopping Alloy or removing this collector stack and its added
security-group rules; the WAL volume remains protected. The role may be removed
by an administrator after its last consumer is gone. Keep Datadog running during
-all pilot work. Query statistics, richer investigations, full Datadog inventory
-and alert-by-alert migration remain later passes.
+all pilot work. Automatic plans, full Datadog inventory and alert-by-alert migration remain
+separate passes.
+
+## Existing PostgreSQL monitoring projects
+
+The dashboard uses Grafana's built-in mixed-source table, joins, filters, logs
+and time-series panels; no additional plugin is required. Existing projects
+provide useful alternatives, with different collection/storage requirements:
+
+- [pgwatch](https://github.com/cybertec-postgresql/pgwatch/) provides a PostgreSQL
+ collector and Grafana dashboard suite. Its dashboards expect pgwatch's metric
+ names and data model; importing them alone does not populate Alloy data.
+- [Crunchy Data pgMonitor](https://github.com/CrunchyData/pgmonitor) provides
+ PostgreSQL health dashboards, metrics and alerting resources using its own
+ collection definitions.
+- [Percona PMM](https://docs.percona.com/percona-monitoring-and-management/3/install-pmm/install-pmm-client/connect-database/aws.html)
+ supports RDS PostgreSQL query analytics using pg_stat_statements. Adopting its
+ Query Analytics application requires PMM Server and its collection components;
+ it is not a standalone panel for this Grafana instance.
+- [Grafana Cloud Database Observability](https://grafana.com/docs/grafana-cloud/observe-and-act/monitor-applications/database-observability/introduction/)
+ provides a purpose-built query investigation UI using Alloy. Those Cloud
+ dashboards/features do not arrive with the open-source collector alone.
+
+This pilot improves the existing self-hosted setup. It does not claim full
+Datadog DBM parity: automatic plans, trace correlation and time-weighted active
+session analysis remain separate work. A plan-visualization plugin cannot supply
+plans that have not been collected.
diff --git a/infra/stacks/database-observability/query-monitoring.sql b/infra/stacks/database-observability/query-monitoring.sql
new file mode 100644
index 0000000000..a59877624d
--- /dev/null
+++ b/infra/stacks/database-observability/query-monitoring.sql
@@ -0,0 +1,24 @@
+-- Run on DEV writer / macrodb after monitoring-role.sql, before starting Alloy.
+-- This never changes RDS parameters, resets statistics, or restarts the database.
+\set ON_ERROR_STOP on
+DO $$
+BEGIN
+ IF current_database() <> 'macrodb' THEN
+ RAISE EXCEPTION 'Connect to macrodb before enabling query monitoring';
+ END IF;
+ IF NOT ('pg_stat_statements' = ANY(string_to_array(
+ replace(current_setting('shared_preload_libraries'), ' ', ''), ','))) THEN
+ RAISE EXCEPTION 'pg_stat_statements must already be preloaded; review the RDS parameter group and restart requirement separately';
+ END IF;
+ IF current_setting('compute_query_id') = 'off' THEN
+ RAISE EXCEPTION 'compute_query_id is off; review the RDS configuration separately';
+ END IF;
+END $$;
+CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
+-- Allow bounded concurrent health, statement and activity collection.
+ALTER ROLE grafana_monitor CONNECTION LIMIT 8;
+-- Confirm statistics visibility without reading application rows.
+SET ROLE grafana_monitor;
+SELECT count(*) AS visible_statement_groups FROM pg_stat_statements;
+SELECT count(*) AS visible_sessions FROM pg_stat_activity;
+RESET ROLE;
diff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.md
index 14b0cc7a35..6727f06624 100644
--- a/infra/stacks/observability/README.md
+++ b/infra/stacks/observability/README.md
@@ -413,8 +413,8 @@ Tempo spans link to the service's logs in a time window and its generated metric
logs carrying an OTLP trace ID link back to the trace. FireLens logs do not always
carry trace IDs, so service/time correlation is not a claim of exact trace-to-log
matching. Service graph edges require correctly related client/server spans.
-Profiles, database query monitoring, RUM/session replay, and complete Datadog
-dashboard/monitor migration remain separate coverage work.
+Profiles, production/full database-query parity, RUM/session replay, and complete
+Datadog dashboard/monitor migration remain separate coverage work.
### Provisioned dashboards and alerts
@@ -508,6 +508,17 @@ stack definitions and the CloudWatch-logged preview gateway need a separate
coverage pass. Lambda CloudWatch logs, RUM/session replay, synthetics and database query monitoring
are separate sources; ECS/edge OTLP duplication alone does not provide parity.
+## PostgreSQL query monitoring
+
+The separate [dev database collector](../database-observability/README.md) sends
+query statistics to Prometheus and normalized SQL/activity samples to Loki. Its
+query dashboard at `/d/macro-postgres-queries` links from `/d/macro-postgres`.
+Deploy the Ohio image with the authenticated POST-only `/loki/api/v1/push` route
+before upgrading the collector. The route reuses the existing ingestion
+credential and exposes no Loki query or administration APIs. SQL records are
+sensitive and use Loki's existing retention/access controls. Database setup and
+collector deployment are manual; merging the PR does not enable collection.
+
## Validation and subsequent passes
From `infra/`, run `bunx biome check stacks/observability` and `bun run check`.
@@ -524,9 +535,10 @@ Follow-up PRs:
independent bounded queues, retries and matching sampling. Check actual
metrics temporality and attribute mapping before enabling all services.
2. Inventory non-OTEL sources: Datadog agents, infrastructure metrics, CloudWatch,
- RDS/Postgres query monitoring and logs, browser RUM, synthetics, profiling and
- security features. They are not covered by duplicating OTLP. Add DB collectors
- with dedicated read-only monitoring identities and evaluate query-level
+ production RDS/Postgres query parity and logs, browser RUM, synthetics,
+ profiling and security features. They are not covered by duplicating OTLP.
+ Extend the dev DB collector to production with dedicated read-only monitoring
+ identities and evaluate query-level
parity before replacing Datadog DBM.
3. Port dashboards/alerts and run representative incidents in both tools. Add
pipeline/queue alerts, ingestion canaries, host/root-disk monitoring and a
diff --git a/infra/stacks/observability/nixos/database-collector/collector.alloy b/infra/stacks/observability/nixos/database-collector/collector.alloy
index 2aec89729a..d17e537f43 100644
--- a/infra/stacks/observability/nixos/database-collector/collector.alloy
+++ b/infra/stacks/observability/nixos/database-collector/collector.alloy
@@ -9,15 +9,29 @@ local.file "replica_dsn" {
}
// A separate exporter per endpoint avoids upstream multi-DSN collector limitations.
-// Statement samples, SQL text and execution plans are deliberately a later pass.
+// Query IDs stay in metrics; statement text and activity samples go to Loki.
prometheus.exporter.postgres "writer" {
data_source_names = [local.file.writer_dsn.content]
- enabled_collectors = ["database", "locks", "replication", "replication_slot", "stat_bgwriter", "stat_database", "stat_progress_vacuum", "stat_user_tables", "statio_user_tables", "wal", "long_running_transactions", "stat_activity_autovacuum"]
+ enabled_collectors = ["database", "locks", "replication", "replication_slot", "stat_bgwriter", "stat_database", "stat_progress_vacuum", "stat_user_tables", "statio_user_tables", "wal", "long_running_transactions", "stat_activity_autovacuum", "stat_statements"]
+
+ stat_statements {
+ include_query = false
+ limit = 200
+ exclude_users = ["grafana_monitor", "rdsadmin"]
+ exclude_databases = ["postgres", "template0", "template1", "rdsadmin"]
+ }
}
prometheus.exporter.postgres "replica" {
data_source_names = [local.file.replica_dsn.content]
- enabled_collectors = ["database", "locks", "replication", "replication_slot", "stat_bgwriter", "stat_database", "stat_progress_vacuum", "stat_user_tables", "statio_user_tables", "wal", "long_running_transactions", "stat_activity_autovacuum", "stat_wal_receiver"]
+ enabled_collectors = ["database", "locks", "replication", "replication_slot", "stat_bgwriter", "stat_database", "stat_progress_vacuum", "stat_user_tables", "statio_user_tables", "wal", "long_running_transactions", "stat_activity_autovacuum", "stat_wal_receiver", "stat_statements"]
+
+ stat_statements {
+ include_query = false
+ limit = 200
+ exclude_users = ["grafana_monitor", "rdsadmin"]
+ exclude_databases = ["postgres", "template0", "template1", "rdsadmin"]
+ }
}
discovery.relabel "writer" {
@@ -111,3 +125,129 @@ prometheus.remote_write "ohio" {
max_keepalive_time = "24h"
}
}
+
+// Reuse the existing exporter; do not autodiscover or connect to other databases.
+database_observability.postgres "writer" {
+ data_source_name = local.file.writer_dsn.content
+ targets = prometheus.exporter.postgres.writer.targets
+ forward_to = [loki.relabel.writer.receiver]
+ disable_collectors = ["explain_plans", "schema_details"]
+ exclude_users = ["grafana_monitor", "rdsadmin", "db-o11y"]
+ exclude_databases = ["postgres", "template0", "template1", "rdsadmin"]
+
+ query_details {
+ collect_interval = "1m"
+ statements_limit = 200
+ }
+
+ query_samples {
+ collect_interval = "15s"
+ disable_query_redaction = false
+ exclude_current_user = true
+ }
+}
+
+loki.relabel "writer" {
+ forward_to = [loki.process.database_queries.receiver]
+
+ rule {
+ target_label = "instance"
+ replacement = "macro-db-dev"
+ }
+
+ rule {
+ target_label = "db_instance"
+ replacement = "macro-db-dev"
+ }
+
+ rule {
+ target_label = "db_role"
+ replacement = "writer"
+ }
+
+ rule {
+ target_label = "deployment_environment"
+ replacement = "dev"
+ }
+
+ rule {
+ target_label = "job"
+ replacement = "integrations/postgres-queries"
+ }
+}
+
+// Reuse the existing exporter; do not autodiscover or connect to other databases.
+database_observability.postgres "replica" {
+ data_source_name = local.file.replica_dsn.content
+ targets = prometheus.exporter.postgres.replica.targets
+ forward_to = [loki.relabel.replica.receiver]
+ disable_collectors = ["explain_plans", "schema_details"]
+ exclude_users = ["grafana_monitor", "rdsadmin", "db-o11y"]
+ exclude_databases = ["postgres", "template0", "template1", "rdsadmin"]
+
+ query_details {
+ collect_interval = "1m"
+ statements_limit = 200
+ }
+
+ query_samples {
+ collect_interval = "15s"
+ disable_query_redaction = false
+ exclude_current_user = true
+ }
+}
+
+loki.relabel "replica" {
+ forward_to = [loki.process.database_queries.receiver]
+
+ rule {
+ target_label = "instance"
+ replacement = "macro-db-dev-read-replica"
+ }
+
+ rule {
+ target_label = "db_instance"
+ replacement = "macro-db-dev-read-replica"
+ }
+
+ rule {
+ target_label = "db_role"
+ replacement = "replica"
+ }
+
+ rule {
+ target_label = "deployment_environment"
+ replacement = "dev"
+ }
+
+ rule {
+ target_label = "job"
+ replacement = "integrations/postgres-queries"
+ }
+}
+
+loki.process "database_queries" {
+ forward_to = [loki.write.ohio.receiver]
+ // pg_stat_statements normalizes DML, but utility statements can retain literals.
+ // Keep only DML associations and activity/wait samples; SQL is never a stream label.
+ stage.match {
+ selector = "{job=\"integrations/postgres-queries\",op=\"query_association\"} !~ \"(?i)querytext=\\\"[[:space:]]*(SELECT|INSERT|UPDATE|DELETE|WITH|MERGE)[[:space:](]\""
+ action = "drop"
+ drop_counter_reason = "postgres_utility_statement"
+ }
+}
+
+loki.write "ohio" {
+ endpoint {
+ url = "https://otlp.macro-internal.com/loki/api/v1/push"
+
+ basic_auth {
+ username = "metrics"
+ password_file = sys.env("CREDENTIALS_DIRECTORY") + "/otlp_token"
+ }
+ }
+
+ wal {
+ enabled = true
+ }
+}
diff --git a/infra/stacks/observability/nixos/database-dashboard.nix b/infra/stacks/observability/nixos/database-dashboard.nix
index ae073d67f9..c99f59b9a4 100644
--- a/infra/stacks/observability/nixos/database-dashboard.nix
+++ b/infra/stacks/observability/nixos/database-dashboard.nix
@@ -39,7 +39,7 @@ let
from = "now-6h";
to = "now";
};
- description = "Dev writer and replica collected directly from PostgreSQL. Start with connection health and collection errors. No query text or plans are collected in this baseline. Missing data means collection needs attention, not that the database is healthy.";
+ description = "Dev writer and replica collected directly from PostgreSQL. Start with connection health and collection errors. Open the query dashboard for statement costs, normalized SQL and sampled waits. Plans are disabled. Missing data means collection needs attention, not that the database is healthy.";
templating.list = [
{
name = "database";
@@ -57,6 +57,11 @@ let
}
];
links = [
+ {
+ title = "Query costs, SQL and sampled waits";
+ type = "link";
+ url = "/d/macro-postgres-queries";
+ }
{
title = "AWS database health";
type = "link";
diff --git a/infra/stacks/observability/nixos/database-queries-dashboard.nix b/infra/stacks/observability/nixos/database-queries-dashboard.nix
new file mode 100644
index 0000000000..3217640a2c
--- /dev/null
+++ b/infra/stacks/observability/nixos/database-queries-dashboard.nix
@@ -0,0 +1,456 @@
+{ lib, pkgs, ... }:
+let
+ prometheus = {
+ type = "prometheus";
+ uid = "prometheus";
+ };
+ loki = {
+ type = "loki";
+ uid = "loki";
+ };
+ selector = ''job="integrations/postgres",deployment_environment="dev",db_instance=~"$database",datname="macrodb",user=~"''${dbuser:regex}",queryid=~"$queryid"'';
+ logs = ''job="integrations/postgres-queries",deployment_environment="dev",db_instance=~"$database"'';
+ panel = id: title: datasource: targets: {
+ inherit
+ id
+ title
+ datasource
+ targets
+ ;
+ type = "timeseries";
+ gridPos = {
+ x = lib.mod (id - 2) 2 * 12;
+ y = 16 + builtins.div (id - 2) 2 * 9;
+ w = 12;
+ h = 9;
+ };
+ };
+ metric =
+ id: title: expr: unit:
+ (panel id title prometheus [
+ {
+ refId = "A";
+ inherit expr;
+ legendFormat = "{{db_instance}} {{user}} {{queryid}}";
+ }
+ ])
+ // {
+ fieldConfig.defaults = { inherit unit; };
+ };
+ logPanel =
+ id: title: expr:
+ (panel id title loki [
+ {
+ refId = "A";
+ inherit expr;
+ queryType = "range";
+ }
+ ])
+ // {
+ type = "logs";
+ options = {
+ showTime = true;
+ showCommonLabels = false;
+ wrapLogMessage = true;
+ sortOrder = "Descending";
+ };
+ };
+ totals =
+ name: "sum by (queryid) (increase(pg_stat_statements_${name}_total{${selector}}[$__range]))";
+ timeTotals = totals "seconds";
+ callTotals = totals "calls";
+ rowTotals = totals "rows";
+ ranking = "topk($top, ${timeTotals} > 0)";
+ instant = refId: expr: {
+ inherit refId expr;
+ datasource = prometheus;
+ instant = true;
+ range = false;
+ format = "table";
+ };
+ field = name: properties: {
+ matcher = {
+ id = "byName";
+ options = name;
+ };
+ inherit properties;
+ };
+ unit = value: {
+ id = "unit";
+ inherit value;
+ };
+ summary =
+ id: title: expr: unitName: x:
+ (metric id title expr unitName)
+ // {
+ type = "stat";
+ gridPos = {
+ inherit x;
+ y = 0;
+ w = 6;
+ h = 4;
+ };
+ targets = [ (instant "A" expr) ];
+ description = "Selected queries within the bounded collected statement set. Missing data is not replaced by zero. These summaries are not health thresholds.";
+ fieldConfig.defaults = {
+ unit = unitName;
+ color = {
+ mode = "fixed";
+ fixedColor = "blue";
+ };
+ };
+ options = {
+ colorMode = "value";
+ graphMode = "none";
+ reduceOptions = {
+ calcs = [ "lastNotNull" ];
+ fields = "";
+ values = false;
+ };
+ };
+ };
+ dashboard = {
+ uid = "macro-postgres-queries";
+ title = "Macro PostgreSQL queries";
+ schemaVersion = 39;
+ version = 2;
+ editable = false;
+ tags = [
+ "macro"
+ "postgres"
+ "queries"
+ ];
+ timezone = "browser";
+ refresh = "1m";
+ time = {
+ from = "now-1h";
+ to = "now";
+ };
+ description = "Dev query statistics and sampled activity. Select an instance and database user, filter the SQL column, then click a query ID to investigate. All workload totals cover the collected statement set, not every query on the server. The collector retains up to 200 statements per endpoint ranked by cumulative execution time; this is not every execution. Means are not percentiles. SQL is sensitive even when normalized. Activity samples can miss short queries; plans are disabled.";
+ links = [
+ {
+ title = "Database health and collection errors";
+ type = "link";
+ url = "/d/macro-postgres";
+ }
+ ];
+ templating.list = [
+ {
+ name = "database";
+ label = "Database instance";
+ type = "custom";
+ query = "macro-db-dev,macro-db-dev-read-replica";
+ current = {
+ text = "macro-db-dev";
+ value = "macro-db-dev";
+ };
+ options = [ ];
+ multi = false;
+ includeAll = false;
+ }
+ {
+ name = "dbuser";
+ label = "Database user";
+ type = "query";
+ datasource = prometheus;
+ query = ''label_values(pg_stat_statements_calls_total{job="integrations/postgres",deployment_environment="dev",db_instance=~"$database",datname="macrodb"}, user)'';
+ refresh = 1;
+ multi = true;
+ includeAll = true;
+ allValue = ".*";
+ current = {
+ text = "All";
+ value = "$__all";
+ };
+ options = [ ];
+ }
+ {
+ name = "top";
+ label = "Query rows";
+ type = "custom";
+ query = "20,50,100,200";
+ current = {
+ text = "20";
+ value = "20";
+ };
+ options = [ ];
+ multi = false;
+ includeAll = false;
+ }
+ {
+ name = "queryid";
+ label = "Query ID";
+ type = "query";
+ datasource = prometheus;
+ query = ''label_values(pg_stat_statements_calls_total{job="integrations/postgres",deployment_environment="dev",db_instance=~"$database",datname="macrodb"}, queryid)'';
+ refresh = 1;
+ multi = false;
+ includeAll = true;
+ allValue = ".*";
+ current = {
+ text = "All";
+ value = "$__all";
+ };
+ options = [ ];
+ }
+ ];
+ panels = [
+ (summary 11 "Observed execution time" "sum(${timeTotals})" "s" 0)
+ (summary 12 "Estimated calls" "sum(${callTotals})" "short" 6)
+ (summary 13 "Average time per call" "1000 * sum(${timeTotals}) / (sum(${callTotals}) > 0)" "ms" 12)
+ (summary 14 "Query shapes observed in period" "count(${callTotals} > 0)" "short" 18)
+ {
+ id = 1;
+ title = "Query overview — filter SQL, sort any metric, click a query ID";
+ type = "table";
+ datasource = {
+ type = "datasource";
+ uid = "-- Mixed --";
+ };
+ gridPos = {
+ x = 0;
+ y = 4;
+ w = 24;
+ h = 12;
+ };
+ description = "Query shapes ranked by observed execution time, bounded by the selected row limit. SQL column filters apply to these ranked rows. SQL is looked up near the end of the selected period and may be missing for evicted statements or collector outages. User filters apply to counters; identical query IDs across users are combined. Share is of the selected collected workload. Counter increases are extrapolated estimates, so call and row counts may be fractional. These are execution durations, not CPU time or latency percentiles.";
+ targets = [
+ (instant "A" ranking)
+ (instant "B" "${callTotals} and on(queryid) ${ranking}")
+ (instant "C" "(1000 * ${timeTotals} / (${callTotals} > 0)) and on(queryid) ${ranking}")
+ (instant "D" "100 * ${ranking} / scalar(sum(${timeTotals}))")
+ (instant "E" "${rowTotals} and on(queryid) ${ranking}")
+ {
+ refId = "SQL";
+ datasource = loki;
+ queryType = "instant";
+ expr = ''topk by (queryid) (1, sum by (queryid, querytext) (count_over_time({${logs},op="query_association"} | logfmt | queryid=~"$queryid" | querytext!="" | __error__="" [5m])))'';
+ }
+ ];
+ transformations = [
+ {
+ id = "labelsToFields";
+ options.mode = "columns";
+ }
+ {
+ id = "joinByField";
+ options = {
+ byField = "queryid";
+ mode = "outerTabular";
+ };
+ }
+ {
+ id = "filterByValue";
+ options = {
+ type = "include";
+ match = "all";
+ filters = [
+ {
+ fieldName = "Value #A";
+ config.id = "isNotNull";
+ }
+ ];
+ };
+ }
+ {
+ id = "organize";
+ options = {
+ excludeByName = {
+ Time = true;
+ "Time 1" = true;
+ "Time 2" = true;
+ "Time 3" = true;
+ "Time 4" = true;
+ "Time 5" = true;
+ "Time 6" = true;
+ "Value #SQL" = true;
+ };
+ indexByName = {
+ queryid = 0;
+ querytext = 1;
+ "Value #A" = 2;
+ "Value #D" = 3;
+ "Value #B" = 4;
+ "Value #C" = 5;
+ "Value #E" = 6;
+ };
+ renameByName = {
+ querytext = "SQL";
+ "Value #A" = "Execution time";
+ "Value #B" = "Calls";
+ "Value #C" = "Average time";
+ "Value #D" = "Workload share";
+ "Value #E" = "Rows";
+ };
+ };
+ }
+ ];
+ options = {
+ showHeader = true;
+ sortBy = [
+ {
+ displayName = "Execution time";
+ desc = true;
+ }
+ ];
+ };
+ fieldConfig = {
+ defaults = {
+ custom = {
+ filterable = true;
+ cellOptions.type = "auto";
+ };
+ noValue = "—";
+ };
+ overrides = [
+ (field "queryid" [
+ {
+ id = "custom.width";
+ value = 200;
+ }
+ {
+ id = "links";
+ value = [
+ {
+ title = "Investigate query";
+ url = "/d/macro-postgres-queries?var-database=$database&\${dbuser:queryparam}&var-top=$top&var-queryid=\${__value.raw}&\${__url_time_range}";
+ }
+ ];
+ }
+ ])
+ (field "SQL" [
+ {
+ id = "custom.width";
+ value = 500;
+ }
+ {
+ id = "custom.cellOptions";
+ value = {
+ type = "auto";
+ wrapText = true;
+ };
+ }
+ ])
+ (field "Execution time" [ (unit "s") ])
+ (field "Average time" [ (unit "ms") ])
+ (field "Workload share" [ (unit "percent") ])
+ (field "Calls" [
+ (unit "short")
+ {
+ id = "decimals";
+ value = 0;
+ }
+ ])
+ (field "Rows" [
+ (unit "short")
+ {
+ id = "decimals";
+ value = 0;
+ }
+ ])
+ ];
+ };
+ }
+ (metric 2 "Average execution time per call"
+ "1000 * rate(pg_stat_statements_seconds_total{${selector}}[$__rate_interval]) / (rate(pg_stat_statements_calls_total{${selector}}[$__rate_interval]) > 0)"
+ "ms"
+ )
+ (metric 3 "Calls per second" "rate(pg_stat_statements_calls_total{${selector}}[$__rate_interval])"
+ "ops"
+ )
+ (metric 4 "Rows per call"
+ "rate(pg_stat_statements_rows_total{${selector}}[$__rate_interval]) / (rate(pg_stat_statements_calls_total{${selector}}[$__rate_interval]) > 0)"
+ "short"
+ )
+ (
+ (metric 5 "Block read time per second"
+ "rate(pg_stat_statements_block_read_seconds_total{${selector}}[$__rate_interval])"
+ "s"
+ )
+ // {
+ description = "Requires track_io_timing on the database; zero can mean timing is disabled. This PR does not change that setting.";
+ }
+ )
+ (metric 6 "Block write time per second"
+ "rate(pg_stat_statements_block_write_seconds_total{${selector}}[$__rate_interval])"
+ "s"
+ )
+ (logPanel 7 "Normalized SQL for selected queries"
+ ''{${logs},op="query_association"} | logfmt | queryid=~"$queryid" | __error__="" | line_format "{{.queryid}}: {{.querytext}}"''
+ )
+ (
+ (logPanel 8 "Sampled query activity"
+ ''{${logs},op="query_sample"} | logfmt | user=~"''${dbuser:regex}" | queryid=~"$queryid" | __error__=""''
+ )
+ // {
+ description = "Activity sampled every 15 seconds, emitted when an observed query ends or changes state. Raw SQL is omitted. These records do not represent every execution.";
+ }
+ )
+ (logPanel 9 "Sampled waits and blocking sessions"
+ ''{${logs},op="wait_event"} | logfmt | user=~"''${dbuser:regex}" | queryid=~"$queryid" | __error__=""''
+ )
+ (
+ (logPanel 10 "Query collector health checks" ''{${logs},op="health_status"}'')
+ // {
+ description = "Hourly upstream checks, including requirements for features we leave disabled. Also inspect database health: a healthy metric exporter alone does not prove query-detail collection works.";
+ }
+ )
+ (
+ (metric 15 "Observed wait events per minute"
+ ''sum by (wait_event_type) (count_over_time({${logs},op="wait_event"} | logfmt | user=~"''${dbuser:regex}" | queryid=~"$queryid" | wait_event_type!="" | __error__="" [1m]))''
+ "short"
+ )
+ // {
+ datasource = loki;
+ gridPos = {
+ x = 0;
+ y = 61;
+ w = 12;
+ h = 9;
+ };
+ targets = [
+ {
+ refId = "A";
+ queryType = "range";
+ expr = ''sum by (wait_event_type) (count_over_time({${logs},op="wait_event"} | logfmt | user=~"''${dbuser:regex}" | queryid=~"$queryid" | wait_event_type!="" | __error__="" [1m]))'';
+ legendFormat = "{{wait_event_type}}";
+ }
+ ];
+ description = "Counts emitted wait observations. This is not time-weighted database load or a count of every wait. Short waits may be missed.";
+ }
+ )
+ (
+ (logPanel 16 "Observed blocking chains — blocked session and blocker PIDs"
+ ''{${logs},op="wait_event"} | logfmt | user=~"''${dbuser:regex}" | queryid=~"$queryid" | blocked_by_pids!="[]" | blocked_by_pids!="" | __error__="" | line_format "Query {{.queryid}} · blocked PID {{.pid}} · blockers {{.blocked_by_pids}} · {{.wait_event_type}}/{{.wait_event}} · observed wait {{.wait_time}}"''
+ )
+ // {
+ gridPos = {
+ x = 12;
+ y = 61;
+ w = 12;
+ h = 9;
+ };
+ description = "Historical sampled blocking relationships, emitted when observed activity changes or ends. Not a live list of all sessions.";
+ }
+ )
+ ];
+ };
+in
+{
+ services.grafana.provision.dashboards.settings.providers = [
+ {
+ name = "macro-postgres-queries";
+ orgId = 1;
+ folder = "Macro";
+ type = "file";
+ disableDeletion = true;
+ allowUiUpdates = false;
+ options.path = pkgs.linkFarm "macro-postgres-queries-dashboard" [
+ {
+ name = "postgres-queries.json";
+ path = pkgs.writeText "postgres-queries.json" (builtins.toJSON dashboard);
+ }
+ ];
+ }
+ ];
+}
diff --git a/infra/stacks/observability/nixos/proxy.nix b/infra/stacks/observability/nixos/proxy.nix
index 3f425a5b17..72f8e34126 100644
--- a/infra/stacks/observability/nixos/proxy.nix
+++ b/infra/stacks/observability/nixos/proxy.nix
@@ -77,6 +77,18 @@ in
proxy_set_header Authorization "";
'';
};
+ "= /loki/api/v1/push" = {
+ proxyPass = "http://127.0.0.1:3100/loki/api/v1/push";
+ basicAuthFile = "/run/credentials/nginx.service/metrics.htpasswd";
+ extraConfig = ''
+ if ($observability_host != otlp) { return 404; }
+ limit_except POST { deny all; }
+ limit_req zone=ingest burst=200 nodelay;
+ limit_req_status 429;
+ proxy_read_timeout 30s;
+ proxy_set_header Authorization "";
+ '';
+ };
};
};
# Grafana Viewers can choose arbitrary datasource proxy paths. Allow query
diff --git a/infra/stacks/observability/nixos/services.nix b/infra/stacks/observability/nixos/services.nix
index beb9485b71..c10a6932b8 100644
--- a/infra/stacks/observability/nixos/services.nix
+++ b/infra/stacks/observability/nixos/services.nix
@@ -18,6 +18,7 @@ in
./dashboards.nix
./ci-dashboard.nix
./database-dashboard.nix
+ ./database-queries-dashboard.nix
./alerting.nix
];
services.grafana = {