Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

feat(observability): add CI percentile panels and a daily CI report (#9919)

teo0e2bbd3095e1
diff --git a/infra/package.json b/infra/package.json
index 811ea5f1ea..ab9e147463 100644
--- a/infra/package.json
+++ b/infra/package.json
@@ -9,8 +9,9 @@
"preinstall": "node -e \"if (process.env.npm_execpath && !process.env.npm_execpath.includes('bun')) { console.error('Please use bun as the package manager'); process.exit(1); }\"",
"lint": "biome check .",
"format": "biome check --write .",
- "check": "tsc --noEmit && tsc --noEmit --skipLibCheck --project stacks/web-app/tsconfig.json && bun run test:web-delivery",
- "test:web-delivery": "bun test stacks/web-app/publication.test.ts && bun test stacks/web-app/app-route.test.ts && bun test stacks/website/app-cache.test.ts"
+ "check": "tsc --noEmit && tsc --noEmit --skipLibCheck --project stacks/web-app/tsconfig.json && bun run test:web-delivery && bun run test:observability",
+ "test:web-delivery": "bun test stacks/web-app/publication.test.ts && bun test stacks/web-app/app-route.test.ts && bun test stacks/website/app-cache.test.ts",
+ "test:observability": "bun test stacks/observability/scripts/ci-report.test.ts"
},
"devDependencies": {
"@biomejs/biome": "2.4.12",
diff --git a/infra/stacks/observability/AGENTS.md b/infra/stacks/observability/AGENTS.md
index 6e9f5de4bb..32bc4eee07 100644
--- a/infra/stacks/observability/AGENTS.md
+++ b/infra/stacks/observability/AGENTS.md
@@ -18,6 +18,8 @@ us-east-1. `README.md` is the full operations guide.
(rules and the `Prod alerts` dashboard panel both read it), and its Datadog
copy in `infra/stacks/monitoring/monitors/` by hand. A new entry takes the next
free `panelId`; never renumber one, since sent notifications link to it.
+- Change a CI percentile, queue or CI hours query: edit `nixos/ci-report.json`;
+ the `Macro CI` dashboard's Loki panels and `scripts/ci-report.ts` both read it.
- Add a runtime setting: `settings.ts` → `render.ts` (bump the schema version) →
`nixos/read-user-data.py` (keys, pattern, version) → the consuming `.nix` file.
- Add a secret: `assets/refresh-secrets.py` (validation) → `LoadCredential` in
diff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.md
index e6248e51fd..7183808b0d 100644
--- a/infra/stacks/observability/README.md
+++ b/infra/stacks/observability/README.md
@@ -44,13 +44,14 @@ that starts after everything on its head has finished came from something else,
as a PR edit, a label or leaving draft. It forms its own line with `rerun=true`,
which also leaves a commit's first full run after draft out of the real PR numbers.
-Real PR numbers exclude experiments, reruns and drafts. Metadata makes an ungrouped
-query return one series per combination, so always group:
+Real PR numbers exclude experiments, reruns and drafts, and pushes where only one
+workflow ran: a lone CLA run means CI didn't run, as on a PR into another branch.
+Metadata makes an ungrouped query return one series per combination, so always group:
```logql
quantile_over_time(0.9, {service_name="ci-telemetry"} | event="push"
| experiment="false" | rerun="false" | pr_draft="false" | conclusion="success"
- | unwrap wall_seconds [1d]) by (pr_type)
+ | workflows > 1 | unwrap wall_seconds [1d]) by (pr_type)
```
Lines never carry branch names, actors, titles or file paths. A line's time and
@@ -62,12 +63,27 @@ shell step takes 30 s or more, an inference from step timing.
Collection and sending are best effort and cannot fail the exporter, and neither
touches the observed PR's jobs. Collection runs before any credential is read.
-`/d/macro-ci` provides workflow traces, failed/cancelled jobs and steps, and jobs
+`nixos/ci-report.json` holds the queries over these events: real PR wall time
+overall and by PR type, what finished last, the slowest PR jobs and steps, queue
+by runner and job, warm and cold cache volumes, CI hours by trigger, and the
+exporter's own cost. The top of `/d/macro-ci` renders them over the selected
+range, and the daily report prints the same tables as Markdown, plus one summary
+row per UTC day:
+
+```sh
+GRAFANA_TOKEN=$(cat <token file>) bun infra/stacks/observability/scripts/ci-report.ts --range 24h --days 7
+```
+
+The token needs only Viewer access. `--loki <url>` points it at another Loki API
+root, such as a local one. Change a query in the JSON and both follow. PR steps
+grouped by workflow, job and step pass Loki's default cap of 500 series in a day,
+so `backend-config.nix` raises `max_query_series` to 2000.
+
+Below them, `/d/macro-ci` provides workflow traces, failed/cancelled jobs and steps, and jobs
running longer than five minutes. Open a trace for its execution waterfall and
GitHub URL. Search tables are limited to 100 traces and 100 matching spans per
trace; narrow the time range to investigate further. These are bounded searches,
-not complete counts or p95 summaries. The action does not emit our old structured
-Loki events. Job durations exclude runner queue time. Workflow root duration
+not complete counts or p95 summaries. Job durations exclude runner queue time. Workflow root duration
includes waiting and, on reruns, starts at the original run creation time because
of an upstream limitation; do not treat it as attempt execution time.
diff --git a/infra/stacks/observability/nixos/backend-config.nix b/infra/stacks/observability/nixos/backend-config.nix
index 68080576ec..dc8d72d857 100644
--- a/infra/stacks/observability/nixos/backend-config.nix
+++ b/infra/stacks/observability/nixos/backend-config.nix
@@ -53,6 +53,8 @@
ingestion_rate_mb = 8;
ingestion_burst_size_mb = 16;
max_query_parallelism = 4;
+ # The CI dashboard groups PR steps by workflow, job and step: about 600 a day.
+ max_query_series = 2000;
};
querier.max_concurrent = 4;
analytics.reporting_enabled = false;
diff --git a/infra/stacks/observability/nixos/ci-dashboard.nix b/infra/stacks/observability/nixos/ci-dashboard.nix
index b368c76e85..487f026844 100644
--- a/infra/stacks/observability/nixos/ci-dashboard.nix
+++ b/infra/stacks/observability/nixos/ci-dashboard.nix
@@ -1,20 +1,250 @@
-{ pkgs, ... }:
+{ lib, pkgs, ... }:
let
tempo = {
type = "tempo";
uid = "tempo";
};
- selector = ''resource.service.name="github-actions" && resource.deployment.environment="ci"'';
+ loki = {
+ type = "loki";
+ uid = "loki";
+ };
+ # Shared with scripts/ci-report.ts, so the daily report and these panels agree.
+ report = builtins.fromJSON (builtins.readFile ./ci-report.json);
+ # Longest names first, as the script does, so no name replaces part of another.
+ filterNames = lib.sort (a: b: lib.stringLength a > lib.stringLength b) (
+ builtins.attrNames report.filters
+ );
+ expand =
+ range:
+ builtins.replaceStrings ([ "$RANGE" ] ++ map (name: "$" + name) filterNames) (
+ [ range ] ++ map (name: report.filters.${name}) filterNames
+ );
+ table =
+ id: lib.findFirst (entry: entry.id == id) (throw "ci-report.json has no table ${id}") report.tables;
+ refId = index: builtins.elemAt (lib.stringToCharacters "ABCDEFGHIJ") index;
+ grid = x: y: w: h: {
+ inherit
+ x
+ y
+ w
+ h
+ ;
+ };
+ # Grafana's "h" unit rolls over into days; CI hours should stay hours.
+ grafanaUnit = unit: if unit == "h" then "suffix: h" else unit;
+ unitOverride = name: unit: {
+ matcher = {
+ id = "byName";
+ options = name;
+ };
+ properties = [
+ {
+ id = "unit";
+ value = grafanaUnit unit;
+ }
+ ];
+ };
+ # Green under three minutes, red from five: the CI target for a push's wall time.
+ wallThresholds = {
+ mode = "absolute";
+ steps = [
+ {
+ color = "green";
+ value = null;
+ }
+ {
+ color = "orange";
+ value = 180;
+ }
+ {
+ color = "red";
+ value = 300;
+ }
+ ];
+ };
+ # One instant query per column over the dashboard's range, merged on the labels.
+ instantTargets =
+ entry:
+ lib.imap0 (index: column: {
+ refId = refId index;
+ datasource = loki;
+ expr = expand "$__range" column.expr;
+ queryType = "instant";
+ legendFormat = column.name;
+ }) entry.columns;
+
+ lokiTable =
+ id: tableId: gridPos:
+ let
+ entry = table tableId;
+ in
+ {
+ inherit id gridPos;
+ inherit (entry) title description;
+ type = "table";
+ datasource = loki;
+ targets = instantTargets entry;
+ transformations = [
+ {
+ id = "merge";
+ options = { };
+ }
+ {
+ id = "organize";
+ options = {
+ excludeByName.Time = true;
+ renameByName = lib.listToAttrs (
+ lib.imap0 (index: column: lib.nameValuePair "Value #${refId index}" column.name) entry.columns
+ );
+ indexByName = lib.listToAttrs (lib.imap0 (index: label: lib.nameValuePair label index) entry.by);
+ };
+ }
+ ]
+ ++ lib.optional (entry ? sort) {
+ id = "sortBy";
+ options.sort = [
+ {
+ field = entry.sort;
+ desc = true;
+ }
+ ];
+ }
+ ++ lib.optional (entry ? limit) {
+ id = "limit";
+ options.limitField = entry.limit;
+ };
+ fieldConfig = {
+ # The default minimum column width hides the last columns of a half-width panel.
+ defaults.custom.minWidth = 60;
+ overrides = map (column: unitOverride column.name column.unit) entry.columns;
+ };
+ options = {
+ showHeader = true;
+ cellHeight = "sm";
+ };
+ };
+
+ lokiStat =
+ id: tableId: gridPos:
+ let
+ entry = table tableId;
+ in
+ {
+ inherit id gridPos;
+ inherit (entry) title description;
+ type = "stat";
+ datasource = loki;
+ targets = instantTargets entry;
+ fieldConfig = {
+ defaults = {
+ color = {
+ mode = "fixed";
+ fixedColor = "text";
+ };
+ };
+ # Instant results carry no series name, so match each query by its refId.
+ overrides = lib.imap0 (index: column: {
+ matcher = {
+ id = "byFrameRefID";
+ options = refId index;
+ };
+ properties = [
+ {
+ id = "displayName";
+ value = column.name;
+ }
+ {
+ id = "unit";
+ value = grafanaUnit column.unit;
+ }
+ ]
+ ++
+ lib.optionals
+ (lib.elem column.name [
+ "p50"
+ "p90"
+ ])
+ [
+ {
+ id = "thresholds";
+ value = wallThresholds;
+ }
+ {
+ id = "color";
+ value.mode = "thresholds";
+ }
+ ];
+ }) entry.columns;
+ };
+ options = {
+ reduceOptions = {
+ calcs = [ "lastNotNull" ];
+ fields = "";
+ values = false;
+ };
+ colorMode = "value";
+ graphMode = "none";
+ textMode = "value_and_name";
+ justifyMode = "center";
+ # A narrow panel lists its values instead of squeezing them side by side.
+ orientation = if gridPos.w < 12 then "horizontal" else "vertical";
+ };
+ };
+
+ lokiSeries =
+ id: gridPos: title: description: unit: interval: targets: extra:
+ lib.recursiveUpdate {
+ inherit
+ id
+ gridPos
+ title
+ description
+ interval
+ ;
+ type = "timeseries";
+ datasource = loki;
+ maxDataPoints = 200;
+ targets = lib.imap0 (
+ index: target:
+ target
+ // {
+ refId = refId index;
+ datasource = loki;
+ queryType = "range";
+ }
+ ) targets;
+ fieldConfig.defaults = {
+ inherit unit;
+ custom = {
+ spanNulls = true;
+ showPoints = "never";
+ };
+ };
+ options.legend = {
+ displayMode = "list";
+ placement = "bottom";
+ };
+ } extra;
+
+ wallTarget = quantile: by: legendFormat: {
+ inherit legendFormat;
+ expr = expand "1d" "quantile_over_time(${quantile}, $PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by (${by})";
+ };
+
+ row = id: y: title: {
+ inherit id title;
+ type = "row";
+ collapsed = false;
+ gridPos = grid 0 y 24 1;
+ panels = [ ];
+ };
+
+ tracesTop = 47;
panel = id: title: description: filter: fields: {
inherit id title description;
type = "table";
datasource = tempo;
- gridPos = {
- x = 0;
- y = (id - 1) * 10;
- w = 24;
- h = 10;
- };
+ gridPos = grid 0 (tracesTop + (id - 1) * 10) 24 10;
targets = [
{
refId = "A";
@@ -52,24 +282,91 @@ let
];
options.showHeader = true;
};
+ selector = ''resource.service.name="github-actions" && resource.deployment.environment="ci"'';
+
dashboard = {
uid = "macro-ci";
title = "Macro CI";
- description = "GitHub Actions workflow, job and step traces exported after completion. Select a trace to open its execution waterfall. Tables are bounded search results, not complete counts or percentile summaries. Narrow the time range when there are many runs.";
+ description = "Where CI time goes. The top section reads the exporter's push, job and step events in Loki, kept 30 days, over the selected range; set it to a day or more for percentiles. The bottom section searches the run traces in Tempo, kept 7 days: open a trace for its execution waterfall. Trace tables are bounded search results, not complete counts.";
schemaVersion = 39;
- version = 2;
+ version = 3;
editable = false;
tags = [
"macro"
"ci"
];
timezone = "browser";
- refresh = "1m";
+ refresh = "5m";
time = {
from = "now-24h";
to = "now";
};
panels = [
+ (row 100 0 "Wall time and where it goes, from Loki events")
+ (lokiStat 5 "summary" (grid 0 1 24 4))
+ (lokiTable 6 "wall" (grid 0 5 12 8))
+ (lokiSeries 7 (grid 12 5 12 8) "Real PR wall time, rolling 24 h"
+ "Each point covers the 24 hours before it. Red line: the 5 minute target."
+ "s"
+ "15m"
+ [
+ (wallTarget "0.5" "" "p50")
+ (wallTarget "0.9" "" "p90")
+ ]
+ {
+ fieldConfig.defaults = {
+ thresholds = {
+ mode = "absolute";
+ steps = [
+ {
+ color = "transparent";
+ value = null;
+ }
+ {
+ color = "red";
+ value = 300;
+ }
+ ];
+ };
+ custom.thresholdsStyle.mode = "line";
+ };
+ }
+ )
+ (lokiSeries 8 (grid 0 13 12 8) "p90 wall time by PR type, rolling 24 h"
+ "Real PR pushes only; a type with few pushes moves in steps."
+ "s"
+ "15m"
+ [ (wallTarget "0.9" "pr_type" "{{pr_type}}") ]
+ { }
+ )
+ (lokiTable 9 "last" (grid 12 13 12 8))
+ (lokiTable 10 "jobs" (grid 0 21 12 10))
+ (lokiTable 11 "steps" (grid 12 21 12 10))
+ (lokiTable 12 "queue_runner" (grid 0 31 8 8))
+ (lokiTable 13 "queue_job" (grid 8 31 8 8))
+ (lokiTable 14 "volumes" (grid 16 31 8 8))
+ (lokiSeries 15 (grid 0 39 12 8) "CI hours by trigger"
+ "Job run time summed per interval, queue excluded. Compare days to measure a change such as running CI only on labelled PRs."
+ "short"
+ "1h"
+ [
+ {
+ legendFormat = "{{event_name}}";
+ expr = expand "$__interval" "sum by (event_name) (sum_over_time($JOB | unwrap run_seconds | __error__=\"\" [$RANGE])) / 3600";
+ }
+ ]
+ {
+ fieldConfig.defaults.custom = {
+ axisLabel = "CI hours";
+ drawStyle = "bars";
+ fillOpacity = 80;
+ stacking.mode = "normal";
+ };
+ }
+ )
+ (lokiTable 16 "hours" (grid 12 39 6 8))
+ (lokiStat 17 "exporter" (grid 18 39 6 8))
+ (row 101 (tracesTop - 1) "Run traces, from Tempo")
(panel 1 "Workflow runs — open a trace for jobs and steps"
"Workflow duration includes waiting; for reruns the upstream exporter starts the root span at the original run creation. Use job durations for execution timing. The url attribute links to GitHub."
"span.workflow_run_id != nil"
diff --git a/infra/stacks/observability/nixos/ci-report.json b/infra/stacks/observability/nixos/ci-report.json
new file mode 100644
index 0000000000..e65811b76e
--- /dev/null
+++ b/infra/stacks/observability/nixos/ci-report.json
@@ -0,0 +1,322 @@
+{
+ "description": "Loki queries over the CI exporter's push, job, step and export events, read by both ci-dashboard.nix and scripts/ci-report.ts. $NAME expands to filters.NAME and $RANGE to the window.",
+ "filters": {
+ "PUSH": "{service_name=\"ci-telemetry\"} | event=\"push\" | experiment=\"false\" | rerun=\"false\" | pr_draft=\"false\" | conclusion=\"success\" | workflows > 1",
+ "PR_JOB": "{service_name=\"ci-telemetry\"} | event=\"job\" | event_name=~\"pull_request|pull_request_target\" | experiment=\"false\"",
+ "PR_STEP": "{service_name=\"ci-telemetry\"} | event=\"step\" | event_name=~\"pull_request|pull_request_target\" | experiment=\"false\"",
+ "JOB": "{service_name=\"ci-telemetry\"} | event=\"job\"",
+ "EXPORT": "{service_name=\"ci-telemetry\"} | event=\"export\""
+ },
+ "tables": [
+ {
+ "id": "summary",
+ "title": "Real PR pushes: are we under 5 minutes?",
+ "description": "Successful pushes on PRs that aren't drafts, experiments or reruns, where more than one workflow ran: a lone CLA run means CI didn't run, as on a PR into another branch. Wall time runs from the push, or opening the PR, to the last check finishing. PR CI hours sum every PR job's run time.",
+ "by": [],
+ "columns": [
+ {
+ "name": "Pushes",
+ "unit": "short",
+ "expr": "sum(count_over_time($PUSH [$RANGE]))"
+ },
+ {
+ "name": "p50",
+ "unit": "s",
+ "expr": "quantile_over_time(0.5, $PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by ()"
+ },
+ {
+ "name": "p90",
+ "unit": "s",
+ "expr": "quantile_over_time(0.9, $PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by ()"
+ },
+ {
+ "name": "Under 5 min",
+ "unit": "percentunit",
+ "expr": "(sum(count_over_time($PUSH | wall_seconds <= 300 [$RANGE])) or 0 * sum(count_over_time($PUSH [$RANGE]))) / sum(count_over_time($PUSH [$RANGE]))"
+ },
+ {
+ "name": "Under 3 min",
+ "unit": "percentunit",
+ "expr": "(sum(count_over_time($PUSH | wall_seconds <= 180 [$RANGE])) or 0 * sum(count_over_time($PUSH [$RANGE]))) / sum(count_over_time($PUSH [$RANGE]))"
+ },
+ {
+ "name": "PR CI hours",
+ "unit": "h",
+ "expr": "sum(sum_over_time($PR_JOB | unwrap run_seconds | __error__=\"\" [$RANGE])) / 3600"
+ }
+ ]
+ },
+ {
+ "id": "wall",
+ "title": "Wall time by PR type",
+ "description": "Real PR pushes as above, by what the PR changed. Rust size counts changed crates: small is up to 2, medium up to 9, large more or a root manifest.",
+ "by": [
+ "pr_type"
+ ],
+ "sort": "p90",
+ "columns": [
+ {
+ "name": "Pushes",
+ "unit": "short",
+ "expr": "sum by (pr_type) (count_over_time($PUSH [$RANGE]))"
+ },
+ {
+ "name": "p50",
+ "unit": "s",
+ "expr": "quantile_over_time(0.5, $PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by (pr_type)"
+ },
+ {
+ "name": "p90",
+ "unit": "s",
+ "expr": "quantile_over_time(0.9, $PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by (pr_type)"
+ },
+ {
+ "name": "Max",
+ "unit": "s",
+ "expr": "max_over_time($PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by (pr_type)"
+ },
+ {
+ "name": "Under 5 min",
+ "unit": "percentunit",
+ "expr": "(sum by (pr_type) (count_over_time($PUSH | wall_seconds <= 300 [$RANGE])) or 0 * sum by (pr_type) (count_over_time($PUSH [$RANGE]))) / sum by (pr_type) (count_over_time($PUSH [$RANGE]))"
+ }
+ ]
+ },
+ {
+ "id": "last",
+ "title": "What finished last on real pushes",
+ "description": "The job each push waited for, leaving out roll-up status checks. Its p50 run time is the job's own; queue and earlier jobs come on top.",
+ "by": [
+ "last_workflow",
+ "last_job"
+ ],
+ "sort": "Pushes",
+ "limit": 15,
+ "columns": [
+ {
+ "name": "Pushes",
+ "unit": "short",
+ "expr": "sum by (last_workflow, last_job) (count_over_time($PUSH [$RANGE]))"
+ },
+ {
+ "name": "Push p50",
+ "unit": "s",
+ "expr": "quantile_over_time(0.5, $PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by (last_workflow, last_job)"
+ },
+ {
+ "name": "Job p50",
+ "unit": "s",
+ "expr": "quantile_over_time(0.5, $PUSH | unwrap last_job_seconds | __error__=\"\" [$RANGE]) by (last_workflow, last_job)"
+ }
+ ]
+ },
+ {
+ "id": "jobs",
+ "title": "PR jobs by CI time",
+ "description": "Run time of each job on PRs, experiments left out, excluding queue time. Sorted by total hours, so a rare slow job doesn't crowd out a common one.",
+ "by": [
+ "workflow",
+ "job"
+ ],
+ "sort": "Hours",
+ "limit": 20,
+ "columns": [
+ {
+ "name": "Runs",
+ "unit": "short",
+ "expr": "sum by (workflow, job) (count_over_time($PR_JOB [$RANGE]))"
+ },
+ {
+ "name": "p50",
+ "unit": "s",
+ "expr": "quantile_over_time(0.5, $PR_JOB | unwrap run_seconds | __error__=\"\" [$RANGE]) by (workflow, job)"
+ },
+ {
+ "name": "p90",
+ "unit": "s",
+ "expr": "quantile_over_time(0.9, $PR_JOB | unwrap run_seconds | __error__=\"\" [$RANGE]) by (workflow, job)"
+ },
+ {
+ "name": "Hours",
+ "unit": "h",
+ "expr": "sum by (workflow, job) (sum_over_time($PR_JOB | unwrap run_seconds | __error__=\"\" [$RANGE])) / 3600"
+ }
+ ]
+ },
+ {
+ "id": "steps",
+ "title": "PR steps by CI time",
+ "description": "Steps of a second or more in PR jobs, sorted by total hours. Open a run's trace from Tempo below for the full waterfall.",
+ "by": [
+ "workflow",
+ "job",
+ "step"
+ ],
+ "sort": "Hours",
+ "limit": 20,
+ "columns": [
+ {
+ "name": "Runs",
+ "unit": "short",
+ "expr": "sum by (workflow, job, step) (count_over_time($PR_STEP [$RANGE]))"
+ },
+ {
+ "name": "p50",
+ "unit": "s",
+ "expr": "quantile_over_time(0.5, $PR_STEP | unwrap run_seconds | __error__=\"\" [$RANGE]) by (workflow, job, step)"
+ },
+ {
+ "name": "p90",
+ "unit": "s",
+ "expr": "quantile_over_time(0.9, $PR_STEP | unwrap run_seconds | __error__=\"\" [$RANGE]) by (workflow, job, step)"
+ },
+ {
+ "name": "Hours",
+ "unit": "h",
+ "expr": "sum by (workflow, job, step) (sum_over_time($PR_STEP | unwrap run_seconds | __error__=\"\" [$RANGE])) / 3600"
+ }
+ ]
+ },
+ {
+ "id": "queue_runner",
+ "title": "Queue by runner",
+ "description": "From a job's creation to a runner picking it up, on PRs. Copied jobs of a partial rerun have no queue time.",
+ "by": [
+ "runner_profile"
+ ],
+ "sort": "p90",
+ "columns": [
+ {
+ "name": "Jobs",
+ "unit": "short",
+ "expr": "sum by (runner_profile) (count_over_time($PR_JOB | queue_seconds != \"\" [$RANGE]))"
+ },
+ {
+ "name": "p50",
+ "unit": "s",
+ "expr": "quantile_over_time(0.5, $PR_JOB | unwrap queue_seconds | __error__=\"\" [$RANGE]) by (runner_profile)"
+ },
+ {
+ "name": "p90",
+ "unit": "s",
+ "expr": "quantile_over_time(0.9, $PR_JOB | unwrap queue_seconds | __error__=\"\" [$RANGE]) by (runner_profile)"
+ },
+ {
+ "name": "Max",
+ "unit": "s",
+ "expr": "max_over_time($PR_JOB | unwrap queue_seconds | __error__=\"\" [$RANGE]) by (runner_profile)"
+ }
+ ]
+ },
+ {
+ "id": "queue_job",
+ "title": "Longest-queued PR jobs",
+ "description": "Queue time per job, on PRs.",
+ "by": [
+ "workflow",
+ "job"
+ ],
+ "sort": "p90",
+ "limit": 15,
+ "columns": [
+ {
+ "name": "p50",
+ "unit": "s",
+ "expr": "quantile_over_time(0.5, $PR_JOB | unwrap queue_seconds | __error__=\"\" [$RANGE]) by (workflow, job)"
+ },
+ {
+ "name": "p90",
+ "unit": "s",
+ "expr": "quantile_over_time(0.9, $PR_JOB | unwrap queue_seconds | __error__=\"\" [$RANGE]) by (workflow, job)"
+ },
+ {
+ "name": "Max",
+ "unit": "s",
+ "expr": "max_over_time($PR_JOB | unwrap queue_seconds | __error__=\"\" [$RANGE]) by (workflow, job)"
+ }
+ ]
+ },
+ {
+ "id": "volumes",
+ "title": "Warm and cold cache volumes",
+ "description": "Jobs that set up the Nix dev shell, on PRs. Cold means the step took 30 s or more, which an empty cache volume causes.",
+ "by": [
+ "runner_profile",
+ "volume_state"
+ ],
+ "sort": "Jobs",
+ "columns": [
+ {
+ "name": "Jobs",
+ "unit": "short",
+ "expr": "sum by (runner_profile, volume_state) (count_over_time($PR_JOB | volume_state != \"\" [$RANGE]))"
+ },
+ {
+ "name": "Dev shell p50",
+ "unit": "s",
+ "expr": "quantile_over_time(0.5, $PR_JOB | unwrap dev_shell_seconds | __error__=\"\" [$RANGE]) by (runner_profile, volume_state)"
+ },
+ {
+ "name": "Dev shell p90",
+ "unit": "s",
+ "expr": "quantile_over_time(0.9, $PR_JOB | unwrap dev_shell_seconds | __error__=\"\" [$RANGE]) by (runner_profile, volume_state)"
+ }
+ ]
+ },
+ {
+ "id": "hours",
+ "title": "CI hours by trigger",
+ "description": "Run time summed over every observed job, queue excluded. Experiment branches are counted separately.",
+ "by": [
+ "event_name",
+ "experiment"
+ ],
+ "sort": "Hours",
+ "columns": [
+ {
+ "name": "Jobs",
+ "unit": "short",
+ "expr": "sum by (event_name, experiment) (count_over_time($JOB [$RANGE]))"
+ },
+ {
+ "name": "Hours",
+ "unit": "h",
+ "expr": "sum by (event_name, experiment) (sum_over_time($JOB | unwrap run_seconds | __error__=\"\" [$RANGE])) / 3600"
+ }
+ ]
+ },
+ {
+ "id": "exporter",
+ "title": "Exporter cost",
+ "description": "The exporter's own collection time and GitHub API use per completed run. The token's limit is shared with the trace export.",
+ "by": [],
+ "columns": [
+ {
+ "name": "Exports",
+ "unit": "short",
+ "expr": "sum(count_over_time($EXPORT [$RANGE]))"
+ },
+ {
+ "name": "Collect p50",
+ "unit": "s",
+ "expr": "quantile_over_time(0.5, $EXPORT | unwrap collect_seconds | __error__=\"\" [$RANGE]) by ()"
+ },
+ {
+ "name": "Collect max",
+ "unit": "s",
+ "expr": "max_over_time($EXPORT | unwrap collect_seconds | __error__=\"\" [$RANGE]) by ()"
+ },
+ {
+ "name": "API calls",
+ "unit": "short",
+ "expr": "sum(sum_over_time($EXPORT | unwrap api_calls | __error__=\"\" [$RANGE]))"
+ },
+ {
+ "name": "Lowest rate remaining",
+ "unit": "short",
+ "expr": "min_over_time($EXPORT | unwrap rate_remaining | __error__=\"\" [$RANGE]) by ()"
+ }
+ ]
+ }
+ ]
+}
diff --git a/infra/stacks/observability/scripts/ci-report.test.ts b/infra/stacks/observability/scripts/ci-report.test.ts
new file mode 100644
index 0000000000..d77298e332
--- /dev/null
+++ b/infra/stacks/observability/scripts/ci-report.test.ts
@@ -0,0 +1,116 @@
+import { describe, expect, test } from 'bun:test';
+import { readFileSync } from 'node:fs';
+import {
+ expand,
+ formatValue,
+ joinColumns,
+ parseRange,
+ renderTable,
+ type Spec,
+ type Table,
+} from './ci-report';
+
+const spec: Spec = JSON.parse(
+ readFileSync(new URL('../nixos/ci-report.json', import.meta.url), 'utf8')
+);
+
+const table: Table = {
+ id: 'jobs',
+ title: 'Jobs',
+ description: '',
+ by: ['workflow', 'job'],
+ sort: 'p90',
+ limit: 2,
+ columns: [
+ { name: 'Runs', unit: 'short', expr: '' },
+ { name: 'p90', unit: 's', expr: '' },
+ ],
+};
+
+describe('the shared spec', () => {
+ test('expands every query to LogQL with no placeholder left', () => {
+ for (const entry of spec.tables) {
+ for (const column of entry.columns) {
+ const query = expand(column.expr, spec.filters, '1d');
+ expect(query).not.toContain('$');
+ expect(query).toContain('{service_name="ci-telemetry"}');
+ }
+ }
+ });
+
+ test('groups every column by its table labels, so rows join', () => {
+ for (const entry of spec.tables) {
+ for (const column of entry.columns) {
+ const grouping = entry.by.length
+ ? `by (${entry.by.join(', ')})`
+ : /by \(\)|^\(?sum\(/;
+ if (typeof grouping === 'string')
+ expect(column.expr).toContain(grouping);
+ else expect(column.expr).toMatch(grouping);
+ }
+ }
+ });
+
+ test('sorts by a column it has, and has unique ids', () => {
+ for (const entry of spec.tables) {
+ if (entry.sort)
+ expect(entry.columns.map((column) => column.name)).toContain(
+ entry.sort
+ );
+ }
+ const ids = spec.tables.map((entry) => entry.id);
+ expect(new Set(ids).size).toBe(ids.length);
+ expect(ids).toContain('summary');
+ });
+});
+
+test('expand replaces the range and longer names before their prefixes', () => {
+ expect(
+ expand('$PUSH_X $PUSH [$RANGE]', { PUSH: 'a', PUSH_X: 'b' }, '1d')
+ ).toBe('b a [1d]');
+});
+
+test('joinColumns joins on labels, sorts, limits and keeps gaps', () => {
+ const rows = joinColumns(table, [
+ [
+ { metric: { workflow: 'W', job: 'a' }, value: 3 },
+ { metric: { workflow: 'W', job: 'b' }, value: 5 },
+ { metric: { workflow: 'W', job: 'c' }, value: 1 },
+ ],
+ [
+ { metric: { workflow: 'W', job: 'b' }, value: 400 },
+ { metric: { workflow: 'W', job: 'c' }, value: 90 },
+ ],
+ ]);
+ expect(rows).toEqual([
+ { labels: ['W', 'b'], values: [5, 400] },
+ { labels: ['W', 'c'], values: [1, 90] },
+ ]);
+ expect(
+ joinColumns({ ...table, limit: undefined }, [
+ [{ metric: { workflow: 'W', job: 'a' }, value: 3 }],
+ [],
+ ])
+ ).toEqual([{ labels: ['W', 'a'], values: [3, null] }]);
+});
+
+test('formatValue writes durations, hours and shares', () => {
+ expect(formatValue(42.4, 's')).toBe('42s');
+ expect(formatValue(387, 's')).toBe('6m 27s');
+ expect(formatValue(12.34, 'h')).toBe('12.3 h');
+ expect(formatValue(0.826, 'percentunit')).toBe('83%');
+ expect(formatValue(null, 's')).toBe('');
+});
+
+test('renderTable writes a markdown table with an empty row when there is no data', () => {
+ expect(renderTable(table, [{ labels: ['W', ''], values: [2, 61] }])).toBe(
+ '| workflow | job | Runs | p90 |\n| --- | --- | ---: | ---: |\n| W | (none) | 2 | 1m 01s |'
+ );
+ expect(renderTable(table, []).split('\n')).toHaveLength(3);
+});
+
+test('parseRange reads Loki durations', () => {
+ expect(parseRange('24h')).toBe(86_400);
+ expect(parseRange('7d')).toBe(604_800);
+ expect(() => parseRange('1w')).toThrow();
+});
diff --git a/infra/stacks/observability/scripts/ci-report.ts b/infra/stacks/observability/scripts/ci-report.ts
new file mode 100644
index 0000000000..05e1b6eaa4
--- /dev/null
+++ b/infra/stacks/observability/scripts/ci-report.ts
@@ -0,0 +1,243 @@
+// Daily CI report from the exporter's Loki events: wall time by PR type, what held
+// pushes up, slowest jobs and steps, queues, cache volumes and CI hours. The Macro CI
+// dashboard reads the same queries from nixos/ci-report.json.
+//
+// GRAFANA_TOKEN=$(cat <token file>) bun infra/stacks/observability/scripts/ci-report.ts [--range 24h] [--days 7]
+// Pass --loki <url> to query another Loki API root, such as a local one.
+
+import { readFileSync } from 'node:fs';
+import { fileURLToPath } from 'node:url';
+
+const HOSTED_LOKI =
+ 'https://grafana.macro-internal.com/api/datasources/proxy/uid/loki';
+const TIMEOUT_MILLISECONDS = 30_000;
+const DAY_SECONDS = 86_400;
+
+export type Column = { name: string; unit: string; expr: string };
+export type Table = {
+ id: string;
+ title: string;
+ description: string;
+ by: string[];
+ sort?: string;
+ limit?: number;
+ columns: Column[];
+};
+export type Spec = { filters: Record<string, string>; tables: Table[] };
+export type Sample = { metric: Record<string, string>; value: number };
+export type Row = { labels: string[]; values: (number | null)[] };
+
+/** Replace `$NAME` with its filter and `$RANGE` with the window, as the dashboard does. */
+export function expand(
+ expr: string,
+ filters: Record<string, string>,
+ range: string
+): string {
+ let result = expr.replaceAll('$RANGE', range);
+ // Longest names first, so a name that prefixes another can't replace part of it.
+ for (const name of Object.keys(filters).sort((a, b) => b.length - a.length)) {
+ result = result.replaceAll(`$${name}`, filters[name]);
+ }
+ return result;
+}
+
+/** One row per label combination, with a value per column; missing values stay null. */
+export function joinColumns(table: Table, results: Sample[][]): Row[] {
+ const rows = new Map<string, Row>();
+ for (const [index, samples] of results.entries()) {
+ for (const sample of samples) {
+ const labels = table.by.map((label) => sample.metric[label] ?? '');
+ const key = JSON.stringify(labels);
+ let row = rows.get(key);
+ if (!row) {
+ row = { labels, values: table.columns.map(() => null) };
+ rows.set(key, row);
+ }
+ row.values[index] = sample.value;
+ }
+ }
+ const sortIndex = table.columns.findIndex(
+ (column) => column.name === table.sort
+ );
+ const ordered = [...rows.values()];
+ if (sortIndex >= 0) {
+ ordered.sort(
+ (a, b) => (b.values[sortIndex] ?? -1) - (a.values[sortIndex] ?? -1)
+ );
+ }
+ return table.limit ? ordered.slice(0, table.limit) : ordered;
+}
+
+export function formatValue(value: number | null, unit: string): string {
+ if (value === null || Number.isNaN(value)) return '';
+ if (unit === 's') {
+ const total = Math.round(value);
+ return total < 60
+ ? `${total}s`
+ : `${Math.floor(total / 60)}m ${String(total % 60).padStart(2, '0')}s`;
+ }
+ if (unit === 'h') return `${value.toFixed(1)} h`;
+ if (unit === 'percentunit') return `${Math.round(value * 100)}%`;
+ return String(Math.round(value));
+}
+
+export function renderTable(
+ table: Table,
+ rows: Row[],
+ labelHeaders: string[] = table.by
+): string {
+ const header = [
+ ...labelHeaders,
+ ...table.columns.map((column) => column.name),
+ ];
+ const lines = [
+ `| ${header.join(' | ')} |`,
+ `| ${header.map((_, index) => (index < labelHeaders.length ? '---' : '---:')).join(' | ')} |`,
+ ];
+ for (const row of rows) {
+ const cells = [
+ ...row.labels.map((label) => label || '(none)'),
+ ...row.values.map((value, index) =>
+ formatValue(value, table.columns[index].unit)
+ ),
+ ];
+ lines.push(`| ${cells.join(' | ')} |`);
+ }
+ if (rows.length === 0) lines.push(`| ${header.map(() => '').join(' | ')} |`);
+ return lines.join('\n');
+}
+
+/** A Loki duration such as `24h` or `7d`, in seconds. */
+export function parseRange(range: string): number {
+ const match = /^(\d+)([mhd])$/.exec(range);
+ if (!match)
+ throw new Error(`Expected a range such as 24h or 7d, got ${range}`);
+ return (
+ Number(match[1]) *
+ { m: 60, h: 3600, d: DAY_SECONDS }[match[2] as 'm' | 'h' | 'd']
+ );
+}
+
+type LokiVector = {
+ status: string;
+ data: {
+ resultType: string;
+ result: { metric: Record<string, string>; value: [number, string] }[];
+ };
+};
+
+class Loki {
+ base: string;
+ token: string | undefined;
+
+ constructor(base: string, token: string | undefined) {
+ this.base = base;
+ this.token = token;
+ }
+
+ async instant(query: string, time: number): Promise<Sample[]> {
+ const url = `${this.base}/loki/api/v1/query?${new URLSearchParams({ query, time: String(time) })}`;
+ const response = await fetch(url, {
+ headers: this.token ? { Authorization: `Bearer ${this.token}` } : {},
+ signal: AbortSignal.timeout(TIMEOUT_MILLISECONDS),
+ });
+ if (!response.ok) {
+ // The body names the query problem; it never echoes the token.
+ throw new Error(
+ `Loki answered HTTP ${response.status}: ${(await response.text()).slice(0, 300)}`
+ );
+ }
+ const body = (await response.json()) as LokiVector;
+ if (body.data.resultType !== 'vector') return [];
+ return body.data.result.map((entry) => ({
+ metric: entry.metric,
+ value: Number(entry.value[1]),
+ }));
+ }
+
+ async table(
+ spec: Spec,
+ table: Table,
+ range: string,
+ time: number
+ ): Promise<Row[]> {
+ const results = await Promise.all(
+ table.columns.map((column) =>
+ this.instant(expand(column.expr, spec.filters, range), time)
+ )
+ );
+ return joinColumns(table, results);
+ }
+}
+
+function argument(args: string[], name: string, fallback: string): string {
+ const index = args.indexOf(name);
+ return index >= 0 && args[index + 1] ? args[index + 1] : fallback;
+}
+
+function utc(seconds: number): string {
+ return new Date(seconds * 1000).toISOString().slice(0, 16).replace('T', ' ');
+}
+
+export async function main(args: string[]) {
+ const spec: Spec = JSON.parse(
+ readFileSync(new URL('../nixos/ci-report.json', import.meta.url), 'utf8')
+ );
+ const range = argument(args, '--range', '24h');
+ const days = Number(argument(args, '--days', '7'));
+ const base = argument(args, '--loki', HOSTED_LOKI);
+ const token = process.env.GRAFANA_TOKEN;
+ if (base === HOSTED_LOKI && !token) {
+ throw new Error(
+ 'Set GRAFANA_TOKEN to a Grafana token that can read the Loki datasource'
+ );
+ }
+ const loki = new Loki(base, token);
+ const now = Math.floor(Date.now() / 1000);
+ const output = [
+ `# CI report, ${utc(now - parseRange(range))} to ${utc(now)} UTC`,
+ ];
+ for (const table of spec.tables) {
+ output.push(`\n## ${table.title}\n\n${table.description}\n`);
+ try {
+ output.push(
+ renderTable(table, await loki.table(spec, table, range, now))
+ );
+ } catch (error) {
+ // One refused query, such as a series cap, shouldn't cost the rest of the report.
+ output.push(
+ `Not available: ${error instanceof Error ? error.message : String(error)}`
+ );
+ }
+ }
+ const summary = spec.tables.find((table) => table.id === 'summary');
+ if (summary && days > 0) {
+ const today = Math.floor(now / DAY_SECONDS) * DAY_SECONDS;
+ const rows: Row[] = [];
+ for (let day = days - 1; day >= 0; day -= 1) {
+ const start = today - day * DAY_SECONDS;
+ const end = Math.min(start + DAY_SECONDS, now);
+ // A day Loki refuses stays blank rather than costing the tables above.
+ const [row] = await loki
+ .table(spec, summary, `${end - start}s`, end)
+ .catch(() => []);
+ rows.push({
+ labels: [
+ utc(start).slice(0, 10) +
+ (end < start + DAY_SECONDS ? ' (so far)' : ''),
+ ],
+ values: row?.values ?? summary.columns.map(() => null),
+ });
+ }
+ output.push(`\n## Per UTC day\n`);
+ output.push(renderTable(summary, rows, ['Day']));
+ }
+ console.log(output.join('\n'));
+}
+
+if (process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]) {
+ main(process.argv.slice(2)).catch((error: unknown) => {
+ console.error(error instanceof Error ? error.message : String(error));
+ process.exit(1);
+ });
+}