Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.
TypeScript agentaiai-agentsall-in-onecrmcrm-systememaillinearmcpmessagingnotesnotionofficerustslackslack-alternativestartupstartup-toolstypescriptworkspace
feat(observability): add CI percentile panels and a daily CI report (#9919)
teo0e2bbd3095e1
diff --git a/infra/package.json b/infra/package.jsonindex 811ea5f1ea..ab9e147463 100644--- a/infra/package.json+++ b/infra/package.json@@ -9,8 +9,9 @@"preinstall": "node -e \"if (process.env.npm_execpath && !process.env.npm_execpath.includes('bun')) { console.error('Please use bun as the package manager'); process.exit(1); }\"","lint": "biome check .","format": "biome check --write .",- "check": "tsc --noEmit && tsc --noEmit --skipLibCheck --project stacks/web-app/tsconfig.json && bun run test:web-delivery",- "test:web-delivery": "bun test stacks/web-app/publication.test.ts && bun test stacks/web-app/app-route.test.ts && bun test stacks/website/app-cache.test.ts"+ "check": "tsc --noEmit && tsc --noEmit --skipLibCheck --project stacks/web-app/tsconfig.json && bun run test:web-delivery && bun run test:observability",+ "test:web-delivery": "bun test stacks/web-app/publication.test.ts && bun test stacks/web-app/app-route.test.ts && bun test stacks/website/app-cache.test.ts",+ "test:observability": "bun test stacks/observability/scripts/ci-report.test.ts"},"devDependencies": {"@biomejs/biome": "2.4.12",diff --git a/infra/stacks/observability/AGENTS.md b/infra/stacks/observability/AGENTS.mdindex 6e9f5de4bb..32bc4eee07 100644--- a/infra/stacks/observability/AGENTS.md+++ b/infra/stacks/observability/AGENTS.md@@ -18,6 +18,8 @@ us-east-1. `README.md` is the full operations guide.(rules and the `Prod alerts` dashboard panel both read it), and its Datadogcopy in `infra/stacks/monitoring/monitors/` by hand. A new entry takes the nextfree `panelId`; never renumber one, since sent notifications link to it.+- Change a CI percentile, queue or CI hours query: edit `nixos/ci-report.json`;+ the `Macro CI` dashboard's Loki panels and `scripts/ci-report.ts` both read it.- Add a runtime setting: `settings.ts` → `render.ts` (bump the schema version) →`nixos/read-user-data.py` (keys, pattern, version) → the consuming `.nix` file.- Add a secret: `assets/refresh-secrets.py` (validation) → `LoadCredential` indiff --git a/infra/stacks/observability/README.md b/infra/stacks/observability/README.mdindex e6248e51fd..7183808b0d 100644--- a/infra/stacks/observability/README.md+++ b/infra/stacks/observability/README.md@@ -44,13 +44,14 @@ that starts after everything on its head has finished came from something else,as a PR edit, a label or leaving draft. It forms its own line with `rerun=true`,which also leaves a commit's first full run after draft out of the real PR numbers.-Real PR numbers exclude experiments, reruns and drafts. Metadata makes an ungrouped-query return one series per combination, so always group:+Real PR numbers exclude experiments, reruns and drafts, and pushes where only one+workflow ran: a lone CLA run means CI didn't run, as on a PR into another branch.+Metadata makes an ungrouped query return one series per combination, so always group:```logqlquantile_over_time(0.9, {service_name="ci-telemetry"} | event="push"| experiment="false" | rerun="false" | pr_draft="false" | conclusion="success"- | unwrap wall_seconds [1d]) by (pr_type)+ | workflows > 1 | unwrap wall_seconds [1d]) by (pr_type)```Lines never carry branch names, actors, titles or file paths. A line's time and@@ -62,12 +63,27 @@ shell step takes 30 s or more, an inference from step timing.Collection and sending are best effort and cannot fail the exporter, and neithertouches the observed PR's jobs. Collection runs before any credential is read.-`/d/macro-ci` provides workflow traces, failed/cancelled jobs and steps, and jobs+`nixos/ci-report.json` holds the queries over these events: real PR wall time+overall and by PR type, what finished last, the slowest PR jobs and steps, queue+by runner and job, warm and cold cache volumes, CI hours by trigger, and the+exporter's own cost. The top of `/d/macro-ci` renders them over the selected+range, and the daily report prints the same tables as Markdown, plus one summary+row per UTC day:++```sh+GRAFANA_TOKEN=$(cat <token file>) bun infra/stacks/observability/scripts/ci-report.ts --range 24h --days 7+```++The token needs only Viewer access. `--loki <url>` points it at another Loki API+root, such as a local one. Change a query in the JSON and both follow. PR steps+grouped by workflow, job and step pass Loki's default cap of 500 series in a day,+so `backend-config.nix` raises `max_query_series` to 2000.++Below them, `/d/macro-ci` provides workflow traces, failed/cancelled jobs and steps, and jobsrunning longer than five minutes. Open a trace for its execution waterfall andGitHub URL. Search tables are limited to 100 traces and 100 matching spans pertrace; narrow the time range to investigate further. These are bounded searches,-not complete counts or p95 summaries. The action does not emit our old structured-Loki events. Job durations exclude runner queue time. Workflow root duration+not complete counts or p95 summaries. Job durations exclude runner queue time. Workflow root durationincludes waiting and, on reruns, starts at the original run creation time becauseof an upstream limitation; do not treat it as attempt execution time.diff --git a/infra/stacks/observability/nixos/backend-config.nix b/infra/stacks/observability/nixos/backend-config.nixindex 68080576ec..dc8d72d857 100644--- a/infra/stacks/observability/nixos/backend-config.nix+++ b/infra/stacks/observability/nixos/backend-config.nix@@ -53,6 +53,8 @@ingestion_rate_mb = 8;ingestion_burst_size_mb = 16;max_query_parallelism = 4;+ # The CI dashboard groups PR steps by workflow, job and step: about 600 a day.+ max_query_series = 2000;};querier.max_concurrent = 4;analytics.reporting_enabled = false;diff --git a/infra/stacks/observability/nixos/ci-dashboard.nix b/infra/stacks/observability/nixos/ci-dashboard.nixindex b368c76e85..487f026844 100644--- a/infra/stacks/observability/nixos/ci-dashboard.nix+++ b/infra/stacks/observability/nixos/ci-dashboard.nix@@ -1,20 +1,250 @@-{ pkgs, ... }:+{ lib, pkgs, ... }:lettempo = {type = "tempo";uid = "tempo";};- selector = ''resource.service.name="github-actions" && resource.deployment.environment="ci"'';+ loki = {+ type = "loki";+ uid = "loki";+ };+ # Shared with scripts/ci-report.ts, so the daily report and these panels agree.+ report = builtins.fromJSON (builtins.readFile ./ci-report.json);+ # Longest names first, as the script does, so no name replaces part of another.+ filterNames = lib.sort (a: b: lib.stringLength a > lib.stringLength b) (+ builtins.attrNames report.filters+ );+ expand =+ range:+ builtins.replaceStrings ([ "$RANGE" ] ++ map (name: "$" + name) filterNames) (+ [ range ] ++ map (name: report.filters.${name}) filterNames+ );+ table =+ id: lib.findFirst (entry: entry.id == id) (throw "ci-report.json has no table ${id}") report.tables;+ refId = index: builtins.elemAt (lib.stringToCharacters "ABCDEFGHIJ") index;+ grid = x: y: w: h: {+ inherit+ x+ y+ w+ h+ ;+ };+ # Grafana's "h" unit rolls over into days; CI hours should stay hours.+ grafanaUnit = unit: if unit == "h" then "suffix: h" else unit;+ unitOverride = name: unit: {+ matcher = {+ id = "byName";+ options = name;+ };+ properties = [+ {+ id = "unit";+ value = grafanaUnit unit;+ }+ ];+ };+ # Green under three minutes, red from five: the CI target for a push's wall time.+ wallThresholds = {+ mode = "absolute";+ steps = [+ {+ color = "green";+ value = null;+ }+ {+ color = "orange";+ value = 180;+ }+ {+ color = "red";+ value = 300;+ }+ ];+ };+ # One instant query per column over the dashboard's range, merged on the labels.+ instantTargets =+ entry:+ lib.imap0 (index: column: {+ refId = refId index;+ datasource = loki;+ expr = expand "$__range" column.expr;+ queryType = "instant";+ legendFormat = column.name;+ }) entry.columns;++ lokiTable =+ id: tableId: gridPos:+ let+ entry = table tableId;+ in+ {+ inherit id gridPos;+ inherit (entry) title description;+ type = "table";+ datasource = loki;+ targets = instantTargets entry;+ transformations = [+ {+ id = "merge";+ options = { };+ }+ {+ id = "organize";+ options = {+ excludeByName.Time = true;+ renameByName = lib.listToAttrs (+ lib.imap0 (index: column: lib.nameValuePair "Value #${refId index}" column.name) entry.columns+ );+ indexByName = lib.listToAttrs (lib.imap0 (index: label: lib.nameValuePair label index) entry.by);+ };+ }+ ]+ ++ lib.optional (entry ? sort) {+ id = "sortBy";+ options.sort = [+ {+ field = entry.sort;+ desc = true;+ }+ ];+ }+ ++ lib.optional (entry ? limit) {+ id = "limit";+ options.limitField = entry.limit;+ };+ fieldConfig = {+ # The default minimum column width hides the last columns of a half-width panel.+ defaults.custom.minWidth = 60;+ overrides = map (column: unitOverride column.name column.unit) entry.columns;+ };+ options = {+ showHeader = true;+ cellHeight = "sm";+ };+ };++ lokiStat =+ id: tableId: gridPos:+ let+ entry = table tableId;+ in+ {+ inherit id gridPos;+ inherit (entry) title description;+ type = "stat";+ datasource = loki;+ targets = instantTargets entry;+ fieldConfig = {+ defaults = {+ color = {+ mode = "fixed";+ fixedColor = "text";+ };+ };+ # Instant results carry no series name, so match each query by its refId.+ overrides = lib.imap0 (index: column: {+ matcher = {+ id = "byFrameRefID";+ options = refId index;+ };+ properties = [+ {+ id = "displayName";+ value = column.name;+ }+ {+ id = "unit";+ value = grafanaUnit column.unit;+ }+ ]+ +++ lib.optionals+ (lib.elem column.name [+ "p50"+ "p90"+ ])+ [+ {+ id = "thresholds";+ value = wallThresholds;+ }+ {+ id = "color";+ value.mode = "thresholds";+ }+ ];+ }) entry.columns;+ };+ options = {+ reduceOptions = {+ calcs = [ "lastNotNull" ];+ fields = "";+ values = false;+ };+ colorMode = "value";+ graphMode = "none";+ textMode = "value_and_name";+ justifyMode = "center";+ # A narrow panel lists its values instead of squeezing them side by side.+ orientation = if gridPos.w < 12 then "horizontal" else "vertical";+ };+ };++ lokiSeries =+ id: gridPos: title: description: unit: interval: targets: extra:+ lib.recursiveUpdate {+ inherit+ id+ gridPos+ title+ description+ interval+ ;+ type = "timeseries";+ datasource = loki;+ maxDataPoints = 200;+ targets = lib.imap0 (+ index: target:+ target+ // {+ refId = refId index;+ datasource = loki;+ queryType = "range";+ }+ ) targets;+ fieldConfig.defaults = {+ inherit unit;+ custom = {+ spanNulls = true;+ showPoints = "never";+ };+ };+ options.legend = {+ displayMode = "list";+ placement = "bottom";+ };+ } extra;++ wallTarget = quantile: by: legendFormat: {+ inherit legendFormat;+ expr = expand "1d" "quantile_over_time(${quantile}, $PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by (${by})";+ };++ row = id: y: title: {+ inherit id title;+ type = "row";+ collapsed = false;+ gridPos = grid 0 y 24 1;+ panels = [ ];+ };++ tracesTop = 47;panel = id: title: description: filter: fields: {inherit id title description;type = "table";datasource = tempo;- gridPos = {- x = 0;- y = (id - 1) * 10;- w = 24;- h = 10;- };+ gridPos = grid 0 (tracesTop + (id - 1) * 10) 24 10;targets = [{refId = "A";@@ -52,24 +282,91 @@ let];options.showHeader = true;};+ selector = ''resource.service.name="github-actions" && resource.deployment.environment="ci"'';+dashboard = {uid = "macro-ci";title = "Macro CI";- description = "GitHub Actions workflow, job and step traces exported after completion. Select a trace to open its execution waterfall. Tables are bounded search results, not complete counts or percentile summaries. Narrow the time range when there are many runs.";+ description = "Where CI time goes. The top section reads the exporter's push, job and step events in Loki, kept 30 days, over the selected range; set it to a day or more for percentiles. The bottom section searches the run traces in Tempo, kept 7 days: open a trace for its execution waterfall. Trace tables are bounded search results, not complete counts.";schemaVersion = 39;- version = 2;+ version = 3;editable = false;tags = ["macro""ci"];timezone = "browser";- refresh = "1m";+ refresh = "5m";time = {from = "now-24h";to = "now";};panels = [+ (row 100 0 "Wall time and where it goes, from Loki events")+ (lokiStat 5 "summary" (grid 0 1 24 4))+ (lokiTable 6 "wall" (grid 0 5 12 8))+ (lokiSeries 7 (grid 12 5 12 8) "Real PR wall time, rolling 24 h"+ "Each point covers the 24 hours before it. Red line: the 5 minute target."+ "s"+ "15m"+ [+ (wallTarget "0.5" "" "p50")+ (wallTarget "0.9" "" "p90")+ ]+ {+ fieldConfig.defaults = {+ thresholds = {+ mode = "absolute";+ steps = [+ {+ color = "transparent";+ value = null;+ }+ {+ color = "red";+ value = 300;+ }+ ];+ };+ custom.thresholdsStyle.mode = "line";+ };+ }+ )+ (lokiSeries 8 (grid 0 13 12 8) "p90 wall time by PR type, rolling 24 h"+ "Real PR pushes only; a type with few pushes moves in steps."+ "s"+ "15m"+ [ (wallTarget "0.9" "pr_type" "{{pr_type}}") ]+ { }+ )+ (lokiTable 9 "last" (grid 12 13 12 8))+ (lokiTable 10 "jobs" (grid 0 21 12 10))+ (lokiTable 11 "steps" (grid 12 21 12 10))+ (lokiTable 12 "queue_runner" (grid 0 31 8 8))+ (lokiTable 13 "queue_job" (grid 8 31 8 8))+ (lokiTable 14 "volumes" (grid 16 31 8 8))+ (lokiSeries 15 (grid 0 39 12 8) "CI hours by trigger"+ "Job run time summed per interval, queue excluded. Compare days to measure a change such as running CI only on labelled PRs."+ "short"+ "1h"+ [+ {+ legendFormat = "{{event_name}}";+ expr = expand "$__interval" "sum by (event_name) (sum_over_time($JOB | unwrap run_seconds | __error__=\"\" [$RANGE])) / 3600";+ }+ ]+ {+ fieldConfig.defaults.custom = {+ axisLabel = "CI hours";+ drawStyle = "bars";+ fillOpacity = 80;+ stacking.mode = "normal";+ };+ }+ )+ (lokiTable 16 "hours" (grid 12 39 6 8))+ (lokiStat 17 "exporter" (grid 18 39 6 8))+ (row 101 (tracesTop - 1) "Run traces, from Tempo")(panel 1 "Workflow runs — open a trace for jobs and steps""Workflow duration includes waiting; for reruns the upstream exporter starts the root span at the original run creation. Use job durations for execution timing. The url attribute links to GitHub.""span.workflow_run_id != nil"diff --git a/infra/stacks/observability/nixos/ci-report.json b/infra/stacks/observability/nixos/ci-report.jsonnew file mode 100644index 0000000000..e65811b76e--- /dev/null+++ b/infra/stacks/observability/nixos/ci-report.json@@ -0,0 +1,322 @@+{+ "description": "Loki queries over the CI exporter's push, job, step and export events, read by both ci-dashboard.nix and scripts/ci-report.ts. $NAME expands to filters.NAME and $RANGE to the window.",+ "filters": {+ "PUSH": "{service_name=\"ci-telemetry\"} | event=\"push\" | experiment=\"false\" | rerun=\"false\" | pr_draft=\"false\" | conclusion=\"success\" | workflows > 1",+ "PR_JOB": "{service_name=\"ci-telemetry\"} | event=\"job\" | event_name=~\"pull_request|pull_request_target\" | experiment=\"false\"",+ "PR_STEP": "{service_name=\"ci-telemetry\"} | event=\"step\" | event_name=~\"pull_request|pull_request_target\" | experiment=\"false\"",+ "JOB": "{service_name=\"ci-telemetry\"} | event=\"job\"",+ "EXPORT": "{service_name=\"ci-telemetry\"} | event=\"export\""+ },+ "tables": [+ {+ "id": "summary",+ "title": "Real PR pushes: are we under 5 minutes?",+ "description": "Successful pushes on PRs that aren't drafts, experiments or reruns, where more than one workflow ran: a lone CLA run means CI didn't run, as on a PR into another branch. Wall time runs from the push, or opening the PR, to the last check finishing. PR CI hours sum every PR job's run time.",+ "by": [],+ "columns": [+ {+ "name": "Pushes",+ "unit": "short",+ "expr": "sum(count_over_time($PUSH [$RANGE]))"+ },+ {+ "name": "p50",+ "unit": "s",+ "expr": "quantile_over_time(0.5, $PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by ()"+ },+ {+ "name": "p90",+ "unit": "s",+ "expr": "quantile_over_time(0.9, $PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by ()"+ },+ {+ "name": "Under 5 min",+ "unit": "percentunit",+ "expr": "(sum(count_over_time($PUSH | wall_seconds <= 300 [$RANGE])) or 0 * sum(count_over_time($PUSH [$RANGE]))) / sum(count_over_time($PUSH [$RANGE]))"+ },+ {+ "name": "Under 3 min",+ "unit": "percentunit",+ "expr": "(sum(count_over_time($PUSH | wall_seconds <= 180 [$RANGE])) or 0 * sum(count_over_time($PUSH [$RANGE]))) / sum(count_over_time($PUSH [$RANGE]))"+ },+ {+ "name": "PR CI hours",+ "unit": "h",+ "expr": "sum(sum_over_time($PR_JOB | unwrap run_seconds | __error__=\"\" [$RANGE])) / 3600"+ }+ ]+ },+ {+ "id": "wall",+ "title": "Wall time by PR type",+ "description": "Real PR pushes as above, by what the PR changed. Rust size counts changed crates: small is up to 2, medium up to 9, large more or a root manifest.",+ "by": [+ "pr_type"+ ],+ "sort": "p90",+ "columns": [+ {+ "name": "Pushes",+ "unit": "short",+ "expr": "sum by (pr_type) (count_over_time($PUSH [$RANGE]))"+ },+ {+ "name": "p50",+ "unit": "s",+ "expr": "quantile_over_time(0.5, $PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by (pr_type)"+ },+ {+ "name": "p90",+ "unit": "s",+ "expr": "quantile_over_time(0.9, $PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by (pr_type)"+ },+ {+ "name": "Max",+ "unit": "s",+ "expr": "max_over_time($PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by (pr_type)"+ },+ {+ "name": "Under 5 min",+ "unit": "percentunit",+ "expr": "(sum by (pr_type) (count_over_time($PUSH | wall_seconds <= 300 [$RANGE])) or 0 * sum by (pr_type) (count_over_time($PUSH [$RANGE]))) / sum by (pr_type) (count_over_time($PUSH [$RANGE]))"+ }+ ]+ },+ {+ "id": "last",+ "title": "What finished last on real pushes",+ "description": "The job each push waited for, leaving out roll-up status checks. Its p50 run time is the job's own; queue and earlier jobs come on top.",+ "by": [+ "last_workflow",+ "last_job"+ ],+ "sort": "Pushes",+ "limit": 15,+ "columns": [+ {+ "name": "Pushes",+ "unit": "short",+ "expr": "sum by (last_workflow, last_job) (count_over_time($PUSH [$RANGE]))"+ },+ {+ "name": "Push p50",+ "unit": "s",+ "expr": "quantile_over_time(0.5, $PUSH | unwrap wall_seconds | __error__=\"\" [$RANGE]) by (last_workflow, last_job)"+ },+ {+ "name": "Job p50",+ "unit": "s",+ "expr": "quantile_over_time(0.5, $PUSH | unwrap last_job_seconds | __error__=\"\" [$RANGE]) by (last_workflow, last_job)"+ }+ ]+ },+ {+ "id": "jobs",+ "title": "PR jobs by CI time",+ "description": "Run time of each job on PRs, experiments left out, excluding queue time. Sorted by total hours, so a rare slow job doesn't crowd out a common one.",+ "by": [+ "workflow",+ "job"+ ],+ "sort": "Hours",+ "limit": 20,+ "columns": [+ {+ "name": "Runs",+ "unit": "short",+ "expr": "sum by (workflow, job) (count_over_time($PR_JOB [$RANGE]))"+ },+ {+ "name": "p50",+ "unit": "s",+ "expr": "quantile_over_time(0.5, $PR_JOB | unwrap run_seconds | __error__=\"\" [$RANGE]) by (workflow, job)"+ },+ {+ "name": "p90",+ "unit": "s",+ "expr": "quantile_over_time(0.9, $PR_JOB | unwrap run_seconds | __error__=\"\" [$RANGE]) by (workflow, job)"+ },+ {+ "name": "Hours",+ "unit": "h",+ "expr": "sum by (workflow, job) (sum_over_time($PR_JOB | unwrap run_seconds | __error__=\"\" [$RANGE])) / 3600"+ }+ ]+ },+ {+ "id": "steps",+ "title": "PR steps by CI time",+ "description": "Steps of a second or more in PR jobs, sorted by total hours. Open a run's trace from Tempo below for the full waterfall.",+ "by": [+ "workflow",+ "job",+ "step"+ ],+ "sort": "Hours",+ "limit": 20,+ "columns": [+ {+ "name": "Runs",+ "unit": "short",+ "expr": "sum by (workflow, job, step) (count_over_time($PR_STEP [$RANGE]))"+ },+ {+ "name": "p50",+ "unit": "s",+ "expr": "quantile_over_time(0.5, $PR_STEP | unwrap run_seconds | __error__=\"\" [$RANGE]) by (workflow, job, step)"+ },+ {+ "name": "p90",+ "unit": "s",+ "expr": "quantile_over_time(0.9, $PR_STEP | unwrap run_seconds | __error__=\"\" [$RANGE]) by (workflow, job, step)"+ },+ {+ "name": "Hours",+ "unit": "h",+ "expr": "sum by (workflow, job, step) (sum_over_time($PR_STEP | unwrap run_seconds | __error__=\"\" [$RANGE])) / 3600"+ }+ ]+ },+ {+ "id": "queue_runner",+ "title": "Queue by runner",+ "description": "From a job's creation to a runner picking it up, on PRs. Copied jobs of a partial rerun have no queue time.",+ "by": [+ "runner_profile"+ ],+ "sort": "p90",+ "columns": [+ {+ "name": "Jobs",+ "unit": "short",+ "expr": "sum by (runner_profile) (count_over_time($PR_JOB | queue_seconds != \"\" [$RANGE]))"+ },+ {+ "name": "p50",+ "unit": "s",+ "expr": "quantile_over_time(0.5, $PR_JOB | unwrap queue_seconds | __error__=\"\" [$RANGE]) by (runner_profile)"+ },+ {+ "name": "p90",+ "unit": "s",+ "expr": "quantile_over_time(0.9, $PR_JOB | unwrap queue_seconds | __error__=\"\" [$RANGE]) by (runner_profile)"+ },+ {+ "name": "Max",+ "unit": "s",+ "expr": "max_over_time($PR_JOB | unwrap queue_seconds | __error__=\"\" [$RANGE]) by (runner_profile)"+ }+ ]+ },+ {+ "id": "queue_job",+ "title": "Longest-queued PR jobs",+ "description": "Queue time per job, on PRs.",+ "by": [+ "workflow",+ "job"+ ],+ "sort": "p90",+ "limit": 15,+ "columns": [+ {+ "name": "p50",+ "unit": "s",+ "expr": "quantile_over_time(0.5, $PR_JOB | unwrap queue_seconds | __error__=\"\" [$RANGE]) by (workflow, job)"+ },+ {+ "name": "p90",+ "unit": "s",+ "expr": "quantile_over_time(0.9, $PR_JOB | unwrap queue_seconds | __error__=\"\" [$RANGE]) by (workflow, job)"+ },+ {+ "name": "Max",+ "unit": "s",+ "expr": "max_over_time($PR_JOB | unwrap queue_seconds | __error__=\"\" [$RANGE]) by (workflow, job)"+ }+ ]+ },+ {+ "id": "volumes",+ "title": "Warm and cold cache volumes",+ "description": "Jobs that set up the Nix dev shell, on PRs. Cold means the step took 30 s or more, which an empty cache volume causes.",+ "by": [+ "runner_profile",+ "volume_state"+ ],+ "sort": "Jobs",+ "columns": [+ {+ "name": "Jobs",+ "unit": "short",+ "expr": "sum by (runner_profile, volume_state) (count_over_time($PR_JOB | volume_state != \"\" [$RANGE]))"+ },+ {+ "name": "Dev shell p50",+ "unit": "s",+ "expr": "quantile_over_time(0.5, $PR_JOB | unwrap dev_shell_seconds | __error__=\"\" [$RANGE]) by (runner_profile, volume_state)"+ },+ {+ "name": "Dev shell p90",+ "unit": "s",+ "expr": "quantile_over_time(0.9, $PR_JOB | unwrap dev_shell_seconds | __error__=\"\" [$RANGE]) by (runner_profile, volume_state)"+ }+ ]+ },+ {+ "id": "hours",+ "title": "CI hours by trigger",+ "description": "Run time summed over every observed job, queue excluded. Experiment branches are counted separately.",+ "by": [+ "event_name",+ "experiment"+ ],+ "sort": "Hours",+ "columns": [+ {+ "name": "Jobs",+ "unit": "short",+ "expr": "sum by (event_name, experiment) (count_over_time($JOB [$RANGE]))"+ },+ {+ "name": "Hours",+ "unit": "h",+ "expr": "sum by (event_name, experiment) (sum_over_time($JOB | unwrap run_seconds | __error__=\"\" [$RANGE])) / 3600"+ }+ ]+ },+ {+ "id": "exporter",+ "title": "Exporter cost",+ "description": "The exporter's own collection time and GitHub API use per completed run. The token's limit is shared with the trace export.",+ "by": [],+ "columns": [+ {+ "name": "Exports",+ "unit": "short",+ "expr": "sum(count_over_time($EXPORT [$RANGE]))"+ },+ {+ "name": "Collect p50",+ "unit": "s",+ "expr": "quantile_over_time(0.5, $EXPORT | unwrap collect_seconds | __error__=\"\" [$RANGE]) by ()"+ },+ {+ "name": "Collect max",+ "unit": "s",+ "expr": "max_over_time($EXPORT | unwrap collect_seconds | __error__=\"\" [$RANGE]) by ()"+ },+ {+ "name": "API calls",+ "unit": "short",+ "expr": "sum(sum_over_time($EXPORT | unwrap api_calls | __error__=\"\" [$RANGE]))"+ },+ {+ "name": "Lowest rate remaining",+ "unit": "short",+ "expr": "min_over_time($EXPORT | unwrap rate_remaining | __error__=\"\" [$RANGE]) by ()"+ }+ ]+ }+ ]+}diff --git a/infra/stacks/observability/scripts/ci-report.test.ts b/infra/stacks/observability/scripts/ci-report.test.tsnew file mode 100644index 0000000000..d77298e332--- /dev/null+++ b/infra/stacks/observability/scripts/ci-report.test.ts@@ -0,0 +1,116 @@+import { describe, expect, test } from 'bun:test';+import { readFileSync } from 'node:fs';+import {+ expand,+ formatValue,+ joinColumns,+ parseRange,+ renderTable,+ type Spec,+ type Table,+} from './ci-report';++const spec: Spec = JSON.parse(+ readFileSync(new URL('../nixos/ci-report.json', import.meta.url), 'utf8')+);++const table: Table = {+ id: 'jobs',+ title: 'Jobs',+ description: '',+ by: ['workflow', 'job'],+ sort: 'p90',+ limit: 2,+ columns: [+ { name: 'Runs', unit: 'short', expr: '' },+ { name: 'p90', unit: 's', expr: '' },+ ],+};++describe('the shared spec', () => {+ test('expands every query to LogQL with no placeholder left', () => {+ for (const entry of spec.tables) {+ for (const column of entry.columns) {+ const query = expand(column.expr, spec.filters, '1d');+ expect(query).not.toContain('$');+ expect(query).toContain('{service_name="ci-telemetry"}');+ }+ }+ });++ test('groups every column by its table labels, so rows join', () => {+ for (const entry of spec.tables) {+ for (const column of entry.columns) {+ const grouping = entry.by.length+ ? `by (${entry.by.join(', ')})`+ : /by \(\)|^\(?sum\(/;+ if (typeof grouping === 'string')+ expect(column.expr).toContain(grouping);+ else expect(column.expr).toMatch(grouping);+ }+ }+ });++ test('sorts by a column it has, and has unique ids', () => {+ for (const entry of spec.tables) {+ if (entry.sort)+ expect(entry.columns.map((column) => column.name)).toContain(+ entry.sort+ );+ }+ const ids = spec.tables.map((entry) => entry.id);+ expect(new Set(ids).size).toBe(ids.length);+ expect(ids).toContain('summary');+ });+});++test('expand replaces the range and longer names before their prefixes', () => {+ expect(+ expand('$PUSH_X $PUSH [$RANGE]', { PUSH: 'a', PUSH_X: 'b' }, '1d')+ ).toBe('b a [1d]');+});++test('joinColumns joins on labels, sorts, limits and keeps gaps', () => {+ const rows = joinColumns(table, [+ [+ { metric: { workflow: 'W', job: 'a' }, value: 3 },+ { metric: { workflow: 'W', job: 'b' }, value: 5 },+ { metric: { workflow: 'W', job: 'c' }, value: 1 },+ ],+ [+ { metric: { workflow: 'W', job: 'b' }, value: 400 },+ { metric: { workflow: 'W', job: 'c' }, value: 90 },+ ],+ ]);+ expect(rows).toEqual([+ { labels: ['W', 'b'], values: [5, 400] },+ { labels: ['W', 'c'], values: [1, 90] },+ ]);+ expect(+ joinColumns({ ...table, limit: undefined }, [+ [{ metric: { workflow: 'W', job: 'a' }, value: 3 }],+ [],+ ])+ ).toEqual([{ labels: ['W', 'a'], values: [3, null] }]);+});++test('formatValue writes durations, hours and shares', () => {+ expect(formatValue(42.4, 's')).toBe('42s');+ expect(formatValue(387, 's')).toBe('6m 27s');+ expect(formatValue(12.34, 'h')).toBe('12.3 h');+ expect(formatValue(0.826, 'percentunit')).toBe('83%');+ expect(formatValue(null, 's')).toBe('');+});++test('renderTable writes a markdown table with an empty row when there is no data', () => {+ expect(renderTable(table, [{ labels: ['W', ''], values: [2, 61] }])).toBe(+ '| workflow | job | Runs | p90 |\n| --- | --- | ---: | ---: |\n| W | (none) | 2 | 1m 01s |'+ );+ expect(renderTable(table, []).split('\n')).toHaveLength(3);+});++test('parseRange reads Loki durations', () => {+ expect(parseRange('24h')).toBe(86_400);+ expect(parseRange('7d')).toBe(604_800);+ expect(() => parseRange('1w')).toThrow();+});diff --git a/infra/stacks/observability/scripts/ci-report.ts b/infra/stacks/observability/scripts/ci-report.tsnew file mode 100644index 0000000000..05e1b6eaa4--- /dev/null+++ b/infra/stacks/observability/scripts/ci-report.ts@@ -0,0 +1,243 @@+// Daily CI report from the exporter's Loki events: wall time by PR type, what held+// pushes up, slowest jobs and steps, queues, cache volumes and CI hours. The Macro CI+// dashboard reads the same queries from nixos/ci-report.json.+//+// GRAFANA_TOKEN=$(cat <token file>) bun infra/stacks/observability/scripts/ci-report.ts [--range 24h] [--days 7]+// Pass --loki <url> to query another Loki API root, such as a local one.++import { readFileSync } from 'node:fs';+import { fileURLToPath } from 'node:url';++const HOSTED_LOKI =+ 'https://grafana.macro-internal.com/api/datasources/proxy/uid/loki';+const TIMEOUT_MILLISECONDS = 30_000;+const DAY_SECONDS = 86_400;++export type Column = { name: string; unit: string; expr: string };+export type Table = {+ id: string;+ title: string;+ description: string;+ by: string[];+ sort?: string;+ limit?: number;+ columns: Column[];+};+export type Spec = { filters: Record<string, string>; tables: Table[] };+export type Sample = { metric: Record<string, string>; value: number };+export type Row = { labels: string[]; values: (number | null)[] };++/** Replace `$NAME` with its filter and `$RANGE` with the window, as the dashboard does. */+export function expand(+ expr: string,+ filters: Record<string, string>,+ range: string+): string {+ let result = expr.replaceAll('$RANGE', range);+ // Longest names first, so a name that prefixes another can't replace part of it.+ for (const name of Object.keys(filters).sort((a, b) => b.length - a.length)) {+ result = result.replaceAll(`$${name}`, filters[name]);+ }+ return result;+}++/** One row per label combination, with a value per column; missing values stay null. */+export function joinColumns(table: Table, results: Sample[][]): Row[] {+ const rows = new Map<string, Row>();+ for (const [index, samples] of results.entries()) {+ for (const sample of samples) {+ const labels = table.by.map((label) => sample.metric[label] ?? '');+ const key = JSON.stringify(labels);+ let row = rows.get(key);+ if (!row) {+ row = { labels, values: table.columns.map(() => null) };+ rows.set(key, row);+ }+ row.values[index] = sample.value;+ }+ }+ const sortIndex = table.columns.findIndex(+ (column) => column.name === table.sort+ );+ const ordered = [...rows.values()];+ if (sortIndex >= 0) {+ ordered.sort(+ (a, b) => (b.values[sortIndex] ?? -1) - (a.values[sortIndex] ?? -1)+ );+ }+ return table.limit ? ordered.slice(0, table.limit) : ordered;+}++export function formatValue(value: number | null, unit: string): string {+ if (value === null || Number.isNaN(value)) return '';+ if (unit === 's') {+ const total = Math.round(value);+ return total < 60+ ? `${total}s`+ : `${Math.floor(total / 60)}m ${String(total % 60).padStart(2, '0')}s`;+ }+ if (unit === 'h') return `${value.toFixed(1)} h`;+ if (unit === 'percentunit') return `${Math.round(value * 100)}%`;+ return String(Math.round(value));+}++export function renderTable(+ table: Table,+ rows: Row[],+ labelHeaders: string[] = table.by+): string {+ const header = [+ ...labelHeaders,+ ...table.columns.map((column) => column.name),+ ];+ const lines = [+ `| ${header.join(' | ')} |`,+ `| ${header.map((_, index) => (index < labelHeaders.length ? '---' : '---:')).join(' | ')} |`,+ ];+ for (const row of rows) {+ const cells = [+ ...row.labels.map((label) => label || '(none)'),+ ...row.values.map((value, index) =>+ formatValue(value, table.columns[index].unit)+ ),+ ];+ lines.push(`| ${cells.join(' | ')} |`);+ }+ if (rows.length === 0) lines.push(`| ${header.map(() => '').join(' | ')} |`);+ return lines.join('\n');+}++/** A Loki duration such as `24h` or `7d`, in seconds. */+export function parseRange(range: string): number {+ const match = /^(\d+)([mhd])$/.exec(range);+ if (!match)+ throw new Error(`Expected a range such as 24h or 7d, got ${range}`);+ return (+ Number(match[1]) *+ { m: 60, h: 3600, d: DAY_SECONDS }[match[2] as 'm' | 'h' | 'd']+ );+}++type LokiVector = {+ status: string;+ data: {+ resultType: string;+ result: { metric: Record<string, string>; value: [number, string] }[];+ };+};++class Loki {+ base: string;+ token: string | undefined;++ constructor(base: string, token: string | undefined) {+ this.base = base;+ this.token = token;+ }++ async instant(query: string, time: number): Promise<Sample[]> {+ const url = `${this.base}/loki/api/v1/query?${new URLSearchParams({ query, time: String(time) })}`;+ const response = await fetch(url, {+ headers: this.token ? { Authorization: `Bearer ${this.token}` } : {},+ signal: AbortSignal.timeout(TIMEOUT_MILLISECONDS),+ });+ if (!response.ok) {+ // The body names the query problem; it never echoes the token.+ throw new Error(+ `Loki answered HTTP ${response.status}: ${(await response.text()).slice(0, 300)}`+ );+ }+ const body = (await response.json()) as LokiVector;+ if (body.data.resultType !== 'vector') return [];+ return body.data.result.map((entry) => ({+ metric: entry.metric,+ value: Number(entry.value[1]),+ }));+ }++ async table(+ spec: Spec,+ table: Table,+ range: string,+ time: number+ ): Promise<Row[]> {+ const results = await Promise.all(+ table.columns.map((column) =>+ this.instant(expand(column.expr, spec.filters, range), time)+ )+ );+ return joinColumns(table, results);+ }+}++function argument(args: string[], name: string, fallback: string): string {+ const index = args.indexOf(name);+ return index >= 0 && args[index + 1] ? args[index + 1] : fallback;+}++function utc(seconds: number): string {+ return new Date(seconds * 1000).toISOString().slice(0, 16).replace('T', ' ');+}++export async function main(args: string[]) {+ const spec: Spec = JSON.parse(+ readFileSync(new URL('../nixos/ci-report.json', import.meta.url), 'utf8')+ );+ const range = argument(args, '--range', '24h');+ const days = Number(argument(args, '--days', '7'));+ const base = argument(args, '--loki', HOSTED_LOKI);+ const token = process.env.GRAFANA_TOKEN;+ if (base === HOSTED_LOKI && !token) {+ throw new Error(+ 'Set GRAFANA_TOKEN to a Grafana token that can read the Loki datasource'+ );+ }+ const loki = new Loki(base, token);+ const now = Math.floor(Date.now() / 1000);+ const output = [+ `# CI report, ${utc(now - parseRange(range))} to ${utc(now)} UTC`,+ ];+ for (const table of spec.tables) {+ output.push(`\n## ${table.title}\n\n${table.description}\n`);+ try {+ output.push(+ renderTable(table, await loki.table(spec, table, range, now))+ );+ } catch (error) {+ // One refused query, such as a series cap, shouldn't cost the rest of the report.+ output.push(+ `Not available: ${error instanceof Error ? error.message : String(error)}`+ );+ }+ }+ const summary = spec.tables.find((table) => table.id === 'summary');+ if (summary && days > 0) {+ const today = Math.floor(now / DAY_SECONDS) * DAY_SECONDS;+ const rows: Row[] = [];+ for (let day = days - 1; day >= 0; day -= 1) {+ const start = today - day * DAY_SECONDS;+ const end = Math.min(start + DAY_SECONDS, now);+ // A day Loki refuses stays blank rather than costing the tables above.+ const [row] = await loki+ .table(spec, summary, `${end - start}s`, end)+ .catch(() => []);+ rows.push({+ labels: [+ utc(start).slice(0, 10) ++ (end < start + DAY_SECONDS ? ' (so far)' : ''),+ ],+ values: row?.values ?? summary.columns.map(() => null),+ });+ }+ output.push(`\n## Per UTC day\n`);+ output.push(renderTable(summary, rows, ['Day']));+ }+ console.log(output.join('\n'));+}++if (process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]) {+ main(process.argv.slice(2)).catch((error: unknown) => {+ console.error(error instanceof Error ? error.message : String(error));+ process.exit(1);+ });+}