Skip to content

Worktrunk is a CLI for Git worktree management, designed for parallel AI agent workflows

ci: run benchmarks on their own daily schedule, split from nightly (#3337)

Maximilian Roos820666741f4f
diff --git a/.claude/skills/running-tend/SKILL.md b/.claude/skills/running-tend/SKILL.md
index 8d13b0ee5..81b7484d7 100644
--- a/.claude/skills/running-tend/SKILL.md
+++ b/.claude/skills/running-tend/SKILL.md
@@ -278,7 +278,7 @@ Pinned third-party versions in CI are invisible to Dependabot — it follows `Ca
For each weekly run, check upstream and bump:
- **`baptiste0928/cargo-install@v3` blocks** in `.github/workflows/ci.yaml`, `.github/workflows/nightly.yaml`, and `.github/actions/{test,claude}-setup/action.yaml` — every `version: "=X.Y.Z"` against `cargo info <crate>`. Today: `cargo-insta`, `cargo-nextest`, `cargo-llvm-cov`, `cargo-msrv`, `cargo-udeps`, `lychee`, `worktrunk`. The `cargo-affected` install has no version pin (follows default branch) — leave it alone. Verify each crate's `rust-version` against the pinned toolchain and note compatibility in the PR body (see PR #1657 for the format).
-- **`hustcer/setup-nu@v3`** `version:` input — latest from `gh api repos/nushell/nushell/releases/latest --jq '.tag_name'`. Three call sites: `ci.yaml` (`code-coverage`), `nightly.yaml` (`benchmarks`), and `actions/test-setup/action.yaml`.
+- **`hustcer/setup-nu@v3`** `version:` input — latest from `gh api repos/nushell/nushell/releases/latest --jq '.tag_name'`. Four call sites: `ci.yaml` (`code-coverage`), `nightly.yaml` (`feature-powerset`), `benchmarks.yaml` (`benchmarks`), and `actions/test-setup/action.yaml`.
- **`taiki-e/[email protected]`** `tool: zola@<ver>` in the `check-docs` job — latest from `gh api repos/getzola/zola/releases/latest --jq '.tag_name'`.
- **Runner images** — `ubuntu-24.04`, `macos-15`, `windows-2022`. Keep `windows-2022` pinned (actions/runner-images#12677 — windows-2025 lacks the D: drive).
diff --git a/.github/benchmark-failure.md b/.github/benchmark-failure.md
new file mode 100644
index 000000000..621939fa7
--- /dev/null
+++ b/.github/benchmark-failure.md
@@ -0,0 +1,6 @@
+---
+title: Benchmarks failed
+labels: ci
+---
+
+Benchmarks [failed on {{ date | date('YYYY-MM-DD') }}]({{ env.LINK }})
diff --git a/.github/workflows/benchmarks.yaml b/.github/workflows/benchmarks.yaml
new file mode 100644
index 000000000..c5cf17597
--- /dev/null
+++ b/.github/workflows/benchmarks.yaml
@@ -0,0 +1,117 @@
+name: benchmarks
+# Full criterion suite plus the time-series gist append. Runs ~80 min and
+# checks performance, not correctness, so it stands apart from the PR/merge
+# flow and from nightly's correctness checks: a slow or failing bench never
+# gates a merge. The daily cron is the perf-history feed (it appends to the
+# gist); workflow_dispatch covers on-demand runs against a chosen branch.
+#
+# Not run on PRs or pushes, so the daily run is the only signal that the bench
+# harness still builds — create-issue-on-benchmark-failure surfaces a break.
+#
+# Runner version pinned; see ci.yaml header comment for rationale.
+
+on:
+ schedule:
+ # 3:47 UTC daily. Off-peak, offset from nightly's 5:37 so the two long
+ # runs don't contend, and off :00 to be a good citizen w.r.t. GitHub's
+ # cron scheduler.
+ - cron: '47 3 * * *'
+ workflow_dispatch:
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.ref }}
+ cancel-in-progress: false
+
+env:
+ # Match ci.yaml/nightly.yaml: Swatinem/rust-cache hashes CARGO*/RUST* into
+ # the cache key, so the shared `test` cache only restores when these agree
+ # (see .github/CLAUDE.md).
+ CARGO_TERM_COLOR: always
+ CARGO_INCREMENTAL: 0
+ RUSTFLAGS: -C debuginfo=0
+
+jobs:
+ benchmarks:
+ runs-on: ubuntu-24.04
+ steps:
+ - name: 📂 Checkout code
+ uses: actions/checkout@v7
+
+ - name: 💰 Cache
+ uses: Swatinem/rust-cache@v2
+ with:
+ # Restore the shared `test` cache (registry + deps), never save —
+ # `cargo bench` is release-profile so it rebuilds its own artifacts,
+ # but the dependency download/extract is still worth restoring.
+ prefix-key: v1-rust
+ shared-key: shared
+ cache-bin: "false"
+ save-if: false
+
+ - name: Install shells (zsh, fish)
+ run: sudo apt-get update && sudo apt-get install -y zsh fish
+
+ - name: Install nushell
+ uses: hustcer/setup-nu@v3
+ with:
+ version: '0.113.1'
+
+ - name: 💰 Rust repo cache
+ uses: actions/cache@v6
+ with:
+ path: target/bench-repos
+ key: bench-repos-rust-${{ runner.os }}
+
+ - name: 📊 Run benchmarks
+ run: cargo bench
+
+ - name: 📦 Upload benchmark results
+ uses: actions/upload-artifact@v7
+ with:
+ name: benchmark-results-${{ github.run_id }}
+ path: target/criterion
+
+ # Time-series benchmark store, owned by worktrunk-bot:
+ # https://gist.github.com/worktrunk-bot/19bb23cb9658722abfe69479d0a4f9bf
+ #
+ # Cron-only: workflow_dispatch runs aren't appended (would pollute the
+ # time series). Skipped on forks: `TEND_BOT_TOKEN` is not exposed there.
+ - name: 💾 Append results to gist
+ if: github.repository_owner == 'max-sixty' && github.event_name == 'schedule'
+ env:
+ GITHUB_TOKEN: ${{ secrets.TEND_BOT_TOKEN }}
+ GIST_ID: 19bb23cb9658722abfe69479d0a4f9bf
+ run: |
+ set -euo pipefail
+ python3 .github/scripts/criterion-to-jsonl.py --sha "$GITHUB_SHA" > new-rows.jsonl
+ git clone "https://x-access-token:${GITHUB_TOKEN}@gist.github.com/${GIST_ID}.git" /tmp/gist
+ cat new-rows.jsonl >> /tmp/gist/results.jsonl
+ git -C /tmp/gist \
+ -c user.name=worktrunk-bot \
+ commit -am "benchmarks: ${GITHUB_SHA::7}"
+ git -C /tmp/gist push
+
+ create-issue-on-benchmark-failure:
+ needs: benchmarks
+ if: always() && contains(needs.*.result, 'failure') && github.repository_owner == 'max-sixty' && github.event_name == 'schedule'
+ runs-on: ubuntu-24.04
+ permissions:
+ contents: read
+ issues: write
+ steps:
+ - name: 📂 Checkout code
+ uses: actions/checkout@v7
+
+ - uses: JasonEtco/create-an-issue@v2
+ env:
+ # Use TEND_BOT_TOKEN for a consistent bot identity (per
+ # .github/CLAUDE.md) and so any future issue-triage automation can
+ # cascade off issue creation — events from the default GITHUB_TOKEN
+ # don't trigger other workflows.
+ GITHUB_TOKEN: ${{ secrets.TEND_BOT_TOKEN }}
+ LINK: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+ with:
+ filename: .github/benchmark-failure.md
+ update_existing: true
+ search_existing: open
diff --git a/.github/workflows/nightly.yaml b/.github/workflows/nightly.yaml
index 175e440c6..a456a3813 100644
--- a/.github/workflows/nightly.yaml
+++ b/.github/workflows/nightly.yaml
@@ -5,7 +5,6 @@ name: nightly
# - full-tests: full nextest suite on the linux/macos/windows matrix — the
# cargo-affected safety net (see the job comment and ci.yaml's affected block)
# - release-target: PTY+shell suite on the release triples ci.yaml doesn't cover
-# - benchmarks: full criterion suite + time-series gist append (cron only)
# - check-unused-dependencies: cargo-udeps on nightly toolchain
# - minimal-versions: cargo check against minimum-version dep resolution
# - nix-flake: nix flake check (packaging-environment bugs, motivated by #2624)
@@ -252,73 +251,6 @@ jobs:
NEXTEST_NO_INPUT_HANDLER: 1
run: cargo nextest run --target ${{ matrix.target }}
- benchmarks:
- # Full criterion suite. Moved here from ci.yaml — runs ~80 min, was
- # already advisory non-required (CLAUDE.md: "Don't wait for CI
- # benchmarks before merging"). The cron run also appends results to the
- # time-series gist so we have daily perf history on main.
- needs: gate
- if: needs.gate.outputs.run == 'true'
- runs-on: ubuntu-24.04
- steps:
- - name: 📂 Checkout code
- uses: actions/checkout@v7
-
- - name: 💰 Cache
- uses: Swatinem/rust-cache@v2
- with:
- # Restore the shared `test` cache (registry + deps), never save —
- # `cargo bench` is release-profile so it rebuilds its own artifacts,
- # but the dependency download/extract is still worth restoring.
- prefix-key: v1-rust
- shared-key: shared
- cache-bin: "false"
- save-if: false
-
- - name: Install shells (zsh, fish)
- run: sudo apt-get update && sudo apt-get install -y zsh fish
-
- - name: Install nushell
- uses: hustcer/setup-nu@v3
- with:
- version: '0.113.1'
-
- - name: 💰 Rust repo cache
- uses: actions/cache@v6
- with:
- path: target/bench-repos
- key: bench-repos-rust-${{ runner.os }}
-
- - name: 📊 Run benchmarks
- run: cargo bench
-
- - name: 📦 Upload benchmark results
- uses: actions/upload-artifact@v7
- with:
- name: benchmark-results-${{ github.run_id }}
- path: target/criterion
-
- # Time-series benchmark store, owned by worktrunk-bot:
- # https://gist.github.com/worktrunk-bot/19bb23cb9658722abfe69479d0a4f9bf
- #
- # Cron-only: PR/push runs aren't appended (would pollute the time
- # series). Skipped on forks: `TEND_BOT_TOKEN` is not exposed there.
- - name: 💾 Append results to gist
- if: github.repository_owner == 'max-sixty' && github.event_name == 'schedule'
- env:
- GITHUB_TOKEN: ${{ secrets.TEND_BOT_TOKEN }}
- GIST_ID: 19bb23cb9658722abfe69479d0a4f9bf
- run: |
- set -euo pipefail
- python3 .github/scripts/criterion-to-jsonl.py --sha "$GITHUB_SHA" > new-rows.jsonl
- git clone "https://x-access-token:${GITHUB_TOKEN}@gist.github.com/${GIST_ID}.git" /tmp/gist
- cat new-rows.jsonl >> /tmp/gist/results.jsonl
- git -C /tmp/gist \
- -c user.name=worktrunk-bot \
- commit -am "benchmarks: ${GITHUB_SHA::7}"
- git -C /tmp/gist push
-
check-unused-dependencies:
# Moved from ci.yaml — `cargo udeps` requires nightly toolchain anyway,
# so it's already in the "nightly concern" bucket; rarely flips between
@@ -453,7 +385,6 @@ jobs:
needs:
- feature-powerset
- release-target
- - benchmarks
- check-unused-dependencies
- minimal-versions
- nix-flake
diff --git a/CLAUDE.md b/CLAUDE.md
index 447e228ff..092539b76 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -143,7 +143,7 @@ Why: wt installs a `signal_hook` SIGINT/SIGTERM handler so it can forward signal
## Benchmarks & Traces
-`cargo bench --bench list <filter>` (Criterion takes a positional substring filter; there's no `--skip`). `cargo run -p wt-perf -- timeline -- <args>` traces one `wt` invocation. Real-repo benchmarks clone rust-lang/rust on first run. The `benchmarks` CI job is non-required — only `test (linux|macos|windows)` block merge; `mergeStateStatus: UNSTABLE` from a still-pending bench run is mergeable. Filter map, expected numbers, and trace queries: `benches/CLAUDE.md`.
+`cargo bench --bench list <filter>` (Criterion takes a positional substring filter; there's no `--skip`). `cargo run -p wt-perf -- timeline -- <args>` traces one `wt` invocation. Real-repo benchmarks clone rust-lang/rust on first run. Benchmarks run as a standalone scheduled workflow (`.github/workflows/benchmarks.yaml`, daily cron plus `workflow_dispatch`), not on PRs, so they never gate a merge; only `test (linux|macos|windows)` block it. Filter map, expected numbers, and trace queries: `benches/CLAUDE.md`.
## Code Quality