Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Keep complete benchmark pairs within isolated package shards

pulse-triage[bot]84eac1a3dff2
diff --git a/.github/workflows/build-and-test.yml b/.github/workflows/build-and-test.yml
index 08b20d0140..59ae3b398b 100644
--- a/.github/workflows/build-and-test.yml
+++ b/.github/workflows/build-and-test.yml
@@ -938,11 +938,17 @@ jobs:
echo "::warning title=Chart SLO verdict not published::This token cannot create check runs; the evidence job carries the verdict."
benchmarks:
- name: Benchmarks
+ name: Benchmarks (shard ${{ matrix.shard }})
needs: changes
if: needs.changes.outputs.code == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 35
+ strategy:
+ fail-fast: false
+ matrix:
+ shard: [0, 1, 2]
+ env:
+ PULSE_BENCH_SHARD: ${{ matrix.shard }}
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -983,10 +989,10 @@ jobs:
run: bash scripts/run-ci-benchmarks.sh
# Only packages whose dependency closure the pull request changes are
- # sampled. A module or shared-model change still samples nearly every
- # package: about 24 minutes, warm-up included, as of October 2026. The
- # script's budget stops the run with per-package costs before the step
- # timeout can kill it without them.
+ # sampled. Each package's complete same-host pair stays in one fixed
+ # shard, including its warm-ups and all ten alternating rounds. The
+ # unchanged budget stops an outgrown shard with per-package costs before
+ # the step timeout can kill it without them.
- name: Paired base and candidate benchmarks (pull requests)
if: github.event_name == 'pull_request'
timeout-minutes: 30
@@ -1012,7 +1018,7 @@ jobs:
if: always()
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
- name: bench-results
+ name: bench-results-${{ matrix.shard }}
path: |
bench-results.txt
bench-baseline.txt
@@ -1020,3 +1026,19 @@ jobs:
bench-metadata.txt
if-no-files-found: warn
retention-days: 7
+
+ benchmarks-verdict:
+ name: Benchmarks
+ needs: [changes, benchmarks]
+ if: always() && needs.changes.outputs.code == 'true'
+ runs-on: ubuntu-24.04
+ timeout-minutes: 5
+ steps:
+ - name: Require every benchmark shard
+ env:
+ SHARD_RESULT: ${{ needs.benchmarks.result }}
+ run: |
+ if [ "$SHARD_RESULT" != success ]; then
+ echo "Benchmark collection or comparison is incomplete: $SHARD_RESULT"
+ exit 1
+ fi
diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.md
index 34ec6da634..fe6bd678cb 100644
--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md
+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md
@@ -1601,13 +1601,33 @@ remain in `scripts/tests/test-ci-benchmarks.sh`.
Paired pull-request runs sample only the packages whose in-module dependency
closure the change reaches, recorded as `selection=` and `packages=` in the
metadata. Module files, unclean or non-Git trees and an unreadable package
-graph select every package; unpaired branch runs always sample every package.
+graph select every package; unpaired branch runs select every package.
An empty selection leaves nothing to compare and is not a passed comparison of
skipped packages. The paired step's wall-clock budget sits below its timeout so
an outgrown suite stops with each package's mean cost per sample, not a bare
timeout; `TestPairedBenchmarksStopInsideTheirBudget` checks that ordering and
the selection and budget cases run in `scripts/tests/test-ci-benchmarks.sh`.
+Build and Test partitions the selected inventory into three fixed, disjoint
+package shards. Each package retains baseline and candidate on one runner,
+both warmups, ten alternating paired rounds, executable hashes and the existing
+duration, budget and regression rules. A package comparison never combines
+measurements from different hosts. Metadata names the shard and its selected
+packages; invalid shards and unassigned packages fail collection. Local runs
+default to the unpartitioned inventory. An empty affected shard invokes no Go
+test rather than inadvertently testing the repository root.
+
+Each shard uploads a distinct `bench-results-<shard>` artifact on success or
+failure. Matrix fail-fast is disabled and the aggregate `Benchmarks` check
+accepts only success from the complete matrix; a failed, cancelled, skipped or
+missing result stays incomplete. `test_benchmark_workflow_contract.py` executes
+the actual verdict shell against those results, while the collector controls
+check complete/disjoint membership, sample counts, alternating order and byte
+provenance. The native workflow contract checks artifact separation and retains
+the budget/deadline ordering. These controls establish collection boundaries,
+not a hosted speedup, less total compute, release acceptance or resolution of
+an earlier incomplete performance observation.
+
### Public Helm exact-package receipt
Helm Pages admission uses the immutable published packet, not the continued
diff --git a/scripts/BENCHMARK_EVIDENCE.md b/scripts/BENCHMARK_EVIDENCE.md
index 7a6cc3613a..ac6a08b53e 100644
--- a/scripts/BENCHMARK_EVIDENCE.md
+++ b/scripts/BENCHMARK_EVIDENCE.md
@@ -4,7 +4,7 @@
The workflow uploads all four files even when the regression gate fails.
Metadata records the candidate and baseline Git HEAD and committed tree IDs,
selected Go versions, platform, GOMAXPROCS setting, sample count, duration,
-package list and paired sample start order/timestamps. It deliberately does
+package list, shard and paired sample start order/timestamps. It deliberately does
not dump environment variables, remote URLs or filesystem paths.
HEAD/tree identify committed source, not a clean-worktree or clean-build
@@ -48,7 +48,26 @@ embeds and testdata. Module or workspace files, trees that are not both clean
Git roots, or an unreadable package graph select every package. Metadata `selection=` names the rule applied and
`packages=` the packages sampled; an empty list leaves empty timing files and
nothing to compare. A benchmark reading files outside that closure at run time
-escapes selection. Unpaired runs always sample every package.
+escapes selection. Unpaired runs select every package before sharding.
+
+CI now partitions that inventory across three isolated hosted jobs using
+`PULSE_BENCH_SHARD=0`, `1` and `2`. The fixed assignment in the collector keeps
+monitoring and unified resources apart. Each package's baseline, candidate,
+warmups and all ten alternating pairs stay on the **same** runner; samples for
+one comparison are never combined across hosts. Each shard still uses the
+existing duration, budget and regression rules. With no shard specified,
+local collection retains the full inventory. Invalid shards and unassigned
+packages fail; an empty affected shard records an explicit empty package list
+without invoking Go on the repository root.
+
+The aggregate `Benchmarks` check requires every shard to succeed, with matrix
+fail-fast disabled so a failed sibling does not cancel another comparison.
+Each job uploads its own `bench-results-0`, `-1` or `-2` artifact even on failure.
+Read **all three** artifacts and their source/shard identities before calling
+collection complete. Source controls and a disposable fixture do not establish
+hosted runtime improvement or clear an older incomplete comparison. Sharding
+adds hosted setup work in exchange for a smaller per-job measurement workload;
+it does not assert a reduction in total compute or an idle worker.
With `PULSE_BENCH_BUDGET_SECONDS` set, a paired run projects its total,
warm-up included, after each round and exits 1 once the projection reaches the
diff --git a/scripts/installtests/build_release_assets_test.go b/scripts/installtests/build_release_assets_test.go
index f1acf951bd..60c1f80590 100644
--- a/scripts/installtests/build_release_assets_test.go
+++ b/scripts/installtests/build_release_assets_test.go
@@ -4122,6 +4122,28 @@ func TestBenchmarkQualificationRetainsProvenance(t *testing.T) {
if !strings.Contains(job, "bash scripts/check-bench-regression.sh bench-comparison.txt") {
t.Fatal("provenance must not replace the benchmark regression gate")
}
+ // Matrix jobs cannot share an artifact name: a successful sibling must
+ // neither overwrite a failed collection nor hide its source-bound samples.
+ for _, required := range []string{
+ "fail-fast: false", "shard: [0, 1, 2]",
+ "PULSE_BENCH_SHARD: ${{ matrix.shard }}",
+ "name: bench-results-${{ matrix.shard }}",
+ } {
+ if !strings.Contains(job, required) {
+ t.Fatalf("benchmark shards must retain complete isolated evidence: missing %q", required)
+ }
+ }
+ verdict := workflowJobBlock(t, string(content), "benchmarks-verdict")
+ for _, required := range []string{
+ "name: Benchmarks", "needs: [changes, benchmarks]",
+ "if: always() && needs.changes.outputs.code == 'true'",
+ "SHARD_RESULT: ${{ needs.benchmarks.result }}",
+ `if [ "$SHARD_RESULT" != success ]; then`, "exit 1",
+ } {
+ if !strings.Contains(verdict, required) {
+ t.Fatalf("benchmark verdict must reject an incomplete matrix: missing %q", required)
+ }
+ }
}
// A paired run names each package's cost only when its own budget stops it
diff --git a/scripts/run-ci-benchmarks.sh b/scripts/run-ci-benchmarks.sh
index 7a20e85128..93dab0afa8 100755
--- a/scripts/run-ci-benchmarks.sh
+++ b/scripts/run-ci-benchmarks.sh
@@ -9,6 +9,9 @@ CURRENT_DIR="${PULSE_BENCH_CURRENT_DIR:-$(pwd)}"
BASELINE_DIR="${PULSE_BENCH_BASELINE_DIR:-}"
SAMPLE_COUNT="${PULSE_BENCH_SAMPLE_COUNT:-10}"
BENCHTIME="${PULSE_BENCH_BENCHTIME:-100ms}"
+# CI partitions packages, never a package's paired samples, across three
+# isolated runners. Ordinary local collection still covers the full inventory.
+SHARD="${PULSE_BENCH_SHARD:-all}"
# Wall-clock seconds a paired run may take, warm-up included; unset means no
# limit. Checked after every round, so a suite that has outgrown its budget
# stops early and names each package's cost instead of meeting a step timeout.
@@ -22,6 +25,10 @@ if [[ -n "${BUDGET_SECONDS}" && ! "${BUDGET_SECONDS}" =~ ^[0-9]+$ ]]; then
echo "PULSE_BENCH_BUDGET_SECONDS must be a whole number of seconds." >&2
exit 2
fi
+if [[ ! "${SHARD}" =~ ^(all|0|1|2)$ ]]; then
+ echo "PULSE_BENCH_SHARD must be all, 0, 1 or 2." >&2
+ exit 2
+fi
if [[ ! -d "${CURRENT_DIR}" ]]; then
echo "Benchmark candidate tree not found: ${CURRENT_DIR}" >&2
exit 2
@@ -134,9 +141,30 @@ select_packages() {
}
if [[ -n "${BASELINE_DIR}" ]]; then
select_packages
- echo "Benchmark packages, ${selection}: ${SELECTED[*]:-none}"
fi
+# Keep the slow monitoring and unified-resource families on different hosts.
+# Membership is fixed before either tree runs; both trees and all ten rounds
+# for a package stay together. Fail closed if the inventory grows without an
+# assignment, rather than silently dropping the new package from every shard.
+shard_packages() {
+ local package owner picked=()
+ for package in "${SELECTED[@]}"; do
+ case "${package}" in
+ ./pkg/auth/|./internal/monitoring/|./internal/dockeragent/) owner=0 ;;
+ ./internal/api/|./internal/unifiedresources/|./internal/hostagent/) owner=1 ;;
+ ./pkg/metrics/|./cmd/pulse-agent/|./internal/hostmetrics/) owner=2 ;;
+ *) echo "Benchmark package has no shard: ${package}" >&2; exit 2 ;;
+ esac
+ if [[ "${SHARD}" == all || "${SHARD}" == "${owner}" ]]; then
+ picked+=("${package}")
+ fi
+ done
+ SELECTED=(${picked[@]+"${picked[@]}"})
+}
+shard_packages
+echo "Benchmark packages, ${selection}, shard ${SHARD}: ${SELECTED[*]:-none}"
+
# Retain provenance without dumping credentials, remote URLs or working paths.
# PR checkouts may be synthetic merges; Go selects a toolchain for each tree.
metadata="${CURRENT_DIR}/bench-metadata.txt"
@@ -150,6 +178,7 @@ revision() {
printf 'platform=%s\n' "$(uname -sm)"
printf 'gomaxprocs=%s\n' "${GOMAXPROCS:-runtime-default}"
printf 'selection=%s\n' "${selection}"
+ printf 'shard=%s\nshard_count=3\n' "${SHARD}"
printf 'packages=%s\n' "${SELECTED[*]:-}"
for label in candidate baseline; do
tree="${CURRENT_DIR}"
@@ -207,7 +236,7 @@ run_unpaired() {
PULSE_DATA_DIR="${data_dir}" go test -exec "${binary_wrapper}" \
-bench=. -benchmem -count="${SAMPLE_COUNT}" -run='^$' \
-benchtime="${BENCHTIME}" -timeout=5m \
- "${PACKAGES[@]}"
+ "${SELECTED[@]}"
) | tee "${output}"
}
diff --git a/scripts/tests/test-ci-benchmarks.sh b/scripts/tests/test-ci-benchmarks.sh
index 6c5086862a..518e26406b 100755
--- a/scripts/tests/test-ci-benchmarks.sh
+++ b/scripts/tests/test-ci-benchmarks.sh
@@ -81,6 +81,8 @@ mapfile -t calls < "${WORK_DIR}/go.log"
metadata="${WORK_DIR}/candidate/bench-metadata.txt"
grep -qFx 'samples=2' "${metadata}"
+grep -qFx 'shard=all' "${metadata}"
+grep -qFx 'shard_count=3' "${metadata}"
grep -qFx 'selection=all (trees are not both clean Git checkouts)' "${metadata}"
grep -qFx 'packages=./pkg/metrics/ ./pkg/auth/ ./internal/api/ ./internal/monitoring/ ./internal/unifiedresources/ ./internal/dockeragent/ ./cmd/pulse-agent/ ./internal/hostagent/ ./internal/hostmetrics/' "${metadata}"
grep -qFx 'benchtime=100ms' "${metadata}"
@@ -148,6 +150,20 @@ for call in "${calls[@]}"; do
[[ "${call}" == *' ./internal/api/' && "${call}" != *pkg/metrics* ]]
done
+# Package selection is intersected with the fixed shard, not expanded. A shard
+# with no affected package must not invoke go test with an empty argument list
+# (which would unexpectedly run the repository root).
+PULSE_BENCH_SHARD=0 run_paired
+grep -qFx 'selection=affected' "${metadata}"
+grep -qFx 'shard=0' "${metadata}"
+grep -qFx 'packages=' "${metadata}"
+[[ ! -s "${WORK_DIR}/go.log" ]]
+[[ ! -s "${WORK_DIR}/candidate/bench-baseline.txt" ]]
+[[ ! -s "${WORK_DIR}/candidate/bench-results.txt" ]]
+PULSE_BENCH_SHARD=1 run_paired
+grep -qFx 'packages=./internal/api/' "${metadata}"
+[[ "$(grep -c '^sample=' "${metadata}")" == 2 ]]
+
# A shared non-test file selects every package that imports its directory.
change internal/shared/shared.go shared
commit candidate
@@ -173,6 +189,86 @@ run_paired
grep -qFx 'selection=all (module files changed)' "${metadata}"
grep -qF 'packages=./pkg/metrics/ ./pkg/auth/' "${metadata}"
+# Every package belongs to exactly one shard. Each shard retains both warmups,
+# all ten base/candidate samples, alternating order and executable provenance.
+: > "${WORK_DIR}/partition"
+for shard in 0 1 2; do
+ case "${shard}" in
+ 0) expected='./pkg/auth/ ./internal/monitoring/ ./internal/dockeragent/' ;;
+ 1) expected='./internal/api/ ./internal/unifiedresources/ ./internal/hostagent/' ;;
+ 2) expected='./pkg/metrics/ ./cmd/pulse-agent/ ./internal/hostmetrics/' ;;
+ esac
+ : > "${WORK_DIR}/go.log"
+ PATH="${WORK_DIR}/bin:${PATH}" FAKE_GO_LOG="${WORK_DIR}/go.log" \
+ PULSE_BENCH_CURRENT_DIR="${WORK_DIR}/candidate" \
+ PULSE_BENCH_BASELINE_DIR="${WORK_DIR}/baseline" PULSE_BENCH_SAMPLE_COUNT=10 \
+ PULSE_BENCH_SHARD="${shard}" \
+ bash "${ROOT_DIR}/scripts/run-ci-benchmarks.sh" >/dev/null
+ grep -qFx "shard=${shard}" "${metadata}"
+ grep -qFx "packages=${expected}" "${metadata}"
+ printf '%s\n' "${expected}" | tr ' ' '\n' >> "${WORK_DIR}/partition"
+ [[ "$(grep -c '^sample=' "${metadata}")" == 20 ]]
+ [[ "$(grep -c '^binary=' "${metadata}")" == 20 ]]
+ [[ "$(grep -c '^BenchmarkExample' "${WORK_DIR}/candidate/bench-baseline.txt")" == 10 ]]
+ [[ "$(grep -c '^BenchmarkExample' "${WORK_DIR}/candidate/bench-results.txt")" == 10 ]]
+ mapfile -t calls < "${WORK_DIR}/go.log"
+ [[ "${#calls[@]}" == 22 ]]
+ for call in "${calls[@]}"; do [[ "${call}" == *" ${expected}" ]]; done
+ for ((round=1; round<=10; round++)); do
+ first=baseline; second=candidate
+ if ((round % 2 == 0)); then first=candidate; second=baseline; fi
+ [[ "${calls[2*round]}" == "${WORK_DIR}/${first}"$'\t'* ]]
+ [[ "${calls[2*round+1]}" == "${WORK_DIR}/${second}"$'\t'* ]]
+ for label in baseline candidate; do
+ grep -qFx "binary=${label},${round},fixture.test,${digest}" "${metadata}"
+ done
+ done
+ # Branch collection partitions the same inventory but still uses Go's ten
+ # counts in a single invocation; it must not fall back to the full package set.
+ : > "${WORK_DIR}/go.log"
+ PATH="${WORK_DIR}/bin:${PATH}" FAKE_GO_LOG="${WORK_DIR}/go.log" \
+ PULSE_BENCH_CURRENT_DIR="${WORK_DIR}/candidate" PULSE_BENCH_BASELINE_DIR='' \
+ PULSE_BENCH_SAMPLE_COUNT=10 PULSE_BENCH_SHARD="${shard}" \
+ bash "${ROOT_DIR}/scripts/run-ci-benchmarks.sh" >/dev/null
+ grep -qFx 'selection=all (no baseline)' "${metadata}"
+ grep -qFx "packages=${expected}" "${metadata}"
+ [[ "$(wc -l < "${WORK_DIR}/go.log")" == 1 ]]
+ grep -qF -- "-count=10 -run=^$ -benchtime=100ms -timeout=5m ${expected}" "${WORK_DIR}/go.log"
+done
+[[ "$(wc -l < "${WORK_DIR}/partition")" == 9 ]]
+[[ -z "$(sort "${WORK_DIR}/partition" | uniq -d)" ]]
+
+# A malformed shard is a collection error, never a silent no-op or an all-tree
+# fallback. This must stop before any Go invocation.
+for invalid in 3 -1 01 x all,0; do
+ : > "${WORK_DIR}/go.log"
+ set +e
+ output="$(PATH="${WORK_DIR}/bin:${PATH}" FAKE_GO_LOG="${WORK_DIR}/go.log" \
+ PULSE_BENCH_CURRENT_DIR="${WORK_DIR}/candidate" PULSE_BENCH_SHARD="${invalid}" \
+ bash "${ROOT_DIR}/scripts/run-ci-benchmarks.sh" 2>&1)"
+ status=$?
+ set -e
+ [[ "${status}" == 2 ]]
+ grep -qF 'PULSE_BENCH_SHARD must be all, 0, 1 or 2.' <<<"${output}"
+ [[ ! -s "${WORK_DIR}/go.log" ]]
+done
+
+# A newly added inventory member without a fixed assignment fails rather than
+# disappearing from all three jobs. Exercise the actual collector, not a model
+# of the partition function.
+sed '/^PACKAGES=(/a\ ./internal/new-benchmark/' \
+ "${ROOT_DIR}/scripts/run-ci-benchmarks.sh" > "${WORK_DIR}/unassigned.sh"
+: > "${WORK_DIR}/go.log"
+set +e
+output="$(PATH="${WORK_DIR}/bin:${PATH}" FAKE_GO_LOG="${WORK_DIR}/go.log" \
+ PULSE_BENCH_CURRENT_DIR="${WORK_DIR}/candidate" PULSE_BENCH_BASELINE_DIR='' \
+ PULSE_BENCH_SHARD=0 bash "${WORK_DIR}/unassigned.sh" 2>&1)"
+status=$?
+set -e
+[[ "${status}" == 2 ]]
+grep -qFx 'Benchmark package has no shard: ./internal/new-benchmark/' <<<"${output}"
+[[ ! -s "${WORK_DIR}/go.log" ]]
+
# A nested archive has no identity of its own; never attribute its parent's SHA.
mkdir -p "${WORK_DIR}/candidate/archive"
PATH="${WORK_DIR}/bin:${PATH}" FAKE_GO_LOG="${WORK_DIR}/go.log" \
diff --git a/scripts/tests/test_benchmark_workflow_contract.py b/scripts/tests/test_benchmark_workflow_contract.py
index 3dce0373a0..ef1238ea50 100755
--- a/scripts/tests/test_benchmark_workflow_contract.py
+++ b/scripts/tests/test_benchmark_workflow_contract.py
@@ -1,28 +1,103 @@
#!/usr/bin/env python3
-"""Guard the paired, adequately sampled benchmark workflow contract."""
+"""Guard complete, adequately sampled same-host benchmark comparisons."""
+from copy import deepcopy
+import os
from pathlib import Path
+import subprocess
import unittest
+import yaml
ROOT = Path(__file__).resolve().parents[2]
WORKFLOW = ROOT / ".github" / "workflows" / "build-and-test.yml"
class BenchmarkWorkflowContractTest(unittest.TestCase):
- def test_prs_compare_base_and_candidate_on_one_runner(self) -> None:
- workflow = WORKFLOW.read_text(encoding="utf-8")
- benchmark_job = workflow[workflow.index(" benchmarks:") :]
-
- self.assertIn("ref: ${{ github.event.pull_request.base.sha }}", benchmark_job)
- self.assertIn("path: benchmark-base", benchmark_job)
- self.assertIn('PULSE_BENCH_SAMPLE_COUNT: "10"', benchmark_job)
- self.assertIn("bash scripts/run-ci-benchmarks.sh", benchmark_job)
- self.assertNotIn("go-bench-baseline-", benchmark_job)
- self.assertNotIn("actions/cache/restore@", benchmark_job)
- self.assertIn("bench-baseline.txt", benchmark_job)
- self.assertIn("bench-comparison.txt", benchmark_job)
- self.assertIn("bench-metadata.txt", benchmark_job)
+ def setUp(self) -> None:
+ self.workflow = yaml.safe_load(WORKFLOW.read_text(encoding="utf-8"))
+
+ def assert_contract(self, workflow: dict) -> None:
+ jobs = workflow["jobs"]
+ collector = jobs["benchmarks"]
+ self.assertEqual(collector["name"], "Benchmarks (shard ${{ matrix.shard }})")
+ self.assertEqual(collector["needs"], "changes")
+ self.assertEqual(collector["if"], "needs.changes.outputs.code == 'true'")
+ self.assertEqual(collector["strategy"]["matrix"], {"shard": [0, 1, 2]})
+ self.assertIs(collector["strategy"]["fail-fast"], False)
+ self.assertFalse(collector.get("continue-on-error", False))
+ self.assertEqual(collector["env"]["PULSE_BENCH_SHARD"], "${{ matrix.shard }}")
+ steps = {step["name"]: step for step in collector["steps"]}
+ base = steps["Checkout pull request base for paired comparison"]
+ self.assertEqual(base["if"], "github.event_name == 'pull_request'")
+ self.assertEqual(base["with"]["ref"], "${{ github.event.pull_request.base.sha }}")
+ self.assertEqual(base["with"]["path"], "benchmark-base")
+ pair = steps["Paired base and candidate benchmarks (pull requests)"]
+ single = steps["Go benchmarks (non-PR)"]
+ self.assertEqual(pair["if"], "github.event_name == 'pull_request'")
+ self.assertEqual(single["if"], "github.event_name != 'pull_request'")
+ for step in [pair, single]:
+ self.assertEqual(step["env"]["PULSE_BENCH_SAMPLE_COUNT"], "10")
+ self.assertEqual(step["run"], "bash scripts/run-ci-benchmarks.sh")
+ self.assertFalse(step.get("continue-on-error", False))
+ self.assertEqual(pair["env"]["PULSE_BENCH_BUDGET_SECONDS"], "1680")
+ compare = steps["Compare benchmarks against baseline"]["run"]
+ self.assertEqual(steps["Compare benchmarks against baseline"]["if"], "github.event_name == 'pull_request'")
+ self.assertIn("benchstat bench-baseline.txt bench-results.txt", compare)
+ self.assertIn("bash scripts/check-bench-regression.sh bench-comparison.txt", compare)
+ self.assertFalse(steps["Compare benchmarks against baseline"].get("continue-on-error", False))
+ upload = steps["Upload benchmark evidence"]
+ self.assertEqual(upload["if"], "always()")
+ self.assertEqual(upload["with"]["name"], "bench-results-${{ matrix.shard }}")
+ self.assertEqual(set(upload["with"]["path"].split()), {
+ "bench-baseline.txt", "bench-results.txt", "bench-comparison.txt", "bench-metadata.txt",
+ })
+ verdict = jobs["benchmarks-verdict"]
+ self.assertEqual(verdict["name"], "Benchmarks")
+ self.assertEqual(verdict["needs"], ["changes", "benchmarks"])
+ self.assertEqual(verdict["if"], "always() && needs.changes.outputs.code == 'true'")
+ self.assertFalse(verdict.get("continue-on-error", False))
+ self.assertEqual(verdict["steps"][0]["env"]["SHARD_RESULT"], "${{ needs.benchmarks.result }}")
+ for step in verdict["steps"]:
+ self.assertNotIn("if", step)
+ self.assertFalse(step.get("continue-on-error", False))
+
+ def test_prs_keep_each_complete_pair_on_one_runner(self) -> None:
+ self.assert_contract(self.workflow)
+ text = WORKFLOW.read_text(encoding="utf-8")
+ benchmark_jobs = text[text.index(" benchmarks:"):]
+ self.assertNotIn("go-bench-baseline-", benchmark_jobs)
+ self.assertNotIn("actions/cache/restore@", benchmark_jobs)
+
+ def test_incomplete_matrix_or_weakened_result_is_rejected(self) -> None:
+ mutations = [
+ lambda w: w["jobs"]["benchmarks"]["strategy"]["matrix"].update(shard=[0, 1]),
+ lambda w: w["jobs"]["benchmarks"]["strategy"].update({"fail-fast": True}),
+ lambda w: w["jobs"]["benchmarks"].update({"continue-on-error": True}),
+ lambda w: w["jobs"]["benchmarks"]["env"].update(PULSE_BENCH_SHARD="all"),
+ lambda w: w["jobs"]["benchmarks-verdict"].update(needs=["changes"]),
+ lambda w: w["jobs"]["benchmarks-verdict"].update({"if": "success()"}),
+ lambda w: w["jobs"]["benchmarks-verdict"]["steps"][0].update({"continue-on-error": True}),
+ ]
+ for index, mutate in enumerate(mutations):
+ with self.subTest(index=index):
+ workflow = deepcopy(self.workflow)
+ mutate(workflow)
+ with self.assertRaises(AssertionError):
+ self.assert_contract(workflow)
+
+ def test_actual_verdict_command_fails_for_every_incomplete_result(self) -> None:
+ command = self.workflow["jobs"]["benchmarks-verdict"]["steps"][0]["run"]
+ for result in ["success", "failure", "cancelled", "skipped", "", "unknown"]:
+ with self.subTest(result=result):
+ process = subprocess.run(
+ ["bash", "--noprofile", "--norc", "-e", "-o", "pipefail", "-c", command],
+ env={**os.environ, "SHARD_RESULT": result},
+ capture_output=True, text=True, check=False,
+ )
+ self.assertEqual(process.returncode, 0 if result == "success" else 1)
+ if result != "success":
+ self.assertIn("incomplete:", process.stdout)
if __name__ == "__main__":