Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.
Go aialertsdashboarddockerdocker-monitoringgohomelabinfrastructure-monitoringkubernetesmonitoringmspproxmoxproxmox-backup-serverproxmox-mail-gatewayproxmox-veself-hostedtruenastypescriptvspherewebhooks
Release completed API capacity to the backend package queue
pulse-triage[bot]32b338c34480
diff --git a/.github/workflows/create-release.yml b/.github/workflows/create-release.ymlindex a6dd7ac739..3e4013d5e6 100644--- a/.github/workflows/create-release.yml+++ b/.github/workflows/create-release.yml@@ -599,6 +599,15 @@ jobs:- name: Run backend testsrun: ./scripts/run-release-backend-tests.sh --data-root "$RUNNER_TEMP/pulse-test-data"+ - name: Retain backend package coverage and CPU leases+ if: always()+ uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2+ with:+ name: release-backend-coverage-${{ github.sha }}-${{ github.run_attempt }}+ path: ${{ runner.temp }}/pulse-test-data/release-backend-run/coverage/+ if-no-files-found: warn+ retention-days: 7+# Alpha and beta builds are feedback checkpoints. RC and stable publication# run the deeper integration gate because an RC must be promotable in intent.diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.mdindex fe6bd678cb..bf910ff33d 100644--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md@@ -4970,9 +4970,9 @@ release is public as a prerelease, while stable/latest remains `v6.3.2`.Future release qualification on `main` must not depend on a warm worker. Onthe canonical 8-vCPU, three-shard PVE plan, the backend runner reserves two-single-CPU package workers for the non-API graph, gives the API shards a-volume-weighted `4/1/1` allocation, passes `-p 2` to `go test`, and sets-`GOMAXPROCS=1` for every non-API test binary. The combined declared width is+single-CPU package workers initially for the non-API graph and gives the API+shards a volume-weighted `4/1/1` allocation. Each whole non-API package runs+with `-p 1` and `GOMAXPROCS=1`. The combined declared width istherefore eight CPUs instead of eight API CPUs plus unbounded non-API packageconcurrency. EachAPI shard has a 45-minute cumulative watchdog, the non-API graph retains its@@ -4983,7 +4983,7 @@ evidence for that outer budget: all three race-enabled API shards passed inpackages until the former 55-minute ceiling canceled it during the expandedsecure-runtime install-test package. The added headroom does not weaken orextend either inner watchdog. Constrained-one-vCPU hosts run the non-API graph before the API shard rather than knowingly+one-vCPU hosts run the API shard and whole packages serially rather than knowinglyoversubscribing the host. The deterministic CPU allocator and release guardspin this partition and reject a job ceiling that could pre-empt the APIwatchdog.@@ -4999,10 +4999,26 @@ failed closed, release `378812881` remained an unpublished draft, andconvergence run `33220806414` changed no customer pointers. This is measuredpartition imbalance rather than a hung product test. Four-vCPU release workersmust therefore use the same three-way cost boundary as the PVE worker, with a-non-oversubscribed `1/1/1` API allocation plus one non-API package worker. The+non-oversubscribed `1/1/1` API allocation plus one initial non-API package worker. Theexisting 10-GiB admission floor still degrades the shard count on constrainedhosts instead of weakening the watchdog.+Beta.3 backend job `123003682684` at `d8402d1da91e` completed all three API+shards in 601, 833 and 1,007 seconds, but the non-API graph retained only one+worker until the enclosing 70-minute job was cancelled. The runner now releases+each completed API shard's actual width to queued whole packages. It never+exceeds the original vCPU budget; API roots, race/vet checks, 45-minute API and+30-minute per-package deadlines, and the 70-minute hosted ceiling are unchanged.+Source `go list` must complete before any runtime child starts. Existing package+weights only order the complete non-API inventory, longest first. Every package+gets a separate data directory, and any failed child fails the gate and cleans+up its siblings. `test_release_backend_queue.py`, discovered by script smoke,+executes the actual runner with controlled children at one, four and eight CPUs,+including partial discovery and both branch failures. The hosted artifact+retains source-listed package/API coverage and actual start/finish CPU leases;+missing terminal entries are not passes. This repairs a scheduling fault, not+a product defect diagnosis or qualification of the interrupted candidate.+Activation-only recovery must distinguish the immutable binary packet fromgoverned release-note presentation sidecars. Release run `33223712880` passedits immutable readiness join and retained draft release `378812881`, but itsdiff --git a/scripts/installtests/build_release_assets_test.go b/scripts/installtests/build_release_assets_test.goindex 60c1f80590..4936643fe5 100644--- a/scripts/installtests/build_release_assets_test.go+++ b/scripts/installtests/build_release_assets_test.go@@ -4859,13 +4859,16 @@ func TestReleaseBackendRaceGateUsesCompleteWorkerPartition(t *testing.T) {"SHARD_GOMAXPROCS",`GOMAXPROCS="$shard_procs"`,"RESERVED_OTHER_PACKAGE_PROCS",- `GOMAXPROCS=1 PULSE_DATA_DIR="$RUN_ROOT/data/other"`,+ `GOMAXPROCS=1 PULSE_DATA_DIR="$package_root"`,`go test -race -p "$OTHER_PACKAGE_PROCS" -timeout 30m`,`API_SHARD_TIMEOUT="${PULSE_BACKEND_API_SHARD_TIMEOUT:-45m}"`,"--shard-boundaries","TestWebSocketOriginAllowsTrustedForwardedHostedOriginIPv6Loopback","TestServerInfoEndpointMethodNotAllowed",`-test.timeout "$API_SHARD_TIMEOUT"`,+ `record_task "$!" "$width" "internal/api:$shard_index"`,+ `free_procs=$((free_procs + task_widths[$completed_pid]))`,+ `wait -n -p completed_pid`,} {if !strings.Contains(backendScript, needle) {t.Fatalf("release backend partition missing coverage contract: %s", needle)diff --git a/scripts/release_control/internal/release_preflight_test.py b/scripts/release_control/internal/release_preflight_test.pyindex e9f757d97f..ee5900b826 100755--- a/scripts/release_control/internal/release_preflight_test.py+++ b/scripts/release_control/internal/release_preflight_test.py@@ -610,7 +610,7 @@ echo CONTINUED)self.assertIn("RESERVED_OTHER_PACKAGE_PROCS", backend)self.assertIn(- 'GOMAXPROCS=1 PULSE_DATA_DIR="$RUN_ROOT/data/other"', backend+ 'GOMAXPROCS=1 PULSE_DATA_DIR="$package_root"', backend)self.assertIn('go test -race -p "$OTHER_PACKAGE_PROCS" -timeout 30m', backenddiff --git a/scripts/run-release-backend-tests.sh b/scripts/run-release-backend-tests.shindex 9b998e1566..90fee9d49b 100755--- a/scripts/run-release-backend-tests.sh+++ b/scripts/run-release-backend-tests.sh@@ -264,30 +264,36 @@ print("\n".join(str(width) for width in widths))PY)RESERVED_OTHER_PACKAGE_PROCS="${CPU_PLAN[0]}"-OTHER_PACKAGE_PROCS="$RESERVED_OTHER_PACKAGE_PROCS"-if [ "$OTHER_PACKAGE_PROCS" -eq 0 ]; then OTHER_PACKAGE_PROCS=1; fiSHARD_GOMAXPROCS=("${CPU_PLAN[@]:1}")if [ "${#SHARD_GOMAXPROCS[@]}" -ne "$API_SHARDS" ]; thenecho "Error: shard GOMAXPROCS plan produced ${#SHARD_GOMAXPROCS[@]} entries for $API_SHARDS shards." >&2exit 4fiecho "API shard GOMAXPROCS: ${SHARD_GOMAXPROCS[*]}"-echo "Non-API package workers: $OTHER_PACKAGE_PROCS (GOMAXPROCS=1 each)"+echo "Initial non-API package workers: $RESERVED_OTHER_PACKAGE_PROCS (GOMAXPROCS=1 each)"API_IMPORT_PATH="$(go list -f '{{.ImportPath}}' ./internal/api)"-mapfile -t OTHER_PACKAGES < <(- go list ./cmd/... ./internal/... ./pkg/... ./scripts/... ./tests/... \- | awk -v api="$API_IMPORT_PATH" '$0 != api'-)+mkdir -p "$RUN_ROOT/coverage"+# Do not hide a failed/partial go list in process substitution. Source defines+# the complete inventory; the existing weights only put slow packages first.+go list ./cmd/... ./internal/... ./pkg/... ./scripts/... ./tests/... > "$RUN_ROOT/packages.txt"+awk -v api="$API_IMPORT_PATH" '$0 != api' "$RUN_ROOT/packages.txt" \+ | python3 .github/scripts/select-go-package-shard.py \+ .github/scripts/go-package-test-seconds.txt 1 0 > "$RUN_ROOT/coverage/non-api-packages.txt"+mapfile -t OTHER_PACKAGES < "$RUN_ROOT/coverage/non-api-packages.txt"if [ "${#OTHER_PACKAGES[@]}" -eq 0 ]; thenecho "Error: no non-API backend test packages were discovered." >&2exit 4fi-run_other_packages() {- echo "Running ${#OTHER_PACKAGES[@]} non-API packages..."- env GOMAXPROCS=1 PULSE_DATA_DIR="$RUN_ROOT/data/other" \- go test -race -p "$OTHER_PACKAGE_PROCS" -timeout 30m "${OTHER_PACKAGES[@]}"+run_other_package() {+ local package_index="$1" package="$2"+ # Separate package processes must not race on the old shared data directory.+ local package_root="$RUN_ROOT/data/other/$package_index"+ local OTHER_PACKAGE_PROCS=1+ mkdir -p "$package_root"+ env GOMAXPROCS=1 PULSE_DATA_DIR="$package_root" \+ go test -race -p "$OTHER_PACKAGE_PROCS" -timeout 30m "$package"}run_api_shard() {@@ -319,23 +325,6 @@ run_api_shard() {echo "Completed internal/api shard $shard_index in $((SECONDS - shard_started_seconds))s."}-pids=()-# A one-vCPU host cannot reserve a concurrent package slot. Complete the-# non-API graph first there; release workers have enough width for concurrent-# execution through the normal path below.-if [ "$RESERVED_OTHER_PACKAGE_PROCS" -eq 0 ]; then- run_other_packages-else- run_other_packages &- pids+=("$!")-fi-for shard_number in $(seq 1 "$API_SHARDS"); do- shard_index="$(printf '%02d' "$shard_number")"- run_api_shard "$shard_index" "${SHARD_GOMAXPROCS[$((shard_number - 1))]}" &- pids+=("$!")-done--remaining="${#pids[@]}"terminate_tree() {local parent_pid="$1"local child_pid@@ -347,17 +336,68 @@ terminate_tree() {kill -TERM "$parent_pid" >/dev/null 2>&1 || true}-while [ "$remaining" -gt 0 ]; do- if wait -n; then- remaining=$((remaining - 1))- else- status=$?- for pid in "${pids[@]}"; do- terminate_tree "$pid"- done- wait "${pids[@]}" >/dev/null 2>&1 || true- exit "$status"+pids=()+declare -A task_widths=() task_names=() task_starts=()+free_procs="$VCPUS"+stop_tasks() {+ local pid+ for pid in "${pids[@]}"; do terminate_tree "$pid"; done+ if [ "${#pids[@]}" -gt 0 ]; then wait "${pids[@]}" >/dev/null 2>&1 || true; fi+}+trap stop_tasks EXIT+trap 'exit 130' INT+trap 'exit 143' TERM+printf 'event\tpid\twidth\tseconds\texit\tpackage\n' > "$RUN_ROOT/coverage/execution.tsv"+cp "$PLAN_DIR/manifest.json" "$RUN_ROOT/coverage/api-plan.json"+printf 'vcpus=%s\napi_widths=%s\ninitial_other=%s\n' \+ "$VCPUS" "${SHARD_GOMAXPROCS[*]}" "$RESERVED_OTHER_PACKAGE_PROCS" > "$RUN_ROOT/coverage/cpu-plan.txt"++record_task() {+ local pid="$1" width="$2" name="$3"+ pids+=("$pid")+ task_widths[$pid]="$width"+ task_names[$pid]="$name"+ task_starts[$pid]="$SECONDS"+ free_procs=$((free_procs - width))+ printf 'start\t%s\t%s\t%s\t-\t%s\n' "$pid" "$width" "$SECONDS" "$name" >> "$RUN_ROOT/coverage/execution.tsv"+}++# Beta.3 job 123003682684 finished all API shards in 17 minutes, but kept+# the other 169 packages on one slot until the 70-minute job was cancelled.+# A completed shard now releases its actual CPU width to whole packages.+# No extra runner, wider CPU envelope, changed root order or timeout is needed.+for shard_number in $(seq 1 "$API_SHARDS"); do+ shard_index="$(printf '%02d' "$shard_number")"+ width="${SHARD_GOMAXPROCS[$((shard_number - 1))]}"+ run_api_shard "$shard_index" "$width" &+ record_task "$!" "$width" "internal/api:$shard_index"+done++next_package=0+echo "Running ${#OTHER_PACKAGES[@]} non-API packages with reusable CPU slots..."+while [ "$next_package" -lt "${#OTHER_PACKAGES[@]}" ] || [ "${#pids[@]}" -gt 0 ]; do+ while [ "$free_procs" -gt 0 ] && [ "$next_package" -lt "${#OTHER_PACKAGES[@]}" ]; do+ package="${OTHER_PACKAGES[$next_package]}"+ run_other_package "$next_package" "$package" &+ record_task "$!" 1 "$package"+ next_package=$((next_package + 1))+ done+ status=0+ wait -n -p completed_pid "${pids[@]}" || status=$?+ if [ -z "${completed_pid:-}" ]; then+ echo "Error: backend child completion was not observed (exit $status)." >&2+ exit 4fi+ printf 'finish\t%s\t%s\t%s\t%s\t%s\n' "$completed_pid" "${task_widths[$completed_pid]}" \+ "$((SECONDS - task_starts[$completed_pid]))" "$status" "${task_names[$completed_pid]}" >> "$RUN_ROOT/coverage/execution.tsv"+ free_procs=$((free_procs + task_widths[$completed_pid]))+ active=()+ for pid in "${pids[@]}"; do+ if [ "$pid" != "$completed_pid" ]; then active+=("$pid"); fi+ done+ pids=("${active[@]}")+ unset 'task_widths[$completed_pid]' 'task_names[$completed_pid]' 'task_starts[$completed_pid]'+ if [ "$status" -ne 0 ]; then exit "$status"; fidoneecho "Race-enabled backend release tests passed with $API_SHARDS API shard(s)."diff --git a/scripts/tests/test_release_backend_queue.py b/scripts/tests/test_release_backend_queue.pynew file mode 100644index 0000000000..44ed5a7626--- /dev/null+++ b/scripts/tests/test_release_backend_queue.py@@ -0,0 +1,167 @@+#!/usr/bin/env python3+"""Execute the release queue with controlled children, not a second scheduler."""++import csv+import json+import os+from pathlib import Path+import shutil+import subprocess+import tempfile+import unittest+++ROOT = Path(__file__).resolve().parents[2]+MODULE = "github.com/rcourtman/pulse-go-rewrite/"+PACKAGES = [MODULE + name for name in (+ "scripts/installtests", "internal/monitoring", "pkg/new-package",+ "cmd/example", "tests/example", "internal/example")]++# All subprocess work is synthetic. The actual shell runner owns discovery,+# API partitioning, CPU leases, command flags, data paths and failure cleanup.+CHILD = r'''#!/usr/bin/env python3+import fcntl, json, os, re, sys, time+from pathlib import Path++root = Path(os.environ["QUEUE_FIXTURE"])+args = sys.argv[1:]+names = ["TestPrefix%02d" % n for n in range(20)] + [+ "TestWebSocketOriginAllowsTrustedForwardedHostedOriginIPv6Loopback",+ "TestServerInfoEndpointMethodNotAllowed", "TestTail"]+if args[:1] == ["list"]:+ if "-f" in args:+ print("github.com/rcourtman/pulse-go-rewrite/internal/api")+ else:+ for name in json.loads((root / "packages.json").read_text()): print(name)+ sys.exit(int(os.environ.get("QUEUE_LIST_EXIT", "0")))+ sys.exit(0)+if args[:2] == ["test", "-c"]:+ target = Path(args[args.index("-o") + 1])+ target.write_bytes(Path(__file__).read_bytes()); target.chmod(0o755)+ sys.exit(0)+if args[:1] == ["build"]: sys.exit(0)+if args[:1] == ["-test.list"]:+ print("\n".join(names)); sys.exit(0)+if args[:1] == ["-test.run"]:+ assert args[2:] == ["-test.timeout", "45m"], args+ selected = [n for n in names if re.fullmatch(args[1], n)]+ assert selected+ name = "api:" + selected[0]+ delay = .6 if selected[0].startswith("TestPrefix") else .15+else:+ assert args[:7] == ["test", "-race", "-p", "1", "-timeout", "30m", args[-1]], args+ assert len(args) == 7 and os.environ["GOMAXPROCS"] == "1"+ selected = []+ name = args[-1]+ # The first ordinary package remains busy after a short API shard ends.+ delay = .7 if name.endswith("scripts/installtests") else .25+assert os.environ["CI"] == os.environ["GITHUB_ACTIONS"] == "true"+def record(event):+ with (root / "children.jsonl").open("a") as f:+ fcntl.flock(f, fcntl.LOCK_EX)+ f.write(json.dumps(dict(event=event, name=name, pid=os.getpid(),+ width=int(os.environ["GOMAXPROCS"]), selected=selected,+ data=os.environ["PULSE_DATA_DIR"], time=time.monotonic())) + "\n")+record("start")+time.sleep(delay)+record("finish")+if os.environ.get("QUEUE_FAIL") in (name, "api" if selected else "other"):+ sys.exit(19)+'''+++class ReleaseBackendQueueTest(unittest.TestCase):+ def run_queue(self, cpus=4, packages=None, **extra):+ with tempfile.TemporaryDirectory() as directory:+ root = Path(directory)+ for name in ("scripts", ".github/scripts", "internal/api", "bin"):+ (root / name).mkdir(parents=True)+ for name in ("scripts/run-release-backend-tests.sh", "scripts/shard_go_tests.py",+ ".github/scripts/select-go-package-shard.py",+ ".github/scripts/go-package-test-seconds.txt"):+ shutil.copyfile(ROOT / name, root / name)+ assets = root / "scripts/ensure_test_assets.sh"+ assets.write_text("#!/bin/bash\nexit 0\n"); assets.chmod(0o755)+ for name, text in {+ "go": CHILD,+ "git": '#!/bin/bash\nprintf "%s\\n" "$QUEUE_FIXTURE"\n',+ "getconf": f"#!/bin/bash\necho {cpus}\n",+ }.items():+ path = root / "bin" / name+ path.write_text(text); path.chmod(0o755)+ (root / "packages.json").write_text(json.dumps(+ packages if packages is not None else PACKAGES + [MODULE + "internal/api"]))+ result = subprocess.run(+ ["bash", str(root / "scripts/run-release-backend-tests.sh"),+ "--data-root", str(root / "data"), "--api-shards", str(min(3, cpus))],+ cwd=root, capture_output=True, text=True, timeout=20,+ env={**os.environ, "PATH": str(root / "bin") + ":" + os.environ["PATH"],+ "QUEUE_FIXTURE": str(root), "PULSE_BACKEND_TEST_SHARD_BOUNDARIES": "",+ "PULSE_BACKEND_TEST_SHARD_WEIGHTS": "", **extra})+ events = root / "children.jsonl"+ events = [json.loads(line) for line in events.read_text().splitlines()] if events.exists() else []+ coverage = root / "data/release-backend-run/coverage"+ receipts = list(csv.DictReader((coverage / "execution.tsv").read_text().splitlines(), delimiter="\t")) if (coverage / "execution.tsv").exists() else []+ selected = (coverage / "non-api-packages.txt").read_text().splitlines() if (coverage / "non-api-packages.txt").exists() else []+ for event in events:+ if event["event"] == "start":+ with self.assertRaises(ProcessLookupError, msg="runner left a child alive"):+ os.kill(event["pid"], 0)+ return result, events, receipts, selected++ def test_complete_source_inventory_reuses_finished_api_width(self):+ for cpus in (1, 4, 8):+ with self.subTest(cpus=cpus):+ result, events, receipts, selected = self.run_queue(cpus)+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)+ self.assertEqual(set(selected), set(PACKAGES))+ starts = [e for e in events if e["event"] == "start"]+ ordinary = [e for e in starts if not e["name"].startswith("api:")]+ self.assertEqual(sorted(e["name"] for e in ordinary), sorted(PACKAGES))+ self.assertEqual(len({e["data"] for e in ordinary}), len(PACKAGES))+ roots = [n for e in starts for n in e["selected"]]+ self.assertEqual(len(roots), len(set(roots)))+ self.assertEqual(len(roots), 23)+ # Validate the parent's actual lease journal as well as the+ # controlled children's independently recorded live widths.+ used = peak = 0+ for row in receipts:+ used += int(row["width"]) * (1 if row["event"] == "start" else -1)+ peak = max(peak, used)+ self.assertLessEqual(used, cpus)+ self.assertGreaterEqual(used, 0)+ if row["event"] == "finish": self.assertEqual(row["exit"], "0")+ self.assertEqual(used, 0)+ self.assertEqual(peak, cpus)+ used = 0+ for event in sorted(events, key=lambda e: e["time"]):+ used += event["width"] * (1 if event["event"] == "start" else -1)+ self.assertLessEqual(used, cpus)+ self.assertEqual(used, 0)+ if cpus == 4:+ prefix_end = next(e["time"] for e in events if e["event"] == "finish" and e["name"].startswith("api:TestPrefix"))+ # Old scheduling leaves just the first ordinary package+ # running here; released API slots must admit the tail.+ self.assertGreater(len([e for e in ordinary if e["time"] < prefix_end]), 1)+ self.assertIn("Race-enabled backend release tests passed", result.stdout)++ def test_failed_partial_or_duplicate_discovery_starts_no_test_child(self):+ for packages, env in [(PACKAGES, {"QUEUE_LIST_EXIT": "23"}),+ (PACKAGES + [PACKAGES[0]], {}), ([], {})]:+ with self.subTest(packages=packages, env=env):+ result, events, _, _ = self.run_queue(packages=packages, **env)+ self.assertNotEqual(result.returncode, 0)+ self.assertEqual(events, [])+ self.assertNotIn("Race-enabled backend release tests passed", result.stdout)++ def test_failure_of_either_branch_fails_and_cleans_up(self):+ for branch in ("api", "other"):+ with self.subTest(branch=branch):+ result, _, receipts, _ = self.run_queue(QUEUE_FAIL=branch)+ self.assertEqual(result.returncode, 19, result.stdout + result.stderr)+ self.assertTrue(any(row["exit"] == "19" for row in receipts))+ self.assertNotIn("Race-enabled backend release tests passed", result.stdout)+++if __name__ == "__main__":+ unittest.main()