Skip to content

armgit / Pulse

Public Imported from GitHub
0

Real-time monitoring dashboard for Proxmox VE, PBS, Docker, Kubernetes, TrueNAS and vSphere. Self-hosted, with alerts for failed backups, full disks, unhealthy containers and offline nodes.

Release completed API capacity to the backend package queue

pulse-triage[bot]32b338c34480
diff --git a/.github/workflows/create-release.yml b/.github/workflows/create-release.yml
index a6dd7ac739..3e4013d5e6 100644
--- a/.github/workflows/create-release.yml
+++ b/.github/workflows/create-release.yml
@@ -599,6 +599,15 @@ jobs:
- name: Run backend tests
run: ./scripts/run-release-backend-tests.sh --data-root "$RUNNER_TEMP/pulse-test-data"
+ - name: Retain backend package coverage and CPU leases
+ if: always()
+ uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
+ with:
+ name: release-backend-coverage-${{ github.sha }}-${{ github.run_attempt }}
+ path: ${{ runner.temp }}/pulse-test-data/release-backend-run/coverage/
+ if-no-files-found: warn
+ retention-days: 7
+
# Alpha and beta builds are feedback checkpoints. RC and stable publication
# run the deeper integration gate because an RC must be promotable in intent.
diff --git a/docs/release-control/v6/internal/subsystems/deployment-installability.md b/docs/release-control/v6/internal/subsystems/deployment-installability.md
index fe6bd678cb..bf910ff33d 100644
--- a/docs/release-control/v6/internal/subsystems/deployment-installability.md
+++ b/docs/release-control/v6/internal/subsystems/deployment-installability.md
@@ -4970,9 +4970,9 @@ release is public as a prerelease, while stable/latest remains `v6.3.2`.
Future release qualification on `main` must not depend on a warm worker. On
the canonical 8-vCPU, three-shard PVE plan, the backend runner reserves two
-single-CPU package workers for the non-API graph, gives the API shards a
-volume-weighted `4/1/1` allocation, passes `-p 2` to `go test`, and sets
-`GOMAXPROCS=1` for every non-API test binary. The combined declared width is
+single-CPU package workers initially for the non-API graph and gives the API
+shards a volume-weighted `4/1/1` allocation. Each whole non-API package runs
+with `-p 1` and `GOMAXPROCS=1`. The combined declared width is
therefore eight CPUs instead of eight API CPUs plus unbounded non-API package
concurrency. Each
API shard has a 45-minute cumulative watchdog, the non-API graph retains its
@@ -4983,7 +4983,7 @@ evidence for that outer budget: all three race-enabled API shards passed in
packages until the former 55-minute ceiling canceled it during the expanded
secure-runtime install-test package. The added headroom does not weaken or
extend either inner watchdog. Constrained
-one-vCPU hosts run the non-API graph before the API shard rather than knowingly
+one-vCPU hosts run the API shard and whole packages serially rather than knowingly
oversubscribing the host. The deterministic CPU allocator and release guards
pin this partition and reject a job ceiling that could pre-empt the API
watchdog.
@@ -4999,10 +4999,26 @@ failed closed, release `378812881` remained an unpublished draft, and
convergence run `33220806414` changed no customer pointers. This is measured
partition imbalance rather than a hung product test. Four-vCPU release workers
must therefore use the same three-way cost boundary as the PVE worker, with a
-non-oversubscribed `1/1/1` API allocation plus one non-API package worker. The
+non-oversubscribed `1/1/1` API allocation plus one initial non-API package worker. The
existing 10-GiB admission floor still degrades the shard count on constrained
hosts instead of weakening the watchdog.
+Beta.3 backend job `123003682684` at `d8402d1da91e` completed all three API
+shards in 601, 833 and 1,007 seconds, but the non-API graph retained only one
+worker until the enclosing 70-minute job was cancelled. The runner now releases
+each completed API shard's actual width to queued whole packages. It never
+exceeds the original vCPU budget; API roots, race/vet checks, 45-minute API and
+30-minute per-package deadlines, and the 70-minute hosted ceiling are unchanged.
+Source `go list` must complete before any runtime child starts. Existing package
+weights only order the complete non-API inventory, longest first. Every package
+gets a separate data directory, and any failed child fails the gate and cleans
+up its siblings. `test_release_backend_queue.py`, discovered by script smoke,
+executes the actual runner with controlled children at one, four and eight CPUs,
+including partial discovery and both branch failures. The hosted artifact
+retains source-listed package/API coverage and actual start/finish CPU leases;
+missing terminal entries are not passes. This repairs a scheduling fault, not
+a product defect diagnosis or qualification of the interrupted candidate.
+
Activation-only recovery must distinguish the immutable binary packet from
governed release-note presentation sidecars. Release run `33223712880` passed
its immutable readiness join and retained draft release `378812881`, but its
diff --git a/scripts/installtests/build_release_assets_test.go b/scripts/installtests/build_release_assets_test.go
index 60c1f80590..4936643fe5 100644
--- a/scripts/installtests/build_release_assets_test.go
+++ b/scripts/installtests/build_release_assets_test.go
@@ -4859,13 +4859,16 @@ func TestReleaseBackendRaceGateUsesCompleteWorkerPartition(t *testing.T) {
"SHARD_GOMAXPROCS",
`GOMAXPROCS="$shard_procs"`,
"RESERVED_OTHER_PACKAGE_PROCS",
- `GOMAXPROCS=1 PULSE_DATA_DIR="$RUN_ROOT/data/other"`,
+ `GOMAXPROCS=1 PULSE_DATA_DIR="$package_root"`,
`go test -race -p "$OTHER_PACKAGE_PROCS" -timeout 30m`,
`API_SHARD_TIMEOUT="${PULSE_BACKEND_API_SHARD_TIMEOUT:-45m}"`,
"--shard-boundaries",
"TestWebSocketOriginAllowsTrustedForwardedHostedOriginIPv6Loopback",
"TestServerInfoEndpointMethodNotAllowed",
`-test.timeout "$API_SHARD_TIMEOUT"`,
+ `record_task "$!" "$width" "internal/api:$shard_index"`,
+ `free_procs=$((free_procs + task_widths[$completed_pid]))`,
+ `wait -n -p completed_pid`,
} {
if !strings.Contains(backendScript, needle) {
t.Fatalf("release backend partition missing coverage contract: %s", needle)
diff --git a/scripts/release_control/internal/release_preflight_test.py b/scripts/release_control/internal/release_preflight_test.py
index e9f757d97f..ee5900b826 100755
--- a/scripts/release_control/internal/release_preflight_test.py
+++ b/scripts/release_control/internal/release_preflight_test.py
@@ -610,7 +610,7 @@ echo CONTINUED
)
self.assertIn("RESERVED_OTHER_PACKAGE_PROCS", backend)
self.assertIn(
- 'GOMAXPROCS=1 PULSE_DATA_DIR="$RUN_ROOT/data/other"', backend
+ 'GOMAXPROCS=1 PULSE_DATA_DIR="$package_root"', backend
)
self.assertIn(
'go test -race -p "$OTHER_PACKAGE_PROCS" -timeout 30m', backend
diff --git a/scripts/run-release-backend-tests.sh b/scripts/run-release-backend-tests.sh
index 9b998e1566..90fee9d49b 100755
--- a/scripts/run-release-backend-tests.sh
+++ b/scripts/run-release-backend-tests.sh
@@ -264,30 +264,36 @@ print("\n".join(str(width) for width in widths))
PY
)
RESERVED_OTHER_PACKAGE_PROCS="${CPU_PLAN[0]}"
-OTHER_PACKAGE_PROCS="$RESERVED_OTHER_PACKAGE_PROCS"
-if [ "$OTHER_PACKAGE_PROCS" -eq 0 ]; then OTHER_PACKAGE_PROCS=1; fi
SHARD_GOMAXPROCS=("${CPU_PLAN[@]:1}")
if [ "${#SHARD_GOMAXPROCS[@]}" -ne "$API_SHARDS" ]; then
echo "Error: shard GOMAXPROCS plan produced ${#SHARD_GOMAXPROCS[@]} entries for $API_SHARDS shards." >&2
exit 4
fi
echo "API shard GOMAXPROCS: ${SHARD_GOMAXPROCS[*]}"
-echo "Non-API package workers: $OTHER_PACKAGE_PROCS (GOMAXPROCS=1 each)"
+echo "Initial non-API package workers: $RESERVED_OTHER_PACKAGE_PROCS (GOMAXPROCS=1 each)"
API_IMPORT_PATH="$(go list -f '{{.ImportPath}}' ./internal/api)"
-mapfile -t OTHER_PACKAGES < <(
- go list ./cmd/... ./internal/... ./pkg/... ./scripts/... ./tests/... \
- | awk -v api="$API_IMPORT_PATH" '$0 != api'
-)
+mkdir -p "$RUN_ROOT/coverage"
+# Do not hide a failed/partial go list in process substitution. Source defines
+# the complete inventory; the existing weights only put slow packages first.
+go list ./cmd/... ./internal/... ./pkg/... ./scripts/... ./tests/... > "$RUN_ROOT/packages.txt"
+awk -v api="$API_IMPORT_PATH" '$0 != api' "$RUN_ROOT/packages.txt" \
+ | python3 .github/scripts/select-go-package-shard.py \
+ .github/scripts/go-package-test-seconds.txt 1 0 > "$RUN_ROOT/coverage/non-api-packages.txt"
+mapfile -t OTHER_PACKAGES < "$RUN_ROOT/coverage/non-api-packages.txt"
if [ "${#OTHER_PACKAGES[@]}" -eq 0 ]; then
echo "Error: no non-API backend test packages were discovered." >&2
exit 4
fi
-run_other_packages() {
- echo "Running ${#OTHER_PACKAGES[@]} non-API packages..."
- env GOMAXPROCS=1 PULSE_DATA_DIR="$RUN_ROOT/data/other" \
- go test -race -p "$OTHER_PACKAGE_PROCS" -timeout 30m "${OTHER_PACKAGES[@]}"
+run_other_package() {
+ local package_index="$1" package="$2"
+ # Separate package processes must not race on the old shared data directory.
+ local package_root="$RUN_ROOT/data/other/$package_index"
+ local OTHER_PACKAGE_PROCS=1
+ mkdir -p "$package_root"
+ env GOMAXPROCS=1 PULSE_DATA_DIR="$package_root" \
+ go test -race -p "$OTHER_PACKAGE_PROCS" -timeout 30m "$package"
}
run_api_shard() {
@@ -319,23 +325,6 @@ run_api_shard() {
echo "Completed internal/api shard $shard_index in $((SECONDS - shard_started_seconds))s."
}
-pids=()
-# A one-vCPU host cannot reserve a concurrent package slot. Complete the
-# non-API graph first there; release workers have enough width for concurrent
-# execution through the normal path below.
-if [ "$RESERVED_OTHER_PACKAGE_PROCS" -eq 0 ]; then
- run_other_packages
-else
- run_other_packages &
- pids+=("$!")
-fi
-for shard_number in $(seq 1 "$API_SHARDS"); do
- shard_index="$(printf '%02d' "$shard_number")"
- run_api_shard "$shard_index" "${SHARD_GOMAXPROCS[$((shard_number - 1))]}" &
- pids+=("$!")
-done
-
-remaining="${#pids[@]}"
terminate_tree() {
local parent_pid="$1"
local child_pid
@@ -347,17 +336,68 @@ terminate_tree() {
kill -TERM "$parent_pid" >/dev/null 2>&1 || true
}
-while [ "$remaining" -gt 0 ]; do
- if wait -n; then
- remaining=$((remaining - 1))
- else
- status=$?
- for pid in "${pids[@]}"; do
- terminate_tree "$pid"
- done
- wait "${pids[@]}" >/dev/null 2>&1 || true
- exit "$status"
+pids=()
+declare -A task_widths=() task_names=() task_starts=()
+free_procs="$VCPUS"
+stop_tasks() {
+ local pid
+ for pid in "${pids[@]}"; do terminate_tree "$pid"; done
+ if [ "${#pids[@]}" -gt 0 ]; then wait "${pids[@]}" >/dev/null 2>&1 || true; fi
+}
+trap stop_tasks EXIT
+trap 'exit 130' INT
+trap 'exit 143' TERM
+printf 'event\tpid\twidth\tseconds\texit\tpackage\n' > "$RUN_ROOT/coverage/execution.tsv"
+cp "$PLAN_DIR/manifest.json" "$RUN_ROOT/coverage/api-plan.json"
+printf 'vcpus=%s\napi_widths=%s\ninitial_other=%s\n' \
+ "$VCPUS" "${SHARD_GOMAXPROCS[*]}" "$RESERVED_OTHER_PACKAGE_PROCS" > "$RUN_ROOT/coverage/cpu-plan.txt"
+
+record_task() {
+ local pid="$1" width="$2" name="$3"
+ pids+=("$pid")
+ task_widths[$pid]="$width"
+ task_names[$pid]="$name"
+ task_starts[$pid]="$SECONDS"
+ free_procs=$((free_procs - width))
+ printf 'start\t%s\t%s\t%s\t-\t%s\n' "$pid" "$width" "$SECONDS" "$name" >> "$RUN_ROOT/coverage/execution.tsv"
+}
+
+# Beta.3 job 123003682684 finished all API shards in 17 minutes, but kept
+# the other 169 packages on one slot until the 70-minute job was cancelled.
+# A completed shard now releases its actual CPU width to whole packages.
+# No extra runner, wider CPU envelope, changed root order or timeout is needed.
+for shard_number in $(seq 1 "$API_SHARDS"); do
+ shard_index="$(printf '%02d' "$shard_number")"
+ width="${SHARD_GOMAXPROCS[$((shard_number - 1))]}"
+ run_api_shard "$shard_index" "$width" &
+ record_task "$!" "$width" "internal/api:$shard_index"
+done
+
+next_package=0
+echo "Running ${#OTHER_PACKAGES[@]} non-API packages with reusable CPU slots..."
+while [ "$next_package" -lt "${#OTHER_PACKAGES[@]}" ] || [ "${#pids[@]}" -gt 0 ]; do
+ while [ "$free_procs" -gt 0 ] && [ "$next_package" -lt "${#OTHER_PACKAGES[@]}" ]; do
+ package="${OTHER_PACKAGES[$next_package]}"
+ run_other_package "$next_package" "$package" &
+ record_task "$!" 1 "$package"
+ next_package=$((next_package + 1))
+ done
+ status=0
+ wait -n -p completed_pid "${pids[@]}" || status=$?
+ if [ -z "${completed_pid:-}" ]; then
+ echo "Error: backend child completion was not observed (exit $status)." >&2
+ exit 4
fi
+ printf 'finish\t%s\t%s\t%s\t%s\t%s\n' "$completed_pid" "${task_widths[$completed_pid]}" \
+ "$((SECONDS - task_starts[$completed_pid]))" "$status" "${task_names[$completed_pid]}" >> "$RUN_ROOT/coverage/execution.tsv"
+ free_procs=$((free_procs + task_widths[$completed_pid]))
+ active=()
+ for pid in "${pids[@]}"; do
+ if [ "$pid" != "$completed_pid" ]; then active+=("$pid"); fi
+ done
+ pids=("${active[@]}")
+ unset 'task_widths[$completed_pid]' 'task_names[$completed_pid]' 'task_starts[$completed_pid]'
+ if [ "$status" -ne 0 ]; then exit "$status"; fi
done
echo "Race-enabled backend release tests passed with $API_SHARDS API shard(s)."
diff --git a/scripts/tests/test_release_backend_queue.py b/scripts/tests/test_release_backend_queue.py
new file mode 100644
index 0000000000..44ed5a7626
--- /dev/null
+++ b/scripts/tests/test_release_backend_queue.py
@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+"""Execute the release queue with controlled children, not a second scheduler."""
+
+import csv
+import json
+import os
+from pathlib import Path
+import shutil
+import subprocess
+import tempfile
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[2]
+MODULE = "github.com/rcourtman/pulse-go-rewrite/"
+PACKAGES = [MODULE + name for name in (
+ "scripts/installtests", "internal/monitoring", "pkg/new-package",
+ "cmd/example", "tests/example", "internal/example")]
+
+# All subprocess work is synthetic. The actual shell runner owns discovery,
+# API partitioning, CPU leases, command flags, data paths and failure cleanup.
+CHILD = r'''#!/usr/bin/env python3
+import fcntl, json, os, re, sys, time
+from pathlib import Path
+
+root = Path(os.environ["QUEUE_FIXTURE"])
+args = sys.argv[1:]
+names = ["TestPrefix%02d" % n for n in range(20)] + [
+ "TestWebSocketOriginAllowsTrustedForwardedHostedOriginIPv6Loopback",
+ "TestServerInfoEndpointMethodNotAllowed", "TestTail"]
+if args[:1] == ["list"]:
+ if "-f" in args:
+ print("github.com/rcourtman/pulse-go-rewrite/internal/api")
+ else:
+ for name in json.loads((root / "packages.json").read_text()): print(name)
+ sys.exit(int(os.environ.get("QUEUE_LIST_EXIT", "0")))
+ sys.exit(0)
+if args[:2] == ["test", "-c"]:
+ target = Path(args[args.index("-o") + 1])
+ target.write_bytes(Path(__file__).read_bytes()); target.chmod(0o755)
+ sys.exit(0)
+if args[:1] == ["build"]: sys.exit(0)
+if args[:1] == ["-test.list"]:
+ print("\n".join(names)); sys.exit(0)
+if args[:1] == ["-test.run"]:
+ assert args[2:] == ["-test.timeout", "45m"], args
+ selected = [n for n in names if re.fullmatch(args[1], n)]
+ assert selected
+ name = "api:" + selected[0]
+ delay = .6 if selected[0].startswith("TestPrefix") else .15
+else:
+ assert args[:7] == ["test", "-race", "-p", "1", "-timeout", "30m", args[-1]], args
+ assert len(args) == 7 and os.environ["GOMAXPROCS"] == "1"
+ selected = []
+ name = args[-1]
+ # The first ordinary package remains busy after a short API shard ends.
+ delay = .7 if name.endswith("scripts/installtests") else .25
+assert os.environ["CI"] == os.environ["GITHUB_ACTIONS"] == "true"
+def record(event):
+ with (root / "children.jsonl").open("a") as f:
+ fcntl.flock(f, fcntl.LOCK_EX)
+ f.write(json.dumps(dict(event=event, name=name, pid=os.getpid(),
+ width=int(os.environ["GOMAXPROCS"]), selected=selected,
+ data=os.environ["PULSE_DATA_DIR"], time=time.monotonic())) + "\n")
+record("start")
+time.sleep(delay)
+record("finish")
+if os.environ.get("QUEUE_FAIL") in (name, "api" if selected else "other"):
+ sys.exit(19)
+'''
+
+
+class ReleaseBackendQueueTest(unittest.TestCase):
+ def run_queue(self, cpus=4, packages=None, **extra):
+ with tempfile.TemporaryDirectory() as directory:
+ root = Path(directory)
+ for name in ("scripts", ".github/scripts", "internal/api", "bin"):
+ (root / name).mkdir(parents=True)
+ for name in ("scripts/run-release-backend-tests.sh", "scripts/shard_go_tests.py",
+ ".github/scripts/select-go-package-shard.py",
+ ".github/scripts/go-package-test-seconds.txt"):
+ shutil.copyfile(ROOT / name, root / name)
+ assets = root / "scripts/ensure_test_assets.sh"
+ assets.write_text("#!/bin/bash\nexit 0\n"); assets.chmod(0o755)
+ for name, text in {
+ "go": CHILD,
+ "git": '#!/bin/bash\nprintf "%s\\n" "$QUEUE_FIXTURE"\n',
+ "getconf": f"#!/bin/bash\necho {cpus}\n",
+ }.items():
+ path = root / "bin" / name
+ path.write_text(text); path.chmod(0o755)
+ (root / "packages.json").write_text(json.dumps(
+ packages if packages is not None else PACKAGES + [MODULE + "internal/api"]))
+ result = subprocess.run(
+ ["bash", str(root / "scripts/run-release-backend-tests.sh"),
+ "--data-root", str(root / "data"), "--api-shards", str(min(3, cpus))],
+ cwd=root, capture_output=True, text=True, timeout=20,
+ env={**os.environ, "PATH": str(root / "bin") + ":" + os.environ["PATH"],
+ "QUEUE_FIXTURE": str(root), "PULSE_BACKEND_TEST_SHARD_BOUNDARIES": "",
+ "PULSE_BACKEND_TEST_SHARD_WEIGHTS": "", **extra})
+ events = root / "children.jsonl"
+ events = [json.loads(line) for line in events.read_text().splitlines()] if events.exists() else []
+ coverage = root / "data/release-backend-run/coverage"
+ receipts = list(csv.DictReader((coverage / "execution.tsv").read_text().splitlines(), delimiter="\t")) if (coverage / "execution.tsv").exists() else []
+ selected = (coverage / "non-api-packages.txt").read_text().splitlines() if (coverage / "non-api-packages.txt").exists() else []
+ for event in events:
+ if event["event"] == "start":
+ with self.assertRaises(ProcessLookupError, msg="runner left a child alive"):
+ os.kill(event["pid"], 0)
+ return result, events, receipts, selected
+
+ def test_complete_source_inventory_reuses_finished_api_width(self):
+ for cpus in (1, 4, 8):
+ with self.subTest(cpus=cpus):
+ result, events, receipts, selected = self.run_queue(cpus)
+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+ self.assertEqual(set(selected), set(PACKAGES))
+ starts = [e for e in events if e["event"] == "start"]
+ ordinary = [e for e in starts if not e["name"].startswith("api:")]
+ self.assertEqual(sorted(e["name"] for e in ordinary), sorted(PACKAGES))
+ self.assertEqual(len({e["data"] for e in ordinary}), len(PACKAGES))
+ roots = [n for e in starts for n in e["selected"]]
+ self.assertEqual(len(roots), len(set(roots)))
+ self.assertEqual(len(roots), 23)
+ # Validate the parent's actual lease journal as well as the
+ # controlled children's independently recorded live widths.
+ used = peak = 0
+ for row in receipts:
+ used += int(row["width"]) * (1 if row["event"] == "start" else -1)
+ peak = max(peak, used)
+ self.assertLessEqual(used, cpus)
+ self.assertGreaterEqual(used, 0)
+ if row["event"] == "finish": self.assertEqual(row["exit"], "0")
+ self.assertEqual(used, 0)
+ self.assertEqual(peak, cpus)
+ used = 0
+ for event in sorted(events, key=lambda e: e["time"]):
+ used += event["width"] * (1 if event["event"] == "start" else -1)
+ self.assertLessEqual(used, cpus)
+ self.assertEqual(used, 0)
+ if cpus == 4:
+ prefix_end = next(e["time"] for e in events if e["event"] == "finish" and e["name"].startswith("api:TestPrefix"))
+ # Old scheduling leaves just the first ordinary package
+ # running here; released API slots must admit the tail.
+ self.assertGreater(len([e for e in ordinary if e["time"] < prefix_end]), 1)
+ self.assertIn("Race-enabled backend release tests passed", result.stdout)
+
+ def test_failed_partial_or_duplicate_discovery_starts_no_test_child(self):
+ for packages, env in [(PACKAGES, {"QUEUE_LIST_EXIT": "23"}),
+ (PACKAGES + [PACKAGES[0]], {}), ([], {})]:
+ with self.subTest(packages=packages, env=env):
+ result, events, _, _ = self.run_queue(packages=packages, **env)
+ self.assertNotEqual(result.returncode, 0)
+ self.assertEqual(events, [])
+ self.assertNotIn("Race-enabled backend release tests passed", result.stdout)
+
+ def test_failure_of_either_branch_fails_and_cleans_up(self):
+ for branch in ("api", "other"):
+ with self.subTest(branch=branch):
+ result, _, receipts, _ = self.run_queue(QUEUE_FAIL=branch)
+ self.assertEqual(result.returncode, 19, result.stdout + result.stderr)
+ self.assertTrue(any(row["exit"] == "19" for row in receipts))
+ self.assertNotIn("Race-enabled backend release tests passed", result.stdout)
+
+
+if __name__ == "__main__":
+ unittest.main()