Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix(ubicloud): reap destroys only the caller's own VMs; pack, usage and reap on macOS and an empty project (port of gstack #3756, #3746)
garrytan994140ea4e08
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6a8161aa25..645924a501 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -40,6 +40,20 @@ identifiers and attribution are available in the pre-removal Git revision- `scripts/pregate-remote-gate.sh`: `--list` refuses (no inventory) for a tree that changes `test.yml`, `e2e.yml`, `ci-reuse.ts`, `ci-full-suite-tree.sh`, `ci-executed-counts.ts` or `lib/gh-issue.ts` relative to `origin/master`, when there is no `origin/master`, and under `GBRAIN_PREGATE_REUSE=0`. The reuse path and `--list` share one check. `test/scripts/pregate-remote-gate.test.ts` pins this.- `docs/ci-event-parity.md`: the profiles and the refusal.+**`ubi-runner.sh reap` now destroys only the caller's own VMs, and pack, usage and reap work on macOS and on an empty project (gstack #3756, #3746).** Reap used to destroy any owner's VM whose TTL had ended. That is how one project's `up` destroyed another lane's VMs (GST-324). Now it destroys only VMs whose owner tag equals `UBI_OWNER` and logs `reap: left N expired VMs of other owners` for the rest. The operator reaper, `reap-expired.ts`, still covers every owner. gbrain keeps `UBI_REAP=0` by default, so `up` still reaps nothing unless asked. On macOS, `pack` exited 1 under bsdtar after the VM was already created. On a project with no VMs, Ubicloud's bare-newline list read as a VM named `-`, and reap died on it. A failed size read made `usage` print a short total, which the quota preflight took as room.++### Itemized changes++- `scripts/ubicloud/ubi-runner.sh`, ported from gstack 74836e41 and 693ee713:+ - `cmd_reap` checks the owner tag first and counts other owners' expired VMs.+ - `vm_rows` skips a blank row.+ - `cmd_usage` exits 1 with no total when the list or a size read fails, and skips a VM whose `show` answers 404.+ - `quota_preflight` retries an unreadable usage while `UBI_QUOTA_WAIT` lasts, then logs it and leaves the decision to the API.+ - `cmd_pack` builds `files.tar`, appends `.git` with `tar -rf` and gzips it, with `COPYFILE_DISABLE=1` on every tar call.+- `test/scripts/ubi-runner-run.test.ts` and `test/scripts/ubi-runner.test.ts` pin the owner-scoped reap, the empty project, the unreadable usage and `COPYFILE_DISABLE`. All eight new cases fail on the old runner. The reap cases that expected another owner's VM destroyed now expect it kept.+- `test/scripts/ubi-runner-bsd-userland.test.ts` runs `pack` under bsdtar. It fails on the old runner and is skipped where no bsdtar is installed. `test/helpers/mock-ubicloud-api.ts` gains `listBody`, `ghosts` and a `show` fault.+- `scripts/ubicloud/README.md`: reap's owner scope.+## [0.60.544.0] - 2026-11-02**The remote gate stops waiting on an artifact GitHub has refused for good, and bench and eval scripts no longer fail pregate for having no test lane (#8370, #7690).** When reading a full run's `executed-identities` artifact failed, `fetchArtifactJson` marked every failure transient, so a 403 or an expired artifact's 410 kept the gate polling for up to 90 minutes before it fell back to VMs. Only no response, a 5xx, a 429 or a rate-limit 403 is transient now. Any other 4xx refuses reuse at once and the gate goes straight to VMs. This never made a gate pass, because a transient read was always `none` and never `reuse`. Separately, `scripts/bench/**` and the system-one eval runners and generators now declare that they need no test lane, so a PR that touches them no longer fails pregate with `PREGATE_NO_LANE`. The unit-lane PostgreSQL arms job also gets real headroom back. Its committed weights had drifted about 24% below what the files now take, so by measured times the busiest of master's 13 shards takes 694 s against its 570 s budget.diff --git a/scripts/ubicloud/README.md b/scripts/ubicloud/README.mdindex f64a24e55a..46e907c0d8 100644--- a/scripts/ubicloud/README.md+++ b/scripts/ubicloud/README.md@@ -42,9 +42,10 @@ two-hour TTL (`--ttl-hours N` changes it, up to 999). `ubi-runner.sh up` and(gstack's default, 3 hours), so give a long dev, benchmark or eval VM`--ttl-hours` above its longest run. With `-n NAME`, `--ttl-hours H` appends`-t<H>`, and a `ubirun-` name without a TTL is refused (nothing could ever reap it).-`ubi-runner.sh reap` destroys every VM whose TTL ended over ten minutes ago,-any owner's, plus the caller's own TTL-less `ubirun-<owner>-*` VMs once they are-`UBI_LEGACY_HOURS` (4) old. gstack's `up` runs that reap first; gbrain's copy+`ubi-runner.sh reap` destroys only the caller's own VMs: its `ubirun-<owner>-*`+VMs whose TTL ended over ten minutes ago, and its TTL-less ones once they are+`UBI_LEGACY_HOURS` (4) old. Another owner's expired VM is only counted in reap's+log; `reap-expired.ts` is the operator reaper that destroys expired VMs of every owner. gstack's `up` runs that reap first; gbrain's copydoes only with `UBI_REAP=1` (or `true`, `yes`, `on`; `0`, `false`, `no` and `off`turn it off, and any other value is refused), and logs one `reap skipped` line otherwise, untilgbrain's TTL-less VMs older than four hours are gone. `run`, `verify` anddiff --git a/scripts/ubicloud/ubi-runner.sh b/scripts/ubicloud/ubi-runner.shindex 5b1d165644..b0033ff355 100755--- a/scripts/ubicloud/ubi-runner.sh+++ b/scripts/ubicloud/ubi-runner.sh@@ -4,7 +4,8 @@# UBICLOUD_API_KEY=... scripts/ubicloud/ubi-runner.sh run --setup S.sh -- 'make test'## Vendored from gstack's scripts/ubicloud/ubi-runner.sh (synced at gstack-# 5532345a, v1.91.341.0). UBICLOUD_API_TOKEN is accepted as an alias for+# 5532345a, v1.91.341.0, plus gstack #3746 at 74836e41 and #3756 at 693ee713:+# macOS pack and usage, empty projects, owner-scoped reap). UBICLOUD_API_TOKEN is accepted as an alias for# UBICLOUD_API_KEY. SSH connections are multiplexed per VM, and `pack` /# `unpack` upload one prebuilt checkout tarball to many VMs.#@@ -62,15 +63,16 @@## TTL and reaper: a name this script generates ends in -t<hours> (UBI_TTL_HOURS,# 1-999, default 3), the creator's own declaration of how long the VM may live.-# `reap` destroys every VM in the project whose TTL expired more than ten-# minutes ago, whichever owner made it, and this owner's own ubirun-<owner>-*-# names with no TTL once they are UBI_LEGACY_HOURS old (default 4; another-# owner's or an untagged TTL-less VM is never reaped); every `up` runs it first when UBI_REAP=1-# (gbrain: off by default until gbrain's TTL-less VMs older than 4 h are gone;-# the operator reaper, scripts/ubicloud/reap-expired.ts, still destroys expired-# TTL VMs), so a runner killed by a sleeping machine leaves nothing behind for-# longer than its TTL. --ttl-hours H (gbrain) sets one VM's TTL.-# Names outside the ubirun- prefix are never reaped.+# `reap` destroys only this owner's VMs: its ubirun-<owner>-* names whose TTL+# expired more than ten minutes ago, and those with no TTL once they are+# UBI_LEGACY_HOURS old (default 4). Another owner's VM, an untagged one and+# names outside the ubirun- prefix are never reaped, expired or not; reap only+# counts other owners' expired VMs in its log (cloud deletes stay owner-scoped,+# GST-573). Every `up` runs it first when UBI_REAP=1 (gbrain: off by default+# until gbrain's TTL-less VMs older than 4 h are gone; the operator reaper,+# scripts/ubicloud/reap-expired.ts, still destroys expired TTL VMs of every+# owner), so a lane killed by a sleeping machine has its leftovers destroyed by+# its own next `up`. --ttl-hours H (gbrain) sets one VM's TTL.## Leftover line: `run` and `verify` end with# UBICLOUD: left=<0|1|unknown> owner=<owner> vm=<name> others=<n>@@ -358,6 +360,7 @@ watched() {vm_rows() {local loc name owner created expirescli vm list -N -f location,name | while read -r loc name; do+ [ -n "$name" ] || continueowner=. created="" expires=-if [[ "$name" =~ ^${PREFIX}-([a-z][a-z0-9]{0,11})-([0-9]{10})- ]]; thenowner=${BASH_REMATCH[1]} created=${BASH_REMATCH[2]}@@ -384,13 +387,12 @@ mine_rows() {done}-# reap: destroy every VM whose name carries a TTL that ended more than-# REAP_GRACE seconds ago, whichever owner created it (the TTL is that owner's-# own declaration), and the caller's own TTL-less ubirun-<UBI_OWNER>-* names at-# least LEGACY_HOURS old. Another owner's TTL-less VM, an untagged one, and names-# outside the prefix are never touched: only a TTL lets reap judge another lane's VM.+# reap: destroy the caller's own VMs whose TTL ended more than REAP_GRACE+# seconds ago, and its TTL-less ubirun-<UBI_OWNER>-* names at least LEGACY_HOURS+# old. Every other VM is never touched, whatever its TTL says: a delete stays+# with the owner that created the VM. Other owners' expired VMs are only counted.cmd_reap() {- local now rows loc name owner expires created n=0 failed=0+ local now rows loc name owner expires created n=0 failed=0 others=0check_seconds REAP_DOWN_TIMEOUT# cmd_down reads DOWN_TIMEOUT; a reap waits at most REAP_DOWN_TIMEOUT per VM.local DOWN_TIMEOUT=$REAP_DOWN_TIMEOUT@@ -399,11 +401,15 @@ cmd_reap() {rows=$(vm_rows) || { log "reap: could not list the project's VMs; nothing destroyed"; return 1; }while read -r loc name owner expires created; do[ -n "$name" ] || continue+ if [ "$owner" != "$OWNER" ]; then+ [ "$expires" = - ] || [ "$now" -lt $(( expires + REAP_GRACE )) ] || others=$(( others + 1 ))+ continue+ fiif [ "$expires" != - ]; then[ "$now" -ge $(( expires + REAP_GRACE )) ] || continuelog "reaping expired $loc/$name (owner $owner; its TTL ended $(( (now - expires) / 60 )) min ago)"else- [ -n "${UBI_OWNER:-}" ] && [ "$owner" = "$OWNER" ] && [ "$created" != - ] || continue+ [ -n "${UBI_OWNER:-}" ] && [ "$created" != - ] || continue[ $(( now - 10#$created )) -ge $(( LEGACY_HOURS * 3600 )) ] || continuelog "reaping TTL-less $loc/$name (owner $owner; $(( (now - 10#$created) / 3600 ))h old, legacy sweep at ${LEGACY_HOURS}h)"fi@@ -411,6 +417,7 @@ cmd_reap() {cmd_down "$name" -l "$loc" || { failed=$(( failed + 1 )); log "failed to reap $loc/$name"; }done <<<"$rows"[ "$n" -gt 0 ] || log "reap: no expired VMs"+ [ "$others" -eq 0 ] || log "reap: left $others expired VMs of other owners; only their owners reap them"[ "$failed" -eq 0 ] || { log "reap: $failed of $n expired VMs could not be destroyed"; return 1; }}@@ -443,15 +450,30 @@ cmd_list() {# usage: VMs and vCPUs per owner, from each VM's size (standard-16 = 16 vCPUs).cmd_usage() {- local tmp loc name owner i=0+ local tmp rows loc name owner f i=0 unread=0+ rows=$(vm_rows) || { log "usage: could not list the project's VMs"; return 1; }tmp=$(mktemp -d)while read -r loc name owner _; do+ [ -n "$name" ] || continuei=$(( i + 1 ))- cli vm "$loc/$name" show 2>/dev/null | sed -n "s/^size: /$owner /p" >"$tmp/$i" &+ {+ case $(api "$tmp/$i.show" vm "$loc/$name" show) in+ 200) sed -n "s/^size: /$owner /p" "$tmp/$i.show" >"$tmp/$i" ;;+ 404) ;;+ *) : >"$tmp/$i" ;;+ esac+ rm -f "$tmp/$i.show"+ } &[ $(( i % 8 )) -ne 0 ] || wait- done < <(vm_rows)+ done <<<"$rows"wait- cat "$tmp"/* 2>/dev/null | awk -v me="$OWNER" '+ for f in "$tmp"/*; do [ ! -e "$f" ] || [ -s "$f" ] || unread=$(( unread + 1 )); done+ if [ "$unread" -gt 0 ]; then+ rm -rf "$tmp"+ log "usage: could not read the size of $unread of $i VMs"+ return 1+ fi+ find "$tmp" -type f -exec cat {} + | awk -v me="$OWNER" '{ n = $2; sub(/^.*-/, "", n); o = ($1 == "-") ? "(untagged)" : ($1 == ".") ? "(other)" : $1vms[o]++; cpu[o] += n; total += n; count++ }END {@@ -542,8 +564,17 @@ quota_preflight() {# between them: a slow first usage read must not shave the printed budget.remaining=$QUOTA_WAITwhile :; do- used=$(cmd_usage 2>/dev/null | awk '$1=="total" {print $3}')- [[ "$used" =~ ^[0-9]+$ ]] || return 0+ if ! used=$(cmd_usage 2>/dev/null | awk '$1=="total" {print $3}') || ! [[ "$used" =~ ^[0-9]+$ ]]; then+ if [ "$remaining" -le 0 ]; then+ log "quota preflight: could not read the project's vCPU usage; the API's own answer decides"+ return 0+ fi+ log "quota: could not read the project's vCPU usage; retrying, up to ${remaining}s more"+ step=$(( remaining < 30 ? remaining : 30 ))+ sleep "$step"+ remaining=$(( remaining - step ))+ continue+ fi[ $(( used + need )) -gt "$max" ] || return 0if [ "$remaining" -le 0 ]; thenlog "quota preflight refused $location/$name ($size): it needs $need vCPUs and the project's VMs already use $used of $max (ceiling from $source; other usage such as managed runners counts on top)"@@ -753,7 +784,12 @@ cmd_down() {# pointer into the main repository, useless on the VM; the same snapshot# replaces it, on the worktree's own branch and index. Not carried: submodule# repositories (.git/modules) and a partial clone's promisor (the fetch pulls-# every missing blob first); gstack has neither.+# every missing blob first); gstack has neither. The archive is built in two+# passes inside the snapshot directory because macOS's bsdtar stops reading+# options at the first operand and resolves -T names after a -C, so no single+# tar command line works on both it and GNU tar (gstack #3680). COPYFILE_DISABLE=1+# keeps Apple's tar from adding ._* AppleDouble entries for files with+# extended attributes, which the VM would see as untracked files.cmd_pack() {local src tmp gitdir branch head entry shared specssrc=$(cd "${1:-.}" && pwd)@@ -800,15 +836,18 @@ cmd_pack() {|| die "pack: could not snapshot the staged blobs of $src"(cd "$src"+ export COPYFILE_DISABLE=1git ls-files -z -co --exclude-standard \| while IFS= read -r -d '' f; doif [ -e "$f" ] || [ -L "$f" ]; then printf '%s\0' "$f"; fidone \- | tar --null -T - -c -C "$tmp" .git -zf -+ | tar --null -T - -cf "$tmp/files.tar" \+ && tar -rf "$tmp/files.tar" -C "$tmp" .git \+ && gzip -c "$tmp/files.tar")rm -rf "$tmp"; trap - EXITelse- tar -C "$src" -czf - .+ COPYFILE_DISABLE=1 tar -C "$src" -czf - .fi}@@ -928,7 +967,7 @@ cmd_run() {trap 'exit 130' INT TERM HUP QUITcmd_up "${up_args[@]}" >/dev/nullif [[ "$name" =~ -t([0-9]+)$ ]]; then- log "$name carries a ${BASH_REMATCH[1]}h TTL (any later run of this script reaps it); if this process dies first: $0 down $name"+ log "$name carries a ${BASH_REMATCH[1]}h TTL (this owner's next up with UBI_REAP=1, or reap, destroys it once it expires); if this process dies first: $0 down $name"elif [[ "$name" =~ ^$PREFIX-$OWNER-[0-9]{10}- ]]; thenlog "$name has no TTL (reap destroys it once it is ${LEGACY_HOURS}h old); if this process dies before its teardown line, destroy it with: $0 down $name"else@@ -1086,8 +1125,8 @@ usage: ubi-runner.sh <command> [args]owner print this caller's owner tag ($OWNER)records VMs this machine created and has not confirmed gone ($VM_RECORD)gc HOURS destroy this owner's VMs older than HOURS (never other owners')- reap destroy every VM whose -t<hours> TTL has ended (any owner) and this- owner's TTL-less ubirun-* VMs older than UBI_LEGACY_HOURS+ reap destroy this owner's VMs whose -t<hours> TTL has ended and its+ TTL-less ubirun-* VMs older than UBI_LEGACY_HOURS (never other owners')cli ARGS... raw Ubicloud CLI passthrough (e.g. cli vm list)VMs are named $PREFIX-<owner>-<epoch>-<suffix>-t<ttl hours>; up, run and verify needdiff --git a/test/helpers/mock-ubicloud-api.ts b/test/helpers/mock-ubicloud-api.tsindex b62a5f63cb..c8ac9d6411 100644--- a/test/helpers/mock-ubicloud-api.ts+++ b/test/helpers/mock-ubicloud-api.ts@@ -4,8 +4,10 @@// the real API), `vm L/N show` (404 once gone) and `vm L/N destroy -f`.// VMs never reach `running`, so every `up` stays mid-provision until killed,// unless `running` is set: then a created VM shows as running at 192.0.2.1-// (pair it with a fake `ssh` on PATH). `faults` makes `destroy` or `list`-// answer HTTP 500 / 503 from the next request on; `destroyFaults` answers the+// (pair it with a fake `ssh` on PATH). `faults` makes `destroy`, `list` or+// `show` answer HTTP 500 / 503 / 503 from the next request on; `listBody`+// replaces the list answer (an empty project answers a bare newline), and+// `ghosts` are listed but answer `show` with 404, like a VM destroyed since; `destroyFaults` answers the// next destroy requests one by one with a status, or holds one past the// client's timeout without acting on it ("hang").@@ -27,6 +29,10 @@ export interface MockUbicloudOptions {running?: boolean;/** Answers for the next destroy requests, in order: an HTTP status, or "hang" (3 s, then a 504 nobody reads). */destroyFaults?: Array<number | "hang">;+ /** The exact `vm list` body, instead of one row per VM. */+ listBody?: string;+ /** Names `vm list` shows after the VMs that `show` answers with 404. */+ ghosts?: string[];seed?: Array<{ location?: string; name: string; size?: string }>;}@@ -35,8 +41,8 @@ export interface MockUbicloud {vms: Map<string, MockVm>;/** "create-start N", "created N", "create-rejected N", "destroy N", "removed N", in order. */events: string[];- /** Flip to make every later `destroy` (500) or `list` (503) fail. */- faults: { destroy: boolean; list: boolean };+ /** Flip to make every later `destroy` (500), `list` (503) or `show` (503) fail. */+ faults: { destroy: boolean; list: boolean; show: boolean };/** Every destroy request received, answered or not. */destroyRequests: number;stop(): void;@@ -45,7 +51,7 @@ export interface MockUbicloud {export function startMockUbicloud(opts: MockUbicloudOptions = {}): MockUbicloud {const vms = new Map<string, MockVm>();const events: string[] = [];- const faults = { destroy: false, list: false };+ const faults = { destroy: false, list: false, show: false };for (const vm of opts.seed ?? []) {vms.set(vm.name, { location: vm.location ?? "eu-central-h1", name: vm.name, size: vm.size ?? "standard-16", state: "creating" });}@@ -59,7 +65,8 @@ export function startMockUbicloud(opts: MockUbicloudOptions = {}): MockUbicloudif (argv[0] !== "vm") return text(`! unsupported: ${argv.join(" ")}`, 400);if (argv[1] === "list") {if (faults.list) return text("! Unexpected response status: 503", 503);- return text([...vms.values()].map((vm) => `${vm.location} ${vm.name}\n`).join(""));+ if (opts.listBody !== undefined) return text(opts.listBody);+ return text([...vms.values(), ...(opts.ghosts ?? []).map((name) => ({ location: "eu-central-h1", name }))].map((vm) => `${vm.location} ${vm.name}\n`).join(""));}const [location, name] = (argv[1] ?? "").split("/") as [string, string];const verb = argv[2];@@ -77,6 +84,7 @@ export function startMockUbicloud(opts: MockUbicloudOptions = {}): MockUbicloud}const vm = vms.get(name);if (verb === "show") {+ if (faults.show) return text("! Unexpected response status: 503", 503);if (!vm) return text("! Unexpected response status: 404", 404);return text(`id: vm-${name}\nname: ${name}\nstate: ${vm.state}\nlocation: ${vm.location}\nsize: ${vm.size}\n${vm.state === "running" ? "ip4: 192.0.2.1\n" : ""}`);}diff --git a/test/scripts/ubi-runner-bsd-userland.test.ts b/test/scripts/ubi-runner-bsd-userland.test.tsnew file mode 100644index 0000000000..259e08f43b--- /dev/null+++ b/test/scripts/ubi-runner-bsd-userland.test.ts@@ -0,0 +1,68 @@+// ubi-runner-bsd-userland.test.ts — scripts/ubicloud/ubi-runner.sh pack under+// the bsdtar macOS ships (ported from gstack's+// test/regression-issue3680-ubicloud-bsd-userland.test.ts, gstack #3746).+//+// Protects: pack writes the whole checkout under bsdtar: tracked, staged,+// untracked, symlinked, spaced and option-like names, a linked worktree and an+// empty file list, with no files.tar left inside the archive.+// Fails when: pack passes tar options after an operand, or resolves its -T list+// after a -C, again (bsdtar then archives .git alone and exits 1).+// Needs: macOS's /usr/bin/tar, or bsdtar on PATH (libarchive-tools); skipped+// without it. usage under the one-true-awk is covered in ubi-runner.test.ts.+// Network: none.++import { describe, expect, it } from "bun:test";+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync } from "node:fs";+import { tmpdir } from "node:os";+import { join, resolve } from "node:path";++const RUNNER = resolve(import.meta.dir, "..", "..", "scripts/ubicloud/ubi-runner.sh");+const bsdTar = process.platform === "darwin" ? "/usr/bin/tar" : Bun.which("bsdtar");+const gitEnv = { GIT_AUTHOR_NAME: "t", GIT_AUTHOR_EMAIL: "t@t", GIT_COMMITTER_NAME: "t", GIT_COMMITTER_EMAIL: "t@t" };++describe(`pack under bsdtar${bsdTar ? "" : ": skipped, no bsdtar (install libarchive-tools)"}`, () => {+ it.skipIf(!bsdTar)("ships tracked, staged, untracked, symlinked, spaced and option-like names, a linked worktree and an empty file list", () => {+ const d = mkdtempSync(join(tmpdir(), "ubi-bsd-pack-"));+ try {+ mkdirSync(join(d, "bsd-bin"));+ symlinkSync(bsdTar!, join(d, "bsd-bin", "tar"));+ const PATH = `${join(d, "bsd-bin")}:${process.env.PATH ?? "/usr/bin:/bin"}`;+ const sh = (cmd: string, cwd: string) => {+ const r = Bun.spawnSync(["bash", "-c", cmd], { cwd, timeout: 30_000, env: { PATH, HOME: d, ...gitEnv } });+ expect(r.exitCode, `${cmd}\n${r.stderr.toString()}`).toBe(0);+ return r.stdout.toString();+ };+ expect(sh("tar --version", d)).toContain("bsdtar");+ const pack = (src: string, out: string) => sh(`bash ${JSON.stringify(RUNNER)} pack ${JSON.stringify(src)} > ${JSON.stringify(out)}`, d);+ const unpack = (tgz: string, to: string) => {+ mkdirSync(join(d, to));+ sh(`tar -xzf ${JSON.stringify(tgz)} -C ${JSON.stringify(to)}`, d);+ return join(d, to);+ };+ const status = (cwd: string) => sh("git status --short", cwd).split("\n").filter(Boolean).sort();++ sh('git init -q plain && cd plain && echo a > a && echo b > gone && echo s > "with space" && ln -s a link && echo o > ./-C && echo o > ./--files-from=a && git add -A && git commit -qm one && rm gone && echo st > staged && git add staged && echo u > untracked && echo x > "untracked space"', d);+ pack("plain", "plain.tgz");+ expect(sh("tar -tzf plain.tgz", d).split("\n").some((l) => l.endsWith("files.tar"))).toBe(false);+ const vm = unpack("plain.tgz", "vm-plain");+ expect(status(vm)).toEqual(status(join(d, "plain")));+ expect(status(vm)).toEqual([" D gone", '?? "untracked space"', "?? untracked", "A staged"]);+ expect(readFileSync(join(vm, "with space"), "utf8")).toBe("s\n");+ expect(sh("readlink link", vm).trim()).toBe("a");+ expect(existsSync(join(vm, "-C")) && existsSync(join(vm, "--files-from=a"))).toBe(true);++ sh("git init -q main && cd main && echo a > a && git add a && git commit -qm one && git branch feat && git worktree add -q ../wt feat", d);+ sh("echo b > b && git add b && git commit -qm two && echo u > u && echo ignored > ig && echo ig > .gitignore", join(d, "wt"));+ pack("wt", "wt.tgz");+ const vmWt = unpack("wt.tgz", "vm-wt");+ expect(existsSync(join(vmWt, "ig"))).toBe(false);+ expect(sh("git rev-parse --abbrev-ref HEAD && git status --short && git log --oneline | wc -l", vmWt).replace(/\s+/g, " ").trim()).toBe("feat ?? .gitignore ?? u 2");++ sh("git init -q empty && cd empty && echo a > a && git add a && git commit -qm one && rm a", d);+ pack("empty", "empty.tgz");+ expect(status(unpack("empty.tgz", "vm-empty"))).toEqual([" D a"]);+ } finally {+ rmSync(d, { recursive: true, force: true });+ }+ }, 60_000);+});diff --git a/test/scripts/ubi-runner-run.test.ts b/test/scripts/ubi-runner-run.test.tsindex 130fb3c217..3763a42006 100644--- a/test/scripts/ubi-runner-run.test.ts+++ b/test/scripts/ubi-runner-run.test.ts@@ -6,8 +6,8 @@//// Protects: up, run and verify refuse without UBI_OWNER before any API call;// gbrain's up reaps nothing unless UBI_REAP=1 and says so in one line; reap-// destroys expired TTL VMs of any owner and only the caller's own TTL-less-// ubirun-<owner>-* VMs past UBI_LEGACY_HOURS; run ends with the UBICLOUD:+// destroys only the caller's own expired TTL VMs and TTL-less+// ubirun-<owner>-* VMs past UBI_LEGACY_HOURS, and counts other owners' expired ones; run ends with the UBICLOUD:// left= line (also in UBI_LEFT_FILE), and a VM it cannot confirm gone fails a// passing command; a silent or overlong remote command is killed with exit// 124, its last output and its remote session ended; --pass/--env and a@@ -132,8 +132,8 @@ describe("owner and reap", () => {// GBRA-108: gbrain still holds TTL-less VMs older than gstack's 4 h legacy// bound, so gbrain's up reaps nothing until UBI_REAP=1, and says so.- it("gbrain's up reaps nothing by default and says so in one line; UBI_REAP=1 reaps expired TTL VMs of any owner first", async () => {- const seed = [{ name: "ubirun-other-1000000000-bbbb-t1" }, { name: "ubirun-t1-1000000000-aaaa" }, { name: `ubirun-peer-${NOW}-cccc-t3` }];+ it("gbrain's up reaps nothing by default and says so in one line; UBI_REAP=1 reaps this owner's expired VMs first and leaves other owners'", async () => {+ const seed = [{ name: "ubirun-other-1000000000-bbbb-t1" }, { name: "ubirun-t1-1000000000-aaaa" }, { name: "ubirun-t1-1000000000-iiii-t1" }, { name: `ubirun-peer-${NOW}-cccc-t3` }];serve({ rejectCreates: true, seed });const def = await run(["up"]);expect(def.code).toBe(1);@@ -146,27 +146,29 @@ describe("owner and reap", () => {expect(creates()).toHaveLength(2);const on = await run(["up"], { UBI_REAP: "1" });expect(on.code).toBe(1);- expect(on.stderr).toMatch(/reaping expired eu-central-h1\/ubirun-other-1000000000-bbbb-t1 \(owner other; its TTL ended \d+ min ago\)/);+ expect(on.stderr).toMatch(/reaping expired eu-central-h1\/ubirun-t1-1000000000-iiii-t1 \(owner t1; its TTL ended \d+ min ago\)/);expect(on.stderr).toMatch(/reaping TTL-less eu-central-h1\/ubirun-t1-1000000000-aaaa \(owner t1; \d+h old, legacy sweep at 4h\)/);+ expect(on.stderr).toContain("reap: left 1 expired VMs of other owners; only their owners reap them");expect(on.stderr).not.toContain("reap skipped");- expect(destroyed()).toEqual(["ubirun-other-1000000000-bbbb-t1", "ubirun-t1-1000000000-aaaa"]);+ expect(destroyed()).toEqual(["ubirun-t1-1000000000-aaaa", "ubirun-t1-1000000000-iiii-t1"]);}, 60_000);- it("reap destroys expired TTLs (and older -x expiries) and this owner's old TTL-less names, never unexpired, recent, another owner's TTL-less, untagged or foreign ones", async () => {+ it("reap destroys this owner's expired TTLs (and older -x expiries) and old TTL-less names, never unexpired, recent, another owner's, untagged or foreign ones", async () => {const seed = [- "ubirun-other-1000000000-bbbb-t1",+ "ubirun-t1-1000000000-bbbb-t1","ubirun-t1-9999999999-dddd-t1","ubirun-t1-1000000000-aaaa","ubirun-1000000000-cccc",`ubirun-t1-${NOW - 3 * 3600}-eeee`,"ubirun-gbra240-1000000000-gggg","team-db-1000000000-ffff",- `ubirun-other-${NOW - 5 * 3600}-x${NOW - 3 * 3600}-hhhh`,+ `ubirun-t1-${NOW - 5 * 3600}-x${NOW - 3 * 3600}-hhhh`,+ "ubirun-other-1000000000-jjjj-t1",].map((name) => ({ name }));const api = serve({ seed });const reap = await run(["reap"]);expect(reap.code).toBe(0);- expect(destroyed()).toEqual(["ubirun-other-1000000000-bbbb-t1", "ubirun-t1-1000000000-aaaa", seed[7]!.name].sort());+ expect(destroyed()).toEqual(["ubirun-t1-1000000000-bbbb-t1", "ubirun-t1-1000000000-aaaa", seed[7]!.name].sort());expect((await run(["reap"])).stderr).toContain("reap: no expired VMs");const lowered = await run(["reap"], { UBI_LEGACY_HOURS: "2" });expect(lowered.code).toBe(0);@@ -174,15 +176,38 @@ describe("owner and reap", () => {const ownerless = await run(["reap"], { UBI_OWNER: "", UBI_LEGACY_HOURS: "1" });expect(ownerless.code).toBe(0);expect(ownerless.stderr).not.toContain("reaping TTL-less");- expect([...api.vms.keys()].sort()).toEqual(["team-db-1000000000-ffff", "ubirun-1000000000-cccc", "ubirun-gbra240-1000000000-gggg", "ubirun-t1-9999999999-dddd-t1"]);+ expect([...api.vms.keys()].sort()).toEqual(["team-db-1000000000-ffff", "ubirun-1000000000-cccc", "ubirun-gbra240-1000000000-gggg", "ubirun-other-1000000000-jjjj-t1", "ubirun-t1-9999999999-dddd-t1"]);+ }, 60_000);++ // GST-573 (gstack #3756): reap destroyed any owner's expired VM; deletes stay with the owner that created the VM.+ it("reap destroys only the caller's own expired and old TTL-less VMs, and only counts other owners' expired ones (GST-573)", async () => {+ const seed = [+ "ubirun-t1-1000000000-aaaa-t1",+ "ubirun-t1-1000000000-bbbb",+ "ubirun-other-1000000000-cccc-t1",+ `ubirun-other-${NOW - 5 * 3600}-x${NOW - 3 * 3600}-dddd`,+ "ubirun-1000000000-eeee-t1",+ `ubirun-peer-${NOW}-ffff-t3`,+ ].map((name) => ({ name }));+ const api = serve({ seed });+ const reap = await run(["reap"]);+ expect(reap.code).toBe(0);+ expect(destroyed()).toEqual(["ubirun-t1-1000000000-aaaa-t1", "ubirun-t1-1000000000-bbbb"]);+ expect(reap.stderr).toContain("reap: left 3 expired VMs of other owners; only their owners reap them");+ expect(reap.stderr).not.toMatch(/reaping \S+\/ubirun-(other|peer|\d)/);+ const other = await run(["reap"], { UBI_OWNER: "other" });+ expect(other.code).toBe(0);+ expect(destroyed()).toEqual(["ubirun-other-1000000000-cccc-t1", seed[3]!.name, "ubirun-t1-1000000000-aaaa-t1", "ubirun-t1-1000000000-bbbb"].sort());+ expect(other.stderr).toContain("reap: left 1 expired VMs of other owners; only their owners reap them");+ expect([...api.vms.keys()].sort()).toEqual(["ubirun-1000000000-eeee-t1", `ubirun-peer-${NOW}-ffff-t3`]);}, 60_000);it("reap reports a failed list and a failed destroy; bad UBI_TTL_HOURS refuses creates and bad UBI_LEGACY_HOURS refuses reap, never read-only commands", async () => {- const api = serve({ seed: [{ name: "ubirun-other-1000000000-bbbb-t1" }] });+ const api = serve({ seed: [{ name: "ubirun-t1-1000000000-bbbb-t1" }] });api.faults.destroy = true;const destroyFails = await run(["reap"], { UBI_REAP_DOWN_TIMEOUT: "2" });expect(destroyFails.code).toBe(1);- expect(destroyFails.stderr).toContain("failed to reap eu-central-h1/ubirun-other-1000000000-bbbb-t1");+ expect(destroyFails.stderr).toContain("failed to reap eu-central-h1/ubirun-t1-1000000000-bbbb-t1");expect(destroyFails.stderr).toContain("reap: 1 of 1 expired VMs could not be destroyed");api.faults.list = true;const listFails = await run(["reap"]);@@ -204,7 +229,7 @@ describe("owner and reap", () => {// #8096 item 3: every UBI_REAP value but 0 used to turn the reap on.it("UBI_REAP=false, no, off (any case) reaps nothing; true, yes, on reap; an unreadable value refuses before any API call", async () => {- serve({ rejectCreates: true, seed: [{ name: "ubirun-other-1000000000-bbbb-t1" }] });+ serve({ rejectCreates: true, seed: [{ name: "ubirun-t1-1000000000-bbbb-t1" }] });for (const value of ["false", "FALSE", "no", "off", "0"]) {const r = await run(["up"], { UBI_REAP: value });expect(r.code).toBe(1);@@ -217,10 +242,10 @@ describe("owner and reap", () => {expect(maybe.stderr).toContain("UBI_REAP must be 1 or 0 (true/false, yes/no and on/off also work), not 'maybe'");expect(creates()).toHaveLength(5);for (const value of ["true", "Yes", "on"]) {- serve({ rejectCreates: true, seed: [{ name: "ubirun-other-1000000000-bbbb-t1" }] });+ serve({ rejectCreates: true, seed: [{ name: "ubirun-t1-1000000000-bbbb-t1" }] });const r = await run(["up"], { UBI_REAP: value });- expect(r.stderr).toMatch(/reaping expired eu-central-h1\/ubirun-other-1000000000-bbbb-t1/);- expect(destroyed()).toEqual(["ubirun-other-1000000000-bbbb-t1"]);+ expect(r.stderr).toMatch(/reaping expired eu-central-h1\/ubirun-t1-1000000000-bbbb-t1/);+ expect(destroyed()).toEqual(["ubirun-t1-1000000000-bbbb-t1"]);}}, 90_000);});@@ -285,13 +310,13 @@ describe("down retries a failed destroy request", () => {}, 60_000);it("reap waits at most UBI_REAP_DOWN_TIMEOUT per VM, not UBI_DOWN_TIMEOUT", async () => {- const api = serve({ seed: [{ name: "ubirun-other-1000000000-bbbb-t1" }, { name: "ubirun-peer-1000000000-cccc-t1" }] });+ const api = serve({ seed: [{ name: "ubirun-t1-1000000000-bbbb-t1" }, { name: "ubirun-t1-1000000000-cccc-t1" }] });api.faults.destroy = true;const started = Date.now();const r = await run(["reap"], { UBI_DOWN_TIMEOUT: "600", UBI_REAP_DOWN_TIMEOUT: "2" });expect(r.code).toBe(1);expect(Date.now() - started).toBeLessThan(30_000);- expect(r.stderr).toContain("could not confirm eu-central-h1/ubirun-other-1000000000-bbbb-t1 is gone after 2s (last destroy request: HTTP 500)");+ expect(r.stderr).toContain("could not confirm eu-central-h1/ubirun-t1-1000000000-bbbb-t1 is gone after 2s (last destroy request: HTTP 500)");expect(r.stderr).toContain("reap: 2 of 2 expired VMs could not be destroyed");for (const bad of ["0", "x"]) expect((await run(["reap"], { UBI_REAP_DOWN_TIMEOUT: bad })).stderr).toContain("UBI_REAP_DOWN_TIMEOUT must be a whole number of seconds from 1");}, 60_000);@@ -594,4 +619,27 @@ describe("quota preflight", () => {expect(name).toMatch(new RegExp(`^${RUN_VM}$`));expect((await run(["down", name])).code).toBe(0);}, 90_000);++ // gstack #3746: an unreadable usage used to read as room and let the create through without a word.+ it("an unreadable usage is retried while UBI_QUOTA_WAIT lasts, then logged, and the create is left to the API", async () => {+ const api = serve({ rejectCreates: true, seed: [{ name: `ubirun-peer-${NOW}-cccc-t3` }] });+ api.faults.list = true;+ const up = await run(["up"], { UBI_QUOTA_VCPUS: "16" });+ expect(up.stderr).toContain("quota preflight: could not read the project's vCPU usage; the API's own answer decides");+ expect(creates()).toHaveLength(1);+ const waited = await run(["up"], { UBI_QUOTA_VCPUS: "16", UBI_QUOTA_WAIT: "2" });+ expect(waited.stderr).toContain("quota: could not read the project's vCPU usage; retrying, up to 2s more");+ expect(waited.stderr).toContain("quota preflight: could not read the project's vCPU usage; the API's own answer decides");+ expect(creates()).toHaveLength(2);+ }, 60_000);++ // gstack #3746 (#3678): the reap before up died on an empty project's bare-newline list.+ it("up with UBI_REAP=1 on an empty project reaps nothing and reaches its create", async () => {+ serve({ rejectCreates: true, listBody: "\n" });+ const up = await run(["up"], { UBI_REAP: "1" });+ expect(up.stderr).toContain("reap: no expired VMs");+ expect(up.stderr).not.toContain("invalid VM name");+ expect(up.stderr).not.toContain("reap failed");+ expect(creates()).toHaveLength(1);+ }, 60_000);});diff --git a/test/scripts/ubi-runner.test.ts b/test/scripts/ubi-runner.test.tsindex 1515c0f9c4..f3d593f4bc 100644--- a/test/scripts/ubi-runner.test.ts+++ b/test/scripts/ubi-runner.test.ts@@ -243,6 +243,42 @@ describe("stale sweep and ownership", () => {});});+// gstack #3746 (#3678): Ubicloud answers `vm list` with a bare newline for a+// project with no VMs, which vm_rows read as a VM named "-"; and a usage whose+// list or size reads failed printed a partial total the quota preflight trusted.+describe("empty project and unreadable usage", () => {+ it("an empty project (vm list answers a bare or blank line) has no VMs: reap destroys nothing, usage prints a zero total, list --mine prints nothing", async () => {+ for (const listBody of ["\n", " \n"]) {+ const api = serve({ listBody });+ const reap = await run(["reap"], { UBI_OWNER: "t1" });+ expect(reap.stderr).toContain("reap: no expired VMs");+ expect(reap.stderr).not.toContain("invalid VM name");+ expect(reap.code).toBe(0);+ const usage = await run(["usage"], { UBI_OWNER: "t1" });+ expect({ code: usage.code, stdout: usage.stdout }).toEqual({ code: 0, stdout: "OWNER VMS VCPUS\ntotal 0 0\n" });+ expect((await run(["list", "--mine"], { UBI_OWNER: "t1" })).stdout).toBe("");+ expect(api.destroyRequests).toBe(0);+ mock?.stop();+ }+ });++ it("usage skips a VM destroyed since the list (404), and exits 1 with no total when a size read or the list fails", async () => {+ const api = serve({ seed: [{ name: `ubirun-peer-${NOW}-cccc-t3` }], ghosts: [`ubirun-gone-${NOW}-dddd-t3`] });+ const ghost = await run(["usage"], { UBI_OWNER: "t1" });+ expect({ code: ghost.code, stdout: ghost.stdout }).toEqual({ code: 0, stdout: "OWNER VMS VCPUS\npeer 1 16\ntotal 1 16\n" });+ api.faults.show = true;+ const noShow = await run(["usage"], { UBI_OWNER: "t1" });+ expect(noShow.code).toBe(1);+ expect(noShow.stdout).not.toContain("total");+ expect(noShow.stderr).toContain("usage: could not read the size of 2 of 2 VMs");+ api.faults.list = true;+ const noList = await run(["usage"], { UBI_OWNER: "t1" });+ expect(noList.code).toBe(1);+ expect(noList.stdout).not.toContain("total");+ expect(noList.stderr).toContain("usage: could not list the project's VMs");+ });+});+// `pack` from a checkout on a branch named like `git init`'s default branch// (master on this repo). slim_git fetches HEAD into that branch inside a fresh// repository whose unborn HEAD names the same branch; without --update-head-ok@@ -276,4 +312,22 @@ describe("pack", () => {expect(execFileSync("git", ["-C", out, "log", "--format=%s"], { encoding: "utf8" }).trim()).toBe("one");expect(() => execFileSync("git", ["-C", out, "cat-file", "-e", dangling], { stdio: "ignore" })).toThrow();});++ // gstack #3746: only Apple's tar reads COPYFILE_DISABLE; without it, files with+ // extended attributes gain ._* entries the VM sees as untracked files.+ it("runs every tar with COPYFILE_DISABLE=1, for a checkout and for a plain directory", () => {+ const realTar = Bun.which("tar");+ expect(realTar).toBeTruthy();+ const bin = join(dir, "tar-shim");+ execFileSync("mkdir", ["-p", bin]);+ writeFileSync(join(bin, "tar"), `#!/usr/bin/env bash\necho "COPYFILE_DISABLE=${"$"}{COPYFILE_DISABLE:-unset}" >> ${JSON.stringify(join(dir, "tar.log"))}\nexec ${JSON.stringify(realTar)} "$@"\n`, { mode: 0o755 });+ const repo = join(dir, "repo");+ execFileSync("git", ["init", "-q", repo]);+ execFileSync("git", ["-C", repo, "-c", "[email protected]", "-c", "user.name=t", "commit", "-q", "--allow-empty", "-m", "one"]);+ writeFileSync(join(repo, "a"), "a\n");+ execFileSync("mkdir", ["-p", join(dir, "plain")]);+ writeFileSync(join(dir, "plain/p"), "p\n");+ for (const src of [repo, join(dir, "plain")]) execFileSync("bash", [RUNNER, "pack", src], { env: env({ PATH: `${bin}:${process.env.PATH ?? "/usr/bin:/bin"}` }), stdio: ["ignore", "ignore", "pipe"] });+ expect(readFileSync(join(dir, "tar.log"), "utf8").trim().split("\n")).toEqual(["COPYFILE_DISABLE=1", "COPYFILE_DISABLE=1", "COPYFILE_DISABLE=1"]);+ });});