Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(ubicloud): reap destroys only the caller's own VMs; pack, usage and reap on macOS and an empty project (port of gstack #3756, #3746)

garrytan994140ea4e08
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6a8161aa25..645924a501 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -40,6 +40,20 @@ identifiers and attribution are available in the pre-removal Git revision
- `scripts/pregate-remote-gate.sh`: `--list` refuses (no inventory) for a tree that changes `test.yml`, `e2e.yml`, `ci-reuse.ts`, `ci-full-suite-tree.sh`, `ci-executed-counts.ts` or `lib/gh-issue.ts` relative to `origin/master`, when there is no `origin/master`, and under `GBRAIN_PREGATE_REUSE=0`. The reuse path and `--list` share one check. `test/scripts/pregate-remote-gate.test.ts` pins this.
- `docs/ci-event-parity.md`: the profiles and the refusal.
+**`ubi-runner.sh reap` now destroys only the caller's own VMs, and pack, usage and reap work on macOS and on an empty project (gstack #3756, #3746).** Reap used to destroy any owner's VM whose TTL had ended. That is how one project's `up` destroyed another lane's VMs (GST-324). Now it destroys only VMs whose owner tag equals `UBI_OWNER` and logs `reap: left N expired VMs of other owners` for the rest. The operator reaper, `reap-expired.ts`, still covers every owner. gbrain keeps `UBI_REAP=0` by default, so `up` still reaps nothing unless asked. On macOS, `pack` exited 1 under bsdtar after the VM was already created. On a project with no VMs, Ubicloud's bare-newline list read as a VM named `-`, and reap died on it. A failed size read made `usage` print a short total, which the quota preflight took as room.
+
+### Itemized changes
+
+- `scripts/ubicloud/ubi-runner.sh`, ported from gstack 74836e41 and 693ee713:
+ - `cmd_reap` checks the owner tag first and counts other owners' expired VMs.
+ - `vm_rows` skips a blank row.
+ - `cmd_usage` exits 1 with no total when the list or a size read fails, and skips a VM whose `show` answers 404.
+ - `quota_preflight` retries an unreadable usage while `UBI_QUOTA_WAIT` lasts, then logs it and leaves the decision to the API.
+ - `cmd_pack` builds `files.tar`, appends `.git` with `tar -rf` and gzips it, with `COPYFILE_DISABLE=1` on every tar call.
+- `test/scripts/ubi-runner-run.test.ts` and `test/scripts/ubi-runner.test.ts` pin the owner-scoped reap, the empty project, the unreadable usage and `COPYFILE_DISABLE`. All eight new cases fail on the old runner. The reap cases that expected another owner's VM destroyed now expect it kept.
+- `test/scripts/ubi-runner-bsd-userland.test.ts` runs `pack` under bsdtar. It fails on the old runner and is skipped where no bsdtar is installed. `test/helpers/mock-ubicloud-api.ts` gains `listBody`, `ghosts` and a `show` fault.
+- `scripts/ubicloud/README.md`: reap's owner scope.
+
## [0.60.544.0] - 2026-11-02
**The remote gate stops waiting on an artifact GitHub has refused for good, and bench and eval scripts no longer fail pregate for having no test lane (#8370, #7690).** When reading a full run's `executed-identities` artifact failed, `fetchArtifactJson` marked every failure transient, so a 403 or an expired artifact's 410 kept the gate polling for up to 90 minutes before it fell back to VMs. Only no response, a 5xx, a 429 or a rate-limit 403 is transient now. Any other 4xx refuses reuse at once and the gate goes straight to VMs. This never made a gate pass, because a transient read was always `none` and never `reuse`. Separately, `scripts/bench/**` and the system-one eval runners and generators now declare that they need no test lane, so a PR that touches them no longer fails pregate with `PREGATE_NO_LANE`. The unit-lane PostgreSQL arms job also gets real headroom back. Its committed weights had drifted about 24% below what the files now take, so by measured times the busiest of master's 13 shards takes 694 s against its 570 s budget.
diff --git a/scripts/ubicloud/README.md b/scripts/ubicloud/README.md
index f64a24e55a..46e907c0d8 100644
--- a/scripts/ubicloud/README.md
+++ b/scripts/ubicloud/README.md
@@ -42,9 +42,10 @@ two-hour TTL (`--ttl-hours N` changes it, up to 999). `ubi-runner.sh up` and
(gstack's default, 3 hours), so give a long dev, benchmark or eval VM
`--ttl-hours` above its longest run. With `-n NAME`, `--ttl-hours H` appends
`-t<H>`, and a `ubirun-` name without a TTL is refused (nothing could ever reap it).
-`ubi-runner.sh reap` destroys every VM whose TTL ended over ten minutes ago,
-any owner's, plus the caller's own TTL-less `ubirun-<owner>-*` VMs once they are
-`UBI_LEGACY_HOURS` (4) old. gstack's `up` runs that reap first; gbrain's copy
+`ubi-runner.sh reap` destroys only the caller's own VMs: its `ubirun-<owner>-*`
+VMs whose TTL ended over ten minutes ago, and its TTL-less ones once they are
+`UBI_LEGACY_HOURS` (4) old. Another owner's expired VM is only counted in reap's
+log; `reap-expired.ts` is the operator reaper that destroys expired VMs of every owner. gstack's `up` runs that reap first; gbrain's copy
does only with `UBI_REAP=1` (or `true`, `yes`, `on`; `0`, `false`, `no` and `off`
turn it off, and any other value is refused), and logs one `reap skipped` line otherwise, until
gbrain's TTL-less VMs older than four hours are gone. `run`, `verify` and
diff --git a/scripts/ubicloud/ubi-runner.sh b/scripts/ubicloud/ubi-runner.sh
index 5b1d165644..b0033ff355 100755
--- a/scripts/ubicloud/ubi-runner.sh
+++ b/scripts/ubicloud/ubi-runner.sh
@@ -4,7 +4,8 @@
# UBICLOUD_API_KEY=... scripts/ubicloud/ubi-runner.sh run --setup S.sh -- 'make test'
#
# Vendored from gstack's scripts/ubicloud/ubi-runner.sh (synced at gstack
-# 5532345a, v1.91.341.0). UBICLOUD_API_TOKEN is accepted as an alias for
+# 5532345a, v1.91.341.0, plus gstack #3746 at 74836e41 and #3756 at 693ee713:
+# macOS pack and usage, empty projects, owner-scoped reap). UBICLOUD_API_TOKEN is accepted as an alias for
# UBICLOUD_API_KEY. SSH connections are multiplexed per VM, and `pack` /
# `unpack` upload one prebuilt checkout tarball to many VMs.
#
@@ -62,15 +63,16 @@
#
# TTL and reaper: a name this script generates ends in -t<hours> (UBI_TTL_HOURS,
# 1-999, default 3), the creator's own declaration of how long the VM may live.
-# `reap` destroys every VM in the project whose TTL expired more than ten
-# minutes ago, whichever owner made it, and this owner's own ubirun-<owner>-*
-# names with no TTL once they are UBI_LEGACY_HOURS old (default 4; another
-# owner's or an untagged TTL-less VM is never reaped); every `up` runs it first when UBI_REAP=1
-# (gbrain: off by default until gbrain's TTL-less VMs older than 4 h are gone;
-# the operator reaper, scripts/ubicloud/reap-expired.ts, still destroys expired
-# TTL VMs), so a runner killed by a sleeping machine leaves nothing behind for
-# longer than its TTL. --ttl-hours H (gbrain) sets one VM's TTL.
-# Names outside the ubirun- prefix are never reaped.
+# `reap` destroys only this owner's VMs: its ubirun-<owner>-* names whose TTL
+# expired more than ten minutes ago, and those with no TTL once they are
+# UBI_LEGACY_HOURS old (default 4). Another owner's VM, an untagged one and
+# names outside the ubirun- prefix are never reaped, expired or not; reap only
+# counts other owners' expired VMs in its log (cloud deletes stay owner-scoped,
+# GST-573). Every `up` runs it first when UBI_REAP=1 (gbrain: off by default
+# until gbrain's TTL-less VMs older than 4 h are gone; the operator reaper,
+# scripts/ubicloud/reap-expired.ts, still destroys expired TTL VMs of every
+# owner), so a lane killed by a sleeping machine has its leftovers destroyed by
+# its own next `up`. --ttl-hours H (gbrain) sets one VM's TTL.
#
# Leftover line: `run` and `verify` end with
# UBICLOUD: left=<0|1|unknown> owner=<owner> vm=<name> others=<n>
@@ -358,6 +360,7 @@ watched() {
vm_rows() {
local loc name owner created expires
cli vm list -N -f location,name | while read -r loc name; do
+ [ -n "$name" ] || continue
owner=. created="" expires=-
if [[ "$name" =~ ^${PREFIX}-([a-z][a-z0-9]{0,11})-([0-9]{10})- ]]; then
owner=${BASH_REMATCH[1]} created=${BASH_REMATCH[2]}
@@ -384,13 +387,12 @@ mine_rows() {
done
}
-# reap: destroy every VM whose name carries a TTL that ended more than
-# REAP_GRACE seconds ago, whichever owner created it (the TTL is that owner's
-# own declaration), and the caller's own TTL-less ubirun-<UBI_OWNER>-* names at
-# least LEGACY_HOURS old. Another owner's TTL-less VM, an untagged one, and names
-# outside the prefix are never touched: only a TTL lets reap judge another lane's VM.
+# reap: destroy the caller's own VMs whose TTL ended more than REAP_GRACE
+# seconds ago, and its TTL-less ubirun-<UBI_OWNER>-* names at least LEGACY_HOURS
+# old. Every other VM is never touched, whatever its TTL says: a delete stays
+# with the owner that created the VM. Other owners' expired VMs are only counted.
cmd_reap() {
- local now rows loc name owner expires created n=0 failed=0
+ local now rows loc name owner expires created n=0 failed=0 others=0
check_seconds REAP_DOWN_TIMEOUT
# cmd_down reads DOWN_TIMEOUT; a reap waits at most REAP_DOWN_TIMEOUT per VM.
local DOWN_TIMEOUT=$REAP_DOWN_TIMEOUT
@@ -399,11 +401,15 @@ cmd_reap() {
rows=$(vm_rows) || { log "reap: could not list the project's VMs; nothing destroyed"; return 1; }
while read -r loc name owner expires created; do
[ -n "$name" ] || continue
+ if [ "$owner" != "$OWNER" ]; then
+ [ "$expires" = - ] || [ "$now" -lt $(( expires + REAP_GRACE )) ] || others=$(( others + 1 ))
+ continue
+ fi
if [ "$expires" != - ]; then
[ "$now" -ge $(( expires + REAP_GRACE )) ] || continue
log "reaping expired $loc/$name (owner $owner; its TTL ended $(( (now - expires) / 60 )) min ago)"
else
- [ -n "${UBI_OWNER:-}" ] && [ "$owner" = "$OWNER" ] && [ "$created" != - ] || continue
+ [ -n "${UBI_OWNER:-}" ] && [ "$created" != - ] || continue
[ $(( now - 10#$created )) -ge $(( LEGACY_HOURS * 3600 )) ] || continue
log "reaping TTL-less $loc/$name (owner $owner; $(( (now - 10#$created) / 3600 ))h old, legacy sweep at ${LEGACY_HOURS}h)"
fi
@@ -411,6 +417,7 @@ cmd_reap() {
cmd_down "$name" -l "$loc" || { failed=$(( failed + 1 )); log "failed to reap $loc/$name"; }
done <<<"$rows"
[ "$n" -gt 0 ] || log "reap: no expired VMs"
+ [ "$others" -eq 0 ] || log "reap: left $others expired VMs of other owners; only their owners reap them"
[ "$failed" -eq 0 ] || { log "reap: $failed of $n expired VMs could not be destroyed"; return 1; }
}
@@ -443,15 +450,30 @@ cmd_list() {
# usage: VMs and vCPUs per owner, from each VM's size (standard-16 = 16 vCPUs).
cmd_usage() {
- local tmp loc name owner i=0
+ local tmp rows loc name owner f i=0 unread=0
+ rows=$(vm_rows) || { log "usage: could not list the project's VMs"; return 1; }
tmp=$(mktemp -d)
while read -r loc name owner _; do
+ [ -n "$name" ] || continue
i=$(( i + 1 ))
- cli vm "$loc/$name" show 2>/dev/null | sed -n "s/^size: /$owner /p" >"$tmp/$i" &
+ {
+ case $(api "$tmp/$i.show" vm "$loc/$name" show) in
+ 200) sed -n "s/^size: /$owner /p" "$tmp/$i.show" >"$tmp/$i" ;;
+ 404) ;;
+ *) : >"$tmp/$i" ;;
+ esac
+ rm -f "$tmp/$i.show"
+ } &
[ $(( i % 8 )) -ne 0 ] || wait
- done < <(vm_rows)
+ done <<<"$rows"
wait
- cat "$tmp"/* 2>/dev/null | awk -v me="$OWNER" '
+ for f in "$tmp"/*; do [ ! -e "$f" ] || [ -s "$f" ] || unread=$(( unread + 1 )); done
+ if [ "$unread" -gt 0 ]; then
+ rm -rf "$tmp"
+ log "usage: could not read the size of $unread of $i VMs"
+ return 1
+ fi
+ find "$tmp" -type f -exec cat {} + | awk -v me="$OWNER" '
{ n = $2; sub(/^.*-/, "", n); o = ($1 == "-") ? "(untagged)" : ($1 == ".") ? "(other)" : $1
vms[o]++; cpu[o] += n; total += n; count++ }
END {
@@ -542,8 +564,17 @@ quota_preflight() {
# between them: a slow first usage read must not shave the printed budget.
remaining=$QUOTA_WAIT
while :; do
- used=$(cmd_usage 2>/dev/null | awk '$1=="total" {print $3}')
- [[ "$used" =~ ^[0-9]+$ ]] || return 0
+ if ! used=$(cmd_usage 2>/dev/null | awk '$1=="total" {print $3}') || ! [[ "$used" =~ ^[0-9]+$ ]]; then
+ if [ "$remaining" -le 0 ]; then
+ log "quota preflight: could not read the project's vCPU usage; the API's own answer decides"
+ return 0
+ fi
+ log "quota: could not read the project's vCPU usage; retrying, up to ${remaining}s more"
+ step=$(( remaining < 30 ? remaining : 30 ))
+ sleep "$step"
+ remaining=$(( remaining - step ))
+ continue
+ fi
[ $(( used + need )) -gt "$max" ] || return 0
if [ "$remaining" -le 0 ]; then
log "quota preflight refused $location/$name ($size): it needs $need vCPUs and the project's VMs already use $used of $max (ceiling from $source; other usage such as managed runners counts on top)"
@@ -753,7 +784,12 @@ cmd_down() {
# pointer into the main repository, useless on the VM; the same snapshot
# replaces it, on the worktree's own branch and index. Not carried: submodule
# repositories (.git/modules) and a partial clone's promisor (the fetch pulls
-# every missing blob first); gstack has neither.
+# every missing blob first); gstack has neither. The archive is built in two
+# passes inside the snapshot directory because macOS's bsdtar stops reading
+# options at the first operand and resolves -T names after a -C, so no single
+# tar command line works on both it and GNU tar (gstack #3680). COPYFILE_DISABLE=1
+# keeps Apple's tar from adding ._* AppleDouble entries for files with
+# extended attributes, which the VM would see as untracked files.
cmd_pack() {
local src tmp gitdir branch head entry shared specs
src=$(cd "${1:-.}" && pwd)
@@ -800,15 +836,18 @@ cmd_pack() {
|| die "pack: could not snapshot the staged blobs of $src"
(
cd "$src"
+ export COPYFILE_DISABLE=1
git ls-files -z -co --exclude-standard \
| while IFS= read -r -d '' f; do
if [ -e "$f" ] || [ -L "$f" ]; then printf '%s\0' "$f"; fi
done \
- | tar --null -T - -c -C "$tmp" .git -zf -
+ | tar --null -T - -cf "$tmp/files.tar" \
+ && tar -rf "$tmp/files.tar" -C "$tmp" .git \
+ && gzip -c "$tmp/files.tar"
)
rm -rf "$tmp"; trap - EXIT
else
- tar -C "$src" -czf - .
+ COPYFILE_DISABLE=1 tar -C "$src" -czf - .
fi
}
@@ -928,7 +967,7 @@ cmd_run() {
trap 'exit 130' INT TERM HUP QUIT
cmd_up "${up_args[@]}" >/dev/null
if [[ "$name" =~ -t([0-9]+)$ ]]; then
- log "$name carries a ${BASH_REMATCH[1]}h TTL (any later run of this script reaps it); if this process dies first: $0 down $name"
+ log "$name carries a ${BASH_REMATCH[1]}h TTL (this owner's next up with UBI_REAP=1, or reap, destroys it once it expires); if this process dies first: $0 down $name"
elif [[ "$name" =~ ^$PREFIX-$OWNER-[0-9]{10}- ]]; then
log "$name has no TTL (reap destroys it once it is ${LEGACY_HOURS}h old); if this process dies before its teardown line, destroy it with: $0 down $name"
else
@@ -1086,8 +1125,8 @@ usage: ubi-runner.sh <command> [args]
owner print this caller's owner tag ($OWNER)
records VMs this machine created and has not confirmed gone ($VM_RECORD)
gc HOURS destroy this owner's VMs older than HOURS (never other owners')
- reap destroy every VM whose -t<hours> TTL has ended (any owner) and this
- owner's TTL-less ubirun-* VMs older than UBI_LEGACY_HOURS
+ reap destroy this owner's VMs whose -t<hours> TTL has ended and its
+ TTL-less ubirun-* VMs older than UBI_LEGACY_HOURS (never other owners')
cli ARGS... raw Ubicloud CLI passthrough (e.g. cli vm list)
VMs are named $PREFIX-<owner>-<epoch>-<suffix>-t<ttl hours>; up, run and verify need
diff --git a/test/helpers/mock-ubicloud-api.ts b/test/helpers/mock-ubicloud-api.ts
index b62a5f63cb..c8ac9d6411 100644
--- a/test/helpers/mock-ubicloud-api.ts
+++ b/test/helpers/mock-ubicloud-api.ts
@@ -4,8 +4,10 @@
// the real API), `vm L/N show` (404 once gone) and `vm L/N destroy -f`.
// VMs never reach `running`, so every `up` stays mid-provision until killed,
// unless `running` is set: then a created VM shows as running at 192.0.2.1
-// (pair it with a fake `ssh` on PATH). `faults` makes `destroy` or `list`
-// answer HTTP 500 / 503 from the next request on; `destroyFaults` answers the
+// (pair it with a fake `ssh` on PATH). `faults` makes `destroy`, `list` or
+// `show` answer HTTP 500 / 503 / 503 from the next request on; `listBody`
+// replaces the list answer (an empty project answers a bare newline), and
+// `ghosts` are listed but answer `show` with 404, like a VM destroyed since; `destroyFaults` answers the
// next destroy requests one by one with a status, or holds one past the
// client's timeout without acting on it ("hang").
@@ -27,6 +29,10 @@ export interface MockUbicloudOptions {
running?: boolean;
/** Answers for the next destroy requests, in order: an HTTP status, or "hang" (3 s, then a 504 nobody reads). */
destroyFaults?: Array<number | "hang">;
+ /** The exact `vm list` body, instead of one row per VM. */
+ listBody?: string;
+ /** Names `vm list` shows after the VMs that `show` answers with 404. */
+ ghosts?: string[];
seed?: Array<{ location?: string; name: string; size?: string }>;
}
@@ -35,8 +41,8 @@ export interface MockUbicloud {
vms: Map<string, MockVm>;
/** "create-start N", "created N", "create-rejected N", "destroy N", "removed N", in order. */
events: string[];
- /** Flip to make every later `destroy` (500) or `list` (503) fail. */
- faults: { destroy: boolean; list: boolean };
+ /** Flip to make every later `destroy` (500), `list` (503) or `show` (503) fail. */
+ faults: { destroy: boolean; list: boolean; show: boolean };
/** Every destroy request received, answered or not. */
destroyRequests: number;
stop(): void;
@@ -45,7 +51,7 @@ export interface MockUbicloud {
export function startMockUbicloud(opts: MockUbicloudOptions = {}): MockUbicloud {
const vms = new Map<string, MockVm>();
const events: string[] = [];
- const faults = { destroy: false, list: false };
+ const faults = { destroy: false, list: false, show: false };
for (const vm of opts.seed ?? []) {
vms.set(vm.name, { location: vm.location ?? "eu-central-h1", name: vm.name, size: vm.size ?? "standard-16", state: "creating" });
}
@@ -59,7 +65,8 @@ export function startMockUbicloud(opts: MockUbicloudOptions = {}): MockUbicloud
if (argv[0] !== "vm") return text(`! unsupported: ${argv.join(" ")}`, 400);
if (argv[1] === "list") {
if (faults.list) return text("! Unexpected response status: 503", 503);
- return text([...vms.values()].map((vm) => `${vm.location} ${vm.name}\n`).join(""));
+ if (opts.listBody !== undefined) return text(opts.listBody);
+ return text([...vms.values(), ...(opts.ghosts ?? []).map((name) => ({ location: "eu-central-h1", name }))].map((vm) => `${vm.location} ${vm.name}\n`).join(""));
}
const [location, name] = (argv[1] ?? "").split("/") as [string, string];
const verb = argv[2];
@@ -77,6 +84,7 @@ export function startMockUbicloud(opts: MockUbicloudOptions = {}): MockUbicloud
}
const vm = vms.get(name);
if (verb === "show") {
+ if (faults.show) return text("! Unexpected response status: 503", 503);
if (!vm) return text("! Unexpected response status: 404", 404);
return text(`id: vm-${name}\nname: ${name}\nstate: ${vm.state}\nlocation: ${vm.location}\nsize: ${vm.size}\n${vm.state === "running" ? "ip4: 192.0.2.1\n" : ""}`);
}
diff --git a/test/scripts/ubi-runner-bsd-userland.test.ts b/test/scripts/ubi-runner-bsd-userland.test.ts
new file mode 100644
index 0000000000..259e08f43b
--- /dev/null
+++ b/test/scripts/ubi-runner-bsd-userland.test.ts
@@ -0,0 +1,68 @@
+// ubi-runner-bsd-userland.test.ts — scripts/ubicloud/ubi-runner.sh pack under
+// the bsdtar macOS ships (ported from gstack's
+// test/regression-issue3680-ubicloud-bsd-userland.test.ts, gstack #3746).
+//
+// Protects: pack writes the whole checkout under bsdtar: tracked, staged,
+// untracked, symlinked, spaced and option-like names, a linked worktree and an
+// empty file list, with no files.tar left inside the archive.
+// Fails when: pack passes tar options after an operand, or resolves its -T list
+// after a -C, again (bsdtar then archives .git alone and exits 1).
+// Needs: macOS's /usr/bin/tar, or bsdtar on PATH (libarchive-tools); skipped
+// without it. usage under the one-true-awk is covered in ubi-runner.test.ts.
+// Network: none.
+
+import { describe, expect, it } from "bun:test";
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, symlinkSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join, resolve } from "node:path";
+
+const RUNNER = resolve(import.meta.dir, "..", "..", "scripts/ubicloud/ubi-runner.sh");
+const bsdTar = process.platform === "darwin" ? "/usr/bin/tar" : Bun.which("bsdtar");
+const gitEnv = { GIT_AUTHOR_NAME: "t", GIT_AUTHOR_EMAIL: "t@t", GIT_COMMITTER_NAME: "t", GIT_COMMITTER_EMAIL: "t@t" };
+
+describe(`pack under bsdtar${bsdTar ? "" : ": skipped, no bsdtar (install libarchive-tools)"}`, () => {
+ it.skipIf(!bsdTar)("ships tracked, staged, untracked, symlinked, spaced and option-like names, a linked worktree and an empty file list", () => {
+ const d = mkdtempSync(join(tmpdir(), "ubi-bsd-pack-"));
+ try {
+ mkdirSync(join(d, "bsd-bin"));
+ symlinkSync(bsdTar!, join(d, "bsd-bin", "tar"));
+ const PATH = `${join(d, "bsd-bin")}:${process.env.PATH ?? "/usr/bin:/bin"}`;
+ const sh = (cmd: string, cwd: string) => {
+ const r = Bun.spawnSync(["bash", "-c", cmd], { cwd, timeout: 30_000, env: { PATH, HOME: d, ...gitEnv } });
+ expect(r.exitCode, `${cmd}\n${r.stderr.toString()}`).toBe(0);
+ return r.stdout.toString();
+ };
+ expect(sh("tar --version", d)).toContain("bsdtar");
+ const pack = (src: string, out: string) => sh(`bash ${JSON.stringify(RUNNER)} pack ${JSON.stringify(src)} > ${JSON.stringify(out)}`, d);
+ const unpack = (tgz: string, to: string) => {
+ mkdirSync(join(d, to));
+ sh(`tar -xzf ${JSON.stringify(tgz)} -C ${JSON.stringify(to)}`, d);
+ return join(d, to);
+ };
+ const status = (cwd: string) => sh("git status --short", cwd).split("\n").filter(Boolean).sort();
+
+ sh('git init -q plain && cd plain && echo a > a && echo b > gone && echo s > "with space" && ln -s a link && echo o > ./-C && echo o > ./--files-from=a && git add -A && git commit -qm one && rm gone && echo st > staged && git add staged && echo u > untracked && echo x > "untracked space"', d);
+ pack("plain", "plain.tgz");
+ expect(sh("tar -tzf plain.tgz", d).split("\n").some((l) => l.endsWith("files.tar"))).toBe(false);
+ const vm = unpack("plain.tgz", "vm-plain");
+ expect(status(vm)).toEqual(status(join(d, "plain")));
+ expect(status(vm)).toEqual([" D gone", '?? "untracked space"', "?? untracked", "A staged"]);
+ expect(readFileSync(join(vm, "with space"), "utf8")).toBe("s\n");
+ expect(sh("readlink link", vm).trim()).toBe("a");
+ expect(existsSync(join(vm, "-C")) && existsSync(join(vm, "--files-from=a"))).toBe(true);
+
+ sh("git init -q main && cd main && echo a > a && git add a && git commit -qm one && git branch feat && git worktree add -q ../wt feat", d);
+ sh("echo b > b && git add b && git commit -qm two && echo u > u && echo ignored > ig && echo ig > .gitignore", join(d, "wt"));
+ pack("wt", "wt.tgz");
+ const vmWt = unpack("wt.tgz", "vm-wt");
+ expect(existsSync(join(vmWt, "ig"))).toBe(false);
+ expect(sh("git rev-parse --abbrev-ref HEAD && git status --short && git log --oneline | wc -l", vmWt).replace(/\s+/g, " ").trim()).toBe("feat ?? .gitignore ?? u 2");
+
+ sh("git init -q empty && cd empty && echo a > a && git add a && git commit -qm one && rm a", d);
+ pack("empty", "empty.tgz");
+ expect(status(unpack("empty.tgz", "vm-empty"))).toEqual([" D a"]);
+ } finally {
+ rmSync(d, { recursive: true, force: true });
+ }
+ }, 60_000);
+});
diff --git a/test/scripts/ubi-runner-run.test.ts b/test/scripts/ubi-runner-run.test.ts
index 130fb3c217..3763a42006 100644
--- a/test/scripts/ubi-runner-run.test.ts
+++ b/test/scripts/ubi-runner-run.test.ts
@@ -6,8 +6,8 @@
//
// Protects: up, run and verify refuse without UBI_OWNER before any API call;
// gbrain's up reaps nothing unless UBI_REAP=1 and says so in one line; reap
-// destroys expired TTL VMs of any owner and only the caller's own TTL-less
-// ubirun-<owner>-* VMs past UBI_LEGACY_HOURS; run ends with the UBICLOUD:
+// destroys only the caller's own expired TTL VMs and TTL-less
+// ubirun-<owner>-* VMs past UBI_LEGACY_HOURS, and counts other owners' expired ones; run ends with the UBICLOUD:
// left= line (also in UBI_LEFT_FILE), and a VM it cannot confirm gone fails a
// passing command; a silent or overlong remote command is killed with exit
// 124, its last output and its remote session ended; --pass/--env and a
@@ -132,8 +132,8 @@ describe("owner and reap", () => {
// GBRA-108: gbrain still holds TTL-less VMs older than gstack's 4 h legacy
// bound, so gbrain's up reaps nothing until UBI_REAP=1, and says so.
- it("gbrain's up reaps nothing by default and says so in one line; UBI_REAP=1 reaps expired TTL VMs of any owner first", async () => {
- const seed = [{ name: "ubirun-other-1000000000-bbbb-t1" }, { name: "ubirun-t1-1000000000-aaaa" }, { name: `ubirun-peer-${NOW}-cccc-t3` }];
+ it("gbrain's up reaps nothing by default and says so in one line; UBI_REAP=1 reaps this owner's expired VMs first and leaves other owners'", async () => {
+ const seed = [{ name: "ubirun-other-1000000000-bbbb-t1" }, { name: "ubirun-t1-1000000000-aaaa" }, { name: "ubirun-t1-1000000000-iiii-t1" }, { name: `ubirun-peer-${NOW}-cccc-t3` }];
serve({ rejectCreates: true, seed });
const def = await run(["up"]);
expect(def.code).toBe(1);
@@ -146,27 +146,29 @@ describe("owner and reap", () => {
expect(creates()).toHaveLength(2);
const on = await run(["up"], { UBI_REAP: "1" });
expect(on.code).toBe(1);
- expect(on.stderr).toMatch(/reaping expired eu-central-h1\/ubirun-other-1000000000-bbbb-t1 \(owner other; its TTL ended \d+ min ago\)/);
+ expect(on.stderr).toMatch(/reaping expired eu-central-h1\/ubirun-t1-1000000000-iiii-t1 \(owner t1; its TTL ended \d+ min ago\)/);
expect(on.stderr).toMatch(/reaping TTL-less eu-central-h1\/ubirun-t1-1000000000-aaaa \(owner t1; \d+h old, legacy sweep at 4h\)/);
+ expect(on.stderr).toContain("reap: left 1 expired VMs of other owners; only their owners reap them");
expect(on.stderr).not.toContain("reap skipped");
- expect(destroyed()).toEqual(["ubirun-other-1000000000-bbbb-t1", "ubirun-t1-1000000000-aaaa"]);
+ expect(destroyed()).toEqual(["ubirun-t1-1000000000-aaaa", "ubirun-t1-1000000000-iiii-t1"]);
}, 60_000);
- it("reap destroys expired TTLs (and older -x expiries) and this owner's old TTL-less names, never unexpired, recent, another owner's TTL-less, untagged or foreign ones", async () => {
+ it("reap destroys this owner's expired TTLs (and older -x expiries) and old TTL-less names, never unexpired, recent, another owner's, untagged or foreign ones", async () => {
const seed = [
- "ubirun-other-1000000000-bbbb-t1",
+ "ubirun-t1-1000000000-bbbb-t1",
"ubirun-t1-9999999999-dddd-t1",
"ubirun-t1-1000000000-aaaa",
"ubirun-1000000000-cccc",
`ubirun-t1-${NOW - 3 * 3600}-eeee`,
"ubirun-gbra240-1000000000-gggg",
"team-db-1000000000-ffff",
- `ubirun-other-${NOW - 5 * 3600}-x${NOW - 3 * 3600}-hhhh`,
+ `ubirun-t1-${NOW - 5 * 3600}-x${NOW - 3 * 3600}-hhhh`,
+ "ubirun-other-1000000000-jjjj-t1",
].map((name) => ({ name }));
const api = serve({ seed });
const reap = await run(["reap"]);
expect(reap.code).toBe(0);
- expect(destroyed()).toEqual(["ubirun-other-1000000000-bbbb-t1", "ubirun-t1-1000000000-aaaa", seed[7]!.name].sort());
+ expect(destroyed()).toEqual(["ubirun-t1-1000000000-bbbb-t1", "ubirun-t1-1000000000-aaaa", seed[7]!.name].sort());
expect((await run(["reap"])).stderr).toContain("reap: no expired VMs");
const lowered = await run(["reap"], { UBI_LEGACY_HOURS: "2" });
expect(lowered.code).toBe(0);
@@ -174,15 +176,38 @@ describe("owner and reap", () => {
const ownerless = await run(["reap"], { UBI_OWNER: "", UBI_LEGACY_HOURS: "1" });
expect(ownerless.code).toBe(0);
expect(ownerless.stderr).not.toContain("reaping TTL-less");
- expect([...api.vms.keys()].sort()).toEqual(["team-db-1000000000-ffff", "ubirun-1000000000-cccc", "ubirun-gbra240-1000000000-gggg", "ubirun-t1-9999999999-dddd-t1"]);
+ expect([...api.vms.keys()].sort()).toEqual(["team-db-1000000000-ffff", "ubirun-1000000000-cccc", "ubirun-gbra240-1000000000-gggg", "ubirun-other-1000000000-jjjj-t1", "ubirun-t1-9999999999-dddd-t1"]);
+ }, 60_000);
+
+ // GST-573 (gstack #3756): reap destroyed any owner's expired VM; deletes stay with the owner that created the VM.
+ it("reap destroys only the caller's own expired and old TTL-less VMs, and only counts other owners' expired ones (GST-573)", async () => {
+ const seed = [
+ "ubirun-t1-1000000000-aaaa-t1",
+ "ubirun-t1-1000000000-bbbb",
+ "ubirun-other-1000000000-cccc-t1",
+ `ubirun-other-${NOW - 5 * 3600}-x${NOW - 3 * 3600}-dddd`,
+ "ubirun-1000000000-eeee-t1",
+ `ubirun-peer-${NOW}-ffff-t3`,
+ ].map((name) => ({ name }));
+ const api = serve({ seed });
+ const reap = await run(["reap"]);
+ expect(reap.code).toBe(0);
+ expect(destroyed()).toEqual(["ubirun-t1-1000000000-aaaa-t1", "ubirun-t1-1000000000-bbbb"]);
+ expect(reap.stderr).toContain("reap: left 3 expired VMs of other owners; only their owners reap them");
+ expect(reap.stderr).not.toMatch(/reaping \S+\/ubirun-(other|peer|\d)/);
+ const other = await run(["reap"], { UBI_OWNER: "other" });
+ expect(other.code).toBe(0);
+ expect(destroyed()).toEqual(["ubirun-other-1000000000-cccc-t1", seed[3]!.name, "ubirun-t1-1000000000-aaaa-t1", "ubirun-t1-1000000000-bbbb"].sort());
+ expect(other.stderr).toContain("reap: left 1 expired VMs of other owners; only their owners reap them");
+ expect([...api.vms.keys()].sort()).toEqual(["ubirun-1000000000-eeee-t1", `ubirun-peer-${NOW}-ffff-t3`]);
}, 60_000);
it("reap reports a failed list and a failed destroy; bad UBI_TTL_HOURS refuses creates and bad UBI_LEGACY_HOURS refuses reap, never read-only commands", async () => {
- const api = serve({ seed: [{ name: "ubirun-other-1000000000-bbbb-t1" }] });
+ const api = serve({ seed: [{ name: "ubirun-t1-1000000000-bbbb-t1" }] });
api.faults.destroy = true;
const destroyFails = await run(["reap"], { UBI_REAP_DOWN_TIMEOUT: "2" });
expect(destroyFails.code).toBe(1);
- expect(destroyFails.stderr).toContain("failed to reap eu-central-h1/ubirun-other-1000000000-bbbb-t1");
+ expect(destroyFails.stderr).toContain("failed to reap eu-central-h1/ubirun-t1-1000000000-bbbb-t1");
expect(destroyFails.stderr).toContain("reap: 1 of 1 expired VMs could not be destroyed");
api.faults.list = true;
const listFails = await run(["reap"]);
@@ -204,7 +229,7 @@ describe("owner and reap", () => {
// #8096 item 3: every UBI_REAP value but 0 used to turn the reap on.
it("UBI_REAP=false, no, off (any case) reaps nothing; true, yes, on reap; an unreadable value refuses before any API call", async () => {
- serve({ rejectCreates: true, seed: [{ name: "ubirun-other-1000000000-bbbb-t1" }] });
+ serve({ rejectCreates: true, seed: [{ name: "ubirun-t1-1000000000-bbbb-t1" }] });
for (const value of ["false", "FALSE", "no", "off", "0"]) {
const r = await run(["up"], { UBI_REAP: value });
expect(r.code).toBe(1);
@@ -217,10 +242,10 @@ describe("owner and reap", () => {
expect(maybe.stderr).toContain("UBI_REAP must be 1 or 0 (true/false, yes/no and on/off also work), not 'maybe'");
expect(creates()).toHaveLength(5);
for (const value of ["true", "Yes", "on"]) {
- serve({ rejectCreates: true, seed: [{ name: "ubirun-other-1000000000-bbbb-t1" }] });
+ serve({ rejectCreates: true, seed: [{ name: "ubirun-t1-1000000000-bbbb-t1" }] });
const r = await run(["up"], { UBI_REAP: value });
- expect(r.stderr).toMatch(/reaping expired eu-central-h1\/ubirun-other-1000000000-bbbb-t1/);
- expect(destroyed()).toEqual(["ubirun-other-1000000000-bbbb-t1"]);
+ expect(r.stderr).toMatch(/reaping expired eu-central-h1\/ubirun-t1-1000000000-bbbb-t1/);
+ expect(destroyed()).toEqual(["ubirun-t1-1000000000-bbbb-t1"]);
}
}, 90_000);
});
@@ -285,13 +310,13 @@ describe("down retries a failed destroy request", () => {
}, 60_000);
it("reap waits at most UBI_REAP_DOWN_TIMEOUT per VM, not UBI_DOWN_TIMEOUT", async () => {
- const api = serve({ seed: [{ name: "ubirun-other-1000000000-bbbb-t1" }, { name: "ubirun-peer-1000000000-cccc-t1" }] });
+ const api = serve({ seed: [{ name: "ubirun-t1-1000000000-bbbb-t1" }, { name: "ubirun-t1-1000000000-cccc-t1" }] });
api.faults.destroy = true;
const started = Date.now();
const r = await run(["reap"], { UBI_DOWN_TIMEOUT: "600", UBI_REAP_DOWN_TIMEOUT: "2" });
expect(r.code).toBe(1);
expect(Date.now() - started).toBeLessThan(30_000);
- expect(r.stderr).toContain("could not confirm eu-central-h1/ubirun-other-1000000000-bbbb-t1 is gone after 2s (last destroy request: HTTP 500)");
+ expect(r.stderr).toContain("could not confirm eu-central-h1/ubirun-t1-1000000000-bbbb-t1 is gone after 2s (last destroy request: HTTP 500)");
expect(r.stderr).toContain("reap: 2 of 2 expired VMs could not be destroyed");
for (const bad of ["0", "x"]) expect((await run(["reap"], { UBI_REAP_DOWN_TIMEOUT: bad })).stderr).toContain("UBI_REAP_DOWN_TIMEOUT must be a whole number of seconds from 1");
}, 60_000);
@@ -594,4 +619,27 @@ describe("quota preflight", () => {
expect(name).toMatch(new RegExp(`^${RUN_VM}$`));
expect((await run(["down", name])).code).toBe(0);
}, 90_000);
+
+ // gstack #3746: an unreadable usage used to read as room and let the create through without a word.
+ it("an unreadable usage is retried while UBI_QUOTA_WAIT lasts, then logged, and the create is left to the API", async () => {
+ const api = serve({ rejectCreates: true, seed: [{ name: `ubirun-peer-${NOW}-cccc-t3` }] });
+ api.faults.list = true;
+ const up = await run(["up"], { UBI_QUOTA_VCPUS: "16" });
+ expect(up.stderr).toContain("quota preflight: could not read the project's vCPU usage; the API's own answer decides");
+ expect(creates()).toHaveLength(1);
+ const waited = await run(["up"], { UBI_QUOTA_VCPUS: "16", UBI_QUOTA_WAIT: "2" });
+ expect(waited.stderr).toContain("quota: could not read the project's vCPU usage; retrying, up to 2s more");
+ expect(waited.stderr).toContain("quota preflight: could not read the project's vCPU usage; the API's own answer decides");
+ expect(creates()).toHaveLength(2);
+ }, 60_000);
+
+ // gstack #3746 (#3678): the reap before up died on an empty project's bare-newline list.
+ it("up with UBI_REAP=1 on an empty project reaps nothing and reaches its create", async () => {
+ serve({ rejectCreates: true, listBody: "\n" });
+ const up = await run(["up"], { UBI_REAP: "1" });
+ expect(up.stderr).toContain("reap: no expired VMs");
+ expect(up.stderr).not.toContain("invalid VM name");
+ expect(up.stderr).not.toContain("reap failed");
+ expect(creates()).toHaveLength(1);
+ }, 60_000);
});
diff --git a/test/scripts/ubi-runner.test.ts b/test/scripts/ubi-runner.test.ts
index 1515c0f9c4..f3d593f4bc 100644
--- a/test/scripts/ubi-runner.test.ts
+++ b/test/scripts/ubi-runner.test.ts
@@ -243,6 +243,42 @@ describe("stale sweep and ownership", () => {
});
});
+// gstack #3746 (#3678): Ubicloud answers `vm list` with a bare newline for a
+// project with no VMs, which vm_rows read as a VM named "-"; and a usage whose
+// list or size reads failed printed a partial total the quota preflight trusted.
+describe("empty project and unreadable usage", () => {
+ it("an empty project (vm list answers a bare or blank line) has no VMs: reap destroys nothing, usage prints a zero total, list --mine prints nothing", async () => {
+ for (const listBody of ["\n", " \n"]) {
+ const api = serve({ listBody });
+ const reap = await run(["reap"], { UBI_OWNER: "t1" });
+ expect(reap.stderr).toContain("reap: no expired VMs");
+ expect(reap.stderr).not.toContain("invalid VM name");
+ expect(reap.code).toBe(0);
+ const usage = await run(["usage"], { UBI_OWNER: "t1" });
+ expect({ code: usage.code, stdout: usage.stdout }).toEqual({ code: 0, stdout: "OWNER VMS VCPUS\ntotal 0 0\n" });
+ expect((await run(["list", "--mine"], { UBI_OWNER: "t1" })).stdout).toBe("");
+ expect(api.destroyRequests).toBe(0);
+ mock?.stop();
+ }
+ });
+
+ it("usage skips a VM destroyed since the list (404), and exits 1 with no total when a size read or the list fails", async () => {
+ const api = serve({ seed: [{ name: `ubirun-peer-${NOW}-cccc-t3` }], ghosts: [`ubirun-gone-${NOW}-dddd-t3`] });
+ const ghost = await run(["usage"], { UBI_OWNER: "t1" });
+ expect({ code: ghost.code, stdout: ghost.stdout }).toEqual({ code: 0, stdout: "OWNER VMS VCPUS\npeer 1 16\ntotal 1 16\n" });
+ api.faults.show = true;
+ const noShow = await run(["usage"], { UBI_OWNER: "t1" });
+ expect(noShow.code).toBe(1);
+ expect(noShow.stdout).not.toContain("total");
+ expect(noShow.stderr).toContain("usage: could not read the size of 2 of 2 VMs");
+ api.faults.list = true;
+ const noList = await run(["usage"], { UBI_OWNER: "t1" });
+ expect(noList.code).toBe(1);
+ expect(noList.stdout).not.toContain("total");
+ expect(noList.stderr).toContain("usage: could not list the project's VMs");
+ });
+});
+
// `pack` from a checkout on a branch named like `git init`'s default branch
// (master on this repo). slim_git fetches HEAD into that branch inside a fresh
// repository whose unborn HEAD names the same branch; without --update-head-ok
@@ -276,4 +312,22 @@ describe("pack", () => {
expect(execFileSync("git", ["-C", out, "log", "--format=%s"], { encoding: "utf8" }).trim()).toBe("one");
expect(() => execFileSync("git", ["-C", out, "cat-file", "-e", dangling], { stdio: "ignore" })).toThrow();
});
+
+ // gstack #3746: only Apple's tar reads COPYFILE_DISABLE; without it, files with
+ // extended attributes gain ._* entries the VM sees as untracked files.
+ it("runs every tar with COPYFILE_DISABLE=1, for a checkout and for a plain directory", () => {
+ const realTar = Bun.which("tar");
+ expect(realTar).toBeTruthy();
+ const bin = join(dir, "tar-shim");
+ execFileSync("mkdir", ["-p", bin]);
+ writeFileSync(join(bin, "tar"), `#!/usr/bin/env bash\necho "COPYFILE_DISABLE=${"$"}{COPYFILE_DISABLE:-unset}" >> ${JSON.stringify(join(dir, "tar.log"))}\nexec ${JSON.stringify(realTar)} "$@"\n`, { mode: 0o755 });
+ const repo = join(dir, "repo");
+ execFileSync("git", ["init", "-q", repo]);
+ execFileSync("git", ["-C", repo, "-c", "[email protected]", "-c", "user.name=t", "commit", "-q", "--allow-empty", "-m", "one"]);
+ writeFileSync(join(repo, "a"), "a\n");
+ execFileSync("mkdir", ["-p", join(dir, "plain")]);
+ writeFileSync(join(dir, "plain/p"), "p\n");
+ for (const src of [repo, join(dir, "plain")]) execFileSync("bash", [RUNNER, "pack", src], { env: env({ PATH: `${bin}:${process.env.PATH ?? "/usr/bin:/bin"}` }), stdio: ["ignore", "ignore", "pipe"] });
+ expect(readFileSync(join(dir, "tar.log"), "utf8").trim().split("\n")).toEqual(["COPYFILE_DISABLE=1", "COPYFILE_DISABLE=1", "COPYFILE_DISABLE=1"]);
+ });
});