Garry's Opinionated OpenClaw/Hermes Agent Brain
Ubicloud fan-out (ci:ubicloud)
scripts/ci-ubicloud.ts runs the ci:local gate on ephemeral Ubicloud VMs
instead of one Docker host. It packs the working tree once (tracked files,
untracked files that are not ignored, and .git) and streams it to every VM, so
uncommitted edits are tested. scripts/ubicloud/ubi-runner.sh creates and
destroys the VMs.
How many full gates
Run one full gate (bun run ci:ubicloud) per PR, at handoff, on the tree you
hand off (the tree rule in the project's shipping rules: a head whose tree
matches an already-gated tree apart from the release stamp files needs no new
gate). During development use bun run ci:ubicloud:diff, --lanes and
targeted suites (bun test --timeout=60000 <files>). The gstack pregate remote stage
(scripts/pregate-remote-gate.sh, .gstack/pregate.json's remote_gate)
runs the :diff form for the same reason, unless the clean tree already passed
GitHub's full Test and E2E runs, when it reuses them and starts no VM
(remote-gate reuse). :diff narrows only doc-only
trees today (doc checks plus gitleaks); any other tree still runs every lane,
so pick --lanes or targeted suites when the change is narrow. In a
multi-lane wave, lanes run ci:ubicloud:diff or targeted suites, and only
the integrator runs the full gate.
Ownership and teardown
scripts/ubicloud/ubi-runner.sh is gstack's runner, vendored (synced at gstack
5532345a); its header lists gbrain's additions, which a sync keeps. Every VM
the runner names is ubirun-<owner>-<epoch>-<suffix>-t<H>, gstack's runner
grammar: its TTL ends H hours (1 to 999) after the creation epoch. The owner
is UBI_OWNER (lowercased, letters and digits only, at most 12 characters,
starting with a letter), and it is required: up, run, verify,
ci:ubicloud and scripts/pregate-remote-gate.sh refuse without it, so no VM
is created untagged (read-only commands fall back to a per-machine id). Set it
to your thread code (UBI_OWNER=gbra41 bun run ci:ubicloud). Ubicloud names
allow 63 characters of [a-z0-9-], so a full name stays under 50.
Ubicloud has no VM tags, so the TTL lives in the name, and gstack's
ubi-runner.sh reap reads the TTL of every name gbrain generates the same way. ci:ubicloud gives every gate VM a
two-hour TTL (--ttl-hours N changes it, up to 999). ubi-runner.sh up and
run take the same --ttl-hours H; without it their VMs get UBI_TTL_HOURS
(gstack's default, 3 hours), so give a long dev, benchmark or eval VM
--ttl-hours above its longest run. With -n NAME, --ttl-hours H appends
-t<H>, and a ubirun- name without a TTL is refused (nothing could ever reap it).
ubi-runner.sh reap destroys only the caller's own VMs: its ubirun-<owner>-*
VMs whose TTL ended over ten minutes ago, and its TTL-less ones once they are
UBI_LEGACY_HOURS (4) old. Another owner's expired VM, including one this machine
created under a different tag (which list --mine and gc do cover), is only counted in reap's
log; reap-expired.ts is the operator reaper that destroys expired VMs of every owner. gstack's up runs that reap first; gbrain's copy
does only with UBI_REAP=1 (or true, yes, on; 0, false, no and off
turn it off, and any other value is refused), and logs one reap skipped line otherwise, until
gbrain's TTL-less VMs older than four hours are gone. run, verify and
ci:ubicloud end with gstack's leftover line, UBICLOUD: left=<n> owner=<owner> ...
(also appended to UBI_LEFT_FILE), counting their own VMs still in the
project's list after teardown; anything but left=0 fails a run that would
otherwise pass, and scripts/pregate-remote-gate.sh treats it, or a missing
line, as PREGATE_REMOTE_INFRA (exit 4, no receipt). scripts/ubicloud/reap-expired.ts is the operator
side of the leak guard: run from the operator's Capy automation, where the
Ubicloud token already lives (never from GitHub Actions), it lists the project's
VMs, selects only names with a TTL (or the older
ubirun-<owner>-<epoch>-x<expiry epoch>-<suffix> form, until none are left)
whose expiry plus a 15-minute grace has passed, from any owner or tool, and prints each ubirun-* VM's owner, age, expiry and action
plus an owner-notice block (expired VMs grouped by owner and the thread code
the owner tag names). It is a dry run unless given --apply, which destroys
each expired VM through ubi-runner.sh down, and it refuses the whole run when
more than --max (40) VMs are expired. It never selects an owner-tagged name
without a TTL, a legacy untagged ubirun-<epoch>-* name or a VM the
runner did not name, whoever owns it.
Each name is recorded before its create request is sent: the runner writes
the VM's state directory first, and ci-ubicloud.ts appends the name to
.context/ci-ubicloud/<run>/vms.txt. On every exit, including Ctrl-C,
SIGTERM, SIGQUIT and SIGHUP (a dropped terminal or a cancelled background
operation), an up still talking to the API finishes the request and destroys
its own VM; further signals are ignored until teardown ends. Then every recorded name gets down, which destroys the VM and polls
until it is confirmed gone. A create whose answer never arrived is waited out
for UBI_CREATE_GRACE seconds (default 180) before the VM counts as never
created. A destroy request that times out or gets a 5xx (or 408, 429, 404) is
retried on the next poll; another 4xx fails at once. down gives up after
UBI_DOWN_TIMEOUT seconds (default 900), naming the last destroy status, and
prints the command to rerun. reap waits at most UBI_REAP_DOWN_TIMEOUT
seconds (default 180) for each VM.
Nothing sweeps stale VMs unless you ask. With UBI_GC_HOURS set to a positive
number, each up first destroys the caller's own VMs older than that many
hours. An operator runs the same sweep with ubi-runner.sh gc HOURS. Neither
ever destroys another owner's VM, an untagged legacy ubirun-<epoch>-* VM, or a
VM the runner did not name. Use down NAME for those after checking who owns
them.
A machine that goes to sleep kills the run outright, with no signal, so
teardown can't run: on Capy, run ci:ubicloud as a watched background
operation so the machine stays awake. After any interrupted run, check
list --mine and reap leftovers with down NAME (or gc HOURS, which only
destroys your own VMs older than HOURS). A gate VM nobody reaps still dies
once its expiry passes and the operator reaper next runs with --apply.
ubi-runner.sh list --mine lists your VMs: those tagged with your owner tag
and every VM this machine created under any tag (its local state stays until
down confirms it gone), so a run started with a different UBI_OWNER is still
listed; stderr names the tags it covered. ubi-runner.sh usage prints VMs
and vCPUs per owner across the project, which shows who holds the shared quota.
Each VM runs scripts/ubicloud/setup-ci-vm.sh: the pinned Bun from
docker-compose.ci.yml, the runner container's test prerequisites plus Node
and a C compiler, frozen dependencies, both PGLite snapshot fixtures, and one
pgvector/pgvector:pg16 server fronted by a transaction-mode PgBouncer per
slot. Every slot's schema is bootstrapped with setupLegacyEmbeddingDB(), the
same step nightly full-corpus E2E workers run, so no E2E file depends on which
file reaches a database first.
Setup takes 60-80 seconds, including VM boot (about 25 seconds). The
orchestrator packs the checkout while the VMs boot, then streams the setup
script over SSH as soon as a VM answers, so apt, the Bun and gitleaks
downloads and the image pulls run while the checkout is still uploading; the
script waits for a marker file before the steps that need the checkout. apt
skips fsync, man-db, needrestart and the translation, AppStream and
command-not-found indexes. Database containers start only after apt finishes,
because a package's systemd reload during docker run fails the container.
The tarball carries a slim .git holding only what HEAD and origin/master
reach, plus the branch, index and config, so a checkout that fetched many
branches uploads about 80 MB instead of 250 MB. The critical path is now apt's
package installs (about 23 seconds of dpkg work); downloads take two seconds.
scripts/ubicloud/build-ci-image.sh builds the gbrain-ci machine image
(apt packages and both database images baked in), and --image gbrain-ci@latest or UBI_CI_IMAGE boots it, which cuts the setup script from
about 40 seconds to 18. It is opt-in until image boots prove as reliable as
stock ones: on its first full gate five of eight VMs sat in waiting for capacity. Stock VMs hit the same wait later (six of ten in one burst), so
eu-central-h1 capacity, not just the quota, can shrink a fleet; the run
then continues on the VMs that did start, and it ends once every item has a
result rather than waiting for a VM that never came up. Rebuild the image when the package
list or images change; setup still pulls the floating image tags, so a stale
image costs time, not correctness.
Scheduling is dynamic. Every unit, serial, slow and E2E file is one item in a
global queue ordered by weight, heaviest first. Each idle slot on any VM takes
the next item, so a slow VM or a mis-weighted file delays only the slot holding
it. Light items leave in same-lane batches to amortize SSH round trips, and the
batch target shrinks as the queue drains. Items of 60 seconds or more are the
run's long poles, so they spread one per VM before any VM takes a second one;
the VM running the machine-level items below counts only once it joins the
pool.
The first VM to finish setup runs the
machine-level work first: gitleaks, verify, then the serial lane's
machine-exclusive files and the slow lane's latency contracts
(run-slow-tests.sh --dry-run-list-exclusive, the files GitHub CI runs on a
dedicated perf runner) one at a time with nothing else on that VM. After that
it joins the pool. Before verify, it uploads the incremental typecheck state
(node_modules/.cache/gbrain-typecheck.tsbuildinfo, which the checkout
tarball never carries). The orchestrator builds that state with
bun run typecheck on the local checkout while the VMs boot
(scripts/ubicloud/typecheck-state.ts). Cold, verify's typecheck took 81-84 s
on a 16-vCPU VM and crossed the 120 s per-check cap on a loaded host; warm, it
takes 16-19 s. tsc revalidates every file by content hash, so the state never
changes the result. When the local checkout's node_modules is a symlink (a
worktree sharing another checkout's install), tsc records node_modules files by
their real path outside the checkout; the uploaded copy
(<run>/typecheck.tsbuildinfo) re-roots them under the checkout's own
node_modules, or the state would not apply on the VM and its typecheck would run
cold. The run log says whether verify's typecheck ran warm or cold, and the local
build's output is in logs/local-typecheck.log. Items run through the ci:local wrappers
(scripts/ubicloud/ci-item.sh): run-unit-shard.sh, run-serial-tests.sh and
run-slow-tests.sh accept explicit file arguments for this purpose, and
run-e2e.sh runs each E2E file against its slot's own server and pooler. The
unit, serial and slow lanes run with database URLs unset. The pg-arms lane
runs every file in test/postgres-unit-arms.txt the way persistence-validation's
unit-postgres-arms step does (bun --no-env-file test --timeout=120000,
GBRAIN_TEST_ALLOW_DATABASE_URL=1, no timeout multiplier) against the slot's
own empty gbrain_arms_test database. The summary takes the two-worker makespan of its
measured durations over that job's ARM_SHARDS partition (24 shards, ARM_WORKERS = 2
files at a time), and a shard over the step's 20-minute timeout
fails the run, since GitHub would cut that shard off mid-list. The committed
weights must also fit each shard twice over with a minute to spare
(ARM_SHARD_BUDGET_MS, 570 s), because a slow runner ran the files 1.3-2x
their mined weights. With the unit and slow lanes, the summary checks
test.yml's unit shards exactly as test/scripts/unit-shard-budget.test.ts
does: test-shard.sh's UNIT_SHARDS partition, each shard's weight from
scripts/test-weights.json, failing a shard over UNIT_SHARD_BUDGET_MS
(unitShardCheck, docs/TESTING.md#keeping-ci-partitions-balanced). It
prints each shard's measured time beside its mined weight but never fails on
it: every item is its own Bun process on a shared VM, so the same partition
measured 1.2-2.5x its mined weight across gates (one shard 1,274 s, about
503 s mined), with every test passing;
evals/ files, which no ci:ubicloud lane runs, count as unmeasured. Tests run natively as
a non-root user on Ubuntu 24.04, the same OS as the CI runners, instead of as
root in the oven/bun container.
Weights come from, in order: .context/ci-ubicloud/weights.json (merged after
every run), the committed scripts/ubicloud/weights.json (refresh it with
--record-weights on a green full run), then the lane weight files mined from
GitHub CI. Unknown files get their lane's p75. Per-item logs, failure logs and
summary.json land in .context/ci-ubicloud/<run>/. The exit status is non-zero
when any item fails, an item never produces a result, or no VM becomes ready.
An item whose SSH batch dies without a result is retried once on any slot, and a
VM with three such infrastructure errors is retired.
The fleet is sized from the quota. Without --vms, the run reads
ubi-runner.sh usage and takes as many standard-16 VMs as fit under 448 of
the project's 512 vCPUs, between the default of four (64 vCPUs) and the burst
maximum of ten (160 vCPUs), and never more than the work can use (each slot
needs at least two minutes of weighted work, so --lanes gitleaks gets one
VM). The decision is logged (fleet: project VMs hold 192 vCPUs, ceiling 448: 10 VM(s) × 16 vCPUs (burst maximum 10)). The 64 vCPUs above the ceiling are
headroom for the managed GitHub runners usage can't see. When usage can't
be read, or the project is busy, the run takes the default. Slots default to
three per eight vCPUs (--slots, six on a standard-16); --size picks the VM shape. A full gate runs only in
eu-central-h1 (the default --location): test timeouts and the mined shard weights are calibrated on
its VMs, and on 2026-10-19 two full gates in us-east-a2 ran about half as fast and timed out tests that
pass there. Another --location is refused unless --lanes names a diagnostic subset, whose summary
says it is not gate evidence. A VM that the quota refuses fails
to provision and the run continues on the VMs that did start, so a busy
project shrinks the fleet instead of failing. The runner prints the refusal
(needed, used and maximum vCPUs) with the project's VMs and vCPUs per owner;
used vCPUs beyond that table are other usage, such as managed GitHub runners. Slow-lane items run test/export-scale.slow.test.ts
at the pull-request scale (GBRAIN_TEST_EXPORT_SCALE_PAGES=10001). --lanes runs a
subset, --keep leaves the VMs up for debugging, and --diff follows
ci:local:diff (a doc-only diff runs the doc checks and gitleaks alone).
A full gate is about 35,000 seconds of test compute at 6 slots per VM, and
the heaviest-first queue keeps every slot busy almost to the end, so wall
time is setup plus compute divided by slots. No single file runs much past
three minutes: the former long poles are split into files of about two
minutes (graduation-cli*, graduation-crash-*, graduation-clients*,
reconcile-crash-*.slow, write-attribution-timeline-10k-postgres), and the
longest single test left is the 1k-page graduation round trip (about 150
seconds). Slots are sized by load, not by vCPU count alone. Each slot's bun process, its child
processes and its own PostgreSQL and PgBouncer keep about 2.5 threads runnable, so on
2026-10-20 one slot per two vCPUs (8 on standard-16) measured a 1-minute load of 1.11 per
vCPU at p50 and 1.39 at p90, ran 45,565 s of test compute and timed out
persistence-sync-options-parity on its PgBouncer pass. Six slots on the same tree
measured 0.79 and 0.95, ran 35,393 s (22% less) and passed; at eight VMs that is about
the same wall time. Every gate prints this load line, and warns when p90 passes 1.1:
test timeouts, run-e2e.sh's per-file cap and the mined shard weights assume a VM
that is not oversubscribed. More slots do not buy wall time; bursts add VMs instead.