Garry's Opinionated OpenClaw/Hermes Agent Brain
Upgrade matrix
.github/workflows/upgrade-matrix.yml installs published gbrain releases exactly as a user does, builds a small brain on each, upgrades it through that release's own updater to one candidate, and checks the candidate on the upgraded brain. The driver is scripts/upgrade-matrix.ts with its modules in scripts/upgrade-matrix/.
It does not gate releases to the latest lane. It gates stable: the daily
stable check dispatches it on the stable candidate's tag and promotes only on
a green run of that exact tag and tree, and holds while the nightly run's
break issue is open (daily stable check,
scripts/stable-matrix.ts).
What a run tests
From-versions. Resolved once per run from published release tags (the
releases API, authenticated with the workflow token) and recorded in the plan:
N-1, N-10 and N-50 (the 1st, 10th and 50th newest release older than the
candidate), the release published closest to 30 days before the candidate, and
the current stable tag when it exists and is older, and the stable
releases in the window (stable-window, versions.ts stableWindow): every
stable-v<version> record older than the candidate that was stable during the
183 days before it, capped at the newest 8 by promotion plus the newest one of
each older calendar month. A role that cannot be filled is recorded as
skipped with its reason. Pull requests run N-1 and N-50 only.
Cells. Every from-version runs on PGLite and Postgres, keyless and keyed,
repo-less and repo brain (32 to 40 cells; one job per from-version, engine and
key mode runs both brain shapes). Pull requests run PGLite and Postgres,
keyless, repo-less only. Keyed cells set a Voyage key (gbrain's default
embedding provider) and the fixture forwards api.voyageai.com to the local
fake provider in scripts/bench/retrieval-scale/mock-providers.ts, so the
brain keeps the provider's real URL and nothing is billed. Repo cells run
gbrain init --git; repo-less cells keep the default content directory
without Git.
Each cell, in its own HOME, GBRAIN_HOME, BUN_INSTALL and global git config:
- Installs the old release with the README command,
bun install -g github:garrytan/gbrain, while the fixture's default branch andlatest-stablepoint at that release, and checksgbrain --version. - Runs
gbrain initfor the cell's shape and seeds pages, a page with a facts fence, a remembered fact, a soft-deleted page, embeddings (keyed) and onegbrain dreampass (recorded, not judged: the old release ran it). It snapshots the canonical content and keeps a copy of the old install. - Starts
gbrain serveas an agent harness does (stdio, stdin held open) and checks it answers MCP as the old version; on Postgres it also startsgbrain jobs supervisor start --detach. - Moves every fixture ref to the candidate and runs
gbrain upgrade(the old release's code). When it refuses withrestart_refusedfor the MCP server it reruns with--stop-mcp, the documented fix. - Checks the install is the candidate:
gbrain --version, the commit Bun pinned in its lockfile, and every installed file against the candidate's git tree. The fixture log must show the candidate tarball and no other commit after the switch. - One build: an old serve still running is restarted with the candidate's
gbrain upgrade --restart-only --stop-mcp(old releases have no restart step); the restarted serve must report the candidate version over MCP and find a seeded page through thesearchtool. On Postgres doctor'sruntime_build_skewmust be ok with serve and the supervisor running. No process for the brain may remain once both are stopped. gbrain apply-migrations --yes --non-interactiveexits 0, doctor'sschema_versionis at the candidate's head, and doctor reports no failing check outside the skill category (see the W4 switch point below).- Preservation: the canonical content snapshot equals the one taken before the upgrade (below).
- A new page is written and found by keyword search; keyed cells embed it,
and
gbrain querymust embed the question and return the matching seed page; onegbrain dreampass exits 0. - Mixed versions: the old install runs
--version,search,doctor --jsonandserve(MCP initialize and one search) on the upgraded brain; the schema catalog and canonical content must not change.
Services the upgrade tries to install go to stub crontab, systemctl,
launchctl and loginctl on the cell's PATH; the receipt lists the calls
(service_calls) and nothing reaches the runner.
Preservation snapshot. Per page: source id, slug, type, kind, title,
content hash, frontmatter, provenance columns (source_kind, source_uri,
ingested_via, source_path) and deletion state. Per fact: entity, claim,
row number, kind, visibility, notability, context, source, confidence, valid
dates, typed-claim columns, and expired or superseded state. Per source: id,
name and archive state. Timestamps, ids, vectors, chunks and bookkeeping are
left out, so a migration that adds columns or re-chunks pages compares equal; a
canonical column the upgrade drops is a difference. The catalog snapshot adds
columns, indexes, constraints, triggers, function bodies, views and the
schema version. A PGLite brain is read with the PGLite build of the release
installed at that moment, never the harness's own.
How the updater is interposed
The old release's updater must only ever see the candidate:
- git:
url.<per-cell bare mirror>.insteadOf https://github.com/garrytan/gbrainin the cell's global git config. The mirror borrows the harness checkout's objects and holds only the fixture refs (master,latest-stable,stable, the old tag andv<candidate version>), sogit ls-remotelists the candidate as the newest release. - Release channel: every cell process runs with
GBRAIN_RELEASE_CHANNEL=alpha. The fixture publishes no stable evidence record, so a release that defaults to stable (v0.60.606.0 on) would refusestable_invalidinstead of upgrading to the candidate; older releases read no such variable and already follow every release. - Bun's tarball fetch:
GITHUB_API_URLpoints at the fixture's loopback server, which answers/repos/garrytan/gbrain/tarball/<ref>withgit archiveof the ref's commit. - Every other HTTPS request (raw.githubusercontent.com for VERSION and the
Bun-floor package.json, api.github.com, codeload, github.com): an
HTTPS_PROXYsends the GitHub hosts to a local TLS server whose certificate a per-run CA signs (NODE_EXTRA_CA_CERTS), tunnels onlyregistry.npmjs.orgto the internet for dependencies, and refuses every other host. Refused hosts are listed in the receipt (interposition.denied).
Every from-version installs and upgrades through these paths today, so no
from-version is listed as un-interposable. The binary updater
(gbrain self-upgrade on a compiled binary) is not exercised: the matrix
installs through Bun as the README says.
Verdicts
A cell is pass, fail (a candidate check failed), error (the old
release could not be installed, initialized or seeded in that shape, so there
is no candidate verdict) or known_exception (every failure matches an entry
below). A run is red when any cell fails, a planned cell left no receipt, a
fault was injected, or a receipt names another candidate or harness; hold
when it is otherwise clean but a cell errored or a door could not run;
green when every planned cell passed. A hold never qualifies a candidate
and never opens an issue.
Known exceptions live in
scripts/upgrade-matrix/known-exceptions.json:
each names one check, an explicit list of old release tags (no wildcards) and
the issue that tracks it. Setup errors cannot be excepted. Old releases that
mutate the brain when started again after the upgrade belong here, linked to
#7776 or a narrower issue; the candidate's own client_behind refusal is
tested in the unit suite, since a frozen old binary cannot carry it.
W4 switch point. Until gbrain doctor --alert ships, the doctor check is
the legacy report (DOCTOR_ARGV and doctorVerdict in
scripts/upgrade-matrix/verdict.ts); a report that carries alert.level is
judged by it. When W4 lands, change DOCTOR_ARGV there and drop the legacy
branch.
Runs and receipts
- Nightly (06:41 UTC): full matrix against the newest release. A red run
opens or updates one issue carrying the
upgrade-breaklabel (found by the marker<!-- gbrain-upgrade-matrix:nightly -->) and a later green run comments and closes it. The issue's<!-- gbrain-upgrade-matrix:red-from <tags> -->line names the from-versions that failed or held. While it is open the daily stable check holds stable (upgrade-matrix/marker.ts). Nightly watch also tracks the workflow like every scheduled run. - Stable candidate: the daily stable check dispatches the full matrix on
its candidate's tag when that release has no run yet (
scripts/stable-daily.ts). - Dispatch:
gh workflow run upgrade-matrix.yml --ref master -f candidate=v0.60.475.0(empty candidate: the newest release;-f from=v0.60.390.0,v0.60.425.0replaces the resolved from-versions;-f fault=corrupt-after-upgradeis the proof that preservation catches a corrupted upgrade, and is always red). - Pull requests touching schema, migrations, self-upgrade,
upgrade.ts,apply-migrationsor the harness: the reduced subset against the head.
The gate runs from master's workflow and harness: dispatch and schedule runs
refuse any other ref, and the candidate is only installed as the subject. The
upgrade-matrix-plan artifact records the harness commit, the workflow ref,
the candidate tag, commit and tree, and the from-versions with their roles; the
upgrade-matrix-receipt artifact adds every cell's verdict and failures. Cell
artifacts keep each cell's receipt, logs and snapshots. A pull-request run's
harness is its own head and is not gate evidence.
Run a cell locally (Postgres cells need a superuser URL, for example the
docs/TESTING.md test container):
bun scripts/upgrade-matrix.ts plan --candidate v0.60.475.0 --out plan.json
bun scripts/upgrade-matrix.ts cell --plan plan.json --from v0.60.474.0 --engine pglite --keys keyless --brain both --out-dir out
UPGRADE_MATRIX_POSTGRES_URL=postgresql://postgres:postgres@localhost:5434/postgres \
bun scripts/upgrade-matrix.ts cell --plan plan.json --from v0.60.474.0 --engine postgres --keys keyed --out-dir out
bun scripts/upgrade-matrix.ts summarize --plan plan.json --receipts out --out summary.json
Coverage
Against the upgrade-breakage classes in the GBRA-224 plan:
| Class | Covered |
|---|---|
| A: schema bring-up from an old version | Yes: every from-version, both engines, through apply-migrations and the schema head check |
| D: the updater targets an unpublished or wrong version | Yes: the old updater runs against the fixture; the installed tree must be the candidate's |
| H: false-healthy signals | Yes: doctor after the upgrade, and the serve answering on the candidate build |
| I: packaged skill and host migrations | Yes: apply-migrations runs every orchestrator on a brain the old release created |
| J: embedding backfill holds created by an upgrade | Yes on keyed cells: embedding a new page and vector query after the upgrade |
| C: mixed versions, stale resident processes | Partly: a stdio serve and (Postgres) a supervisor across the upgrade, one build afterwards, and the old binary on the upgraded brain; not long-lived multi-host fleets |
| E: runtime floor, Bun | Partly: the real Bun-floor read and global install on the runner's Bun; not older Bun versions |
| G: migration orchestrator chain on managed brains | Partly: the chain on fresh small brains; not managed multi-writer brains |
Not covered: scale (classes B and F stay with the production soak and the scale tier), multi-host timing, launchd and systemd restart behaviour (service calls go to stubs), custom database roles, the compiled-binary updater, and older Bun versions.
Cost
A cell takes 40 to 70 seconds on a 4-vCPU runner (install from the Bun cache,
seed, upgrade, checks, mixed-version probe); a job adds about two minutes of
checkout, bun install and cache restore. A full run is 4 cell jobs per
from-version, each about four minutes on ubicloud-standard-4, plus the
plan, summary and issue jobs on ubicloud-standard-2. With the stable window
(on 2026-11-16, candidate v0.60.729.0: 14 from-versions) that is 56 cell jobs
and 112 cells, about 900 vCPU-minutes, against 20 cell jobs (5 from-versions,
about 330 vCPU-minutes) before it. The window adds at most one from-version
per stable promotion until it holds 8 recent ones plus one per older month.
The pull-request subset is 4 jobs, about 50 vCPU-minutes.