Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Merge remote-tracking branch 'origin/capy/release-checks-fail-closed' into capy/release-per-batch

garrytan9b1a1915322c
diff --cc docs/architecture/key-files/ci-health-and-test-guards.md
index e5c682030b,b0dabad443..ab176986e0
--- a/docs/architecture/key-files/ci-health-and-test-guards.md
+++ b/docs/architecture/key-files/ci-health-and-test-guards.md
@@@ -9,13 -9,13 +9,13 @@@ Current behavior and load-bearing invar
- `scripts/lib/master-red.ts` — master-red reconciliation for push-to-master runs of Test and E2E Tests (`PUSH_ALLOWLIST`). Each watch evaluates every completed push run above the high-water mark (`evaluated`: run number, then attempt) stored in the newest `Master red: <workflow>` issue's `master-red:json` block (`applyRun` folds runs; cancelled runs change nothing); a missing or malformed block is rebuilt from the actions API (`pendingRuns`). It opens on red with the suspect range (`suspectRange`: compare plus commit-to-PR lookups), updates while red, closes when every previously failing job (E2E Tests: every failing file, from the manifest) ran and passed with complete evidence, records an incident that opened and closed between watches, and keeps a not-run-since failure open with the manual-close step. A newest issue closed by hand while its block is red ends that incident (`closed_by_hand`): later runs are recorded on it as green, and only a new red run opens a new issue. `externalPass` lets a scheduled run or an issue-bound E2E replay clear failures. Closing without a merged linked repair PR opens `flake` issues (`flakeFor`, `flake:json`); `closeFlakes` closes one after its repair PR merged and a later complete run passed its file and arm. Writes re-read the issue first and re-plan when it moved; a create race closes the newer duplicate. Tests: `test/scripts/nightly-issue.test.ts`. Docs: `docs/ci-red-runbook.md`.
- `scripts/lib/ci-run.ts` — one run's failure evidence (`collectEvidence`): jobs, failing jobs' annotations and, for test.yml and e2e.yml, the `ci-manifest` artifact, returned as structured records (job, step, every failing test file and identity) apart from sanitized excerpts; a cancelled job is not run (listed in `cancelled`) unless its annotation says it timed out, and a red run whose only failures are aggregators of cancelled jobs is `ignored`; with `complete`/`problems` (an unreadable page, annotation or manifest is incomplete evidence, never zero failures). A push run with a succeeded `reused-full-suite` job is judged by the run its `reuse.json` names (`checkRecord` from `scripts/ci-reuse.ts`, recorded attempt, identical commit tree): that run's succeeded jobs and complete `ci-manifest` stand in for the push, and any gap is incomplete. `reproduceCommand` builds `bun run test:stress <file> --iterations 10 [--postgres]` from a validated test path only. `exemptionRows` turns failing identities with an exact test name and signature into the stress gate's `<!-- gbrain-stress-exemption -->` records (backend per `stressBackend`, owner `agent on release duty`), written into master-red, flake and nightly-red bodies; `test/fixtures/nightly-watch/stress-exemption-contract.ts` vendors the gate's parser so the tests prove it accepts them.
- `scripts/ci-manifest.ts` — the versioned failure and execution manifest (`gbrain-ci-manifest/v1`). The `executed-receipts` job of test.yml and e2e.yml builds it from every lane's receipts (via `scripts/ci-executed-counts.ts`, declared skips from the expected-deltas TSV), binds it to the workflow, event, SHA, run id and attempt, and uploads it as `ci-manifest`; each failure carries the JUnit message (credentials redacted) and the stress gate's `failureSignature` (omitted when the message holds a credential); `checkManifest` refuses a manifest for another run or attempt; `passedFiles` lists files that executed tests and failed none. Docs: `docs/ci-red-runbook.md#ci-failure-manifest`.
- - `scripts/release-gate.ts` — release.yml's `ci-gate` job. Polls the push-to-master runs of Test and E2E Tests on the release commit and writes `publish`/`sha`: both green publishes; a failed run publishes nothing and fails with the next step; a cancelled or skipped run publishes nothing and fails with `gh run rerun <id>` (master push runs are keyed by sha, and between releases many commits share one VERSION, so it never follows to a later commit); non-master refs are refused; it fails with the backfill command after 110 minutes. Build, release, `latest-stable` and the publish jobs use the gated commit. Tests: `test/scripts/release-gate.test.ts`. Docs: `docs/RELEASING.md#release-ci-gate`.
+ - `scripts/release-gate.ts` — release.yml's `ci-gate` job. Polls the push-to-master runs of Test and E2E Tests on the release commit and writes `publish`/`sha`: both green publishes; a failed run publishes nothing and fails with the next step; a cancelled or skipped run publishes nothing and fails with `gh run rerun <id>` (master push runs are keyed by sha, and between releases many commits share one VERSION, so it never follows to a later commit); non-master refs are refused; a `workflow_dispatch` (`--event`) gates only the release commit (the first-parent commit that last changed VERSION) and refuses a later master HEAD with the re-run of that commit's push Release run; it fails with the backfill command after 110 minutes. Build, release, `latest-stable` and the publish jobs use the gated commit. Tests: `test/scripts/release-gate.test.ts`. Docs: `docs/RELEASING.md#release-ci-gate`.
-- `.github/workflows/promote-stable.yml` + `scripts/promote-stable.ts` — stable channel promotion (`workflow_dispatch`: `version`, `soak_run_url`, `scoreboard_url`, `rollback`, `reason`, `withdraw`; concurrency group `promote-stable`). `promote()` refuses unless the actor's `role_name` is admin or maintain (`authorize()`; the one exception is `PromoteDeps.scheduled`, set by `scheduleAuthorized()` only for a schedule event of `stable-daily.yml` on the default branch, which also requires the release commit on master and records `promoted_by: schedule (.github/workflows/stable-daily.yml)`), `v<VERSION>` exists with a `VERSION` file naming it and a published (non-draft) GitHub release, the soak run URL names a completed, successful run of this repository's `STABLE_SOAK_WORKFLOW` workflow (repository variable; bare file name or path), the run's head commit has the release tag's exact tree (fetched when missing), and the version is newer than the current stable (the highest `v*` tag on `refs/tags/stable^{commit}`), or older with `rollback` and a reason. Re-promoting the current stable (no rollback) moves no tag: `recordCurrentStable()` writes `stable.json` from the stable tag's message (`parseTagMessage()`, which must name the same commit, tree and soak run) when the published record does not name that release, leasing `stable` unchanged and the record branch, and refuses as a no-op otherwise (GBRA-346 backfill for releases promoted before the record existed). It creates the immutable annotated `stable-v<VERSION>` record (refused when it exists, except on a rollback re-promotion of the same commit) and moves the annotated `stable` tag, both with `renderTagMessage()`'s `key: value` body, commits the evidence record (`renderRecord()`, `stable.json`, `known_bad` carried forward and merged with `withdraw` by `mergeKnownBad()`) on the `stable-record` branch, then pushes all three in one `--atomic` push (`stable` and `stable-record` leased on the values it read, so a failed push changes nothing) and writes the step summary. Side effects come in through `PromoteDeps`. Tests: `test/scripts/promote-stable.test.ts`. Docs: `docs/RELEASING.md#stable-channel-promotion`.
-- `.github/workflows/stable-daily.yml` + `scripts/stable-daily.ts` — the daily stable check (05:17 and 13:17 UTC; dispatch defaults to `dry_run`, and `dry_run=false` needs admin or maintain; concurrency group `promote-stable`). Off when `STABLE_AUTO_PROMOTE` is `off`, paused while the stable record says `rollback: true`. Acts only on a reason: a merged `stable:important` PR released but not in stable, or stable older than `WEEKLY_FLOOR_DAYS` (7) with a newer release. Promotes, through `promote()`, the newest release on master newer than stable whose exact tree passed a full `SOAK_SHAPE` `STABLE_SOAK_WORKFLOW` run with every `INTEGRITY_GATES` gate passing in every cell (on the important-only path it must contain the fix); otherwise dispatches one soak on the newest release's tag, never while a soak runs or after a soak on a newer release failed to qualify. Tests: `test/scripts/stable-daily.test.ts`. Docs: `docs/RELEASING.md#daily-stable-check`.
+- `.github/workflows/promote-stable.yml` + `scripts/promote-stable.ts` — stable channel promotion (`workflow_dispatch`: `version`, `soak_run_url`, `scoreboard_url`, `rollback`, `reason`, `withdraw`, `not_promotable`, `not_promotable_reason`, `not_promotable_fixed_in`, `not_promotable_remove`; concurrency group `promote-stable`). `promote()` refuses unless the actor's `role_name` is admin or maintain (`authorize()`; the one exception is `PromoteDeps.scheduled`, set by `scheduleAuthorized()` only for a schedule event of `stable-daily.yml` on the default branch, which also requires the release commit on master and records `promoted_by: schedule (.github/workflows/stable-daily.yml)`), `v<VERSION>` exists with a `VERSION` file naming it and a published (non-draft) GitHub release, the soak run URL names a completed, successful run of this repository's `STABLE_SOAK_WORKFLOW` workflow (repository variable; bare file name or path), the run's head commit has the release tag's exact tree (fetched when missing), and the version is newer than the current stable (the highest `v*` tag on `refs/tags/stable^{commit}`), or older with `rollback` and a reason. Re-promoting the current stable (no rollback) moves no tag: `recordCurrentStable()` writes `stable.json` from the stable tag's message (`parseTagMessage()`, which must name the same commit, tree and soak run) when the published record does not name that release, leasing `stable` unchanged and the record branch, and refuses as a no-op otherwise (GBRA-346 backfill for releases promoted before the record existed). It creates the immutable annotated `stable-v<VERSION>` record (refused when it exists, except on a rollback re-promotion of the same commit) and moves the annotated `stable` tag, both with `renderTagMessage()`'s `key: value` body, commits the evidence record (`renderRecord()`, `stable.json`, `known_bad` carried forward and merged with `withdraw` by `mergeKnownBad()`, `not_promotable` carried forward, added to and removed from by `mergeNotPromotable()`, which caps a reason at 500 characters; a record over `STABLE_RECORD_MAX_BYTES`, the client's `RECORD_MAX_BYTES`, is refused before the push) on the `stable-record` branch, then pushes all three in one `--atomic` push (`stable` and `stable-record` leased on the values it read, so a failed push changes nothing) and writes the step summary. Side effects come in through `PromoteDeps`. Tests: `test/scripts/promote-stable.test.ts`. Docs: `docs/RELEASING.md#stable-channel-promotion`.
+- `.github/workflows/stable-daily.yml` + `scripts/stable-daily.ts` — the daily stable check (05:17 and 13:17 UTC; dispatch defaults to `dry_run`, and `dry_run=false` needs admin or maintain; concurrency group `promote-stable`). Off when `STABLE_AUTO_PROMOTE` is `off`, paused while the stable record says `rollback: true`. Acts only on a reason: a merged `stable:important` PR released but not in stable, or stable older than `WEEKLY_FLOOR_DAYS` (7) with a newer release. Promotes, through `promote()`, the newest release on master newer than stable whose exact tree passed a full `SOAK_SHAPE` `STABLE_SOAK_WORKFLOW` run with every `INTEGRITY_GATES` gate passing in every cell (on the important-only path it must contain the fix), skipping a release with a `stable-v<version>` tag (stable before, then rolled back) or on stable.json's `known_bad` (#8792) or `not_promotable` list; otherwise dispatches one soak on the newest release's tag, never while a soak runs or after a soak on a newer release failed to qualify. Tests: `test/scripts/stable-daily.test.ts`. Docs: `docs/RELEASING.md#daily-stable-check`.
- `scripts/check-install-ref.ts` (`bun run check:install-ref`, in verify) — every install pin, bootstrap runbook fetch and setup-helper fetch in tracked files (not CHANGELOG.md or test/) names `INSTALL_REF` from `src/core/install-ref.ts`; exact `v<version>` and commit pins are left alone; TypeScript must import the constant; `--write` rewrites the text surfaces. `scripts/check-bootstrap-tag.sh` reads its sanctioned ref from the same constant. Tests: `test/install-ref.test.ts`.
- `scripts/regen-all.ts` — one entry point for every generated artifact (`ARTIFACTS`: regen and check command per artifact). `bun run regen:all` regenerates offline and keyless (provider keys and database URLs removed) and lists what changed; a second run changes nothing. `--check` is read-only and runs in `verify` as `check:regen-all`; `--goldens` also regenerates the offline contract goldens. Postgres goldens are named with their own commands, and `llms.txt` regenerates last because it inlines other generated docs. Every `bun test` it spawns (two checks and the goldens) carries `--timeout=60000` through one `BUN_TEST` prefix, because Bun does not apply bunfig.toml's `[test] timeout`. Tests: `test/scripts/regen-all.test.ts`.
- - `scripts/build-behavior-changes.ts` — writes (or `--check`s) `src/core/behavior-changes.generated.ts` from `behavior-changes.d/<id>.md` and the `<!-- behavior-change: <id> -->` markers in released CHANGELOG sections (and in `changelog.d` fragments while VERSION is newer than the top heading, i.e. inside a release PR): `RELEASED_BEHAVIOR_CHANGES` rows dated by their section, `BEHAVIOR_CHANGE_RELEASES` for `releasedIn(id)` rows. Refuses unmarked files, dangling markers, duplicate ids, bad names and texts over 2,000 characters. The release PR's `release_tool`; in `regen:all`. Tests: `test/scripts/build-behavior-changes.test.ts`.
- - `scripts/check-behavior-since.ts` — refuses a literal `since: 'TBD'` row under `merge: "direct"`; `--stamped` (release.yml ci-gate) also refuses an unfolded `changelog.d` fragment and a stale behavior-change table. Tests: `test/scripts/check-behavior-since.test.ts`.
+ - `scripts/build-behavior-changes.ts` — writes (or `--check`s) `src/core/behavior-changes.generated.ts` from `behavior-changes.d/<id>.md` and the `<!-- behavior-change: <id> -->` markers in released CHANGELOG sections (and in `changelog.d` fragments while VERSION is newer than the top heading, i.e. inside a release PR): `RELEASED_BEHAVIOR_CHANGES` rows dated by their section, `BEHAVIOR_CHANGE_RELEASES` for `releasedIn(id)` rows. Refuses unmarked files, dangling markers, duplicate ids, bad names, texts over 2,000 characters and a `changelog.d` fragment whose `bump:` is not `patch`. The release PR's `release_tool`; in `regen:all`. Tests: `test/scripts/build-behavior-changes.test.ts`.
+ - `scripts/check-behavior-since.ts` — refuses a literal `since: 'TBD'` row under `merge: "direct"`; `--stamped` (release.yml ci-gate) also refuses an unfolded `changelog.d` fragment, a stale behavior-change table and a missing or unparsable `.gstack/ship-policy.json`. Tests: `test/scripts/check-behavior-since.test.ts`.
- `scripts/release-restamp.ts` + `scripts/lib/restamp.ts` — `bun run release:restamp`, run when a PR is next to merge. The CLI refuses a dirty tree, fetches `origin/master`, captures the branch CHANGELOG entry and migration inventory, merges master with a merge commit (version-only stamp hunks, `CHANGELOG.md` and the `GENERATED_PATHS` files resolve mechanically; anything else stops), sets VERSION to master's PATCH + 1, renumbers branch-only migrations from master's latest + 1 (filename, export name and `version:` literal only; `migrationPayloadKey` proves the rest is token-identical and refuses a copy of a published migration or an edited published one), rewrites `STAMPS` (the CLAUDE.md "Version locations" Required rows) plus CHANGELOG and TODOS, runs `DEFAULT_STEPS`, lists branch-added lines that still name an old number (`findLeftoverRefs`, never rewritten), runs `DEFAULT_CHECKS` and commits once. Stops save `.git/gbrain-restamp.json` for `--continue` / `--abort`; `--dry-run` prints the plan, `--no-commit` stages. Generators run through `offlineEnv` (no database URLs, no provider keys). Modes for releases per batch (`docs/RELEASING.md#releases-per-batch`): `--stamp-only <version>` (the release PR's `restamp_command`: every `STAMPS` row plus the plugin, variant and template stamp lines; no fetch, merge or commit), `--unstamp` (`bun run release:unstamp`: merge master, stamps back to master's version, the entry to `changelog.d/<branch-slug>.md`, appended plain-text notice rows to `behavior-changes.d/`; refuses multi-entry branches, released-section edits and rows it cannot move) and `--renumber` (`bun run migrations:renumber`); plain restamp refuses when master's policy says `merge: "direct"`. Tests: `test/scripts/release-restamp.test.ts` (fixture repos with the real migration generator; also pins CLAUDE.md table coverage and docs anchors).
- `scripts/ci-convergence-scope.sh` — #6712 PR 0: classifies a pull request's changed files (stdin) for the managed-sync convergence gate (`persistence-validation.yml` `convergence`): `run` for persistence, sync commands and sync core, page projections and storage, `bun.lock` and `vendor/postgres/`, the harness (`test/sync-convergence.serial.test.ts`, `test/fixtures/convergence-fault-kill.ts`), this script and its workflows (`persistence-validation.yml`, `convergence-nightly.yml`); `skip` otherwise, and never selected by `package.json` alone, so a release-stamp PR does not run it; an empty list runs. test.yml `changes` calls it for pull requests only (step `convergence`, passed as the `convergence` input). Tests: `test/scripts/ci-pr-scope.test.ts`.
- `scripts/ci-manifest-diff.sh` — parsed-JSON classification of manifest diffs for CI scoping: `version-only` (only the top-level `version` differs), `dependency-change` (any dependency field differs), `dependency-scope` (whether a changed-file list touches lockfiles, `patches/` or dependency fields) and `drop-version-only` (filters version-only `package.json` and `openclaw.plugin.json` entries out of a changed-file list, so a release bump does not widen the native scope). Used by test.yml's `changes` job (before `scripts/ci-native-scope.sh`) and its dependency-audit scope step. Tests: `test/scripts/ci-manifest-diff.test.ts`.