Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

ci(harness-check): pin harness-evals v0.0.59 (0a5246d1); v0.0.9's image build no longer runs against master's hermes setup

garrytan5ec254104bd0
diff --git a/.github/workflows/harness-check.yml b/.github/workflows/harness-check.yml
index 485401e2d7..47dff14986 100644
--- a/.github/workflows/harness-check.yml
+++ b/.github/workflows/harness-check.yml
@@ -37,7 +37,7 @@ concurrency:
cancel-in-progress: true
env:
- HARNESS_EVALS_PIN: 2c45a247f67285e693ed6d63cb16710272927a19
+ HARNESS_EVALS_PIN: 0a5246d1678e6773c5f39d40c9eecc65c369e09d
jobs:
# Event parity: docs/ci-event-parity.md#classification (harness-check.yml): a fork head is refused, a missing token skips.
diff --git a/docs/ci-event-parity.md b/docs/ci-event-parity.md
index 4450073bf4..d9aeaaea59 100644
--- a/docs/ci-event-parity.md
+++ b/docs/ci-event-parity.md
@@ -181,8 +181,8 @@ a week, the agent on release duty records it on the CI health issue.
## Harness check
-`.github/workflows/harness-check.yml` runs on every pull request to master from a branch of this repository, on a 4-vCPU Ubicloud runner with a 45-minute limit. It is not a required check. It clones [harness-evals](https://github.com/garrytan/harness-evals) at the commit `HARNESS_EVALS_PIN` names (`2c45a247f6`, harness-evals v0.0.9, main on 2026-10-26), builds the gbrain sidecar image for the pull request's head commit with harness-evals' `scripts/build-gbrain-image.sh` (the keyword target: gbrain at that commit run from source, four keyword PGLite brains over the messy-company-v1 corpus, the install bundle from `gbrain skillpack scaffold`, `compile-context` and `gbrain setup`; no key anywhere), and runs the keyless rungs of `scripts/gbrain-check.ts` against that image with `--no-build`: the fake harness through harness-evals' real runner and its world's gbrain tap, one trial on the starter surface, one on the verbs surface, one with the gbrain-docs bundle mounted, one on the company brain under scope `sources` with two members, plus a direct probe of the company brain that searches and queries once per member and query under that member's `source_ids` (15 members, 344 calls, 1,062 rows in the sample run). It gates on one number: the count of pages or rows returned to a member whose grant does not hold their source (cross-scope). The job fails when that count is not zero, when the scoped probe did not run, or when no `check.json` was written (the image did not build or the check did not finish); every other failed assertion of gbrain-check (a tool count off harness-evals' pin, a missing search tool or verb, a retrieval that misses the known dev page, an install bundle or view digest, the Hermes context, the installer's brain-first block or per-turn hook files) is a `::warning::` in the job log and a line in the job summary, so a pull request that changes the tool list or the install output is told and not blocked. The only secret it reads is `HARNESS_EVALS_READ_TOKEN`, a fine-grained token with Contents: read on the private harness-evals repository; it reads no model key and runs no paid trial (the metering proxy runs with placeholder keys the fake harness never uses). When the secret is not set, the job's first step writes a `::notice::` and a "skipped" job summary naming the missing secret, every later step is skipped, and the job succeeds. A pull request from a fork is refused before that check: the image build fetches the commit from this repository, where a fork's commit does not exist. The artifact `harness-check-keyless` holds `summary.md`, `keyless/check.json` (every assertion and the evidence behind it, including `evidence.scoped` the gate reads) and `trials.jsonl`; never a trial's state or a token file.
+`.github/workflows/harness-check.yml` runs on every pull request to master from a branch of this repository, on a 4-vCPU Ubicloud runner with a 45-minute limit. It is not a required check. It clones [harness-evals](https://github.com/garrytan/harness-evals) at the commit `HARNESS_EVALS_PIN` names (`0a5246d167`, harness-evals v0.0.59, main on 2026-10-30), builds the gbrain sidecar image for the pull request's head commit with harness-evals' `scripts/build-gbrain-image.sh` (the keyword target: gbrain at that commit run from source, four keyword PGLite brains over the messy-company-v1 corpus, the install bundle from `gbrain skillpack scaffold`, `compile-context` and `gbrain setup`; no key anywhere), and runs the keyless rungs of `scripts/gbrain-check.ts` against that image with `--no-build`: the fake harness through harness-evals' real runner and its world's gbrain tap, one trial on the starter surface, one on the verbs surface, one with the gbrain-docs bundle mounted, one on the company brain under scope `sources` with two members, plus a direct probe of the company brain that searches and queries once per member and query under that member's `source_ids` (15 members, 344 calls, 1,062 rows in the sample run). It gates on one number: the count of pages or rows returned to a member whose grant does not hold their source (cross-scope). The job fails when that count is not zero, when the scoped probe did not run, or when no `check.json` was written (the image did not build or the check did not finish); every other failed assertion of gbrain-check (a tool count off harness-evals' pin, a missing search tool or verb, a retrieval that misses the known dev page, an install bundle or view digest, the Hermes context, the installer's brain-first block or per-turn hook files) is a `::warning::` in the job log and a line in the job summary, so a pull request that changes the tool list or the install output is told and not blocked. The only secret it reads is `HARNESS_EVALS_READ_TOKEN`, a fine-grained token with Contents: read on the private harness-evals repository; it reads no model key and runs no paid trial (the metering proxy runs with placeholder keys the fake harness never uses). When the secret is not set, the job's first step writes a `::notice::` and a "skipped" job summary naming the missing secret, every later step is skipped, and the job succeeds. A pull request from a fork is refused before that check: the image build fetches the commit from this repository, where a fork's commit does not exist. The artifact `harness-check-keyless` holds `summary.md`, `keyless/check.json` (every assertion and the evidence behind it, including `evidence.scoped` the gate reads) and `trials.jsonl`; never a trial's state or a token file.
-Measured by hand on 2026-10-26 against master at `b06ac32b` (0.60.465.0) on a 2-core, 7 GB Linux machine (Docker 29.9.2, bun 1.4.4): image build 359 s, check 59 s (30 assertions passed, 0 failed, 0 pin diffs, cross-scope pages 0), 457 s end to end. A CI runner is not measured yet.
+Measured by hand on 2026-10-26 against master at `b06ac32b` (0.60.465.0) on a 2-core, 7 GB Linux machine (Docker 29.9.2, bun 1.4.4): image build 359 s, check 59 s (30 assertions passed, 0 failed, 0 pin diffs, cross-scope pages 0), 457 s end to end. A CI runner is not measured yet. The pin moved to v0.0.59 on 2026-10-31: v0.0.9 failed to build its image against master once `gbrain setup hermes --instructions-only` began writing the shell-hooks allowlist lock; v0.0.59 against the train R head (`32f1748a`) built and passed (37 assertions, 0 failed, 0 pin diffs, cross-scope pages 0, check 62 s).
**Bumping the harness-evals pin.** The pin is one line in the workflow, `HARNESS_EVALS_PIN`, and it must be a commit on harness-evals main. To move it, read the new head with `git ls-remote https://github.com/garrytan/harness-evals refs/heads/main`, run the workflow's steps once by hand against master with the new pin (clone at the pin, `bun install --frozen-lockfile`, `scripts/build-gbrain-image.sh`, `scripts/gbrain-check.ts --no-build`, then the gate step's shell), and read its last lines (`gate passed` and the warning count). Then change the line and open the bump as its own pull request, so the first run under the new pin is on a commit whose gbrain did not change. A pin diff reported as a warning after a deliberate gbrain change is fixed on the harness-evals side (its `gbrain/pin.json`), and the bump that follows picks it up.