Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Merge master ff808fe1 into capy/effect-git-secret-gate

garrytand183681109cb
diff --cc CHANGELOG.md
index dcf805eb86,8bffe1445c..c33772c3eb
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@@ -10,21 -10,51 +10,66 @@@ credits are retained; no result has bee
identifiers and attribution are available in the pre-removal Git revision
`6040075c6cb95be5881cc2e1b76ef7d71f4e5d29` (retained on 2026-09-23).
+## [Unreleased]
+
+**The Git effects of page writes no longer publish a script's secret.** A script that commits into a managed checkout with plain `git commit` never passes the write path's secret scan. The refresh publish refused such a commit with `publish_secret_found`, but the next page write's Git effect pushed it anyway. Its plain push published every unpublished commit unscanned, and when the remote had moved its keep-checkout publish ran without the scan. Both paths now scan the exact commits they push, except the ones gbrain made itself for page writes (a Git effect's commit, recorded only after checking it holds exactly the bytes the effect wrote, or a hosted repository's door commit, as its commit ledger records them): page bodies are stored raw by design and their Git backups keep going, and the refresh publish now skips them the same way. A value that looks like a secret, added text too large to scan, a commit that changes `.gbrain-scan-allow` or a `.gitattributes` holds the effects with `publish_secret_found`; commits the scan cannot read in full hold them with `publish_scan_unreadable`. A first push to a remote that has no such branch yet scans the branch's whole history and creates the branch only when that scan passes. A commit that changes `.gbrain-scan-allow` or a `.gitattributes` is never published automatically, so the user reviews it and pushes it. Nothing is pushed, and the page writes stay safe in the database and in local commits. `gbrain sync status` and doctor name the user's choice, which is to remove the value from those commits or to allowlist a false positive by fingerprint. Then `gbrain sources writer retry-effects <source> --parked-pushes` runs the effects that parked. Every scanned push is also leased on the remote tip it scanned against. Before, a remote rewound past a pushed secret (an admin removing it with a force-push) left a stale tracking ref that hid the commit from the scan, and the next fast-forward put it back; now the scan runs against the remote's real tip. The in-process scan on the owner's event loop is capped at 10 seconds, inside the writer heartbeat's live window; a longer range goes to the keep-checkout publish, whose child process scans with the full budget.
+
+### Itemized changes
+
+- `src/core/persistence/effect-git.ts`: `pushGitRoot` reads the remote branch tip with `git ls-remote`, scans `<tip>..<HEAD sha>` and pushes that sha only as a fast-forward of the tip, leased on it; a tip that is not an ancestor, or a remote that moves before the push lands (`stale info`), goes to the keep-checkout publish, which now runs with `--scan-secrets`. The in-process scan gets 10 s; a range it cannot finish goes to the keep-checkout publish. A remote without the branch gets HEAD's whole history scanned and the branch created with a lease that it does not exist; one too large to scan in process holds with `publish_scan_unreadable` naming `git ls-remote --heads`. A refusal from either path holds the effects as `publish_secret_found` with the user's fix.
+- `src/core/git/commit-push.ts`: a `scanSecrets` publish scans `<base>..<candidate>` before every push and leases the push on that base, so each merge with a newly fetched remote tip is scanned again. It also scans `<remote tip>..<candidate>` before merging with a newly fetched tip, so a remote rewound past a pushed secret that also changed a file this machine changed refuses `publish_secret_found` rather than `integrate_conflict`, and a conflict's resolve command keeps `--scan-secrets`.
+- `src/core/git/range-secret-scan.ts`: `rangeSecretRefusal`, the one refusal both callers throw (`publish_scan_unreadable` for a scan that could not read the range, `publish_secret_found` otherwise; a gate-file change says the user pushes it). `scanRange` marks a scan that ran past its budget (`timed_out`) and scans a whole history when the base is empty. It skips commits the commit ledger records as page writes (`effect`, `door`, `hosted_repo`); `gbrain git commit-push` commits, adopted outside edits and commits the ledger no longer holds are still scanned, and an unreadable ledger means everything is scanned. A Git effect now makes its commit by plumbing (`commit-tree` over its staged paths, then `update-ref HEAD <new> <old>`, which refuses when HEAD moved, then `git maintenance run --auto --detach` as `git commit` does, without `--detach` on git before 2.47), so the ledger records a sha gbrain created before any ref names it. The scan skips an effect commit only by that sha, recorded `verified` when every staged file holds the bytes whose sha256 the effect recorded (the page walk passes the hash it checked too). An amend, reset or reword of it is a different commit and is scanned, a page file changed before it was staged leaves it unverified, and records written before this check existed are scanned. Its time budget is also checked while the diff is parsed.
+- `src/commands/doctor/checks/git-convergence.ts`, `src/core/persistence/worktree-refresh.ts`, `src/core/persistence/git-push-holds.ts`: every publish fix they name for a managed checkout (stranded, after committing its modified pages, diverged, `refresh_diverged`, and a held Git effect's conflict resolve) runs `gbrain git commit-push --push-only --keep-checkout --scan-secrets`, so a script's commit is scanned there too. This changes what the user sees. A commit that changes `.gbrain-scan-allow` or a `.gitattributes` now refuses `publish_secret_found` when the user runs the diverged fix or the `refresh_diverged` publish, where before their agreement published it.
+- `src/core/git/commit-push.ts`, `src/core/persistence/git-conversion.ts`, `src/commands/git.ts`: the publish commands the other messages hand back keep the scan too: a scanned publish's rerun hints (a push the remote keeps rejecting, a busy publish lock, commits outside the source root), the refresh's publish-by-hand line when it cannot find its CLI, the conversion's pending-push notice and a managed `commit-push -m` that leaves commits unpublished all name `--keep-checkout --scan-secrets`.
+- `src/core/persistence/git-push-holds.ts`: `publish_secret_found` and `publish_scan_unreadable` are held push codes, so doctor, `gbrain sync status` and `retry-effects --parked-pushes` cover them, with an ask-the-user fix (the refused commits, or `git fsck`).
+- Docs: `docs/guides/shared-checkout-git.md` (`publish_secret_found` and `publish_scan_unreadable` rows), `docs/guides/live-sync.md`.
+- Tests: `test/effect-git-secret-gate.test.ts` (an ahead checkout, a diverged one, a remote rewound past a pushed secret, a `.gitattributes` change and a remote without the branch whose history holds a secret each refuse with nothing pushed; a clean ahead checkout pushes, a clean history creates a missing branch, and a scan past the in-process budget still scans in the child; through the effect worker the page write's effect holds, sync status names the fix, and an allowlisted false positive releases it, parked or not; a page write whose body holds a credential-shaped value still backs up; a script commit, amend or reword racing the effect commit, a page file changed after the effect checked it, and an unverified ledger record each hold with nothing pushed; a source-scan walk's page with a credential-shaped body still backs up; a repaired unreadable commit releases its hold); `test/persistence-git-removal-push-7089.test.ts` and `test/persistence-git-subdir-root-7091.test.ts` (doctor's push fixes scan); `test/guarded-publish.test.ts` (a remote rewound past a pushed secret is not published again); `test/git-literal-pathspec-hooks-7759.test.ts` publishes its probe `.gitattributes` commit first, since the effect's push now refuses one; `test/cli-flag-equals.test.ts` lists `range-secret-scan.ts` as a git child argv reader.
+
+ ## [0.60.660.0] - 2026-11-11
+
+ **Agent reads answer "nothing here" instead of refusing, and say why a list came back empty.** These fixes come from agent transcripts where models kept getting refused or got bare empty answers.
+
+ ### Itemized changes
+
+ - `get_backlinks` with a `type` that other pages in the brain have, but no page linking to this one has, now returns the normal empty answer plus a note listing the types the linking pages do have. Before, it refused with `unknown type`, so an agent walking several accounts by type was refused once per account. A type no page has is still refused, and the refusal now also lists the brain's own page types. For a remote caller, both checks count only pages that caller can read. A type found only on private pages, pages whose readers leave the caller out, deleted pages or another source is treated as a type that does not exist and is never named.
+ - `list_pages` with a `type` that no page has still returns the same empty list, but now adds a note naming the page types the brain does have, most common first (up to 12, with counts). Before, a guessed type such as `account` on a brain that files accounts as `crm` came back as a bare `[]`, which reads like "no such pages". A remote caller's note names only types of pages it can read.
+ - `search` with `match: "keyword"` now labels every row that matches the query as written `keyword_exact`, including rows from your own `"A" OR "B"` query or a one-word query. Their match scores sit below the cutoff the label used to need, so they were labelled `weak_semantic`, which told agents the batched lookup the instructions recommend had found nothing solid. `create_safety` (the "don't write a duplicate" hint) is unchanged for these rows. Rows found only by gbrain's own fallback of loosening a query to any-word matching stay `weak_semantic`. A query that is an entity's name, which keyword search also runs under the entity's other names, keeps the labels it had before, and default (hybrid) search is unchanged.
+ - `request_tools` accepts `tools` and `surface` in one call. It changes the tool surface exactly as a `surface`-only call does (same limits and refusals), then returns the requested tool schemas, with the surface answer under `surface_change`. Before, the pair was refused, and some models send it every night. Calls with only one of the two answer as before.
+
+ ## [0.60.659.0] - 2026-11-11
+
+ ### A page purge removes the page's name from the volunteer log and the mention gazetteer
+
+ **Purging a page now also deletes the rows that kept its name next to its slug.** Before, `delete_page purge:true` and the retention purge left two stores behind, because neither has a foreign key to `pages`: `context_volunteer_events` kept the slug with a rationale that quotes the matched name (`alias match "<name>"`) for up to 90 days, and `mention_gazetteer_entries` kept the page's normalized title and aliases until that source's next mention pass. The purge receipt called the page gone while both were still readable. Reported and first fixed by @alanshurafa (#9072).
+
+ | What you did | Before | Now |
+ |---|---|---|
+ | `delete_page purge:true`, the expired-tombstone purge, or a restore that replays a page purge | the slug and name stayed in the volunteer log and the mention gazetteer | removed in the purge transaction, and the page receipt checks both stores |
+ | `purge_fact` on a claim that is also another page's name or quoted in a volunteer rationale | the gazetteer was not checked | both stores are probed and a match is listed as a residual |
+
+ - `src/core/persistence/purge-name-stores.ts` (new): `scrubPurgedNameStores` deletes the source's rows for the purged slugs from both stores. `purgePageInTransaction`, `purgeDeletedPages`, `expireRecordedTombstones` and the restore-time purge replay call it beside the retrieval-feedback scrub. A soft delete keeps both, so `restore_page` still brings the page back whole.
+ - `src/core/deletion-inventory.ts`: both tables are `swept` (`page_names`); the fact receipt still probes them, since a fact purge sweeps neither.
+ - Tests: `test/purge-name-stores-9068.test.ts` (PGLite, and Postgres through `test/postgres-unit-arms.txt`).
+
+ ### Removing a source, a connector delete and a late eval capture no longer leave a deleted page's slug behind
+
+ **Four more ways a deleted page's slug outlived the page are closed, and the federation warning stops counting private pages.** Before, only a page purge scrubbed the stores that name a page without a foreign key to it.
+
+ | What you did | Before | Now |
+ |---|---|---|
+ | A query or search with eval capture on, whose capture runs after the response, racing a page purge | the capture could write the purged slug back into `eval_candidates.retrieved_slugs` | the insert keeps only slugs whose page still exists, checked under the same row lock the feedback writers take |
+ | A volunteered-context answer whose background volunteer-log write lands after a page purge (or while it is open) | the write put the purged slug and its name-quoting rationale back into `context_volunteer_events` | the write keeps only rows whose page still exists, under the same row lock |
+ | A connector, GitHub or Google sync deletes a page (`deletePages`), or a transcript re-ingest deletes a stale part | retrieval logs, ratings, weights, the volunteer log, the gazetteer and captured eval slugs kept naming it | scrubbed in the delete's transaction |
+ | `gbrain sources remove`, `sources purge`, the expired-archive sweep, or a managed `remove`/`purge` | the source's retrieval logs, ratings, volunteer rows and captured eval slugs stayed | deleted in the statement that deletes the source, and swept again right after for a write that raced it, unless a source with the same id was created in between |
+ | `gbrain sources federate` or `sources restore` on a source with a remote-written `visibility: private` page and no readers | the warning named that page, though no-grant connections still cannot read it | the warning applies the read paths' privacy rules, unless `search.remote_private_pages` or `GBRAIN_REMOTE_PRIVATE_PAGES` relaxes them |
+
+ - `src/core/feedback/store.ts`: `insertEvalCandidate` is the one eval capture insert both engines run. A facts-arm result (`facts/<id>`) names a fact, not a page, so it stays while that fact exists.
+ - `src/core/context/volunteer-events.ts`: `insertVolunteerEvents`, behind the fire-and-forget volunteer-log sink, inserts only rows whose page exists, probed the same way.
+ - `src/core/engine-sql/pages.ts`: `deletePages` runs `scrubPurgedSlugs` and `scrubPurgedNameStores` after the delete, in one transaction. The transcript importer's stale-part delete now calls `deletePages`. `deletePage` is unchanged: the page purge, its main caller, scrubs and records the stores in its receipt.
+ - `src/core/source-delete.ts`: `removedSourceScrubCtes` runs inside `deleteSourceRow` and the managed lifecycle's source delete, then once more after it. `deleteSourceRow` skips that second pass when a source with the same id exists again, so a re-created source keeps its rows. `retrieval_weights` and `mention_gazetteer_entries` already cascaded from `sources`.
+ - `src/core/access/federation-exposure.ts`: the exposure read adds `privacyPartSql` unless `remotePrivacyRelaxed`.
+ - Tests: `test/purge-scrub-followups-9068.test.ts` and `test/federation-exposure-private-9068.test.ts` (PGLite, and Postgres through `test/postgres-unit-arms.txt`). `test/eval-candidates.test.ts`, the volunteer round-trip in `test/migrate.test.ts` and the fact-purge case in `test/purge-name-stores-9068.test.ts` now create the pages they log, since a capture or volunteer write keeps only live pages.
+
## [0.60.658.0] - 2026-11-11
**Pull requests now run the tests that check gbrain's lists, in about 4 minutes, instead of finding those failures in the 30-minute full run.** Some tests fail whenever a change adds an operation, command, `--flag`, error code, doctor check or repair kind without updating the list that tracks it, or changes output a golden file pins. Before this release, pull requests skipped those tests and their first failure came in the full-ci run. A new check, `enumeration and census tests`, runs them on every pull request that changes more than docs. It is advisory: no merge rule reads it. `bash scripts/ci-census-tests.sh --run` runs the same tests on your own machine without Postgres.