Garry's Opinionated OpenClaw/Hermes Agent Brain
TypeScript
fix: the filter-attribute read fetches nothing in a partial clone
garrytan6da840150ece
diff --git a/docs/architecture/key-files/files-and-sync-2.md b/docs/architecture/key-files/files-and-sync-2.mdindex 8f4e3d3a50..a1b8b2fa72 100644--- a/docs/architecture/key-files/files-and-sync-2.md+++ b/docs/architecture/key-files/files-and-sync-2.md@@ -26,7 +26,7 @@ Current behavior and load-bearing invariants; history belongs in Git and CHANGEL- `src/core/persistence/sync-prescreen.ts` — #6712 PR 1 (L1): the discovery prescreen and drift baseline. After `discoverManagedSync` freezes identities, `prescreenSyncEntries` reads the pinned blobs (`readTreeBlobs`, one recursive listing for a large set) and the import provenance of every Git import of an existing page, and `classifyBaseline` marks each `prescreened` (disk blob, revision and `syncImportParserFor` fingerprint all match; bypassed under `--full` and for held paths) or `drift` (`database`: the page moved, the file did not; `both`/`unknown` in a tree walk or for a `file_drift` hold). A page whose current revision a committed managed-sync request wrote, or whose writer's publication wrote to its file (a write-through `git` effect), is not guarded (`fileBackedRevisions`), and drifted pages under `storage.keep_db` prefixes leave the manifest as drift candidates (`releaseKeepDbDrift`, `recordDriftSkip`). The run passes prescreened entries through `waivePrescreenedRun` (`sync-waivers.ts`); `freezeEntry` holds a drifted import `file_drift` (`fileDriftHold`, `sync-page-fault.ts`) unless `sameContentAtOrigin` proves it a no-op. `movedDriftHolds` re-screens drift holds whose page moved. Tests: `test/sync-prescreen-drift-6712.test.ts` (both engines).- `src/core/persistence/sync-image.ts` — #6529: managed sync's image branch. With multimodal embedding on, `freezeEntry` (`sync-run.ts`) freezes an image entry as its bytes in base64 (`frozenImageIntent`; `rawHash` stays the file sha256, so the working-tree check and the hash no-op are unchanged) and `prepareManagedSyncMutation` (`sync-prepare.ts`) hands it to `prepareManagedSyncImage`, which runs `importImageFile` with `prepare` (the coordinated path `gbrain import` uses in `import-prepare.ts`): the importer decides the hash no-op, OCRs and embeds the image at preparation inside the sync budget (`--no-embed` skips both; the caption chunk's text vector is deferred to `embed --stale`), and the apply records `source_path`, releases the path's hold and re-reads the snapshot. A company plan refuses (`profile_incompatible`), a screening preparation (`sameContentAtOrigin`, the no-op waiver in `sync-waivers.ts` and `unchangedManagedImport` in `import-mutations.ts` return early for images) is refused rather than paid for, newer working-tree bytes or a vanished file refuse `source_changed`, and a Git rename publishes the new slug and soft-deletes the old page. `import-file.ts` records `source_path` on the legacy importer's hash-skip path too (`refreshSourcePath`), so no image import leaves it NULL. Tests: `test/managed-sync-images-5493.test.ts` (both engines), `test/embed-stale-images-managed-6529.test.ts`.- `src/core/persistence/sync-streamed.ts` — Wave B PR 2: managed sync of a Markdown file over the 24 MiB single-string ceiling. With `pages.segmented_storage` on, `freezeEntry` (`sync-run.ts`) freezes such a file by reference instead of holding it by its size: `streamedSyncBodyRef` hashes the sha256 and the Git blob id as a stream (sha1 or sha256 by the pinned commit's length) into `intent.body_ref` (`git_blob`, or `input_file` for an uncommitted working-tree file); the journal carries no content, and a committed entry whose working tree does not hold its pinned blob, a file over `pages.max_bytes` or a non-Markdown file is held as before. `prepareManagedSyncMutation` (`sync-prepare.ts`) hands it to `prepareManagedSyncStreamed`, which streams it through `importStreamedMarkdownFile` (`page-body/streamed-import.ts`, `coordinated`; `moved` on a rename keeps an implicit type following the new path; `slug` is the admitted page, the one recording the file as its origin, as the inline import targets) and publishes database-only, recording the origin through `bindOrigin` (B-14: a renamed page keeping its origin takes the file, no twin at the path's slug); a rename moves the page first and re-seals an unchanged body (`sealSegmentedPage`). A file the streamed path refuses (a facts fence) fails its request with the refusal's code and the run holds it (`streamedRefusalHold` in `sync-screen.ts`); a dry run lists it in `screen_skipped` as `page_body_segmented`; `groupableIntent` (`sync-group.ts`) keeps it out of bulk groups. Tests: `test/large-pages/managed-sync-streamed.test.ts` (both engines), `managed-sync-streamed-crash.test.ts` (owner killed at `consumer:prepared`), `managed-sync-streamed-memory.slow.test.ts` (VmHWM), `streamed-sync-origin-b14.test.ts` (both engines).-- `src/core/persistence/sync-entry-fault.ts` — GBRA-133 (#6712 PR 4b): the per-entry faults of a managed sync freeze, so one odd file never stops its source. `nullBytesRefusal` holds a Markdown text with U+0000 `content_null_bytes` (line of the first NUL; a UTF-16 byte order mark says save as UTF-8), from `screenFrozenImport` with holds on and as a typed run stop under `sync.holds=fail`. `filterOwnedRefusal` (GST-2312, #9024), checked first on the same two paths, holds `content_filter_owned` a blob starting with git-crypt's header or a committed blob whose path names a Git filter driver other than `lfs`, configured here or not (`pinnedFilterDriver` in `sync-screen.ts`, at the pinned commit or a company plan's revision, read with `readFilterDrivers`' batched `git check-attr --source` on git 2.40+, or `read-tree` into a temporary index and `check-attr --cached` on an older git; `NO_REPOSITORY_PROGRAMS` keeps `core.fsmonitor` and hooks off these reads). `committedContentOrMissing` (called by `pinnedContent` when the pinned read fails) imports a committed file whose blob object is gone from working-tree bytes that hash to that blob id, else throws `git_object_missing`; a committed name with U+FFFD that names nothing on disk (a name that is not UTF-8) throws `path_not_utf8` first. `directoryAt` lets a deletion whose name became a directory apply (freeze and the owner's raw-hash check). `freezeFaultHold` is `freezeEntry`'s catch (`sync-run.ts` wraps `freezeEntryOnce`): `git_object_missing` and `path_not_utf8` are held under their codes; a page-body fault (`PAGE_BODY_FAULT_CODES`: `page_body_segmented`, `page_body_rows_unreadable`, e.g. a segmented page whose segment rows don't cover its body) is held `page_write_failed` with `write_failure.code` set to that code, which `sync status` classifies by the `freeze_page_body_*` rows of `SYNC_FAULT_TABLE`; any other `OperationError` and every run-scoped error (`classifyItemFailure` scope run, connection, statement timeout, lock, back-pressure, abort) rethrows, and an unexpected throw is held `page_write_failed` with `write_failure.origin: 'freeze'` and `request_id: null`, re-tested by every sync until held `HOLD_ATTEMPTS_NEEDS_HUMAN` times; a run with `ITEM_BREAKER_MIN_ITEMS` faults and no more clean freezes than faults is systemic and rethrows.+- `src/core/persistence/sync-entry-fault.ts` — GBRA-133 (#6712 PR 4b): the per-entry faults of a managed sync freeze, so one odd file never stops its source. `nullBytesRefusal` holds a Markdown text with U+0000 `content_null_bytes` (line of the first NUL; a UTF-16 byte order mark says save as UTF-8), from `screenFrozenImport` with holds on and as a typed run stop under `sync.holds=fail`. `filterOwnedRefusal` (GST-2312, #9024), checked first on the same two paths, holds `content_filter_owned` a blob starting with git-crypt's header or a committed blob whose path names a Git filter driver other than `lfs`, configured here or not (`pinnedFilterDriver` in `sync-screen.ts`, at the pinned commit or a company plan's revision, read with `readFilterDrivers`' batched `git check-attr --source` on git 2.40+, or `read-tree` into a temporary index and `check-attr --cached` on an older git; with `HARDENED_GIT_ARGS` and `GIT_NO_LAZY_FETCH=1`, so the attribute read runs no repository program and fetches nothing; it first checks with `ls-tree` and `cat-file --batch-check` that each `.gitattributes` that applies is present, and a path under a missing one makes `pinnedFilterDriver` throw `git_object_missing` via `missingAttributesRefusal`; `NO_REPOSITORY_PROGRAMS` keeps `core.fsmonitor` and hooks off the module's other reads, whose blob reads this change leaves as they were). `committedContentOrMissing` (called by `pinnedContent` when the pinned read fails) imports a committed file whose blob object is gone from working-tree bytes that hash to that blob id, else throws `git_object_missing`; a committed name with U+FFFD that names nothing on disk (a name that is not UTF-8) throws `path_not_utf8` first. `directoryAt` lets a deletion whose name became a directory apply (freeze and the owner's raw-hash check). `freezeFaultHold` is `freezeEntry`'s catch (`sync-run.ts` wraps `freezeEntryOnce`): `git_object_missing` and `path_not_utf8` are held under their codes; a page-body fault (`PAGE_BODY_FAULT_CODES`: `page_body_segmented`, `page_body_rows_unreadable`, e.g. a segmented page whose segment rows don't cover its body) is held `page_write_failed` with `write_failure.code` set to that code, which `sync status` classifies by the `freeze_page_body_*` rows of `SYNC_FAULT_TABLE`; any other `OperationError` and every run-scoped error (`classifyItemFailure` scope run, connection, statement timeout, lock, back-pressure, abort) rethrows, and an unexpected throw is held `page_write_failed` with `write_failure.origin: 'freeze'` and `request_id: null`, re-tested by every sync until held `HOLD_ATTEMPTS_NEEDS_HUMAN` times; a run with `ITEM_BREAKER_MIN_ITEMS` faults and no more clean freezes than faults is systemic and rethrows.- `src/core/url-clone.ts` — gbrain-owned `sources add --url` clones (#6946, GBRA-130 C-2). `isOwnedClone` (config `managed_clone` or a `local_path` under the default clone dir) and `defaultCloneDir`; `urlCloneReadOnly` is the one predicate (owned clone on a brain without managed persistence) the fence writer (`fence-write.ts`, `databaseOnlyReason: 'url_clone'`), write-through (`skipped: 'url_clone'`), timeline write-through, unfenced-facts adoption, the empty-fence guard and `sourceMirrorReadOnly` consult, so nothing gbrain writes lands in a checkout it updates with `git pull --ff-only`. `urlClonePullBlock` reads the tracked changes of a clone whose pull failed and returns `pull_blocked_local_changes` with an `ask_user` fix naming the preview `gbrain repair url-clone-fences --source <id>`; `formatUrlClonePullBlock` renders it for sync's stderr (`sync/preflight.ts`) and `cycle.ts` puts it in the sync phase detail as `pull_blocked`. Test: `test/pathological/c2-url-clone-fences.test.ts`.- `src/core/url-source-origin.ts`, `src/commands/sources-record-origin.ts` and `src/commands/doctor/checks/url-source-origin.ts` — who added a URL source (`sources.config.added_by_remote_caller`). Both add paths write it (`true` for a remote caller, `false` for the local operator); only an explicit `true` reads as remote. URL sources added before that record existed carry none, and nothing stored tells a remote caller's add from the local operator's (same row, same default clone directory); a clone outside `$GBRAIN_HOME/clones/<id>` (or a `clones/<id>` under an earlier home) proves a local add. A URL clone counts when `isOwnedClone` accepts it, so an unmarked clone at the default directory is listed too. `urlSourcesWithoutOrigin` (read-only) feeds the wave check `url_source_origin` (host-only, `resolution: 'operator'`, `warn` with an `ask_user` fix whose argv is the preview). `recordSourceOrigin(engine, ids, { origin, apply, remote })` backs `gbrain sources record-origin <id>... --remote|--local [--apply] [--json]`: it throws `trusted_local_only` unless `remote === false`, previews unless `apply`, writes `true` (`--remote`) or `false` (`--local`) in one transaction that re-reads the row `FOR UPDATE` and refuses `source_changed` if its path or URL moved; on a managed brain that transaction runs inside `withTopologyLocks` after `lockTopologyRows`, the locks a reclone takes (a held lock refuses the id `source_busy`), and `runManagedSourceClone` takes the config it clones with from its own locked admission read, so a record and a reclone never interleave. `--remote --apply` also writes `LFS_OFF_CLONE_CONFIG` into the repository's shared config (`--git-common-dir`; `git config --file`, not run in the checkout) before the row, reads the effective values back (`lockDownGaps`) and throws when a worktree-scoped or included setting overrides one, and locks down an already-remote source whose checkout lacks it (no git-lfs refusal for it unless Git cannot place the checkout, `lfs_unknown`); each key's existing values are removed before its one value is written, in a `config.lock` copy renamed over the config in one step; a throw is reported for its id as `write_failed` and the batch continues. It never removes a `true`, and refuses `--remote` for a checkout with any tracked `filter=lfs` path or a committed `.gitattributes` declaring `filter=lfs` outside a comment line (`lfs_tracked`, worded per case; read with `hardenedGitSync` `ls-files -- ':(attr:filter=lfs)'` plus every committed `.gitattributes` blob from `ls-tree -z` records split at their first tab, then `cat-file`, so clone-local attributes cannot hide one, and read again under the locks), or one it cannot inspect (`lfs_unknown`, whose reason names the cause: checkout missing, not its own repository's top level by a realpath comparison, owned by another user (`safe.directory`), or a Git read failed), because a lock-down reclone would not match the stored writer manifest there. An unborn HEAD with an empty index (a clone of an empty repository) has no committed attributes and passes. Without git-lfs files the tracked bytes are the same, so no manifest is written. Pinned by `test/url-source-origin.test.ts` (PGLite and Postgres).diff --git a/docs/guides/write-refusals.md b/docs/guides/write-refusals.mdindex 13b2e26e2e..6ea18cfd0d 100644--- a/docs/guides/write-refusals.md+++ b/docs/guides/write-refusals.md@@ -108,8 +108,8 @@ of making a duplicate.| `file_unreadable` (sync hold) | `file_unreadable` | A managed sync held one file because it is not a readable regular file inside the source root: a symlink (to a file outside the root or inside it), a non-regular file, or a permission or loop error. Sync never reads or follows it. Until GBRA-130 the whole sync stopped at that file on every run and the rest of the source never imported; now only the file waits. A source root that is itself replaced by a symlink still refuses the whole sync. | Make the file a readable regular file in the checkout (replace the symlink with the file, fix its permissions) or exclude it with `sync.exclude`, and commit. A commit that changes it re-screens the hold on the next sync; after a permission fix run `gbrain sources retry-held <source>`, then the same sync. || <a id="page_write_failed"></a>`page_write_failed` (sync hold) | `page_write_failed` | A managed sync held one file because its write failed the same way three times (the same code, origin, error class and SQLSTATE in this source), with an error gbrain cannot classify: an owner exception (for example Postgres 54000 when a frontmatter tag is too long for its index), or a refusal the receipt does not explain. Until GBRA-130 every sync stopped `blocked_by_failures` at that file and the rest of the source never imported; now the file is held and the rest of the source syncs. The first two failures still stop the run, so a one-off defect never holds a file. | Read the receipt the hold names (`gbrain write-request <request_id>`, read-only). Fix the file when the error names its content (shorten the tag, split the page), or report the receipt with the gbrain version. Then `gbrain sources retry-held <source>` and the same sync. An upgrade re-tests the file on its own. A hold whose `meta.write_failure.origin` is `freeze` (GBRA-133) is different: sync could not even prepare the file (an unexpected error while reading or screening it, before any request was admitted), so it holds it on the first failure and names the error in the hold message; the next syncs re-test it on their own until it has been held 3 times, so a transient fault clears without help. When most files of one run fail that way the run stops instead (the cause is not the files). || <a id="content_null_bytes"></a>`content_null_bytes` (sync hold) | `content_null_bytes` | A managed sync held one Markdown file because its text holds NUL bytes (U+0000). Page text is stored as UTF-8 without NUL bytes. The usual cause is a UTF-16 file (a Windows "Unicode" save; the hold says so when the file starts with a UTF-16 byte order mark), a binary file named `.md` or a crash-padded file. Until GBRA-133 the whole managed sync stopped at that file on every run (`storage_error`, SQLSTATE 22P05) and the rest of the source never imported; now only the file waits. With `sync.holds=fail` the run stops with `content_null_bytes` instead of holding. | Save a UTF-16 file as UTF-8, remove the NUL bytes, or remove a binary file (or leave it out with `sync.exclude`), then commit and `gbrain sync --source <source> --no-pull`. The hold re-screens when the file changes. |-| <a id="content_filter_owned"></a>`content_filter_owned` (sync hold) | `content_filter_owned` | A managed sync held one committed file because Git stores it through a filter: its path has a `filter` attribute at the synced commit naming a clean/smudge driver (whether or not this checkout's Git config defines it, since the commit may have been cleaned on another machine), or its bytes start with git-crypt's header. The committed bytes are what the filter's clean program wrote (git-crypt ciphertext, for one), not the text in your checkout, and gbrain does not run a repository's filter programs during sync, so it cannot turn them into text. Sync reads the attribute at the synced commit with `git check-attr --source` (git 2.40 or newer) or, on an older git, from a temporary index of that commit with `check-attr --cached`, with `core.fsmonitor` and hooks switched off, so no program runs. A plain-text file under a `filter` attribute nobody uses is held too: the hold is the safe side and loses nothing. Before, a filtered file imported its stored bytes as the page text, and a git-crypt file was held `content_null_bytes`. A page imported from such a file before this release is left as it is until the file changes. Git LFS pointers are not this hold. With `sync.holds=fail`, and on a company-brain source, the run stops with `content_filter_owned` instead of holding. | Leave the file out with `sync.exclude`, or commit it without the filter so Git stores its text, then `gbrain sync --source <source> --no-pull`. The hold re-screens when the file changes. For a plain-text file under a filter attribute nobody uses, remove the attribute from `.gitattributes`, commit, and run `gbrain sources retry-held <source>`: a commit that changes only `.gitattributes` does not re-screen the file. |-| <a id="git_object_missing"></a>`git_object_missing` (sync hold) | `git_object_missing` | A managed sync held one committed file because the commit names a Git object the checkout no longer has (an interrupted `git gc`, a partial clone without its promisor remote, disk corruption) and the working-tree copy differs from it or is gone. When the working-tree bytes hash to the committed blob id, sync imports them and holds nothing. Until GBRA-133 the whole managed sync stopped at that file on every run. | Run `git fsck --connectivity-only` in the checkout to see what is missing, restore it with `git fetch` (a partial or shallow clone) or from a backup, then `gbrain sync --source <source> --no-pull`. Every sync re-tests the hold. |+| <a id="content_filter_owned"></a>`content_filter_owned` (sync hold) | `content_filter_owned` | A managed sync held one committed file because Git stores it through a filter: its path has a `filter` attribute at the synced commit naming a clean/smudge driver (whether or not this checkout's Git config defines it, since the commit may have been cleaned on another machine), or its bytes start with git-crypt's header. The committed bytes are what the filter's clean program wrote (git-crypt ciphertext, for one), not the text in your checkout, and gbrain does not run a repository's filter programs during sync, so it cannot turn them into text. Sync reads the attribute at the synced commit with `git check-attr --source` (git 2.40 or newer) or, on an older git, from a temporary index of that commit with `check-attr --cached`. Those reads run with fsmonitor, hooks and every Git transport switched off and with lazy fetching off, so reading the attribute runs no repository program and fetches nothing; when a partial clone lacks a `.gitattributes` that applies to the file, the file is held `git_object_missing` instead. This release does not change how sync reads the file's own blob; object fetches in partial clones are handled in a follow-up. A plain-text file under a `filter` attribute nobody uses is held too: the hold is the safe side and loses nothing. Before, a filtered file imported its stored bytes as the page text, and a git-crypt file was held `content_null_bytes`. A page imported from such a file before this release is left as it is until the file changes. Git LFS pointers are not this hold. With `sync.holds=fail`, and on a company-brain source, the run stops with `content_filter_owned` instead of holding. | Leave the file out with `sync.exclude`, or commit it without the filter so Git stores its text, then `gbrain sync --source <source> --no-pull`. The hold re-screens when the file changes. For a plain-text file under a filter attribute nobody uses, remove the attribute from `.gitattributes`, commit, and run `gbrain sources retry-held <source>`: a commit that changes only `.gitattributes` does not re-screen the file. |+| <a id="git_object_missing"></a>`git_object_missing` (sync hold) | `git_object_missing` | A managed sync held one committed file because the commit names a Git object the checkout no longer has (an interrupted `git gc`, a partial clone without its promisor remote, disk corruption) and the working-tree copy differs from it or is gone, or because a partial clone lacks a `.gitattributes` that applies to the file at that commit, so sync cannot read whether a Git filter owns it (sync fetches nothing to find out). When the working-tree bytes hash to the committed blob id, sync imports them and holds nothing. Until GBRA-133 the whole managed sync stopped at that file on every run. | Run `git fsck --connectivity-only` in the checkout to see what is missing, restore it with `git fetch` (a partial or shallow clone) or from a backup, then `gbrain sync --source <source> --no-pull`. Every sync re-tests the hold. || <a id="path_not_utf8"></a>`path_not_utf8` (sync hold) | `path_not_utf8` | A managed sync held one committed file because its name is not valid UTF-8 (a Latin-1 name from an older system, an archive or a network share). Git hands the name back with U+FFFD for the bytes that are not UTF-8, and no Git read or file matches that decoded name, so nothing is imported from it. Until GBRA-133 the managed sync either stopped at that file on every run or skipped it with no hold; now only the file waits and the hold names it. With `sync.holds=fail` the run stops with `path_not_utf8` instead of holding. | Rename the file to a UTF-8 name (`git -c core.quotePath=true ls-files` shows its exact bytes; `convmv -f latin1 -t utf8` converts a Latin-1 name), commit, then `gbrain sync --source <source> --no-pull`. The rename clears the hold. || <a id="write_interrupted"></a>`write_interrupted` | `write_interrupted` | A transient interruption kept ending this managed write: the database ended its own statements (SQLSTATE `57014`, a statement timeout or cancel) at preparation or inside its publication transaction, or its publication ran past `persistence.publication_ceiling_ms` and its grace and the owner cancelled it. Each interruption throws the preparation away, but it is load or latency, not the write's content, so it has its own budget: the request is claimed again after a backoff (1 s, 2 s, 4 s, 8 s, then 15 s, scheduled on the request as `next_attempt_at`; no claim, root lock or connection is held while it waits, and `gbrain sources writer status` names the waiting head `transient_backoff`) up to `persistence.max_transient_retries` times (default 5, 0 to 20), and the next interruption finishes it `failed` instead of preparing it again without limit. Interruptions never spend `persistence.max_preparation_attempts`, which preparations cut off at their deadline keep ([`preparation_stalled`](#preparation_stalled)). Nothing was published: a worktree-backed write's canonical file is restored before the failure is recorded. Contention with other transactions (`55P03` lock timeout, `40001` serialization failure, `40P01` deadlock) is released as `database_contention` uncounted, and a dropped connection or a database that is restarting or out of connections is never counted. The receipt's message and owner-only `error_detail` name the interruptions (`attempts`), the retry limit (`limit`, `limit_key`), the `cause` (`statement_cancel` or `publication_ceiling`) and the SQLSTATE (`sqlstate`, public). A managed sync member is held like [`preparation_stalled`](#write_interrupted-hold): the run moves past it and the rest of the source lands, `gbrain sync status` lists the hold (class `page`, safe action `retry`, [runbook row](sync-unblock-runbook.md)), and `gbrain sources retry-held` re-admits it. A group's shared read the database cancels is charged once, to the first member that settles with it, never to every member; and the charge commits with the claim's release, so a crash between them never counts it twice. With `persistence.preparation_deadlines` off nothing is counted and no backoff applies. | Run `gbrain sources writer status --source <source> --json` (read-only: the owner, its pool, what its writes wait on), then submit the write again with a new `request_id` once the database is less loaded; for a held sync file, run `gbrain sources retry-held <source>` and the same sync with its options. A `publication_ceiling` cause on a page that legitimately takes longer to publish is the user's call: ask before raising `persistence.publication_ceiling_ms` (or the database `statement_timeout` for a `statement_cancel`). If the same write ends this way again, report its request id with `gbrain doctor --json`. Do not raise `persistence.max_transient_retries` to get past it. || <a id="write_outcome_unknown"></a>`write_outcome_unknown` | `write_outcome_unknown` | The database session dropped while the write was being admitted (a pooler reap, a failover, `pg_terminate_backend`), and it kept dropping through the re-reads, so whether the write was accepted is unknown (#6355). A single drop never reaches you: the admission attempt is re-run after 100, 300 and 900 ms, and every attempt begins by reading the retained `request_id`, so a re-run replays an admitted row instead of admitting again. Before this code existed the raw socket error (`write CONNECTION_CLOSED …`, `57P01`) reached the caller while the write went on to commit, and a receipt oracle took it for a refusal. | Read the request first: `gbrain write-request -- <request_id>` or `get_write_request { request_id }` (the error's `fix` names it). A committed or pending row means the write was accepted: replay the same `request_id` to wait for it. No row means it was not: submit it again with the same `request_id`. Never resubmit under a new `request_id` without that read. |diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.tsindex ae17074ae7..bf3b3e34a9 100644--- a/src/core/behavior-change-notice.ts+++ b/src/core/behavior-change-notice.ts@@ -1501,7 +1501,7 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;{ since: '0.60.635.0', id: 'read_scope_params_gbra431', text: 'An MCP read now refuses an unknown parameter whose name is, or is nearest to, a source scope or a filter (`source` for `source_id`, `sources` for `source_ids`, `type` for `types`, `tag`, `since`, `entity`, `visibility`, `min_trust`) with `invalid_params` naming the declared parameter, whatever `mcp.strict_params` says; it used to ignore it and answer unnarrowed. Other unknown read parameters keep the warn grace period. `gbrain serve --http` now applies `GBRAIN_HTTP_RATE_LIMIT_IP`, `GBRAIN_HTTP_RATE_LIMIT_TOKEN` and `GBRAIN_HTTP_MAX_BODY_BYTES` to `/mcp` when they are set (429 with Retry-After, 413); unset, it keeps no rate limit and a 4 MiB body cap.' },{ since: '0.60.637.0', id: 'sync_restored_file_gst2201', text: 'On a managed source, a page sync deleted because its file was gone comes back on the next sync once the file returns (a `git revert`, a branch or detached checkout and back, a later commit that re-adds it with the same or edited bytes, a file missing from disk for one working-tree sync). It used to stay deleted (`deleted_same`, or a `file_drift` hold whose suggested fix removed the file) while `gbrain sync status` said synced. The first sync after the upgrade reads the whole source once, so pages an older version left deleted beside their file come back and such holds clear without a manual step. A page a person deleted keeps its deletion.' },{ since: '0.60.637.0', id: 'purge_keeps_committed_file_gst2201', text: 'The retention purge (the autopilot purge phase, the `purge` job and `gbrain pages purge-deleted`) never removes a soft-deleted page\'s file that is committed at its source\'s last synced commit or at HEAD. A page sync deleted whose file is back is left for the sync that restores it (counted as deferred); any other such page is removed database-only (receipt `write_through.skipped: committed_file`) and its file stays. A page whose file is gone, or a person\'s delete that never removed its file (for example because another page shared it), purges as before. A page purge (`gbrain delete <slug> --purge` and the retention purge) now also redacts the timeline text the page\'s `add_timeline_entry` receipts kept (the entry\'s summary and source, `bullet`), compacted ones included.' },- { since: 'TBD', id: 'sync_filter_owned_hold_9024', text: 'A managed sync now holds a committed file that Git stores through a filter (a path whose `filter` attribute at the synced commit names a driver other than Git LFS, whether or not this checkout configures that driver, or a file starting with git-crypt\'s header) as `content_filter_owned`, and the rest of the source syncs. Before, such a file imported its stored bytes (git-crypt ciphertext, for one) as the page text, or a git-crypt file was held `content_null_bytes`. gbrain still never runs a repository\'s filter programs (or `core.fsmonitor`) during sync; the hold names git-crypt when the header matches and says to leave the file out with `sync.exclude` or commit it without the filter, and for a plain-text file under a filter attribute nobody uses, to remove the attribute and run `gbrain sources retry-held`. A page an older version already imported from such a file is left as it is until the file changes; then the file is held. With `sync.holds=fail`, and on a company-brain source (which never holds), the run stops with `content_filter_owned`. Classic (unmanaged) sync is unchanged.' },+ { since: 'TBD', id: 'sync_filter_owned_hold_9024', text: 'A managed sync now holds a committed file that Git stores through a filter (a path whose `filter` attribute at the synced commit names a driver other than Git LFS, whether or not this checkout configures that driver, or a file starting with git-crypt\'s header) as `content_filter_owned`, and the rest of the source syncs. Before, such a file imported its stored bytes (git-crypt ciphertext, for one) as the page text, or a git-crypt file was held `content_null_bytes`. gbrain still never runs a repository\'s filter programs (or `core.fsmonitor`) during sync, and reading the filter attribute fetches nothing, so a partial clone that lacks a `.gitattributes` that applies to the file holds it `git_object_missing`; the hold names git-crypt when the header matches and says to leave the file out with `sync.exclude` or commit it without the filter, and for a plain-text file under a filter attribute nobody uses, to remove the attribute and run `gbrain sources retry-held`. A page an older version already imported from such a file is left as it is until the file changes; then the file is held. With `sync.holds=fail`, and on a company-brain source (which never holds), the run stops with `content_filter_owned`. Classic (unmanaged) sync is unchanged.' },];/** The newest disclosed change's release: the notice id moves only when a release adds rows. */diff --git a/src/core/persistence/sync-blobs.ts b/src/core/persistence/sync-blobs.tsindex c517b2de18..cfd198ea92 100644--- a/src/core/persistence/sync-blobs.ts+++ b/src/core/persistence/sync-blobs.ts@@ -12,6 +12,7 @@ import { mkdtempSync, rmSync } from 'node:fs';import { tmpdir } from 'node:os';import { join } from 'node:path';import { gitChildEnv } from '../git-env.ts';+import { HARDENED_GIT_ARGS } from '../hardened-git.ts';import { PAGE_MAX_BYTES } from '../page-size-policy.ts';/** The sync read bound (`readSyncFile`): bigger files are held without being read. The page ceiling (page-size-policy.ts), which also covers MAX_IMAGE_BYTES. */@@ -86,29 +87,50 @@ export function __setCheckAttrSourceForTests(value: boolean | undefined): () =>return () => { checkAttrSource = previous; };}+/** GST-2312 (#9024): a `.gitattributes` of the commit that this repository lacks (a partial clone that never fetched it). */+export interface MissingAttributes { attributes: string; oid: string }+/*** GST-2312 (#9024): the Git filter driver each path's `filter` attribute names at commit `target`, null where none does,* whether or not this checkout's Git config defines the driver (the commit may have been cleaned on another machine). A* committed blob a filter owns holds what the filter's clean program stored (git-crypt ciphertext, for one), not the text* of the checkout, and sync never runs a repository's filters. Git 2.40 and newer read the attributes with* `check-attr --source=<target>`; an older git reads `target` into a temporary index (`read-tree`) and asks- * `check-attr --cached` there. Neither runs a program (`NO_REPOSITORY_PROGRAMS`).+ * `check-attr --cached` there. Every call takes `HARDENED_GIT_ARGS` and `GIT_NO_LAZY_FETCH=1`, so no repository+ * program runs and a partial clone fetches nothing. Git then reads a `.gitattributes` the repository lacks as empty, so+ * the ones that apply are checked first, and a path under a missing one maps to it instead of to a driver.*/-export function readFilterDrivers(gitRoot: string, target: string, gitPaths: string[]): Map<string, string | null> {- const out = new Map<string, string | null>();+export function readFilterDrivers(gitRoot: string, target: string, gitPaths: string[]): Map<string, string | null | MissingAttributes> {+ const out = new Map<string, string | null | MissingAttributes>();if (gitPaths.length === 0) return out;+ const read = (args: string[], input?: string, env: NodeJS.ProcessEnv = {}) => git(gitRoot, [...HARDENED_GIT_ARGS, ...args], input, undefined, { GIT_NO_LAZY_FETCH: '1', ...env });if (checkAttrSource === undefined) {- const [major, minor] = (/git version (\d+)\.(\d+)/.exec(git(gitRoot, ['version']).toString('utf8')) ?? ['', '0', '0']).slice(1).map(Number);+ const [major, minor] = (/git version (\d+)\.(\d+)/.exec(read(['version']).toString('utf8')) ?? ['', '0', '0']).slice(1).map(Number);checkAttrSource = major! > 2 || major === 2 && minor! >= 40;}+ const dirsOf = (path: string) => path.split('/').slice(0, -1).map((_, at, parts) => parts.slice(0, at + 1).join('/')).concat('');+ const candidates = [...new Set(gitPaths.flatMap(dirsOf))].map(dir => dir ? `${dir}/.gitattributes` : '.gitattributes');+ const present: MissingAttributes[] = [];+ for (let i = 0; i < candidates.length; i += 1000) {+ for (const record of read(['ls-tree', '-z', '--full-tree', target, '--', ...candidates.slice(i, i + 1000)]).toString('utf8').split('\0')) {+ const match = /^\d+ blob ([0-9a-f]+)\t(.*)$/s.exec(record);+ if (match) present.push({ attributes: match[2]!, oid: match[1]! });+ }+ }+ const missing = new Map<string, MissingAttributes>();+ const checked = present.length ? read(['cat-file', '--batch-check'], present.map(file => file.oid).join('\n') + '\n').toString('utf8').split('\n') : [];+ present.forEach((file, at) => { if (checked[at]?.endsWith(' missing')) missing.set(file.attributes.slice(0, -'.gitattributes'.length).replace(/\/$/, ''), file); });const scratch = checkAttrSource ? null : mkdtempSync(join(tmpdir(), 'gbrain-attributes-'));const env = scratch ? { GIT_INDEX_FILE: join(scratch, 'index') } : {};try {- if (scratch) git(gitRoot, ['read-tree', target], undefined, undefined, env);+ if (scratch) read(['read-tree', target], undefined, env);for (let i = 0; i < gitPaths.length; i += 2000) {- const fields = git(gitRoot, ['check-attr', ...(scratch ? ['--cached'] : [`--source=${target}`]), '-z', '--stdin', 'filter'],- gitPaths.slice(i, i + 2000).join('\0') + '\0', undefined, env).toString('utf8').split('\0');- for (let at = 0; at + 2 < fields.length; at += 3) out.set(fields[at]!, ['unspecified', 'unset', 'set'].includes(fields[at + 2]!) ? null : fields[at + 2]!);+ const fields = read(['check-attr', ...(scratch ? ['--cached'] : [`--source=${target}`]), '-z', '--stdin', 'filter'],+ gitPaths.slice(i, i + 2000).join('\0') + '\0', env).toString('utf8').split('\0');+ for (let at = 0; at + 2 < fields.length; at += 3) {+ const lost = dirsOf(fields[at]!).map(dir => missing.get(dir)).find(Boolean);+ out.set(fields[at]!, lost ?? (['unspecified', 'unset', 'set'].includes(fields[at + 2]!) ? null : fields[at + 2]!));+ }}} finally {if (scratch) rmSync(scratch, { recursive: true, force: true });diff --git a/src/core/persistence/sync-entry-fault.ts b/src/core/persistence/sync-entry-fault.tsindex b8e5e43af0..f65e3b48d7 100644--- a/src/core/persistence/sync-entry-fault.ts+++ b/src/core/persistence/sync-entry-fault.ts@@ -32,7 +32,7 @@ import { VERSION } from '../../version.ts';import { isWriteCapacityWait } from './admission-retry.ts';import { PAGE_BODY_FAULT_CODES } from './sync-fault-class.ts';import { gitBlobOid } from './digest.ts';-import { readBlobBytes, type TreeBlob } from './sync-blobs.ts';+import { readBlobBytes, type MissingAttributes, type TreeBlob } from './sync-blobs.ts';import { readSyncFile, type SyncEntry } from './sync-discovery.ts';import type { GitHoldRecord } from './sync-holds.ts';@@ -65,7 +65,20 @@ export function filterOwnedRefusal(path: string, content: string, driver: stringreturn { code: 'content_filter_owned', message: `${path} is ${gitCrypt ? 'encrypted with git-crypt' : `stored through the Git filter "${driver}"`}, so its committed bytes are not the text in your checkout. `+ `gbrain does not run a repository's filter programs during sync, so ${held ? 'sync held this file and the rest of the source synced' : 'this sync stopped at this file'}. `+ 'Leave it out with sync.exclude, or commit it without the filter so Git stores its text, then sync again.'- + (gitCrypt ? '' : ` If no filter "${driver}" is in use and the file is plain text, remove the filter attribute from .gitattributes, commit, and run gbrain sources retry-held ${sourceId}: a commit that changes only .gitattributes does not re-screen the file.`) };+ + (gitCrypt || !held ? '' : ` If no filter "${driver}" is in use and the file is plain text, remove the filter attribute from .gitattributes, commit, and run gbrain sources retry-held ${sourceId}: a commit that changes only .gitattributes does not re-screen the file.`) };+}++/**+ * GST-2312 (#9024): the `git_object_missing` refusal of a committed file whose `.gitattributes` at the synced commit the+ * repository lacks (`readFilterDrivers`), so whether a Git filter owns it cannot be read; sync fetches nothing to find out.+ */+export function missingAttributesRefusal(cursor: { gitRoot: string; sourceId: string }, path: string, lost: MissingAttributes): OperationError {+ return opError('git_object_missing', `The .gitattributes that applies to ${path} at the synced commit is missing from the repository.`,+ `${lost.attributes} is committed as blob ${lost.oid}, but the repository of source ${cursor.sourceId} does not have that object, as in a partial clone (git clone --filter) that never fetched it. `+ + `Sync reads a file's Git filter attribute without fetching anything, so it cannot tell whether a filter owns ${path}, and nothing was imported from it. `+ + `Fetch the object in the checkout (git cat-file -p ${lost.oid} fetches it from the partial clone's remote) or use a clone without --filter, then run gbrain sync --source ${cursor.sourceId} --no-pull.`,+ { detail: lost.oid, fix: { argv: ['git', '-C', cursor.gitRoot, 'cat-file', '-p', lost.oid], consent: [], actor: 'user', requires_exclusive: false,+ why: 'Fetches the missing .gitattributes from the partial clone\'s remote, so the next sync can read the attribute.', verify: { argv: ['gbrain', 'sources', 'status', cursor.sourceId, '--json'] } } });}/**diff --git a/src/core/persistence/sync-screen.ts b/src/core/persistence/sync-screen.tsindex 1f8e940929..148af0afa7 100644--- a/src/core/persistence/sync-screen.ts+++ b/src/core/persistence/sync-screen.ts@@ -36,9 +36,9 @@ import { materializedSnapshot } from './segmented-apply.ts';import { readPagePurgeTombstones } from './page-purge.ts';import { listTrustAllowRules } from '../trust/allow-rules.ts';import { readSyncContent, readSyncFile, syncGitPath, UNREADABLE_SYNC_FILE, type SyncDiscovery, type SyncEntry } from './sync-discovery.ts';-import { pinnedWindowBlock, readBlobBytes, readBlobContents, readFilterDrivers, readPinnedBlob, readPinnedBytes, readPinnedContent, readTreeBlobs, SYNC_READ_BOUND, type TreeBlob } from './sync-blobs.ts';+import { pinnedWindowBlock, readBlobBytes, readBlobContents, readFilterDrivers, readPinnedBlob, readPinnedBytes, readPinnedContent, readTreeBlobs, SYNC_READ_BOUND, type MissingAttributes, type TreeBlob } from './sync-blobs.ts';import { screenSyncImport, type SyncImportScreenInput } from './sync-prepare.ts';-import { committedContentOrMissing, filterOwnedRefusal, nullBytesRefusal } from './sync-entry-fault.ts';+import { committedContentOrMissing, filterOwnedRefusal, missingAttributesRefusal, nullBytesRefusal } from './sync-entry-fault.ts';import type { SyncProcessingOptions } from './sync-authority.ts';import { gitHoldItem, readSyncHoldPolicy, readSyncImportProvenance, type GitHoldCode, type GitHoldItem, type GitHoldRecord, type PreparationStallMeta, type SyncHoldPolicy } from './sync-holds.ts';import type { WriteRequest } from './model.ts';@@ -82,13 +82,14 @@ export function pinnedBlob(cursor: PinnedCursor, path: string): TreeBlob | nullreturn readPinnedBlob(cursor.gitRoot, cursor.target, syncGitPath(cursor, path), upcomingGitPaths(cursor));}-let filterDrivers: { key: string; drivers: Map<string, string | null> } | null = null;+let filterDrivers: { key: string; drivers: Map<string, string | null | MissingAttributes> } | null = null;/*** GST-2312 (#9024): the Git filter driver of one entry's path at the pinned commit (`readFilterDrivers`), or null; a* company plan's at its approved revision. A miss reads it with up to 2,000 of the committed imports after it in one- * `check-attr`; answers are kept per Git root and commit.+ * `check-attr`; answers are kept per Git root and commit. A path under a `.gitattributes` the repository lacks throws+ * `git_object_missing` (`missingAttributesRefusal`), and that answer is dropped once used, so the next run reads again.*/-export function pinnedFilterDriver(cursor: Pick<SyncDiscovery, 'root' | 'gitRoot' | 'target' | 'entries' | 'companyPlan'> & { index?: number }, path: string): string | null {+export function pinnedFilterDriver(cursor: Pick<SyncDiscovery, 'root' | 'gitRoot' | 'target' | 'entries' | 'companyPlan' | 'sourceId'> & { index?: number }, path: string): string | null {const revision = cursor.companyPlan?.revision, gitRoot = revision?.git_root ?? cursor.gitRoot, target = revision?.commit ?? cursor.target;const gitPathOf = (next: string) => revision ? (revision.scope ? `${revision.scope}/${next}` : next) : syncGitPath(cursor, next);const gitPath = gitPathOf(path), key = `${gitRoot}\0${target}`;@@ -100,7 +101,10 @@ export function pinnedFilterDriver(cursor: Pick<SyncDiscovery, 'root' | 'gitRootfor (const [next, driver] of readFilterDrivers(gitRoot, target, [gitPath, ...upcoming])) known.set(next, driver);if (!known.has(gitPath)) known.set(gitPath, null);}- return known.get(gitPath) ?? null;+ const driver = known.get(gitPath) ?? null;+ if (typeof driver !== 'object' || driver === null) return driver;+ known.delete(gitPath);+ throw missingAttributesRefusal({ gitRoot, sourceId: cursor.sourceId }, path, driver);}/**diff --git a/test/sync-filter-owned-9024.test.ts b/test/sync-filter-owned-9024.test.tsindex e70ea42042..69871229b4 100644--- a/test/sync-filter-owned-9024.test.ts+++ b/test/sync-filter-owned-9024.test.ts@@ -1,9 +1,11 @@import { afterAll, beforeAll, describe, expect, test } from 'bun:test';import { execFileSync } from 'node:child_process';-import { chmodSync, existsSync, mkdirSync, writeFileSync } from 'node:fs';+import { chmodSync, existsSync, mkdirSync, rmSync, writeFileSync } from 'node:fs';import { join } from 'node:path';import type { BrainEngine } from '../src/core/engine.ts';import { __setCheckAttrSourceForTests, __setPinnedWindowForTests } from '../src/core/persistence/sync-blobs.ts';+import { retryHeld } from '../src/commands/sources-retry-held.ts';+import { filterOwnedRefusal } from '../src/core/persistence/sync-entry-fault.ts';import { readGitSourceHolds } from '../src/core/persistence/sync-holds.ts';import { performManagedSync } from '../src/core/persistence/sync-run.ts';import { managedBrain } from './helpers/managed-brain.ts';@@ -123,8 +125,48 @@ for (const backend of testBackends()) describe('[' + backend + '] a committed fiexpect(held[0]!.message).toContain('remove the filter attribute from .gitattributes');expect(held[0]!.message).toContain('gbrain sources retry-held default');expect(await live(engine)).not.toContain('notes/plain');+ writeFileSync(join(root, '.gitattributes'), ''); commit(root, 'drop the attribute');+ await performManagedSync(engine, OPTS);+ expect((await holds(engine)).filter(hold => hold.path === 'notes/plain.md').map(hold => hold.code)).toEqual(['content_filter_owned']);+ await retryHeld(engine, 'default');+ await performManagedSync(engine, OPTS);+ expect((await holds(engine)).filter(hold => hold.path === 'notes/plain.md')).toEqual([]);+ expect(await live(engine)).toContain('notes/plain');}), 240_000);+ test('the stop path (nothing held) does not tell the user to run retry-held', () => {+ expect(filterOwnedRefusal('a.md', 'text', 'secret', 'default', false)!.message).not.toContain('retry-held');+ expect(filterOwnedRefusal('a.md', 'text', 'secret', 'default', true)!.message).toContain('gbrain sources retry-held default');+ });++ for (const source of [true, false]) {+ test('a partial clone missing the commit\'s .gitattributes holds the file git_object_missing and fetches nothing (' + (source ? 'check-attr --source' : 'git before 2.40: a temporary index') + ')', () => brain(async ({ engine, root }) => {+ const restore = __setCheckAttrSourceForTests(source);+ try {+ expect((await performManagedSync(engine, OPTS)).status).toBe('first_sync');+ const marker = filtered(root, false);+ const server = join(root, '..', 'origin.git'), uploadPack = join(root, '..', 'bin', 'upload-pack');+ execFileSync('git', ['clone', '-q', '--bare', root, server]);+ git(server, 'config', 'uploadpack.allowFilter', 'true'); git(server, 'config', 'uploadpack.allowAnySHA1InWant', 'true');+ writeFileSync(uploadPack, '#!/bin/sh\ntouch ' + marker + '.upload-pack\nexec git-upload-pack "$@"\n'); chmodSync(uploadPack, 0o755);+ for (const [key, value] of [['core.repositoryformatversion', '1'], ['extensions.partialClone', 'origin'], ['remote.origin.url', 'file://' + server],+ ['remote.origin.promisor', 'true'], ['remote.origin.partialclonefilter', 'blob:none'], ['remote.origin.uploadpack', uploadPack]]) git(root, 'config', key!, value!);+ const oid = git(root, 'rev-parse', 'HEAD:.gitattributes');+ rmSync(join(root, '.git', 'objects', oid.slice(0, 2), oid.slice(2)));+ await performManagedSync(engine, OPTS);+ expect(existsSync(marker + '.upload-pack')).toBe(false);+ // The root .gitattributes applies to every path, so both files of the commit are held; the seed page stays.+ const held = (await holds(engine)).filter(hold => hold.path === 'secret/one.md' || hold.path === 'ok.md').sort((a, b) => a.path.localeCompare(b.path));+ expect(held.map(hold => [hold.path, hold.code])).toEqual([['ok.md', 'git_object_missing'], ['secret/one.md', 'git_object_missing']]);+ expect(held[1]!.message).toContain('partial clone');+ expect(held[1]!.message).toContain(oid);+ expect(await live(engine)).toEqual(['seed']);+ expect(existsSync(marker)).toBe(false);+ expect(existsSync(marker + '.fsmonitor')).toBe(false);+ } finally { restore(); }+ }), 240_000);+ }+for (const source of [true, false]) {test('the filter attribute is read at the synced commit, not from an uncommitted .gitattributes (' + (source ? 'check-attr --source' : 'git before 2.40: a temporary index') + ')', () => brain(async ({ engine, root }) => {const restore = __setCheckAttrSourceForTests(source);