Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

repair private-labels: review fixes for batched applies

garrytan9b5bf4c72404
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6814092e0f..e5148fc77d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -61,7 +61,7 @@ Restore no longer lists a page whose label was removed by hand after a restore l
### Faster private-labels windows that finish despite declined pages
-`gbrain repair private-labels --apply --expect <hash>` on a brain with managed persistence now admits a window's pages together and publishes them in groups, instead of one page and one wait at a time. On a 1,000-page window with Git commits and a second queue consumer (`scripts/bench/private-labels-apply.ts`), it went from 181.4 to 1365.3 pages a minute with no added database latency, and from 19.9 to 37.1 pages a minute with 20 ms added to each database round trip. Git commits for the window dropped from 990 to 22. The preview no longer claims a re-embed for every page: only pages stored under per-chunk synopsis count.
+`gbrain repair private-labels --apply --expect <hash>` on a brain with managed persistence now admits a window's pages together and publishes them in groups, instead of one page and one wait at a time. On a 1,000-page window with Git commits and a second queue consumer (`scripts/bench/private-labels-apply.ts`), it went from 181.4 to 1365.3 pages a minute with no added database latency, and from 19.9 to 37.1 pages a minute with 20 ms added to each database round trip. Git commits for the window dropped from 990 to 22. The preview no longer claims a re-embed for every page: only pages stored under per-chunk synopsis, or with a chunk that has no vector yet, count.
A page whose own state declines the write (its file changed outside gbrain, `source_changed`, or its body is stored in segments) fails alone and the rest of its group lands. A preview-bound apply of any repair kind that completes with only such declines now exits 0 and counts them under `declined`. The next private-labels preview lists pages an earlier apply tried and that did not change after every page not tried yet, and such a page declining again no longer counts toward the `item_failures_systemic` stop, so windows that carry declines forward keep finishing; pages not tried yet that all fail the same way still stop the run. The apply result names its ledger (`details.ledger`, `details.ledger_rows_written`).
diff --git a/docs/guides/repair.md b/docs/guides/repair.md
index 300467a031..0154bae978 100644
--- a/docs/guides/repair.md
+++ b/docs/guides/repair.md
@@ -2163,6 +2163,9 @@ Clearing a declined page is a separate choice per page: see
run's systemic-failure stop (`item_failures_systemic`), so a window whose tail,
or whole listing, is carried declines completes; pages not tried yet that all
fail the same way, as a dirty checkout makes them, still stop the run.
+A script that previews and applies windows in a loop should stop when every
+listed page was tried before (`details.window.tried_before` equals
+`details.window.listed`): what's left is pages that keep declining.
An apply that meets a `gbrain sources refresh` waits for the refresh to
finish, for up to 10 minutes per step, instead of being refused after 10
diff --git a/scripts/module-size-limits.tsv b/scripts/module-size-limits.tsv
index b61436032a..f474632007 100644
--- a/scripts/module-size-limits.tsv
+++ b/scripts/module-size-limits.tsv
@@ -47,4 +47,4 @@ src/commands/doctor/checks/extraction-sync.ts 1500 ratchet +7 GBRA-130 A-1: link
src/core/search/evidence-delivery.ts 1759 ratchet +5 W5: EvidencePlan.rankCut resolved with the plan (logic in search/page-rank-cut.ts); +20 W5 whole_pages: wholeCost prices a block as delivery would emit it whole (needs the private Block/enrichBlock internals) and the cut is wired before allocate (cutWholePages lives in search/page-rank-cut.ts); new row: crossed the 1500 unlisted cap at the #6367 merge (master 1479, its explicit-budget cap) by the C1 date header threaded through every block path, cap included; pagePlanHits (search/page-plan-hits.ts) and the header loader (evidence-date.ts blockDateHeaders) were peeled first; peel candidate: allocateCapped; +8 at the #6396 merge: page evidence text cut through the fence-chunk trust overlay and the caller minTrust floor (re-summed: our 1504 + master +8); +119 Cat 40 next wave F2: the search.long_page_unit policy (withLongPagePolicy, selectLongPages size preflight with page/byte/deadline limits) and the long-page fetch and fallback in deliverEvidence, kept in the shared seam the plan names (gbrain-evals docs/plans/2026-10-11-cat40-next-wave, F2); +17 F2 rework to the frozen-comparison policy: search.long_page_min_chars (rendered-body characters, default 20,000) read in withLongPagePolicy, a character count in the preflight, the rendered-body confirmation in deliverEvidence, and the long-page row exemption in capDeliveredSnippets (re-summed at the 0.60.225.0 merge: master 1537 + F2 136); +39 Wave B PR 2 lane 2b: planSegmentedBlock joins the windows search/segmented-evidence.ts reads around a segmented page's hits into one run-marked document (needs the private Doc/Piece/buildDoc/locateAnchor internals), planBlock's unit cut split into blockFromDoc for both, sections bounded by their run, selectionCost counts a gap between runs, and the fetch skips the whole-body fence overlay for a segmented page; +4 segmented pages read at once before planning (segmentedReads; the read is search/segmented-evidence.ts, the statements page-body/read-range.ts); +17 GBRA-156: tail trim plan flag, DeliverOptions.query, DeliveryMeta.tail_trim, the trim call and the frozen-hit score/query library fields (logic in search/evidence-tail-trim.ts; interface fields cannot move); +7 GBRA-205: EvidencePlan.titleHeader and DeliveryMeta.title_header (interface fields cannot move) and the titles option on the header loader (logic in search/evidence-date.ts; the query budget key lives in search/evidence-packing.ts); +11 #7600: assemble_evidence quotes conversation rows and capEvidenceToBudget counts the quote fence (logic in search/transcript-quote.ts); +2 GBRA-156 /ship review: the cut-path admission and title room count the quote fence; +2 GBRA-156 /ship review pass 2: capEvidenceToBudget counts the fence only for the paths that fence (quote flag; recall counts the text as returned); +4 GST-687: page evidence and the extractive fallback quote below-owner-tier text for untrusted readers (untrusted flag; logic in search/transcript-quote.ts)
src/commands/config.ts 1526 ratchet +1 GBRA-216: recall.* config set value refusal (facts/recall-config.ts); +1 GBRA-216: config set refuses a dream.claim_conflicts.mode other than propose|off (refusal in facts/claim-conflicts.ts); new row: crossed the 1500 unlisted cap in wave 15 P7.4 #5461: set-time validation of think.pages_budget_chars / think.excerpt_ceiling_chars (range + parser live in core/think/pages-budget.ts), mirroring the cycle.consolidate.cluster_threshold hook; +1 wave 16 G5.1 (#6495): set-time refusal hook for search.keyword_statement_timeout_ms (check lives in commands/config/keyword-timeout-value.ts); +1 wave 18b lane 5 (#5012 item 6): search.contextual_retrieval prints the existing-pages hint after the write (text lives in commands/config/contextual-mode-hint.ts); +1 #6712 PR 2: retired/rollback sync key dispatch line (logic in core/sync-config-keys.ts handleSyncKeyConfig); +1 #6865: sync.slug_conflicts.auto_remove boolean validation at set time (refusal text in repair/slug-autoheal.ts); +1 #6948: sync.origin_twins.auto_resolve boolean validation at set time (refusal text in repair/origin-twins-heal.ts); +2 #6795: config set chat_model/expansion_model help line and the model-pin import (shadow report and file-pin unset live in commands/config/model-pin-shadow.ts); +1 #6712 PR 4b: sync.admission set-time hook (value logic in commands/config/sync-admission-value.ts); +4 #7130: set-time refusal hook for sync.max_deletes / sync.max_delete_ratio (and .source.<id>) / purge.retention_hours (check lives in core/sync-reconcile.ts deleteSafetyConfigRefusal); +1 GBRA-150: sync.metafile_pages set-time route (body in config/sync-metafile-pages-value.ts); re-summed at the v0.60.330.0 merge (1520 measured); +2 #7242 minions.terminal_hook file-plane key and its set branch (logic in minions/terminal-hook.ts); +1 GBRA-161: generations.* set-time route (refusal in commands/config/generations-values.ts); +1 GBRA-207 #7630: sync.repo_path set-time refusal hook (refusal in core/sync-config-keys.ts syncRepoPathRefusal)
src/commands/mcp-expose.ts 1519 ratchet hosted_option: Session carries the gbrain.io coaching notice on the first-receipt run and Tailscale/service friction exits (copy, refs and gating live in core/onboard/hosted-option.ts); +7 GBRA-199: --surface help lines, surfaceChosen option field and the advertise pass-through (registrationAdvertise in core/mcp-registration.ts)
-src/core/behavior-change-notice.ts 1696 ratchet +4 train R: #7491/#7492 keyword_identifier_probe and keyword_slug_handles rows and their comment, #7175 keyword phrase row; the BEHAVIOR_CHANGES table grows by one row per disclosed change: +4 the #7891 provenance-drift and #7917/#7918 maintenance-lowering rows and their comments (re-summed at the 0.60.495.0 merge: 1547 + 4); +1 GBRA-156 #7593: private_pages_granted_sources row (re-summed at the 0.60.527.0 merge: 1574 + 1); +3 GBRA-200 comment, write-grant refusal and degraded-stamp rows (re-summed at the 0.60.562.0 merge: 1615 + 3); +2 GBRA-280 get_page selector-precedence row and comment (re-summed at the 0.60.569.0 merge: 1619 + 2); the BEHAVIOR_CHANGES table grows by one row per disclosed change: +4 the #7891 provenance-drift and #7917/#7918 maintenance-lowering rows and their comments (re-summed at the 0.60.495.0 merge: 1547 + 4); +2 GBRA-205 saved_facts_stems row and its comment (re-summed at the #7901 0.60.492.0 merge: 1541 + 2); +2 GBRA-205 search.query_facts_block default-on row and its approval comment (re-summed at the #7610 0.60.490.0 merge: 1539 + 2); +2 the #7900 content checkpoint host-move row (since TBD until the merge slot), +2 the #7897 private-hold sweep row (since TBD until the merge slot), +2 the #7880 ingest_capture and company-brain private-hold row (since TBD until the merge slot), +5 the frozen-source wave comment and four rows (#7868, #7850, #7869/#7870, #7871), +2 the #7848 visibility_noncanonical hold row (since TBD until the merge slot), +2 the #7846 extract_atoms session-source row (re-summed at the 0.60.481.0 merge: master 1524 + 2), +2 the #7805 synthesize session-source and #7841 visibility_not_applied rows (re-summed at the 0.60.480.0 merge: master 1522 + 2), +2 strict door planned-state row (#7705, re-summed at the 0.60.470.0 merge: master 1514 + 2), +2 Bun floor 1.4.4 row (#7644), the GBRA-95 RLS backfill row (+2), +3 rows for #7596 pass_interrupted, #7599 read-only serve and the --access=read-only ceiling, +1 G5 lane C trust_release row, +1 #7619 cli_flag_equals, +3 GBRA-207 #7630 relative-anchor and stalled-write rows, +2 the #7632 function search_path disclosure row and +2 the #7679 orphaned worktree refresh row (re-summed at the 0.60.471.0 merge: master 1516 + 2), +2 the #7814 retired v0.13.1 grandfather row, +2 the #7618 conversion withhold/progress/push row (master 1520 + 2); the behavior-change table split (#7607) retires this row; +4 GBRA-188 #7489: the wave comment and three rows (brain-first instructions, starter surface default, per-turn rule); +1 GBRA-188 plugin_turn_hook_gbra188 row; +1 GBRA-242 provider_key_rules_7959 row (#7959), +2 the GBRA-199 registration_lists_verbs row (since TBD until the merge slot); +2 GBRA-133 #7652: the #7611 pass_abandoned reader row and the lean read-only serve socket row (re-summed at the 0.60.503.0 merge: master 1555 + 2); +1 GBRA-243: the per-turn hook disclosure row (rows are data and cannot move); +4 GBRA-156 #7598 harness wave comment and rows (remote page cap, sources_not_readable refusal, sources add next step); +4 GBRA-156 #7601 search_quotes_transcript_rows comment and row; +1 GBRA-175 #7475 autopilot_restarts_serve_7475 row (re-summed at the #8012 merge: 1566 + 1); +3 #7997/#8014/#8015 rows (redaction markers, company db_only keep, ignored supabase_only scope); +2 #7199 (re-port of #7228) classic_drift_rescreen_7199 comment and row; +2 #7931 file_drift_agrees_7931 comment and row; +1 #7950 file_drift_slug_release_7950 row; +1 #7974 file_drift_followups_7974 row; +1 #8059 quoted_fence_marker_8059 row; +2 GBRA-247 #8009 provenance_repair_held_8009 and #7982 pass_transient_streak_7982 rows; +4 GBRA-247 #7973 plain_directory_sync_7973, #7988 managed_marker_own_directory_7988, #7985 former_binding_parked_group_7985 and settle_effects_retire_7985 rows Train S merge: 1570 base + 5 (master since the #8077 base) + 13 (#8077) = 1588.; +2 GST-687 remote_quotes_unverified_text_gst687 row and its comment (re-summed at the gate-ahead on train S 0.60.534.0: 1588 + 2); +2 #8377: the quote fence follow-ups row and its comment; +2 #8384: the quote fence follow-ups 2 row and its comment; +2 #8384 wave 3: the quote fence follow-ups 3 row and its comment; +5 GBRA-218 train M rows (A1 member_read_access_gbra218 and member_access_hatch_scope_gbra218, B shared_agent_audience_gbra231, A2 folder_access_rules_gbra218, folder_access_scoped_role_gbra218), re-summed at the train M merge of master v0.60.541.0: 1596 + 5 = 1601 measured; +3 GST-359 train C rows (C1 qm_attested_actor_gst359, C2 member_facts_gst359, D qm_memory_provider_gst359), re-summed at the train C merge of master v0.60.543.0: 1601 + 3 = 1604 measured; +2 #8424: hook_launcher_8424 row and its comment (rows are data and cannot move), re-summed at the #8418 v0.60.546.0 merge: 1604 + 2 = 1606 measured; +2 #8426: mcp_lane_launcher_8426 row and its comment (rows are data and cannot move); +2 GST-927: empty_nonstring_args_gst927 row and its comment, re-summed at the gate-ahead on master v0.60.550.0 plus #8420: 1610 measured; +2 GBRA-283 Wave A (#8460): engine_keyed_caches_gbra283 row and its comment (rows are data and cannot move); +2 GBRA-287: segmented_freeze_hold_gbra287 and segmented_reexport_gbra287 rows (rows are data and cannot move); +1 GST-1209: segmented_tiling_gst1209 row (rows are data and cannot move); +1 GST-1251: skillopt_loop_deprecated_gst1251 row (rows are data and cannot move); +1 GST-1209 follow-up: untiled_hold_repair_gst1209 row (rows are data and cannot move); +1 GBRA-283 B3a: embedded_scope_required_gbra283 row (rows are data and cannot move); re-summed at the train R merge with v0.60.574.0: 1623 + 4; +2 GST-1387: refresh_fence_refusal_gst1387 row and its comment (rows are data and cannot move); re-summed at the memory train #8481 merge with v0.60.578.0 (1643 measured); +3 GBRA-253 #8109 rows (frontmatter_drift_write_8109, contacts_activity_advance_8109, contact_rows_dedupe_8109; rows are data and cannot move), re-summed at the master v0.60.581.0 merge: 1643 + 3 = 1646 measured; +3 #8123 part C rows: sync_write_through_off_writeback_7084, managed_sync_dry_run_counts_7087, frontmatter_missing_source_dir_7889 (since TBD until the merge slot; re-summed at the 0.60.583.0 merge: 1646 + 3); +4 PR B commit_ledger_prb, doctor_foreign_writers_prb and sources_guard_prb rows and their comment (rows are data and cannot move; since TBD until the merge slot); +1 PR B #8615: file_drift_held_needs_prefer_8615 row; +1 PR B #8617: refresh_source_changed_retry_8617 row; +1 PR B #8662: commit_push_checkout_not_a_source_8662 row (since TBD until the merge slot; rows are data and cannot move); +2 PR B refresh_barrier_prb and commit_push_base_prb rows (re-summed: 1658 measured); +3 convergence PR A: the wave comment and two rows (conversion_content_evidence_pr_a, git_gap_live_pages_pr_a; since TBD until the merge slot; rows are data and cannot move); re-summed at the gate-ahead on master v0.60.592.0 plus #8711: 1658 + 3 = 1661 measured; +4 GST-1094/GST-900 rows (memory_recall scope-read, member-bound acting_user, recall max_chars, unknown_param _meta-only) and +1 qm_scope_writes_gst1113 and +1 recall_question_terms_gst1304 rows (rows are data and cannot move), re-summed at the 0.60.595.0 slot merge: 1661 + 6; +1 GST-1582 refresh_barrier_import_gst1582 row (since TBD until the merge slot; rows are data and cannot move); +2 GST-1601 source_changed_origin_gst1601 and adopt_local_edits_gst1601 rows (since TBD until the merge slot; rows are data and cannot move), re-summed at the master v0.60.596.0 merge: 1668 + 2; +1 GST-1623 refresh_barrier_capacity_gst1623 row (since TBD until the merge slot; rows are data and cannot move), re-summed at the master v0.60.599.0 merge: 1670 + 1; +2 GBRA-224 W1 release_channel_stable_default row and its comment; +1 GST-1623 refresh_barrier_later_write_gst1623 row (#8765; rows are data and cannot move); +5 #8766 GST-1623 items 4/6-9 and #8755: the wave comment and four rows (doctor_foreign_writers_partial_gst1623, commit_push_enclosing_repo_gst1623, owner_guard_fail_closed_gst1623, adopt_local_edits_hardening_8755) and +1 managed_sync_kind_reserved_8765 row (since TBD until the merge slot; rows are data and cannot move), re-summed at the gate-ahead on master v0.60.608.0 plus #8798: 1674 + 6; +1 GST-1853 refresh_barrier_aborted_refresh_sync_gst1853 row (since TBD until the merge slot; rows are data and cannot move), re-summed at the master v0.60.613.0 merge: 1680 + 1; +1 GST-1674 refresh_refused_merge_recovery_gst1674 row (since TBD until the merge slot; rows are data and cannot move), re-summed at the master v0.60.614.0 merge: 1681 + 1; +1 #8824/#8826/#8767: adopt_local_edits_replay_markers_8824 row (since TBD until the merge slot; rows are data and cannot move), re-summed at the gate-ahead on #8837: 1682 + 1; +1 url_source_origin row (since TBD until the merge slot; rows are data and cannot move), re-summed at the gate-ahead on #8834: 1683 + 1; +1 #8828: the refresh_replan_own_commits_8828 row (1684 + 1 at the 0.60.618.0 slot); +1 the managed commit-push unpublished-count row; +1 the refresh publish-diverged default row; +1 GST-2051 ungranted_member_writes_gst2051 row (since TBD until the merge slot; rows are data and cannot move); +1 GST-731 think_window_trajectory_gst731 row and +1 GST-1602 purge_cancels_deferred_write_gst1602 row (since TBD until the merge slot; rows are data and cannot move), re-summed at the gate-ahead on #8915 v0.60.621.0: 1688 + 2; +1 GST-2059: owner_reads_private row; +1 #8888: the record-origin checkout lock-down row (re-summed at the Q6 stack on #8934 v0.60.625.0: 1691 + 1); +1 GST-2112: ingest_keeps_readers_gst2112 row (re-summed at the Q7 v0.60.627.0 merge: 1692 + 1); +1 #8982 federated_member_writes_8982 row (since TBD until the merge slot; rows are data and cannot move); +2 GST-2089 owner_private_facts_gst2089 and GST-2092 private_labels_write_ahead_gst2092 rows (since TBD until the merge slot; rows are data and cannot move), re-summed at the access and privacy wave on #8991: 1694 + 2
+src/core/behavior-change-notice.ts 1697 ratchet +4 train R: #7491/#7492 keyword_identifier_probe and keyword_slug_handles rows and their comment, #7175 keyword phrase row; the BEHAVIOR_CHANGES table grows by one row per disclosed change: +4 the #7891 provenance-drift and #7917/#7918 maintenance-lowering rows and their comments (re-summed at the 0.60.495.0 merge: 1547 + 4); +1 GBRA-156 #7593: private_pages_granted_sources row (re-summed at the 0.60.527.0 merge: 1574 + 1); +3 GBRA-200 comment, write-grant refusal and degraded-stamp rows (re-summed at the 0.60.562.0 merge: 1615 + 3); +2 GBRA-280 get_page selector-precedence row and comment (re-summed at the 0.60.569.0 merge: 1619 + 2); the BEHAVIOR_CHANGES table grows by one row per disclosed change: +4 the #7891 provenance-drift and #7917/#7918 maintenance-lowering rows and their comments (re-summed at the 0.60.495.0 merge: 1547 + 4); +2 GBRA-205 saved_facts_stems row and its comment (re-summed at the #7901 0.60.492.0 merge: 1541 + 2); +2 GBRA-205 search.query_facts_block default-on row and its approval comment (re-summed at the #7610 0.60.490.0 merge: 1539 + 2); +2 the #7900 content checkpoint host-move row (since TBD until the merge slot), +2 the #7897 private-hold sweep row (since TBD until the merge slot), +2 the #7880 ingest_capture and company-brain private-hold row (since TBD until the merge slot), +5 the frozen-source wave comment and four rows (#7868, #7850, #7869/#7870, #7871), +2 the #7848 visibility_noncanonical hold row (since TBD until the merge slot), +2 the #7846 extract_atoms session-source row (re-summed at the 0.60.481.0 merge: master 1524 + 2), +2 the #7805 synthesize session-source and #7841 visibility_not_applied rows (re-summed at the 0.60.480.0 merge: master 1522 + 2), +2 strict door planned-state row (#7705, re-summed at the 0.60.470.0 merge: master 1514 + 2), +2 Bun floor 1.4.4 row (#7644), the GBRA-95 RLS backfill row (+2), +3 rows for #7596 pass_interrupted, #7599 read-only serve and the --access=read-only ceiling, +1 G5 lane C trust_release row, +1 #7619 cli_flag_equals, +3 GBRA-207 #7630 relative-anchor and stalled-write rows, +2 the #7632 function search_path disclosure row and +2 the #7679 orphaned worktree refresh row (re-summed at the 0.60.471.0 merge: master 1516 + 2), +2 the #7814 retired v0.13.1 grandfather row, +2 the #7618 conversion withhold/progress/push row (master 1520 + 2); the behavior-change table split (#7607) retires this row; +4 GBRA-188 #7489: the wave comment and three rows (brain-first instructions, starter surface default, per-turn rule); +1 GBRA-188 plugin_turn_hook_gbra188 row; +1 GBRA-242 provider_key_rules_7959 row (#7959), +2 the GBRA-199 registration_lists_verbs row (since TBD until the merge slot); +2 GBRA-133 #7652: the #7611 pass_abandoned reader row and the lean read-only serve socket row (re-summed at the 0.60.503.0 merge: master 1555 + 2); +1 GBRA-243: the per-turn hook disclosure row (rows are data and cannot move); +4 GBRA-156 #7598 harness wave comment and rows (remote page cap, sources_not_readable refusal, sources add next step); +4 GBRA-156 #7601 search_quotes_transcript_rows comment and row; +1 GBRA-175 #7475 autopilot_restarts_serve_7475 row (re-summed at the #8012 merge: 1566 + 1); +3 #7997/#8014/#8015 rows (redaction markers, company db_only keep, ignored supabase_only scope); +2 #7199 (re-port of #7228) classic_drift_rescreen_7199 comment and row; +2 #7931 file_drift_agrees_7931 comment and row; +1 #7950 file_drift_slug_release_7950 row; +1 #7974 file_drift_followups_7974 row; +1 #8059 quoted_fence_marker_8059 row; +2 GBRA-247 #8009 provenance_repair_held_8009 and #7982 pass_transient_streak_7982 rows; +4 GBRA-247 #7973 plain_directory_sync_7973, #7988 managed_marker_own_directory_7988, #7985 former_binding_parked_group_7985 and settle_effects_retire_7985 rows Train S merge: 1570 base + 5 (master since the #8077 base) + 13 (#8077) = 1588.; +2 GST-687 remote_quotes_unverified_text_gst687 row and its comment (re-summed at the gate-ahead on train S 0.60.534.0: 1588 + 2); +2 #8377: the quote fence follow-ups row and its comment; +2 #8384: the quote fence follow-ups 2 row and its comment; +2 #8384 wave 3: the quote fence follow-ups 3 row and its comment; +5 GBRA-218 train M rows (A1 member_read_access_gbra218 and member_access_hatch_scope_gbra218, B shared_agent_audience_gbra231, A2 folder_access_rules_gbra218, folder_access_scoped_role_gbra218), re-summed at the train M merge of master v0.60.541.0: 1596 + 5 = 1601 measured; +3 GST-359 train C rows (C1 qm_attested_actor_gst359, C2 member_facts_gst359, D qm_memory_provider_gst359), re-summed at the train C merge of master v0.60.543.0: 1601 + 3 = 1604 measured; +2 #8424: hook_launcher_8424 row and its comment (rows are data and cannot move), re-summed at the #8418 v0.60.546.0 merge: 1604 + 2 = 1606 measured; +2 #8426: mcp_lane_launcher_8426 row and its comment (rows are data and cannot move); +2 GST-927: empty_nonstring_args_gst927 row and its comment, re-summed at the gate-ahead on master v0.60.550.0 plus #8420: 1610 measured; +2 GBRA-283 Wave A (#8460): engine_keyed_caches_gbra283 row and its comment (rows are data and cannot move); +2 GBRA-287: segmented_freeze_hold_gbra287 and segmented_reexport_gbra287 rows (rows are data and cannot move); +1 GST-1209: segmented_tiling_gst1209 row (rows are data and cannot move); +1 GST-1251: skillopt_loop_deprecated_gst1251 row (rows are data and cannot move); +1 GST-1209 follow-up: untiled_hold_repair_gst1209 row (rows are data and cannot move); +1 GBRA-283 B3a: embedded_scope_required_gbra283 row (rows are data and cannot move); re-summed at the train R merge with v0.60.574.0: 1623 + 4; +2 GST-1387: refresh_fence_refusal_gst1387 row and its comment (rows are data and cannot move); re-summed at the memory train #8481 merge with v0.60.578.0 (1643 measured); +3 GBRA-253 #8109 rows (frontmatter_drift_write_8109, contacts_activity_advance_8109, contact_rows_dedupe_8109; rows are data and cannot move), re-summed at the master v0.60.581.0 merge: 1643 + 3 = 1646 measured; +3 #8123 part C rows: sync_write_through_off_writeback_7084, managed_sync_dry_run_counts_7087, frontmatter_missing_source_dir_7889 (since TBD until the merge slot; re-summed at the 0.60.583.0 merge: 1646 + 3); +4 PR B commit_ledger_prb, doctor_foreign_writers_prb and sources_guard_prb rows and their comment (rows are data and cannot move; since TBD until the merge slot); +1 PR B #8615: file_drift_held_needs_prefer_8615 row; +1 PR B #8617: refresh_source_changed_retry_8617 row; +1 PR B #8662: commit_push_checkout_not_a_source_8662 row (since TBD until the merge slot; rows are data and cannot move); +2 PR B refresh_barrier_prb and commit_push_base_prb rows (re-summed: 1658 measured); +3 convergence PR A: the wave comment and two rows (conversion_content_evidence_pr_a, git_gap_live_pages_pr_a; since TBD until the merge slot; rows are data and cannot move); re-summed at the gate-ahead on master v0.60.592.0 plus #8711: 1658 + 3 = 1661 measured; +4 GST-1094/GST-900 rows (memory_recall scope-read, member-bound acting_user, recall max_chars, unknown_param _meta-only) and +1 qm_scope_writes_gst1113 and +1 recall_question_terms_gst1304 rows (rows are data and cannot move), re-summed at the 0.60.595.0 slot merge: 1661 + 6; +1 GST-1582 refresh_barrier_import_gst1582 row (since TBD until the merge slot; rows are data and cannot move); +2 GST-1601 source_changed_origin_gst1601 and adopt_local_edits_gst1601 rows (since TBD until the merge slot; rows are data and cannot move), re-summed at the master v0.60.596.0 merge: 1668 + 2; +1 GST-1623 refresh_barrier_capacity_gst1623 row (since TBD until the merge slot; rows are data and cannot move), re-summed at the master v0.60.599.0 merge: 1670 + 1; +2 GBRA-224 W1 release_channel_stable_default row and its comment; +1 GST-1623 refresh_barrier_later_write_gst1623 row (#8765; rows are data and cannot move); +5 #8766 GST-1623 items 4/6-9 and #8755: the wave comment and four rows (doctor_foreign_writers_partial_gst1623, commit_push_enclosing_repo_gst1623, owner_guard_fail_closed_gst1623, adopt_local_edits_hardening_8755) and +1 managed_sync_kind_reserved_8765 row (since TBD until the merge slot; rows are data and cannot move), re-summed at the gate-ahead on master v0.60.608.0 plus #8798: 1674 + 6; +1 GST-1853 refresh_barrier_aborted_refresh_sync_gst1853 row (since TBD until the merge slot; rows are data and cannot move), re-summed at the master v0.60.613.0 merge: 1680 + 1; +1 GST-1674 refresh_refused_merge_recovery_gst1674 row (since TBD until the merge slot; rows are data and cannot move), re-summed at the master v0.60.614.0 merge: 1681 + 1; +1 #8824/#8826/#8767: adopt_local_edits_replay_markers_8824 row (since TBD until the merge slot; rows are data and cannot move), re-summed at the gate-ahead on #8837: 1682 + 1; +1 url_source_origin row (since TBD until the merge slot; rows are data and cannot move), re-summed at the gate-ahead on #8834: 1683 + 1; +1 #8828: the refresh_replan_own_commits_8828 row (1684 + 1 at the 0.60.618.0 slot); +1 the managed commit-push unpublished-count row; +1 the refresh publish-diverged default row; +1 GST-2051 ungranted_member_writes_gst2051 row (since TBD until the merge slot; rows are data and cannot move); +1 GST-731 think_window_trajectory_gst731 row and +1 GST-1602 purge_cancels_deferred_write_gst1602 row (since TBD until the merge slot; rows are data and cannot move), re-summed at the gate-ahead on #8915 v0.60.621.0: 1688 + 2; +1 GST-2059: owner_reads_private row; +1 #8888: the record-origin checkout lock-down row (re-summed at the Q6 stack on #8934 v0.60.625.0: 1691 + 1); +1 GST-2112: ingest_keeps_readers_gst2112 row (re-summed at the Q7 v0.60.627.0 merge: 1692 + 1); +1 #8982 federated_member_writes_8982 row (since TBD until the merge slot; rows are data and cannot move); +2 GST-2089 owner_private_facts_gst2089 and GST-2092 private_labels_write_ahead_gst2092 rows (since TBD until the merge slot; rows are data and cannot move), re-summed at the access and privacy wave on #8991: 1694 + 2; +1 GBRA-443 private_labels_batch_exit_gbra443 row (since TBD until the merge slot; rows are data and cannot move)
diff --git a/scripts/page-body-readers-baseline.tsv b/scripts/page-body-readers-baseline.tsv
index d8efd29ede..8f0d5df33a 100644
--- a/scripts/page-body-readers-baseline.tsv
+++ b/scripts/page-body-readers-baseline.tsv
@@ -571,7 +571,7 @@ src/core/repair/overwritten-edits.ts previewEdit call:serializePageToMarkdown SE
src/core/repair/overwritten-edits.ts versionMarkdown call:serializePageToMarkdown SEGW inline versions only: a segmented version has no lost_digest and is never previewed; the restore goes through revert_version, which refuses a segmented page or version
src/core/repair/overwritten-edits.ts versionMarkdown field SEGW inline versions only: overwriteContent returns null for a segmented version, which answers version_changed
src/core/repair/overwritten-edits.ts versionMarkdown sql SEGW selects body_storage; a segmented version answers version_changed
-src/core/repair/private-labels.ts privateLabelsRepair call:serializePageToMarkdown SEGW a segmented page returns segmented_page before the read and keeps its label
+src/core/repair/private-labels.ts prepareWrite call:serializePageToMarkdown SEGW a segmented page returns segmented_page before the read and keeps its label
src/core/repair/redacted-facts.ts pageTexts field INLINE inline branch only (a segmented page's body is read through readBodyRange in segmentedText)
src/core/repair/redacted-facts.ts pageTexts sql INLINE page_versions rows that hold a body copy (compiled_truth IS NOT NULL); a segmented version has none and is not read
src/core/repair/redacted-facts.ts restoreFencedRows call:serializePageToMarkdown INLINE inline pages only: a segmented page returns before the read, and its fenced rows preview as class segmented
diff --git a/src/core/behavior-change-notice.ts b/src/core/behavior-change-notice.ts
index 852f1d3ccc..f927f3d447 100644
--- a/src/core/behavior-change-notice.ts
+++ b/src/core/behavior-change-notice.ts
@@ -1498,7 +1498,7 @@ export const BEHAVIOR_CHANGES: ReadonlyArray<{ since: string; text: ChangeText;
{ since: '0.60.628.0', id: 'federated_member_writes_8982', text: 'A token or OAuth client bound to a member whose write source is a federated source (`gbrain sources federate <id>`) now writes the pages it creates with `readable_by: [member:<id>]`, as one writing to `default` does, because every credential without a source grant reads federated sources too; `POST /ingest` refuses such a credential. Before this, other members\' no-grant tokens and unbound tokens read those pages. A write source that is not federated is unchanged, and pages written before the upgrade keep their readers (`gbrain access who-can-read <slug>`).' },
{ since: 'TBD', id: 'owner_private_facts_gst2089', text: 'The brain owner\'s remote credentials now get private facts in the `query` facts arm, `think` trajectories, hot memory, `delta` and `context_pack` hot facts, and from `recall` with a `question`, under the same rule as its private pages: no audience, or an audience of the owner alone. Facts taken from a page the owner cannot read and facts outside its source grant stay hidden. Facts on `entity` and `context_pack` cards stay world-only. Every other remote caller is unchanged.' },
{ since: 'TBD', id: 'private_labels_write_ahead_gst2092', text: '`gbrain repair private-labels` writes its undo ledger ahead of each page: an `intent` line before the write and a `committed` line after it, so a run that stops between them can still be undone with `--class restore`, and a ledger that cannot take the intent line refuses the apply (`storage_error`) before anything changes. Restore re-adds only `access` and `visibility: private` from well-formed ledger lines (others are counted unreadable) and no longer lists a page whose label was removed after a restore landed (`removed_after_restore`). The preview counts the derived_from pages and life-chronicle events each window would expose along with its pages; on a brain without managed persistence a page whose file sits in its source\'s checkout is held (`file_backed`) instead of being cleared in the database only; and an apply that ends with a failure keeps its approved preview, so the same `--apply --expect` resumes it.' },
- { since: 'TBD', id: 'private_labels_batch_exit_gbra443', text: '`gbrain repair private-labels --apply --expect` on a brain with managed persistence admits a window\'s pages together and publishes them in groups instead of one page and one wait at a time; a page whose write is declined fails alone. `gbrain repair <kind> --apply --expect <hash>` (every preview-bound kind) now exits 0 when the run is complete, nothing stopped, and every failed item is its page declining the write (`source_changed`, `page_body_segmented`); the result counts them in `declined` by reason, and any other failure or a stop still exits 1. An apply bound to no preview keeps exit 1 for them. The private-labels preview lists pages an earlier apply tried and that did not change after every page not tried yet (`details.window.tried_before`), no longer claims a re-embed for every page (only pages stored under per-chunk synopsis count), and an apply result carries `details.ledger`, `details.ledger_rows_written` and `details.ledger_rows`. A page an earlier apply tried that declines again no longer counts toward `item_failures_systemic`, so a window of carried declines completes; pages not tried yet still trip it. A preview-bound repair apply waits out a `gbrain sources refresh` for up to 10 minutes per step instead of 10 s, and past that ends with `worktree_refreshing` (exit 1) naming the refresh and its running time; rerunning the same apply resumes. Other interactive writes keep 10 s.' },
+ { since: 'TBD', id: 'private_labels_batch_exit_gbra443', text: '`gbrain repair private-labels --apply --expect` on a brain with managed persistence admits a window\'s pages together and publishes them in groups instead of one page and one wait at a time; a page whose write is declined fails alone. `gbrain repair <kind> --apply --expect <hash>` (every preview-bound kind) now exits 0 when the run is complete, nothing stopped, and every failed item is its page declining the write (`source_changed`, `page_body_segmented`); the result counts them in `declined` by reason, and any other failure or a stop still exits 1. An apply bound to no preview keeps exit 1 for them. The private-labels preview lists pages an earlier apply tried and that did not change after every page not tried yet (`details.window.tried_before`), no longer claims a re-embed for every page (only pages stored under per-chunk synopsis, or with a chunk that has no vector yet, count), and an apply result carries `details.ledger`, `details.ledger_rows_written` and `details.ledger_rows`. A page an earlier apply tried that declines again no longer counts toward `item_failures_systemic`, so a window of carried declines completes; pages not tried yet still trip it. A preview-bound repair apply waits out a `gbrain sources refresh` for up to 10 minutes per step instead of 10 s, and past that ends with `worktree_refreshing` (exit 1) naming the refresh and its running time; rerunning the same apply resumes. Other interactive writes keep 10 s.' },
];
/** The newest disclosed change's release: the notice id moves only when a release adds rows. */
diff --git a/src/core/repair/core.ts b/src/core/repair/core.ts
index c507f7ab53..4ff284fece 100644
--- a/src/core/repair/core.ts
+++ b/src/core/repair/core.ts
@@ -486,12 +486,19 @@ async function runRepairScoped(ctx: OperationContext, handler: RepairHandler, sc
return unresolved.length <= target;
};
const batched = new Map<number, RepairSettlement>();
+ let batchCap = Infinity;
absorbBatched = () => {
for (const [index, settled] of [...batched].sort(([a], [b]) => a - b)) {
batched.delete(index);
if ('error' in settled) {
const receipt = acceptedPendingReceipt(settled.error);
- if (receipt) unresolved.push({ index, item: pending[index]!, request_id: receipt.request_id });
+ if (receipt) { unresolved.push({ index, item: pending[index]!, request_id: receipt.request_id }); continue; }
+ if (settled.error instanceof OperationError && settled.error.code === 'revision_conflict') { result.skipped++; continue; }
+ // A page the batch already tried and the run stopped before reaching: reported as failed, never counted by the breaker.
+ const item = pending[index]!, failure = { ...classifyItemFailure(settled.error), key: `${item.source_id}:${item.slug}` };
+ result.failed.push(failedEntry(handler.kind, item, failure));
+ failedItems.push({ item, failure });
+ result.outcomes = { ...result.outcomes, failed: (result.outcomes?.failed ?? 0) + 1 };
continue;
}
const value = settled.value;
@@ -558,20 +565,16 @@ async function runRepairScoped(ctx: OperationContext, handler: RepairHandler, sc
+ `${remaining} item(s) still need ${remaining * perItem} more). Run: gbrain config set ${full.config_key} ${needed} — then rerun \`${result.apply_command}\` to resume.` });
}
// GBRA-443: the items from here that fit the admission window, the capacity stop line and one write per page, applied as one run.
- if (handler.applyBatch && !llm) {
- const room = Math.min(Math.max(1, window - unresolved.length), ...counters.map(c => Math.floor((c.stop_at - c.used) / (c.resource === 'lifetime_ids' ? 1 : RECEIPT_RESERVATION_FLOOR))));
- const pages = new Set([...unresolved.map(open => open.item), item].map(other => other.source_id + '\0' + other.slug));
- let end = index + 1;
- for (; end < pending.length && end - index < room; end++) {
- const key = pending[end]!.source_id + '\0' + pending[end]!.slug;
- if (pages.has(key) || (opts.expect && end === plan.items.length - 1 && unresolved.length)) break;
- pages.add(key);
- }
- const run = pending.slice(index, end);
+ if (handler.applyBatch && !llm && batchCap > 1) {
+ const room = Math.min(batchCap, Math.max(1, window - unresolved.length), ...counters.map(c => Math.floor((c.stop_at - c.used) / (c.resource === 'lifetime_ids' ? 1 : RECEIPT_RESERVATION_FLOOR))));
+ const run = batchRun(pending, index, room, unresolved.map(open => open.item), opts.expect && unresolved.length ? plan.items.length - 1 : null);
// A thrown batch is the first item's failure, classified below like a single apply's; the rest run again from the next item.
+ // A batch the writer's capacity refused whole (admission is atomic) halves the batches after it, and this item applies alone.
const settled = await step(() => handler.applyBatch!({ ...ctx, sourceId: item.source_id }, run, { embed: opts.embed === true, runId, ...(opts.expect ? { expect: opts.expect } : {}) }))
.catch((error: unknown): RepairSettlement[] => [{ error }]);
- settled.slice(0, run.length).forEach((entry, offset) => batched.set(index + offset, entry));
+ const refusedWhole = run.length > 1 && settled.length > 0 && settled.every(entry => 'error' in entry && entry.error instanceof OperationError && entry.error.code === 'queue_capacity');
+ if (refusedWhole) batchCap = Math.max(1, Math.floor(run.length / 2));
+ else settled.slice(0, run.length).forEach((entry, offset) => batched.set(index + offset, entry));
}
}
const prefetched = batched.get(index);
@@ -641,7 +644,14 @@ async function runRepairScoped(ctx: OperationContext, handler: RepairHandler, sc
fix: readFix(`Previews ${handler.kind} again (read-only) to show whether the database accepts work again.`, { argv: ['gbrain', 'repair', handler.kind, ...(opts.sourceFlag ? ['--source', opts.sourceFlag] : []), '--json'] }) });
}
// An apply bound to an approved preview (`--expect`) applies that set or answers why not; only the kinds' own refusals continue it.
- if (opts.expect && !(error instanceof OperationError && REPAIR_ITEM_FAILURE_CODES.has(error.code))) throw error;
+ if (opts.expect && !(error instanceof OperationError && REPAIR_ITEM_FAILURE_CODES.has(error.code))) {
+ if (![...batched.values()].some(entry => 'value' in entry)) throw error;
+ // GBRA-443: later pages of this item's batch were already written; stop with this error so the result still counts them.
+ const code = error instanceof OperationError ? error.code : classifyItemFailure(error).code;
+ return stopAfterSettle({ reason: code, message: redactConnectionInfo(`Stopped at ${item.source_id}:${item.slug} (${code}): ${(error instanceof Error ? error.message : String(error)).slice(0, 300)} `
+ + `Pages of its batch after it that were already published are counted in this result. Fix this page, then rerun \`${result.apply_command}\`.`),
+ fix: readFix(`Previews ${handler.kind} again (read-only) to show this page's current state.`, { argv: ['gbrain', 'repair', handler.kind, ...(opts.sourceFlag ? ['--source', opts.sourceFlag] : []), '--json'] }) });
+ }
const carried = item.retried === true && error instanceof OperationError && REPAIR_DECLINE_CODES.has(error.code);
const { failure, systemic: stop } = carried ? { failure: { ...classifyItemFailure(error), key: `${item.source_id}:${item.slug}` }, systemic: null }
: items.record(`${item.source_id}:${item.slug}`, error);
@@ -652,9 +662,7 @@ async function runRepairScoped(ctx: OperationContext, handler: RepairHandler, sc
// A per-item failure: recorded, nothing written, no cursor for it; the next full scan retries it (PR #6459, GBRA-130).
const message = redactConnectionInfo(failure.message);
const failedItem = `${item.source_id}:${item.slug}`;
- result.failed.push({ item: failedItem, source_id: item.source_id, slug: item.slug, reason: failure.code, message, retry: 'next_full_scan',
- fix: readFix(`Previews ${handler.kind} for source ${item.source_id} (read-only); it lists ${item.slug} again with its current state. A run with no stored cursor retries it.`,
- { argv: ['gbrain', 'repair', handler.kind, '--source', item.source_id, '--json'] }) });
+ result.failed.push(failedEntry(handler.kind, item, failure));
failedItems.push({ item, failure });
result.outcomes = { ...result.outcomes, failed: (result.outcomes?.failed ?? 0) + 1 };
if ((result.outcome_items ??= []).length < (handler.outcomeItemsLimit ?? SAMPLE * 2)) {
@@ -674,6 +682,28 @@ async function runRepairScoped(ctx: OperationContext, handler: RepairHandler, sc
return finish();
}
+/** One item-scoped failure as `result.failed` reports it: nothing written, and the next full scan retries the item. */
+function failedEntry(kind: string, item: RepairItem, failure: ItemFailure): RepairResult['failed'][number] {
+ return { item: `${item.source_id}:${item.slug}`, source_id: item.source_id, slug: item.slug, reason: failure.code, message: redactConnectionInfo(failure.message), retry: 'next_full_scan',
+ fix: readFix(`Previews ${kind} for source ${item.source_id} (read-only); it lists ${item.slug} again with its current state. A run with no stored cursor retries it.`,
+ { argv: ['gbrain', 'repair', kind, '--source', item.source_id, '--json'] }) };
+}
+
+/**
+ * GBRA-443: the items from `index` one batch applies: at most `room`, one write per page (a page already pending or earlier in
+ * the batch ends it), and never the item at `stopBefore` (a preview-bound run's last approved item while writes are unresolved).
+ */
+function batchRun(pending: RepairItem[], index: number, room: number, busy: RepairItem[], stopBefore: number | null): RepairItem[] {
+ const pages = new Set([...busy, pending[index]!].map(other => other.source_id + '\0' + other.slug));
+ let end = index + 1;
+ for (; end < pending.length && end - index < room; end++) {
+ const key = pending[end]!.source_id + '\0' + pending[end]!.slug;
+ if (pages.has(key) || end === stopBefore) break;
+ pages.add(key);
+ }
+ return pending.slice(index, end);
+}
+
/** Cursor order shared by the kinds' SQL: (phase, id) strictly after `after`. */
export function afterCursor(item: RepairCursor, after: RepairCursor | null): boolean {
return !after || item.phase > after.phase || (item.phase === after.phase && item.id > after.id);
diff --git a/src/core/repair/private-labels.ts b/src/core/repair/private-labels.ts
index 545f06fcd4..200f0b9ba6 100644
--- a/src/core/repair/private-labels.ts
+++ b/src/core/repair/private-labels.ts
@@ -58,6 +58,7 @@ import { isTerminal } from '../persistence/model.ts';
import { requestPrincipalForContext } from '../persistence/page-mutations.ts';
import { digest } from '../persistence/digest.ts';
import { modeRequiresSynopsis } from '../embedding-context.ts';
+import { chunkGenerationFilter } from '../search/safe-chunks.ts';
import { importFromContent } from '../import-file.ts';
import { isSegmentedPage } from '../page-body/page-shape.ts';
import { managedPersistenceEnabled } from '../persistence/ownership.ts';
@@ -303,13 +304,11 @@ async function importAtRevision(engine: BrainEngine, entry: Approved, revision:
});
}
-/** GBRA-443: lines this run appended to each ledger, by phase (the apply result's ledger_rows_written and ledger_rows). */
-const ledgerRows = new WeakMap<PrivateLabelsLedger, { intent: number; committed: number }>();
-function recordLine(ledger: PrivateLabelsLedger, line: LedgerLine & { phase: 'intent' | 'committed' }): void {
+/** GBRA-443: lines this run appended to its ledger, by phase (the apply result's ledger_rows_written and ledger_rows), kept on the ledger it was handed. */
+type CountedLedger = PrivateLabelsLedger & { rows?: { intent: number; committed: number } };
+function recordLine(ledger: CountedLedger, line: LedgerLine & { phase: 'intent' | 'committed' }): void {
ledger.record(line);
- const rows = ledgerRows.get(ledger) ?? { intent: 0, committed: 0 };
- rows[line.phase]++;
- ledgerRows.set(ledger, rows);
+ (ledger.rows ??= { intent: 0, committed: 0 })[line.phase]++;
}
/**
@@ -326,13 +325,17 @@ function triedBefore(ledger: PrivateLabelsLedger | undefined, brainId: string):
return new Set([...latest].flatMap(([key, line]) => line.removed && line.phase === 'intent' ? [key] : []));
}
-/** GBRA-443: the listed items whose write re-embeds: a label-only edit keeps every stored vector except on a page stored under per-chunk synopsis. */
+/** GBRA-443: the listed items whose write embeds: a label-only edit keeps every stored vector except on a page stored under per-chunk synopsis; a page with a chunk that has no vector yet is counted too. */
async function reembedding(engine: BrainEngine, items: LabelsItem[]): Promise<LabelsItem[]> {
if (!items.length) return [];
+ const ids = items.map(item => item.entry.page_id);
const rows = await engine.executeRaw<{ id: number; mode: string }>(`SELECT id, contextual_retrieval_mode AS mode FROM pages
- WHERE id = ANY($1::integer[]) AND contextual_retrieval_mode IS NOT NULL`, [items.map(item => item.entry.page_id)], bounded);
- const synopsis = new Set(rows.filter(row => modeRequiresSynopsis(row.mode as Parameters<typeof modeRequiresSynopsis>[0])).map(row => Number(row.id)));
- return items.filter(item => synopsis.has(item.entry.page_id));
+ WHERE id = ANY($1::integer[]) AND contextual_retrieval_mode IS NOT NULL`, [ids], bounded);
+ const unembedded = await engine.executeRaw<{ id: number }>(`SELECT DISTINCT cc.page_id AS id FROM content_chunks cc JOIN pages p ON p.id = cc.page_id
+ WHERE cc.page_id = ANY($1::integer[]) AND cc.embedding IS NULL AND ${chunkGenerationFilter('p', 'cc')}`, [ids], bounded);
+ const embeds = new Set([...rows.filter(row => modeRequiresSynopsis(row.mode as Parameters<typeof modeRequiresSynopsis>[0])).map(row => Number(row.id)),
+ ...unembedded.map(row => Number(row.id))]);
+ return items.filter(item => embeds.has(item.entry.page_id));
}
const CHANGED: RepairItemOutcome = { applied: false, outcome: 'changed_since_preview' };
@@ -440,7 +443,7 @@ export const privateLabelsRepair: RepairHandler = {
}
const approvedExposes = new Set(approved.items.flatMap(entry => entry.exposes ?? []));
if ([...(await windowDependents(engine, approved.items)).exposes.values()].flat().some(id => !approvedExposes.has(id))) throw previewChangedError(opts.expect, command);
- ledgerRows.set(ledger!, { intent: 0, committed: 0 });
+ (ledger as CountedLedger).rows = { intent: 0, committed: 0 };
const items = toItems(approved.items).filter(item => afterCursor(item.cursor, after));
return { items, preview_hash: opts.expect, residuals: {}, embedding_items: await reembedding(engine, items) };
}
@@ -515,7 +518,9 @@ export const privateLabelsRepair: RepairHandler = {
async applyBatch(ctx, raw, opts) {
const items = raw as LabelsItem[];
const out: RepairSettlement[] = new Array(items.length);
- const settle = (index: number, write: () => Promise<RepairItemOutcome>) => write().then(value => { out[index] = { value }; }, error => { out[index] = { error }; });
+ // Once any page of this call may have been written, a later throw settles the pages it left instead of discarding the batch.
+ let wrote = false;
+ const settle = (index: number, write: () => Promise<RepairItemOutcome>) => { wrote = true; return write().then(value => { out[index] = { value }; }, error => { out[index] = { error }; }); };
const ready: Array<Ready | null> = [];
for (const [index, item] of items.entries()) {
const prepared = await prepareWrite(ctx, item).catch((error: unknown) => ({ error }));
@@ -527,36 +532,43 @@ export const privateLabelsRepair: RepairHandler = {
for (const sourceId of new Set(items.map(item => item.entry.source_id))) {
const indices = items.flatMap((item, index) => item.entry.source_id === sourceId && ready[index] ? [index] : []);
if (!indices.length) continue;
- const preflight = await maintenancePreflight(ctx.engine, sourceId).then(authority => ({ authority }), (error: unknown) => ({ error }));
- if ('error' in preflight) { for (const index of indices) out[index] = { error: preflight.error }; continue; }
- const { authority } = preflight;
- const batch: Array<{ index: number; line: LedgerLine }> = [];
- for (const index of indices) {
- const line = await ledgerLine(ctx, items[index]!, ready[index]!);
- // An unmanaged source, or a request a single-page apply admitted before batching (its intent has no batch id), writes on its own.
- const prior = authority ? await getWriteRequest(ctx.engine, principal, line.request_id) : null;
- if (!authority || (prior && !isTerminal(prior) && typeof prior.intent?.maintenance_batch !== 'string')) {
- await settle(index, () => writeOne(ctx, items[index]!, ready[index]!, line, authority, opts.embed));
- continue;
+ try {
+ const preflight = await maintenancePreflight(ctx.engine, sourceId).then(authority => ({ authority }), (error: unknown) => ({ error }));
+ if ('error' in preflight) { for (const index of indices) out[index] = { error: preflight.error }; continue; }
+ const { authority } = preflight;
+ const batch: Array<{ index: number; line: LedgerLine }> = [];
+ for (const index of indices) {
+ const line = await ledgerLine(ctx, items[index]!, ready[index]!);
+ // An unmanaged source, or a request a single-page apply admitted before batching (its intent has no batch id), writes on its own.
+ const prior = authority ? await getWriteRequest(ctx.engine, principal, line.request_id) : null;
+ if (!authority || (prior && !isTerminal(prior) && typeof prior.intent?.maintenance_batch !== 'string')) {
+ await settle(index, () => writeOne(ctx, items[index]!, ready[index]!, line, authority, opts.embed));
+ continue;
+ }
+ batch.push({ index, line });
}
- batch.push({ index, line });
- }
- if (!batch.length || !authority) continue;
- const admitted: typeof batch = [];
- for (const [position, entry] of batch.entries()) {
- try { recordIntent(items[entry.index]!, entry.line); admitted.push(entry); } catch (error) {
- // A ledger that refuses one intent line refuses the rest too: nothing after it is written.
- for (const rest of batch.slice(position)) out[rest.index] = { error };
- break;
+ if (!batch.length || !authority) continue;
+ const admitted: typeof batch = [];
+ for (const [position, entry] of batch.entries()) {
+ try { recordIntent(items[entry.index]!, entry.line); admitted.push(entry); } catch (error) {
+ // A ledger that refuses one intent line refuses the rest too: nothing after it is written.
+ for (const rest of batch.slice(position)) out[rest.index] = { error };
+ break;
+ }
}
- }
- if (!admitted.length) continue;
- const published = await publishMaintenancePages(ctx.engine, authority, admitted.map(({ index, line }) => ({ slug: items[index]!.entry.slug, content: ready[index]!.content,
- expectedRevision: ready[index]!.revision, requestId: line.request_id, keepTrust: true as const })), { batch: digest(['private-labels-batch-v1', opts.expect ?? opts.runId ?? '', sourceId]) });
- for (const [position, { index, line }] of admitted.entries()) {
- const settled = published[position]!;
- if ('error' in settled) { out[index] = { error: settled.error }; continue; }
- out[index] = { value: committedOutcome(items[index]!, line, await writtenRevision(ctx.engine, items[index]!.entry)) };
+ if (!admitted.length) continue;
+ const published = await publishMaintenancePages(ctx.engine, authority, admitted.map(({ index, line }) => ({ slug: items[index]!.entry.slug, content: ready[index]!.content,
+ expectedRevision: ready[index]!.revision, requestId: line.request_id, keepTrust: true as const })), { batch: digest(['private-labels-batch-v1', opts.expect ?? opts.runId ?? '', sourceId]) });
+ wrote = true;
+ for (const [position, { index, line }] of admitted.entries()) {
+ const settled = published[position]!;
+ if ('error' in settled) { out[index] = { error: settled.error }; continue; }
+ // A failed read of the written revision still records the committed line (without revision_after).
+ out[index] = { value: committedOutcome(items[index]!, line, await writtenRevision(ctx.engine, items[index]!.entry).catch(() => null)) };
+ }
+ } catch (error) {
+ if (!wrote) throw error;
+ for (const index of indices) out[index] ??= { error };
}
}
return out;
@@ -580,7 +592,7 @@ export const privateLabelsRepair: RepairHandler = {
}
const ledger = opts.privateLabelsLedger;
if (result.mode !== 'apply' || !ledger) return {};
- const rows = ledgerRows.get(ledger) ?? { intent: 0, committed: 0 };
+ const rows = (ledger as CountedLedger).rows ?? { intent: 0, committed: 0 };
return { details: { ...result.details, ledger: ledger.path, ledger_rows_written: rows.intent + rows.committed, ledger_rows: { ...rows } } };
},
render(details) {
diff --git a/test/postgres-unit-arms.txt b/test/postgres-unit-arms.txt
index 90425a9231..ad8a73cc58 100644
--- a/test/postgres-unit-arms.txt
+++ b/test/postgres-unit-arms.txt
@@ -488,6 +488,7 @@ test/repair-file-drift-provenance-gbra133.test.ts
test/repair-frontmatter-keys-sweep-6454.test.ts
test/repair-frontmatter.test.ts
test/repair-ghost-database-ahead-gbra133.test.ts
+test/repair-private-labels-batch.test.ts
test/repair-quoted-markers-8059.test.ts
test/repair-receipt-compaction.test.ts
test/repair-rename-identity-6729.test.ts
diff --git a/test/repair-private-labels-batch.test.ts b/test/repair-private-labels-batch.test.ts
index 6d0a542338..3e6ee64cf3 100644
--- a/test/repair-private-labels-batch.test.ts
+++ b/test/repair-private-labels-batch.test.ts
@@ -108,15 +108,20 @@ for (const backend of testBackends()) {
};
const pageId = async (source: string, slug: string) => Number((await engine.readPageSnapshot(slug, { sourceId: source }))!.page.id);
- test('the preview claims no re-embeds for a label-only edit, except for a page stored under per-chunk synopsis', async () => {
+ test('the preview claims no re-embeds for a label-only edit, except for a page stored under per-chunk synopsis or with a chunk that has no vector yet', async () => {
await seed('source-e', 'notes/e0');
await seed('source-e', 'notes/e1');
+ const vector = '[' + Array(1536).fill(0.01).join(',') + ']';
+ await engine.executeRaw("UPDATE content_chunks SET embedding=$1::text::vector WHERE page_id IN (SELECT id FROM pages WHERE source_id='source-e')", [vector]);
const plain = (await cli(engine, ['private-labels', '--source', 'source-e'], env)).result;
expect(plain.listing.length).toBe(2);
expect(plain.cost.embedding_pages).toBe(0);
await engine.executeRaw("UPDATE pages SET contextual_retrieval_mode='per_chunk_synopsis' WHERE source_id='source-e' AND slug='notes/e1'");
const synopsis = (await cli(engine, ['private-labels', '--source', 'source-e'], env)).result;
expect(synopsis.cost.embedding_pages).toBe(1);
+ await engine.executeRaw("UPDATE content_chunks SET embedding=NULL WHERE page_id=(SELECT id FROM pages WHERE source_id='source-e' AND slug='notes/e0')");
+ const unembedded = (await cli(engine, ['private-labels', '--source', 'source-e'], env)).result;
+ expect(unembedded.cost.embedding_pages).toBe(2);
});
test('pages an earlier apply tried and that did not change are listed after every page not tried yet; the apply names its ledger and the lines it appended', async () => {
@@ -181,7 +186,8 @@ for (const backend of testBackends()) {
await settleManagedGitEffects(engine);
const preview = (await cli(engine, ['private-labels', '--limit', '4'])).result;
expect(preview.listing.map((entry: { item: string }) => entry.item)).toEqual(slugs.slice(0, 4).map(slug => 'default:' + slug));
- expect(preview.cost.embedding_pages).toBe(0);
+ // No provider here, so these pages have no vectors yet: their write embeds them, and the preview counts them.
+ expect(preview.cost.embedding_pages).toBe(4);
const drifted = join(root, slugs[0] + '.md');
writeFileSync(drifted, readFileSync(drifted, 'utf8') + NL + 'An edit made outside gbrain.' + NL);
installFaultHook(async (point, detail) => {
@@ -262,9 +268,48 @@ for (const backend of testBackends()) {
const w4 = await cli(engine, ['private-labels', '--apply', '--expect', four.preview_hash]);
expect(w4.result.stopped?.reason).toBe('item_failures_systemic');
expect(w4.result.applied).toBe(0);
+ // Every page the batch tried is reported, the ones past the stop included; the page the breaker stopped at is named by the stop.
+ expect(w4.result.failed.map((f: { slug: string }) => f.slug).sort()).toEqual([...fresh.slice(0, 4), fresh[5]!, ...declined].sort());
expect(w4.exit).toBe(1);
}, { databaseUrl, setup }), 240_000);
+ test('a page refused with a non-item error inside a batch stops the run with its code, and the pages after it, already published, are counted', () => managedBrain(async ({ engine, ctx }) => {
+ const slugs = [0, 1, 2, 3].map(i => 'notes/stop-' + i);
+ for (const slug of slugs) await put(ctx, slug, labelled(slug));
+ await settleManagedGitEffects(engine);
+ const preview = (await cli(engine, ['private-labels'])).result;
+ installFaultHook(async (point, detail) => {
+ if (point !== 'consumer:preparing') return;
+ const [row] = await engine.executeRaw<{ slug: string }>('SELECT slug FROM persistence_requests WHERE id=$1::uuid', [detail.writeId]);
+ if (row?.slug === slugs[1]) throw new OperationError('invalid_fence', 'Forced fence refusal inside the batch.', 'Fixture.');
+ });
+ let run;
+ try { run = await cli(engine, ['private-labels', '--apply', '--expect', preview.preview_hash]); } finally { installFaultHook(undefined); }
+ const { result, exit } = run;
+ expect(result.stopped?.reason).toBe('invalid_fence');
+ expect(result.stopped?.message).toContain(slugs[1]);
+ expect(exit).toBe(1);
+ expect(result.applied).toBe(3);
+ expect((await frontmatter(engine, slugs[1])).access).toBe('private');
+ for (const slug of [slugs[0], slugs[2], slugs[3]]) expect('access' in await frontmatter(engine, slug)).toBe(false);
+ expect(result.details.ledger_rows).toEqual({ intent: 4, committed: 3 });
+ expect(lines(result.details.ledger).length).toBe(result.details.ledger_rows_written);
+ }, { databaseUrl, setup }), 180_000);
+
+ test('a writer whose brain-wide outstanding cap is below the batch still applies every page: a refused batch halves the next ones and its page applies alone', () => managedBrain(async ({ engine, ctx }) => {
+ const slugs = [0, 1, 2].map(i => 'notes/cap-' + i);
+ for (const slug of slugs) await put(ctx, slug, labelled(slug));
+ await settleManagedGitEffects(engine);
+ await engine.setConfig('persistence.limits.brain_outstanding', '1');
+ const preview = (await cli(engine, ['private-labels'])).result;
+ const { result, exit } = await cli(engine, ['private-labels', '--apply', '--expect', preview.preview_hash]);
+ expect(result.stopped).toBeUndefined();
+ expect(result.applied).toBe(3);
+ expect(result.complete).toBe(true);
+ expect(exit).toBe(0);
+ for (const slug of slugs) expect('access' in await frontmatter(engine, slug)).toBe(false);
+ }, { databaseUrl, setup }), 180_000);
+
test('every page of a run has its intent line before any of them is published, and a crash after the group published still restores each', () => managedBrain(async ({ engine, ctx, root }) => {
const slugs = [0, 1, 2].map(i => 'notes/crash-' + i);
for (const slug of slugs) await put(ctx, slug, labelled(slug));