Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

Keep an ordinary job claim on the claim index when no watchdog-marked job is overdue (GST-2529)

garrytanb2d313a085aa
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 9d18ad2b1b..cc4e72594a 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -17,7 +17,7 @@ identifiers and attribution are available in the pre-removal Git revision
- A large source that the background syncer could not finish no longer retries forever with the same short budget. A resident gives its first run 30 seconds, and every resident restart or change of the process holding the notifier counted as "first" again, so each retry stopped at the same point having saved nothing. That short budget now goes only to a source with no failed attempts on record, and a retry after a run that saved nothing gets the full sync budget.
- A source the background syncer marked stalled (`notifier_stalled`, "retrying every 300 s") now really is retried every five minutes. With `sync.on_commit` off (the default), the retry only happened while the original sync request was still on record, and a run that stopped early could clear that request, so the source sat stalled with nothing retrying it and nothing in the log. The retry now continues until a run completes, also after a restart or when another resident takes over the notifier, which now reads the state the previous one recorded. Turning `sync.on_commit` off, or excluding the source, still ends a retry that a HEAD move started.
- `gbrain sync --skip-failed`, and the automatic skip after three failed syncs, no longer skip files that were refused because of how the source is set up rather than because anything is wrong with them (`strict_requires_managed`: a `canonical: git` source on a brain without managed persistence). Skipping them moved the sync checkpoint to the latest commit with no pages imported, after which `sync status` said synced and later syncs found nothing to do, even once the setup was fixed. The sync now stays blocked, says why, and names the two fixes: turn on managed persistence, or remove `canonical: git`, after which the next plain sync imports every page. A brain that was already skipped past this way needs one `gbrain sync --source <id> --full`. A scheduled `gbrain sync --skip-failed && …` on such a source now stops at the blocked sync until the setup is fixed.
-- A background job that went back to the queue because its worker was killed while running other jobs no longer waits for as long as the next worker stays busy. Such a job only runs alone, so a memory-watchdog stop can be blamed on the right job, and a worker that always had another job running never picked it up: a sync job requeued this way after a restart waited more than ten minutes. Once it has waited a minute, a worker now takes no new jobs until the ones it is running finish, then runs it first.
+- A background job that went back to the queue because its worker was killed while running other jobs no longer waits for as long as the next worker stays busy. Such a job only runs alone, so a memory-watchdog stop can be blamed on the right job, and a worker that always had another job running never picked it up: a sync job requeued this way after a restart waited more than ten minutes. Once it has waited a minute, a worker now takes no new jobs until the ones it is running finish, then runs it first. A worker checks for such a job every five seconds, so a queue with none claims exactly as fast as before.
**Every push gbrain makes from an unmanaged checkout now scans the commits it publishes for secrets, and a long first push can finish.** `gbrain git commit-push` scanned only with `--push-only --scan-secrets`, so its commit mode, `--pending`, the post-commit hook `gbrain sources harden` installs and `scripts/brain-commit-push.sh` pushed earlier local commits unscanned; they now refuse `publish_secret_found` or `publish_scan_unreadable` like the four pushes below, and the refusal is logged in `brain-push.log`. The helper's fallback for a machine without gbrain commits and no longer pushes, since it cannot scan. A scan that runs out of its 300 s budget refuses naming the same command with `--scan-timeout <seconds>`, which `gbrain git commit-push`, `gbrain sources push` and `gbrain bootstrap repo` now take; the scan reads every commit either way, prints its progress on stderr and keeps its temporary disk use to one batch of commits. No push follows tags any more, so `push.followTags` cannot publish an annotated tag's message unscanned. `gbrain sources push` reads origin's branch tip from origin itself when the branch tracks another remote (it used to fail the push, never skip commits), prints its refusal with `--json` and exits 5, and the access-policy commit reports a refusal as `committed_not_pushed` instead of failing.
diff --git a/scripts/module-size-limits.tsv b/scripts/module-size-limits.tsv
index 655f2c9e07..51a589ec98 100644
--- a/scripts/module-size-limits.tsv
+++ b/scripts/module-size-limits.tsv
@@ -24,7 +24,7 @@ src/core/pglite-engine.ts 3103 ratchet +9 #7191: findPages/countPagesBy delegate
src/core/postgres-engine.ts 3180 ratchet +9 #7191: findPages/countPagesBy scoped-read delegates and imports (SQL in engine-sql/page-lookup.ts); +6 GBRA-283 B1 (#8455): an engine opened from settings routes its direct URL and statement-description cache from its settings, and initSchema seeds settings.embedding and refuses a mismatch (logic in embedded-engine.ts; engine methods cannot move); +1 System One v1 evals: search arms clamp to searchLimitCap() (import of search/eval-pool-depth.ts; eval-only --eval-pool-depth, production cap unchanged); refactor wave 1 split target (W1 engine-sql); ceiling rationale: docs/architecture/key-files/engines-2.md entry for src/core/postgres-engine.ts; evidence delivery: getChunkWindows scoped-read delegate + type import (+5; SQL in engine-sql/chunks.ts); #5761: +4 markPagesAttendanceBlocked delegate (SQL in engine-sql/pages.ts); +10 eval wave N1-2/N1-3: mergeOntologyFact same-stint noop check + ontology visibility filter in the grandfathered ontology methods; lowered to 3242 #5824: searchVector SQL moved to search/vector-statement.ts, +explainVectorSearch/runVectorAttempt (-156); +6 #5730: onpoisoned wiring on the instance/module/connection-manager pools + poisonedDiscards in getPoolDiagnostics (counter in pool-gauge.ts); +5 #5801: onCheckout observer seam (fires after reserve resolution / transaction-callback entry; listeners in pool-gauge.ts); lowered to 3119 #5889: searchTitles SQL moved to engine-sql/titles.ts (-133); lowered to 2914 F4a: getHealth SQL moved to engine-sql/health.ts (-216); +2 #5984: transactions track held page guards like PGLite (withHeldPageKeys), so repeated guards in one transaction cost no round trips; +1 #5984: traceSqlOptions import for the env-gated GBRAIN_SQL_TRACE bench hook (src/core/sql-trace.ts); v0.60.48.0 merge: #5984 growth re-summed over v0.60.47.0 (2917 measured); +2 engine graduation (GBRA-50 G2a): initSchema sizing parameter (the source embedding layout sizes a fresh target); +1 #5984: upsertChunks guards pass the per-transaction config memo (transactionMemo in page-state/transactions.ts); v0.60.58.0 merge: #5984 diet growth re-summed over v0.60.57.0 (2920 measured); +4 P7 relationalChainHop delegate; +8 contributor audit wave P0: listFactsKeyset delegate (SQL in engine-sql/facts.ts); +1 #5375: get_versions limit/include_body type import (GetVersionsOpts, PageVersionRows); 6 #6085: initSchema pins advisory lock 42, bootstrap, schema replay and unlock to one reserved backend when the DDL pool has more than one slot (gauge only, no long-hold permit), released in finally; wave 10 collector: lane E (#6085), lane G (#5375) and master compose at 2939; +2 #6110: search transactions pass jitOff to withScopedReadTransaction (SET LOCAL jit = off in postgres-engine/search-settings.ts); +2 #5984 lane apply diet: putPage/setPageAliases write inline in the caller's page transaction (engine methods cannot move); +1 #5984 single-write fast path composes with the lane apply diet at 2944; +2 #5984 shared parameter types: the instance pool passes shared_types and initSchema clears them after migrations (helpers in db.ts); +2 #6276: the remote alias-resolving readPageSnapshot runs with JIT off (withScopedReadTransaction jitOff); +5 #6278: executeRaw timeoutMs delegate (statement in postgres-engine/bounded-statement.ts); +23 #6278 PR 1.5: cancelSettleMs() and the settle timer in runUnsafe that discards a reserved connection whose cancel a transaction-mode pooler never completed; +3 #6317 lane 2: getPoolDiagnostics().pool from the vendored driver (postgres-engine/pool-stats.ts) and the prepare mark; +4 GBRA-69: replaceDerivedLinksBatch delegate (logic in derived-links.ts); +1 GBRA-69 extract batching: replaceDerivedLinksBatch delegate (batch logic in derived-links.ts); +9 GBRA-75: getBacklinkCounts and getHealth statements run JIT-off through the private jitOffRead transaction helper (engine methods cannot move); +3 GBRA-75 wave 7: searchKeyword and searchVector start the once-per-engine background search statistics guard (logic in search/projection-statistics.ts); +7 GBRA-67: readPageSnapshotsBatch scoped-read delegate (batch logic in page-snapshot-batch.ts); +1 GBRA-75 wave 10: findDuplicatePages scoped-read delegate (SQL in engine-sql/pages.ts); +7 #6383: poolIncidents hooks on the instance/module/connection-manager pools (counter and lines in pool-gauge.ts, budget in db.ts) and stuckDiscards/buildFailures in getPoolDiagnostics; +3 GBRA-76: transaction/transactionDirect take TransactionOptions; transactionOn attaches the begin handle to withTransactionAbort (logic in postgres-engine/transaction-abort.ts); +2 W14 P3.12 (#4772): entityTypes HealthDeps delegate + import on top of W13; +1 #5227: schema.sql replay under withSchemaLockTimeout (wave 14 PR1); +1 driver desync tripwire: desyncDiscards in getPoolDiagnostics (counter and line in pool-gauge.ts); +11 Cat 40 Hard fix wave F2 (#6271) on top: countKeywordPages/searchKeywordPages scoped-read delegations (SQL in engine-sql/keyword-pages.ts) and the rankQuery score parameter (3027 measured); +3 W15 P1.2 (#6454): three one-line stale-frontmatter-key delegates, reads scoped (SQL in engine-sql/pages.ts); +12 wave 16 G5.1 (#6495): keywordStatementTimeout memo field, its connect/disconnect resets and the keywordStatementTimeoutSetting accessor, plus one awaited setting per keyword/chunk/CJK transaction (resolver in postgres-engine/keyword-statement-timeout.ts); wave 16 G5.1 gate fix: keywordStatementTimeoutSetting resolves the knob once per connection on the first keyword statement from the saved connect config (connect issues no statement for it; 3042 measured); +7 large pages Wave A sub-a: getChunkPage scoped-read delegate (SQL in engine-sql/chunks.ts); +11 #6696 B3: readEmbeddingStateCounts/listEmbeddingStateRows delegations + type import; +22 Wave B lane 1a: segmented page storage delegations (allocateBodyOccurrenceIds..sweepSegments, SQL in engine-sql/page-body.ts) and their imports; +3 wave 18b lane 2 (#6495): keywordTimeouts negative-cache field and the searchKeyword runner wrap (logic in postgres-engine/keyword-timeout-cache.ts); integrated at wave 18b: master 3082 + 3; +3 PR2 fix lane B: logIngest stamps writer_version/writer_host_id/writer_segment_reader and getIngestLog names its columns (engine methods cannot move); integrated at the Wave B PR 2 master merge (v0.60.264.0): master 3085 + 3; +3 PR2 fix lane E (E1): getChunks/getChunkPage/getChunkWindows run the manifest privilege probe in their scoped transaction (chunkManifestReads in search/safe-chunks.ts); integrated at the lane E merge onto b75a69a6d: 3088 + 3; +6 PR2 fix lane X: searchVector, explainVectorSearch, the hasMore witness and runVectorAttempt pick the chunk-manifest statement form inside each bounded attempt transaction, so its statement_timeout bounds the probe (form memo vectorStatementForms in search/vector-statement.ts; engine methods cannot move): 3091 + 6; +5 PR2 fix lane X: keywordBoundedRead runs the read in withScopedReadTransaction under the keyword statement timeout setting (engine methods cannot move): 3097 + 5; +2 GBRA-130 B-6 (#6860): putDreamVerdict passes its payload through wellFormedJson (engine methods cannot move); +1 GBRA-130 A-1: held option on the stale-page list/count signatures (SQL in engine-sql/pages.ts); +1 GBRA-130 integration: the A-1 (links_extract_failure) and B-6 (wellFormedJson putDreamVerdict) engine hunks composed at the lane merge; +2 #6948: putPage runs assertOriginUnclaimed in its page transaction and restorePage runs assertRevivalUnclaimed (logic in persistence/origin-guard.ts; engine methods cannot move): 3102 + 2; GBRA-130 composed with #6948 at the master 0e0cd27b merge: re-summed to the merged line count (3107); +3 #7159: the frontmatter where clause in the two keyword typesClause sites and its import (SQL in search/frontmatter-where.ts; engine methods cannot move); GBRA-156 retrieval wave (#7201) composed with #7194 at the merge: re-summed (3114 measured); +2 warm CLI socket (#7218): connectedDatabaseUrl getter (the URL the engine connected with, bound into the socket identity), composed over the GBRA-156 retrieval wave (3116 measured); +1 GBRA-133 PR 2b: transactionOn hands the outermost transaction its commit checks (page-state/transactions.ts withCommitChecks), composed with the warm CLI socket at the merge (3117 measured); +18 GBRA-156: the private readTransaction helper behind withScopedReadTransaction and jitOffRead answers a top-level read before its COMMIT settles (engine methods cannot move; 3135 measured); +1 GBRA-156: transactionOn and runUnsafe hold a per-search connection slot (import of search/connection-budget.ts; 3136 measured); +2 GBRA-156: transactionOn passes TransactionOptions.pipelineFirst to the pool begin as { first } (driver gate in vendor/postgres index.js begin); an engine method cannot move; composed with the warm CLI socket and GBRA-133 PR 2b at the master v0.60.344.0 merge (3119 measured); GBRA-151 #7183/#7187/#7188 re-summed at the v0.60.402.0 merge (3135 measured); +2 #7183 (GBRA-151) at the v0.60.412.0 merge: read-write-transactions import and transactionOn wraps its connection in readWriteTransactions (logic in postgres-engine/read-write-transactions.ts; engine methods cannot move): 3135 + 2; +7 GBRA-156: transactionOn opens readOnly transactions as BEGIN READ ONLY and a top-level read inherits its request signal, runUnsafe refuses an aborted request (logic in request-signal.ts; engine methods cannot move): 3137 + 7; +3 GBRA-156 keyword OR rarity: the import and the searchKeyword OR retry through rankOrFallbackByRarity (logic in search/keyword-or-rarity.ts; engine methods cannot move); +2 wave-b PR 4 merged with #7218 (both sides added imports); re-summed at the Wave B PR 4 merge with v0.60.452.0 (3155 measured); +2 GBRA-283 Wave A (#8460): the ENGINE_ROOT import and field (engine identity for per-engine caches; class fields cannot move); +7 GBRA-283 B1 review: the instance pool passes settings connection fields explicitly, initSchema reads the embedding identity again under the schema lock, and reconnect ends the superseded manager (engine methods cannot move); re-summed at the train R merge with v0.60.574.0: 3170 + 4; +4 #6318: withScopedReadTransaction runs a bounded preparation's read under its deadline (read-deadline.ts; reserved-connection envelope runDeadlineRead in postgres-engine/bounded-statement.ts) and adds statement_timeout to the transaction settings (engine methods cannot move): re-summed at the 0.60.583.0 merge: 3174 + 4; +2 TLS default: the pg-tls import and the instance pool builds its options through withPgTls and turns a no-TLS refusal into postgres_tls_required (logic in pg-tls.ts; engine methods cannot move): 3178 + 2
src/core/search/hybrid.ts 1562 ratchet +5 GBRA-81 item 5 per-stage search timings: HybridSearchOpts._timings (interface field) and the timer bindings and import in hybridSearch/hybridSearchCached (collector logic in search/stage-timings.ts); +2 W5 page-unit ranking cut: HybridSearchOpts.pageRankCut (logic in search/page-rank-cut.ts); +9 W3 confidence-gated reranking: HybridSearchOpts.rerankGate and the gate/lookup threading through rerankAndPin and sizeReturnPool (logic in search/hybrid/rerank-gate.ts, crag.ts gradePreRerank); re-summed at the W3 merge onto master v0.60.168.0 (1641 measured); +2 System One v1: HybridSearchOpts.decide (search/decide-stage.ts); refactor wave 1 split target (W4 hybrid, cut line); ceiling rationale: docs/architecture/key-files/core-search-1.md entry for src/core/search/hybrid.ts; +10 #5691: per-request embedding_query_prefix read once and threaded to the embeds and cache key; +5 green master wave (embedding opt-out): HybridSearchOpts._embeddingOptedOut (interface field) read once per request and threaded to the cache-lookup embed and the keyword-only gate; +24 P2 explain RRF attribution stamp in rrfFusionWeighted, backlink_count stamp, explain_target stage observations, explain/explainTarget opts; +14 fix wave 13 P4.8 #6400: lazyQueryEmbedDeadline (the query-embed deadline starts when the embed starts) and the _queryEmbedDeadline getter; +1 query facts arm: HybridSearchOpts.onQueryEmbedding (interface field; its one-line call is paid inside the ceiling) after the vector arms, so the facts arm reuses the query embedding without a second model call; re-summed at the master a08578ca5 merge (1591 master lines + 3 integration lines: the onQueryEmbedding hook); lowered to 1562: LHF wave 2 item 3 peeled cosineReScore into search/cosine-rescore.ts
src/core/types.ts 2357 ratchet +2 #8664 agent label: WriteAttributionView.agent_label (+ its doc line; the header, record and read live in persistence/attribution.ts and ops/attribution.ts) +3 train R: #7491 SearchOpts identifier-probe field and #7175 keywordPhrase option (interface fields); +2 GBRA-154 #7175: SearchOpts.keywordPhrase (opt-in phrase factor); +4 Wave B PR 1 (1c): Page.body_storage/body_bytes (interface fields; guard logic in page-body/page-shape.ts); +8 W3 confidence-gated reranking: HybridSearchMeta.rerank_gate (RerankGateMeta in search/hybrid/rerank-gate.ts); re-summed at the W3 merge onto master v0.60.168.0 (2279 measured); +7 fix wave 11 W6.2/W6.3 (#5989, #6132): SearchOpts.hnswIterativeScan and cjkKeyword, the keyword_candidates_incomplete stage and HybridSearchMeta.keyword_candidates (logic in engine-sql/cjk-search.ts); +5 System One v1 retrieval lane: SearchResult injection_p/injection_suspected, HybridSearchMeta answerability, egress_denied degraded reason; +4 System One v1: SearchResult rerank_score_kind/decide_evidence, HybridSearchMeta decide/rerank; +3 wave review: PAGE_SORT_SQL.updated_asc source_id + id tiebreakers (total order under federated scopes); +5 #4881: OntologyReadOpts extends PageReadScope doc; +3 #4857: PAGE_SORT_SQL.slug total-order tiebreaker comment; grown v0.46.26.0 megawave: EvidenceOpts, pacing/usage result fields, PageFilters growth (#4352 #4218); wave-k: +9 SalienceOpts/AnomaliesOpts source scope (#3867); +20 2026-09 adversarial batch 2: keyword_relaxed_carried stage + relaxed_dropped meta field docs; +16 2026-09 fix wave: SearchResult.keyword_relaxed field + full fusion-demotion doc; (#3867); +9: #4648 rerank_passthrough degraded stage + empty_result_set/malformed_shape reasons | merged v0.48.1.0 true-up; +master v0.48 merge true-up (reranker succession: no_key preflight, reranker_skipped stage, readiness); corrective release: shared internal PageReadScope and PageReadPolicy; ranker wave v0.48.4.0: HybridSearchMeta relational_rerank_pin / keyword_arm_confidence / metadata_boost_gate + SearchResult.relational_pinned (+48 over master's peel); +12: #4681 Page.updated_at_iso + keyset cursor precision docs; #5105: canonical page state shared module and transaction/snapshot delegates; #5105: public page revision fields; upstream integration: preserve independently reviewed master additions with branch facade extraction; #5252 wave: typed vector pool and readiness metadata; merged company ingestion/schema routing changes; retired provider integration removed; +1 write-path wave: 'created' effective-date source; +3 read-path wave: rerank_failed degraded stage + budget reason (closed vocabularies) and its recall classification note; +8 fw3-search: SearchOpts.source_boosts + keyword_arm_failed/title_arm_failed stages; +2 wave-2 W4: ChunkInput.embedding_input_hash (#5553); fix wave 3 x fix wave 2 (v0.60.5.0) composition: +2 at the merge; evidence delivery: optional SearchResult.delivered (+2; shape in search/evidence-delivery.ts); +2 eval wave N1-3: OntologyReadOpts.visibility (remote world-only ontology reads); +4 #5827: GraphNode.source_id and GraphPath from_source_id/to_source_id; +2 #5824: SearchOpts.vectorLegacyGuard rollback switch; +7 #5154: PageFilters.listColumnsOnly (interface field); +11 Foundations 1 F1b: WriteAttributionView + AttributedPageVersion (get_versions / get_write_attribution read shapes); +1 #5988: SearchResult.stale carries the held-file detail (interface field); +6 retrieval feedback: SearchResult feedback_boost/content_hash/relational_path_edges stamps and RelationalFanoutRow.path_edges (interface fields); +2 RelationalFanoutOpts.wide (interface field); 1 GBRA-49: temporal annotation field on link rows; 1 GBRA-49: temporal annotation field on link rows; 1 GBRA-49: temporal annotation field on link rows; +44 P7 ChainHopOpts/ChainHopEdge chain-hop contract types; P7 SearchResult.relational evidence + RelationalEvidence type + relational_plan meta + planner per-call overrides; +2 P7 SearchOpts.relationalChainSlots per-call override; +1 P7 ChainHopOpts.temporal (relationship-validity policy per hop); P3 x P7 (v0.60.60.0) composition: deltas re-summed at the merge; 1 GBRA-49: temporal annotation field on link rows; +44 P7 ChainHopOpts/ChainHopEdge chain-hop contract types; P7 SearchResult.relational evidence + RelationalEvidence type + relational_plan meta + planner per-call overrides; +2 P7 SearchOpts.relationalChainSlots per-call override; +1 P7 ChainHopOpts.temporal (relationship-validity policy per hop); +1 green master wave: embedding_disabled degraded reason (closed vocabulary); +2 #5951: Chunk.source_id (get_chunks names each row's source); 1 P2 SearchResult explain fields (rrf attribution, blend_norm_rrf, score_details, backlink_count) + RrfAttribution (interface fields); +11 #5575 (L1b): PageReadScope minTrust/suppressFlagged, SearchResult trust_tier/origin/unconfirmed, TimelineOpts eligibility (interface fields cannot move); +8 #4772 (W14 PR3): BrainHealth.entity_types_status field (interface fields cannot move); +1 C4: question page type row; +6 entity anchoring: optional SearchResult.entity_anchored row stamp (logic in search/entity-anchor.ts); +4 query facts arm: optional SearchResult.fact_row and superseded_claim row fields (logic in search/facts-arm.ts); +2 query facts arm: SearchResult result_type/fact_id/page_slug/follow_up (fact rows are never page-shaped); +9 Cat 40 Hard fix wave (#6271): SearchResult matched_alias/evidence_omitted/provenance, SearchOpts.rankQuery, keyword_count_unavailable stage and rerank_failed reasons; +12 fix wave 15 P2.1 (#6500): PageFilters.slugAfter (single-source slug keyset, interface field + doc) and the updated_desc tiebreaker comment (the keyset walk lives in facts/conversation-enumeration.ts); +2 wave 16 G5.1 (#6495): EngineConfig.keyword_statement_timeout_ms (interface field + doc); +2 GBRA-92: EngineConfig.pglite_shared_buffers (interface field + doc); +11 large pages (Wave A): Chunk/ChunkInput projected_byte_start/end interface fields (span logic in chunkers/recursive.ts, page-window/document.ts); +4 wave 18b lane 2 (#6495): SearchOpts.keywordTimeoutCache, the skipped_after_timeout degraded reason and the useCache doc line; integrated at wave 18b: master 2319 + 4; +2 LHF wave 2 item 3: retrieval meta cosine_rescore field (js_fallback disclosure; logic in search/cosine-rescore.ts); +14 GBRA-154 ranking wave (#4612 #7153 #7156 #7162 #7164): SearchResult collapsed/redirect_to/image_page/rank_prior/rank_score/fused_score stamps and HybridSearchMeta.collapse (interface fields; logic in search/collapse.ts, rank-priors.ts, score-contract.ts); GBRA-151 retrieval P0 x operability wave merge: both sides grew the file, re-summed (2343 measured); GBRA-156 retrieval wave (#7201) composed with #7194 at the merge: re-summed (2345 measured); GBRA-151 #7147 follow-up re-summed at the v0.60.338.0 merge (2347 measured); +2 GBRA-95 G7a follow-up: Page.withheld_source_path field; +2 #7600: SearchResult.quoted (interface field; logic in search/transcript-quote.ts); +1 #8377: ChronicleTimelineRow.trust_tier (interface field; chronicle reads quote rows by tier); re-summed at the train R merge with v0.60.574.0: 2354 + 1
-src/core/minions/queue.ts 2558 ratchet grown #4078 + megawave probe-liveness unlatch (#4310) + master private-queue lifecycle: recovery classifier + monotonic lease; security: durable queued authority and confined filesystem dispatch; master integration: grown #4078 + megawave probe-liveness unlatch (#4310) + master private-queue lifecycle: recovery classifier + monotonic lease; harness support: boundary wiring to shared grants, isolated setup, withdrawal and delegated guards; merged v0.49: durable authority and delegated admission both enforced; typed readiness, configuration-blocked lifecycle and bounded release; +9: prune archives completed dream synthesis keys; +18 Lane D: deferJob (a deferral returns a job to delayed with no attempt counted and no stacktrace growth); the ceiling had 17 lines of slack; +5 agent operator follow-up Item 5 merged over master: claim-time spend fence token and legacy_default stamp (logic in minions/spend-authorization.ts); +5 GBRA-130 A-30: claim excludeWatchdogSuspects option (signature, doc and the mark parameter), the releaseWatchdogClaim delegate, and the co-runner stall call site in handleStalled (logic in minions/watchdog-attribution.ts); +5 GBRA-158 #7239/#7240: failJob result parameter (the #7216 hunk), jsonbSafeJson import and its call in add (sanitizer in text-safe.ts); +2 #7242 lane reservation: claim excludeNames option (doc line and the bound array; policy in minions/lanes.ts); +1 #7241 submitter label import (job-insight.ts) + 1 queue-families type import; #7252 x #7249 merge: both sides grew the file, re-summed (2544 measured); +3 GBRA-133: the stall sweep deletes a drain-lease holder record whose lease lock is gone or expired; +1 GST-2260: retryOnStalePlan import for completeJob/failJob (helper in retry-matcher.ts); +1 persistence train: add refuses a child under a finished parent (#8909); +9 GST-2529: the claim builds its parameters once for the overdue-suspect drain (a marked job past the worker's wait stops a busy worker claiming and leads an idle one's order)
+src/core/minions/queue.ts 2574 ratchet grown #4078 + megawave probe-liveness unlatch (#4310) + master private-queue lifecycle: recovery classifier + monotonic lease; security: durable queued authority and confined filesystem dispatch; master integration: grown #4078 + megawave probe-liveness unlatch (#4310) + master private-queue lifecycle: recovery classifier + monotonic lease; harness support: boundary wiring to shared grants, isolated setup, withdrawal and delegated guards; merged v0.49: durable authority and delegated admission both enforced; typed readiness, configuration-blocked lifecycle and bounded release; +9: prune archives completed dream synthesis keys; +18 Lane D: deferJob (a deferral returns a job to delayed with no attempt counted and no stacktrace growth); the ceiling had 17 lines of slack; +5 agent operator follow-up Item 5 merged over master: claim-time spend fence token and legacy_default stamp (logic in minions/spend-authorization.ts); +5 GBRA-130 A-30: claim excludeWatchdogSuspects option (signature, doc and the mark parameter), the releaseWatchdogClaim delegate, and the co-runner stall call site in handleStalled (logic in minions/watchdog-attribution.ts); +5 GBRA-158 #7239/#7240: failJob result parameter (the #7216 hunk), jsonbSafeJson import and its call in add (sanitizer in text-safe.ts); +2 #7242 lane reservation: claim excludeNames option (doc line and the bound array; policy in minions/lanes.ts); +1 #7241 submitter label import (job-insight.ts) + 1 queue-families type import; #7252 x #7249 merge: both sides grew the file, re-summed (2544 measured); +3 GBRA-133: the stall sweep deletes a drain-lease holder record whose lease lock is gone or expired; +1 GST-2260: retryOnStalePlan import for completeJob/failJob (helper in retry-matcher.ts); +1 persistence train: add refuses a child under a finished parent (#8909); +9 GST-2529: the claim builds its parameters once for the overdue-suspect drain (a marked job past the worker's wait stops a busy worker claiming and leads an idle one's order); +16 GST-2529: hasOverdueWatchdogSuspect and overdueSuspectSql, the probe the worker runs every 5 s so a claim with no overdue marked job keeps the claim index order
src/commands/init.ts 1892 ratchet +2 fix wave 9 lane F #5253: leaving deferred setup also clears the DB-plane embedding_disabled row (both init branches); grown v0.46.x keyless capability notices; grown db-availability loop: initPostgresCore typed-failure extraction + --prefer-postgres ladder hook + pgvector rethrow fix; +1 at megawave/wave-k merge; +master-merge true-up; wave-k: merge; #3753 --force re-init engine-preservation guard; +1 wave-k merge-union brace (--prefer-postgres close); +10 ambient-writeback WP8: runWritebackNudge consent ask in both engine epilogues (logic lives in core/onboard/writeback-nudge.ts); +4: v0.48.2 reranker readiness at init + test seam; +1: v0.48.2 provider normalization; +3: v0.48.2 review fixes; master integration: +2: preserve typed PGLite busy errors in schema-only init; shared-skills integration on company-ingestion base; retired provider integration removed; retain stored-vector identity on re-init; -14: first-run bundle + quickstart registration helpers live in init-first-run.ts (writeback/skills asks folded into the bundle); +5 agent operator follow-up Item 2: --surface flag (value flag, parse, help line, module-level registration surface, type import) for the printed harness registration; +2 fix wave lane B3: both engine paths stamp a database init created as a fresh install for the behavior_changes notice (core/behavior-change-notice.ts); +4 GBRA-81 item 1: --search-mode value flag, its parse line, module-level choice, help line (picker logic lives in init-mode-picker.ts); +2 D8 funnel: a fresh init counts one clean_install when the opt-in funnel is on (core/funnel.ts); +1 GBRA-199: --surface help line names the default verbs listing; +7 GBRA-248 W0b: the implicit-keyless marker on both zero-key fallbacks, the deferred-hint gate on both engine paths and the kept-identity call in both (logic in commands/init-embedding-plane.ts)
src/commands/integrations.ts 1739 ratchet grown v0.46.26.0 megawave (#4039 connector compat) + 1 nosemgrep dataflow annotation (path-traversal audit); wave-k: exported heartbeatPath seam (#3900)
src/commands/import.ts 1507 ratchet new row: crossed the 1500 unlisted cap with GST-2462 (+7): the RunImportResult.privacy_label_notice interface field, the noticeRun option, the notice import and call, and its JSON and result fields (a static import: a dynamic one yields a macrotask that changes what a CLI import leaves in a checkout; the notice lives in core/search/ignored-privacy-labels.ts); next growth should peel the import summary into a sibling module
diff --git a/src/core/minions/queue.ts b/src/core/minions/queue.ts
index f104143801..6c03a26faf 100644
--- a/src/core/minions/queue.ts
+++ b/src/core/minions/queue.ts
@@ -187,6 +187,11 @@ async function coalesceReturn(
}
+/** SQL true for a job with the watchdog mark (`mark`) whose last requeue is at least `waitMs` milliseconds old (GST-2529). */
+function overdueSuspectSql(mark: string, waitMs: string): string {
+ return `(${markedJobSql(mark)} AND updated_at <= now() - (${waitMs}::double precision * interval '1 millisecond'))`;
+}
+
export class MinionQueue {
readonly maxSpawnDepth: number;
readonly maxAttachmentBytes: number;
@@ -1478,8 +1483,7 @@ export class MinionQueue {
const params: unknown[] = [lockToken, lockDurationMs, queue, registeredNames, HANDLER_DEFAULT_TIMEOUT_MS, HANDLER_DEFAULT_LOCK_DURATION_MS, ...legacyDefaultClaimParams()];
const param = (value: unknown) => `$${params.push(value)}`;
const mark = opts.excludeWatchdogSuspects || opts.suspectWaitMs !== undefined ? param(WATCHDOG_SUSPECT_MARK) : '';
- const overdue = opts.suspectWaitMs === undefined ? null
- : `(${markedJobSql(mark)} AND updated_at <= now() - (${param(opts.suspectWaitMs)}::double precision * interval '1 millisecond'))`;
+ const overdue = opts.suspectWaitMs === undefined ? null : overdueSuspectSql(mark, param(opts.suspectWaitMs));
const suspects = !opts.excludeWatchdogSuspects ? '' : ` AND ${unmarkedJobSql(mark)}${overdue ? ` AND NOT EXISTS (SELECT 1 FROM minion_jobs s WHERE s.queue = $3
AND s.status = 'waiting' AND s.submission_authority IS NOT NULL AND s.name = ANY($4) AND ${overdue})` : ''}`;
const names = excludeNames ? ` AND NOT (name = ANY(${param(excludeNames)}))` : '';
@@ -1513,6 +1517,18 @@ export class MinionQueue {
return rows.length > 0 ? rowToMinionJob(rows[0]) : null;
}
+ /** GST-2529: whether a marked job this worker could claim has waited `suspectWaitMs` since its requeue. The worker asks at most every
+ * few seconds and passes `suspectWaitMs` to `claim` only when one has, so a claim with no such job keeps the claim index's order. */
+ async hasOverdueWatchdogSuspect(queue: string, registeredNames: string[], suspectWaitMs: number): Promise<boolean> {
+ if (registeredNames.length === 0) return false;
+ const rows = await this.engine.executeRaw<{ overdue: boolean }>(
+ `SELECT EXISTS (SELECT 1 FROM minion_jobs WHERE queue = $1 AND status = 'waiting' AND submission_authority IS NOT NULL AND name = ANY($2)
+ AND ${overdueSuspectSql('$3', '$4')}) AS overdue`,
+ [queue, registeredNames, WATCHDOG_SUSPECT_MARK, suspectWaitMs],
+ );
+ return rows[0]?.overdue === true;
+ }
+
/** A co-runner of a watchdog drain back to waiting, marked, no attempt burned (token-fenced). */
releaseWatchdogClaim(id: number, lockToken: string, errorText: string): Promise<boolean> { return releaseWatchdogClaim(this.engine, id, lockToken, errorText); }
diff --git a/src/core/minions/types.ts b/src/core/minions/types.ts
index 89860b5d72..6c99b78cea 100644
--- a/src/core/minions/types.ts
+++ b/src/core/minions/types.ts
@@ -270,7 +270,8 @@ export interface MinionWorkerOpts {
longJobNames?: readonly string[];
/** #7242: slots kept free of long jobs when concurrency >= 2 (at least one slot still takes long jobs). Default 1; 0 disables. */
shortJobReservedSlots?: number;
- /** GST-2529: how long a watchdog-marked job waits before this worker stops claiming to drain and run it alone. Default 60000. */
+ /** GST-2529: how long a watchdog-marked job waits before this worker stops claiming to drain and run it alone. Default 60000.
+ * The worker looks for such a job at most every 5 s and otherwise claims in plain priority order. */
watchdogSuspectWaitMs?: number;
}
diff --git a/src/core/minions/worker.ts b/src/core/minions/worker.ts
index a9c060f3f1..895dd523af 100644
--- a/src/core/minions/worker.ts
+++ b/src/core/minions/worker.ts
@@ -64,6 +64,9 @@ import { isRetryableConnError } from '../retry-matcher.ts';
import { reconnectAfterConnectionError as reconnectEngineAfterConnError } from './reconnect.ts';
import { LocalConfigurationError, isLocalConfigurationError } from './configuration-error.ts';
+/** GST-2529: how often the worker asks whether a watchdog-marked job is past its wait (never less often than the wait itself). */
+const SUSPECT_PROBE_EVERY_MS = 5_000;
+
/**
* Abort reasons that signal infrastructure failure (PgBouncer outage,
* connection drop, lock reclaimed by another worker) — NOT a job
@@ -206,6 +209,9 @@ export class MinionWorker extends EventEmitter {
private inFlight = new Map<number, InFlightJob>();
private executions = new Map<string, OwnedExecution>();
private pendingClaim: { lockToken: string; promise: Promise<MinionJob | null> } | null = null;
+ /** GST-2529: when the next overdue watchdog-suspect probe is due, and whether the last one found such a job. */
+ private suspectProbeDueAt = 0;
+ private suspectOverdue = false;
private _configurationError: LocalConfigurationError | null = null;
private configurationDeadline = 0;
private configurationDrainFinished = false;
@@ -811,6 +817,12 @@ export class MinionWorker extends EventEmitter {
}
}
+ if (Date.now() >= this.suspectProbeDueAt) {
+ this.suspectProbeDueAt = Date.now() + Math.min(this.opts.watchdogSuspectWaitMs, SUSPECT_PROBE_EVERY_MS);
+ this.suspectOverdue = await this.queue.hasOverdueWatchdogSuspect(this.opts.queue, this.registeredNames, this.opts.watchdogSuspectWaitMs)
+ .catch(() => true);
+ }
+
if (!this.running) break;
// Claim jobs up to concurrency limit — unless the system-wide pause
@@ -841,7 +853,7 @@ export class MinionWorker extends EventEmitter {
this.opts.lockDuration,
this.opts.queue,
this.registeredNames,
- { excludeWatchdogSuspects: this.inFlight.size > 0, suspectWaitMs: this.opts.watchdogSuspectWaitMs, excludeNames: claimExclusions(this.opts, [...this.inFlight.values()].map(f => f.job.name)) },
+ { excludeWatchdogSuspects: this.inFlight.size > 0, suspectWaitMs: this.suspectOverdue ? this.opts.watchdogSuspectWaitMs : undefined, excludeNames: claimExclusions(this.opts, [...this.inFlight.values()].map(f => f.job.name)) },
).then(claimed => {
if (claimed) {
this.executions.set(lockToken, { job: claimed, lockToken, stopped: true });
diff --git a/test/fixtures/goldens/exports/types.json b/test/fixtures/goldens/exports/types.json
index 6040a1b39c..8d2e026d9b 100644
--- a/test/fixtures/goldens/exports/types.json
+++ b/test/fixtures/goldens/exports/types.json
@@ -6489,6 +6489,7 @@
"handleTimeouts: () => Promise<MinionJob[]>",
"handleWaitingTTL: (opts?: { maxPerTick?: number; }) => Promise<{ cancelled: number; by_name: Record<string, number>; }>",
"handleWallClockTimeouts: (lockDurationMs: number) => Promise<MinionJob[]>",
+ "hasOverdueWatchdogSuspect: (queue: string, registeredNames: string[], suspectWaitMs: number) => Promise<boolean>",
"listAttachments: (jobId: number) => Promise<Attachment[]>",
"makeThrottledLeaseRenewer: (queueName: string, ownerToken: string, onRenewed?: () => Promise<void> | void, throttleMs?: number) => () => Promise<void>",
"pauseJob: (id: number) => Promise<MinionJob | null>",
@@ -6581,6 +6582,8 @@
"private releaseClaimForPause",
"private running",
"private shutdownAbort",
+ "private suspectOverdue",
+ "private suspectProbeDueAt",
"private wakeDrain",
"private workerId",
"rawListeners: <E extends string | symbol>(eventName: string | symbol) => ((...args: any[]) => void)[]",
diff --git a/test/worker-configuration-release.test.ts b/test/worker-configuration-release.test.ts
index c3a8f3347c..de596e06db 100644
--- a/test/worker-configuration-release.test.ts
+++ b/test/worker-configuration-release.test.ts
@@ -118,6 +118,7 @@ describe('worker configuration admission and actual execution', () => {
ensureSchema: async () => { schemaEntered.resolve(); await schemaReady.promise; events.push('schema'); },
promoteDelayed: async () => { events.push('promote'); },
claim: async () => { events.push('claim'); worker.stop(); return null; },
+ hasOverdueWatchdogSuspect: async () => false,
};
worker.register('fixture', async () => {});
worker.on('ready', () => { events.push('ready'); });
@@ -145,6 +146,7 @@ describe('worker configuration admission and actual execution', () => {
ensureSchema: async () => { events.push('schema'); },
promoteDelayed: async () => { events.push('promote'); },
claim: async () => { events.push('claim'); return null; },
+ hasOverdueWatchdogSuspect: async () => false,
};
worker.register('fixture', async () => {});
worker.on('ready', () => { throw new Error('fixture status publication failed'); });
@@ -210,6 +212,7 @@ describe('worker configuration admission and actual execution', () => {
ensureSchema: async () => { events.push('schema'); },
promoteDelayed: async () => { events.push('promote'); },
claim: async () => { events.push('claim'); worker.stop(); return null; },
+ hasOverdueWatchdogSuspect: async () => false,
};
worker.register('fixture', async () => {});
await worker.start();
diff --git a/test/worker-conn-resilience-1720.test.ts b/test/worker-conn-resilience-1720.test.ts
index 9053baba32..ae02c75899 100644
--- a/test/worker-conn-resilience-1720.test.ts
+++ b/test/worker-conn-resilience-1720.test.ts
@@ -80,6 +80,7 @@ describe('stall-detection interval reconnect (#1720 gap 1)', () => {
setQueue(worker, {
ensureSchema: async () => {},
promoteDelayed: async () => [],
+ hasOverdueWatchdogSuspect: async () => false,
claim: async () => {
claims += 1;
// Stop once the first stall tick fully ran (or bail out after a bound
@@ -149,6 +150,7 @@ describe('failJob failure-recording resilience (#1720 gap 2)', () => {
setQueue(worker, {
ensureSchema: async () => {},
promoteDelayed: async () => [],
+ hasOverdueWatchdogSuspect: async () => false,
handleStalled: async () => ({ requeued: [], dead: [] }),
handleTimeouts: async () => [],
handleWallClockTimeouts: async () => [],
diff --git a/test/worker-promote-reconnect.test.ts b/test/worker-promote-reconnect.test.ts
index aca361c3cf..00672c58ed 100644
--- a/test/worker-promote-reconnect.test.ts
+++ b/test/worker-promote-reconnect.test.ts
@@ -38,11 +38,13 @@ describe('MinionWorker connection recovery', () => {
ensureSchema: () => Promise<void>;
promoteDelayed: () => Promise<unknown[]>;
claim: () => Promise<null>;
+ hasOverdueWatchdogSuspect: () => Promise<boolean>;
handleStalled: () => Promise<{ requeued: unknown[]; dead: unknown[] }>;
handleTimeouts: () => Promise<unknown[]>;
handleWallClockTimeouts: () => Promise<unknown[]>;
} }).queue = {
ensureSchema: async () => {},
+ hasOverdueWatchdogSuspect: async () => false,
promoteDelayed: async () => {
if (!promoted) {
promoted = true;