Skip to content

Garry's Opinionated OpenClaw/Hermes Agent Brain

TypeScript

fix(facts): keep committed partial counts when a conversation-facts backfill is interrupted (#7068)

garrytan63dd3464ad79
diff --git a/scripts/module-size-limits.tsv b/scripts/module-size-limits.tsv
index b2e11fd8fb..25e08fb0f1 100644
--- a/scripts/module-size-limits.tsv
+++ b/scripts/module-size-limits.tsv
@@ -7,7 +7,7 @@ src/cli/main.ts 2959 ratchet +3 GBRA-156: thin-client page-cap line on stderr in
src/commands/autopilot.ts 1662 ratchet refactor wave 1 split target (W4 autopilot, cut line); ceiling rationale: docs/architecture/key-files/commands-1.md entry for src/commands/autopilot.ts; +2 fix wave 5: nosemgrep justifications on the per-brain job launchctl/systemctl calls (#5195); +2 #5813: boot reconcile names attempted and running versions in log + audit; +2 #5855: Bun floor gate before the swap and the swap-refusal hold (logic in core/bun-floor.ts); +3 agent operator wave E9: --dry-run and the persistent_install consent gate before anything is written (logic in autopilot/install-consent.ts); +8 GBRA-130 C-6/A-10: the --retry-global mode row, help line and --phase value flag, and --status reading the brain-wide maintenance verdict into its JSON, text and exit code (logic in core/minions/global-maintenance-stall.ts and autopilot-retry-global.ts); +12 GBRA-133 #7003: --status grades from the daemon's heartbeat sidecar (two report fields and their doc line, the classifier's daemon input and threshold branch, the import and the showStatus read; logic in core/autopilot-heartbeat.ts)
src/commands/doctor.ts 700 ratchet +2 Wave B decision #10: Check.skipped_segmented field (logic in doctor/check-fix.ts withSkippedSegmented); +19 #6696 doctor tiers: Check.elapsed_ms and DoctorReport tier/complete/unknown_sections/elapsed_ms/budget_ms fields, buildChecks/runDoctor deadline params (logic in doctor/budget.ts); +1 agent operator gate fixes: --probe consent gate (logic in doctor/probe-consent.ts); refactor wave 1 split target (W4 doctor); ceiling rationale: docs/architecture/key-files/commands-2.md entry for src/commands/doctor.ts; +4 #7142: DoctorReport.last_known_sections field and the buildChecks last-known/refresh scope params (logic in doctor/last-known.ts); +8 #7276: DoctorReport.unknown_count field and the unknown category/count lines of the text headline (scoring logic in doctor/budget.ts scoredChecks)
src/commands/embed.ts 2306 ratchet +2 agent operator wave D2: --json failure envelope via writeCliError; lowered v0.46.26.0: embedBatchWithBackoff + retry cluster peeled to core/embed-retry.ts (core/commands layering fix); grown wave-g (#4530): per-model input-limit wiring; wave-k #3622 reimpl: --stale zero-progress failure quarantine; +29 #4668 (#4530): oversize-heal call sites in both --stale drains — kept low by peeling the healed-row remap to core/embed-oversize-heal.ts (shared with embed-stale.ts); +35 #4647 (refs #4599): per-tick timeout+abort on the single-flight lock heartbeat refresh (interval deliberately NOT unref'd); +122 refs #4599: stall-watchdog race around the drain + shared idempotent flushAndRelease + CLI stall exit mapping (watchdog machinery itself lives in core/embed-stall.ts); +12: adversarial-review hardening (healed_splits progress key + heal-tick comment); #5105: guarded derived projection preparation and read seals; #5105: conditional embedding installer wiring; archived eligibility: active-only dispatch with honest stranded-work reporting; explicit archived admission: live-source snapshots and honest refused-installation failures; +2 fw3: embed --stale --images dispatch (sweep lives in core/embed-stale-images.ts); +5 #5289: dry run reports restamp-only chunks apart from chunks to embed (count lives in core/embedding-invalidation.ts); +6 #4616: keep a partially refused batch in embedPageTexts and name the recovery command in failure samples (guard lives in core/ai/embedding-guard.ts); +5 #5226: embed summary counts distinct pages; +3 #5804: null projection snapshot counted as a failure with one batch summary in embed --stale; +18 wave 8 lane D1: #5527 mixed-space warning subtracts current-model chunks (+4), #5885 takes option/result fields and the stale-drain takes call (pass lives in core/embed-takes.ts); +9 Foundations 1 F4d: time-budget stop result fields, stdout verdict and stop recording in embed --stale (logic in core/embed-budget-stop.ts); +8 green-master lane A: chunkless install re-reads through retryProjectionConflict instead of recording an unavailable page (retry lives in core/page-state/projections.ts, failure line in core/agent-output.ts); +8 fix wave 13 P1.18: runEmbed consent parameter, cost-cap stop wiring and result fields, --slugs parsing to the next flag (cap, verdict and resume logic live in core/embed-cost-cap.ts); +28 large pages A9/T3/T16: --held listing and per-page cap metering on every embed path (logic in core/page-embedding-budget.ts meterPageBatch/spendGrant/heldEmbedArgv), approvedPageCapUsd from --max-usd on --slugs; +1 #6722: an explicit slug embed skips an embed_skip page (it keeps keyword chunks, never vectors); merge v0.60.240.0: large pages and master growth compose at 2265; +39 large pages Wave B lane 1e: segmented pages route to the keyset sweep on the slug, --all and --stale paths, with a dry-run count and provenance stamp (logic in core/page-body/segmented-embedding.ts); +2 agent operator wave D2: --json failure envelope via writeCliError
-src/commands/extract-conversation-facts.ts 1949 ratchet grown v0.46.x per-provider gateway-unavailable copy; grown wave-g (#4482): expected budget/deadline caps classified apart from error halts; wave-k: save-time resolution caller (#4052/#3729 — supersedes wave-g's per-row resolveEntitySlug mapper); +1: key-aware llm-fallback default (#3813); +1: pricingOverrides wiring into the owned BudgetTracker (#4633 adoption); +3: #4869 snapshot-mismatch counts as pages_failed; merged reliability/shared-skills source integration; +19: additive skip reasons and model-free dry-run (#5364); +21 #5757: managed extraction buffers a page's rows and replaces the prior batch in one coordinator transaction (replacement helper lives in core/persistence/derived-facts.ts); +18 #5823: result diagnostics, strict explicit-cap parsing and CLI recovery/help; default-cap decision lives in facts/conversation-budget.ts; +18 fix wave 7 (#5704, #5448): --json envelope on stdout (parser flag, help line, summary block gated behind the human-output branch); +2 fix wave 10 (#6037, issue #6033): explicit-slug page failures are contained per page through one bounded failure logger (recordPageFailure) shared with the pooled path; +4 fix wave 10 (#5120): facts.default_visibility resolved once per run (resolveDefaultVisibility) and applied to extracted rows only; 2 P2 segmentObservationDate + segmentValidFrom helpers (date-grounding observation time per segment; extractor event date wins over segment start); +4: ExtractConversationFactsResult.failed_pages (interface field) and the help text exit-status paragraph; partial-success logic lives in core/facts/conversation-run-status.ts; re-summed at the master 6ccf30c3e merge (2179 master lines + 20 integration lines over base); re-summed at the #6020 merge on actual lines (as rehearsed on capy/mpw-integration-p2); re-summed at the wave 10 merge on actual lines (+1 recordPageFailure fills failed_pages); lowered at the wave-9 follow-ups merge: the three zero-result literals became emptyExtractConversationFactsResult(); lowered to 2075 at wave 15 P2.5 (#6267): segment tunables, splitIntoSegments and renderSegmentForExtraction moved to core/facts/conversation-windows.ts, the message/segment types to core/facts/conversation-types.ts (re-exported here); +12 large pages T2: the oversized-sidecar snapshot and the durable size-skip receipt (skippedForSize; the cap and the bounded sidecar read live in core/conversation-parser/body.ts, the receipt in core/extraction-watermark.ts); merge v0.60.240.0: large pages and master growth compose at 2087; -6 wave 18 lane 3 (#6731 #6764): the two extractor call sites share one input object, inference lives in core/facts/conversation-subject-infer.ts; Wave B PR 2 (kept at the master 4e29d0904 merge): +7 large pages PR 2 lane 2e2: a segmented conversation over the window cap (snapshot.window, the tail_window receipt, the window fallback date; the window read lives in core/page-body/segment-conversation.ts); +9 lane 2e2 follow-up: the lossless window rule (a page over the window with conversation facts keeps them and is skipped, the receipt names the rule; the count lives in core/facts/conversation-publication.ts); integrated at the master 4e29d0904 merge: 2097 measured; lowered to 2044 by #7013: a page's rows publish once through publishConversationFacts (core/facts/conversation-publication.ts), so the delete-first cleanup, the per-segment inserts and the row-number peek left; +1 GBRA-159: facts from a private page are private (tightenToPage per page; helper in core/facts/visibility.ts); +3 #7438: ExtractConversationFactsResult.facts_skipped_purged (interface field) and one notePurgedSkips call (the count and log live in core/facts/conversation-publication.ts); grown v0.46.x per-provider gateway-unavailable copy; re-summed at the Wave B PR 4 merge with v0.60.452.0 (1949 measured)
+src/commands/extract-conversation-facts.ts 1955 ratchet grown v0.46.x per-provider gateway-unavailable copy; grown wave-g (#4482): expected budget/deadline caps classified apart from error halts; wave-k: save-time resolution caller (#4052/#3729 — supersedes wave-g's per-row resolveEntitySlug mapper); +1: key-aware llm-fallback default (#3813); +1: pricingOverrides wiring into the owned BudgetTracker (#4633 adoption); +3: #4869 snapshot-mismatch counts as pages_failed; merged reliability/shared-skills source integration; +19: additive skip reasons and model-free dry-run (#5364); +21 #5757: managed extraction buffers a page's rows and replaces the prior batch in one coordinator transaction (replacement helper lives in core/persistence/derived-facts.ts); +18 #5823: result diagnostics, strict explicit-cap parsing and CLI recovery/help; default-cap decision lives in facts/conversation-budget.ts; +18 fix wave 7 (#5704, #5448): --json envelope on stdout (parser flag, help line, summary block gated behind the human-output branch); +2 fix wave 10 (#6037, issue #6033): explicit-slug page failures are contained per page through one bounded failure logger (recordPageFailure) shared with the pooled path; +4 fix wave 10 (#5120): facts.default_visibility resolved once per run (resolveDefaultVisibility) and applied to extracted rows only; 2 P2 segmentObservationDate + segmentValidFrom helpers (date-grounding observation time per segment; extractor event date wins over segment start); +4: ExtractConversationFactsResult.failed_pages (interface field) and the help text exit-status paragraph; partial-success logic lives in core/facts/conversation-run-status.ts; re-summed at the master 6ccf30c3e merge (2179 master lines + 20 integration lines over base); re-summed at the #6020 merge on actual lines (as rehearsed on capy/mpw-integration-p2); re-summed at the wave 10 merge on actual lines (+1 recordPageFailure fills failed_pages); lowered at the wave-9 follow-ups merge: the three zero-result literals became emptyExtractConversationFactsResult(); lowered to 2075 at wave 15 P2.5 (#6267): segment tunables, splitIntoSegments and renderSegmentForExtraction moved to core/facts/conversation-windows.ts, the message/segment types to core/facts/conversation-types.ts (re-exported here); +12 large pages T2: the oversized-sidecar snapshot and the durable size-skip receipt (skippedForSize; the cap and the bounded sidecar read live in core/conversation-parser/body.ts, the receipt in core/extraction-watermark.ts); merge v0.60.240.0: large pages and master growth compose at 2087; -6 wave 18 lane 3 (#6731 #6764): the two extractor call sites share one input object, inference lives in core/facts/conversation-subject-infer.ts; Wave B PR 2 (kept at the master 4e29d0904 merge): +7 large pages PR 2 lane 2e2: a segmented conversation over the window cap (snapshot.window, the tail_window receipt, the window fallback date; the window read lives in core/page-body/segment-conversation.ts); +9 lane 2e2 follow-up: the lossless window rule (a page over the window with conversation facts keeps them and is skipped, the receipt names the rule; the count lives in core/facts/conversation-publication.ts); integrated at the master 4e29d0904 merge: 2097 measured; lowered to 2044 by #7013: a page's rows publish once through publishConversationFacts (core/facts/conversation-publication.ts), so the delete-first cleanup, the per-segment inserts and the row-number peek left; +1 GBRA-159: facts from a private page are private (tightenToPage per page; helper in core/facts/visibility.ts); +3 #7438: ExtractConversationFactsResult.facts_skipped_purged (interface field) and one notePurgedSkips call (the count and log live in core/facts/conversation-publication.ts); grown v0.46.x per-provider gateway-unavailable copy; re-summed at the Wave B PR 4 merge with v0.60.452.0 (1949 measured); +6 #7068: ExtractConversationFactsCoreOpts.onInterrupted (interface field and its doc) and its one call before the rethrow (the cycle keeps the committed counters in core/cycle/conversation-facts-backfill.ts)
src/commands/extract.ts 2700 ratchet +8 GBRA-130 A-1: extract --stale --retry-held flag, help lines and the managed result failure fields (logic in core/persistence/links-maintenance.ts and links-extraction-holds.ts); grown v0.46.26.0 megawave: FS-walk stamp snapshot + legacy timeline repair wiring (#3957 D4) + PR#4486 rebase onto current master; grown wave-g (#4542): --from-meetings zero-match warning + default-pass note; +29: #3478 federated-only gate on the cross-source 'default' link fallback; +44: #3908 cross_source opt-in — counted drops + deterministic cross-source pick; +36: #4611 configured-sources.default fallback (resolveLinkFallbackDefault); +14: #4890 extractStaleFromDB quiet opt + extractForSlugs jsonMode batch_error stderr fallback; +5: #4855 stroke-letter fold in link extraction; +5: #4793 by-mention scan honours link_resolution.cross_source; +18: wave review — ExtractOpts.quiet threaded through the fs-walk extractors (stdout silence decoupled from jsonMode); +77: shared buildSlugPathIndex — every per-slug extractor resolves the real on-disk path, and the sync hooks report which slugs they actually read; +10 #4999: includeFrontmatter default-resolved inside extractLinksForSlugs + extractStaleFromDB (resolveIncludeFrontmatter), --stale flag absent -> configured knob; +20: pre-landing review of #4967 — resolveSlugRelPath: index miss falls back to the legacy slug+'.md' path so admitted _-prefixed / include_hidden pages keep their edges; T2: source-qualified inference and atomic origin-scoped derived links; standard DB extraction fences captured endpoint revisions; shared authoritative attendance/source reconciliation moved to the core module; +7 A2 preview-bound attendance repair import, help and authority dispatch; bounded incomplete-attendance diagnostics and consistent basename resolution; fail closed before graph mutation when active ontology is unavailable; +16 source-selected ontology cache, type-scoped preflight and non-starving failed stale drain; +7 write-path wave: per-slug extractors replace page links + retract removed timeline rows + rename-alias resolution; +5 fw3 derived-sync: timeline --prune-orphans CLI (#4649) and full-walk page link replacement; +13 fw3 A16/A15: sync link hook reads scoped page metadata and reports per-page extraction errors; +1 fw3 managed-lifecycle: extract --stale dispatches managed brains to the coordinator-safe path; +4 fw3 integration: managed brains refuse the direct timeline orphan prune; +26 #5609: managed extract --stale publishes links, canonical timeline and watermark per page in one coordinated transaction; fix wave 3 x fix wave 2 (v0.60.5.0) composition: +4 at the merge; +6 fix wave 3 lane B: a managed brain routes the extract phase file walk through extractManagedStaleLinks instead of the refused raw timeline batch, keeping the extract.links_fs progress phase; +6 #5761: the unmanaged extract --stale sweep marks unresolved-attendance skips attendance-blocked; +2 eval-wave N9-3/N12-7: a page-type-bound pack attended rule defers to canonical attendance on the filesystem path (ownsAttendanceInference); lowered to 2528 #5904: the DB timeline walk moved to commands/extract-timeline-db.ts (coordinated publication on managed brains, honest refusal + non-zero exit); +13 entity mention index: extract --stale runs the mention pass after link work (half the budget reserved while mention-due pages exist), dry-run and JSON report mention counts, and the help documents the pass and `extract mentions --explain` (logic in core/mentions/); cat40 entity wave merge with v0.60.61.0 (2541 measured); +6 fix wave 10 lane L1 #5556: quarantined pages skipped by the file walks, slug lists and DB walks (slug lookup in quarantine.ts quarantinedSlugs); +3 #5556: the abort check stays its own statement in each quarantine-aware scan loop (cycle-abort coverage pin); 2 P5 wanted pages: both DB extract paths record unresolved authored refs with each page's links; +5 P5 line grammar: stated relation types on the filesystem link scan; +3 PGLite stale link drains refresh planner statistics before and during the drain (logic in planner-stats.ts plannerStatsForLinkDrain); +10 fix wave 12 W2.1 S4 (#6228): extract --stale flushes and stamps the pages a batch finished before an aborting page; +12 P5 wanted pages: both DB extract paths record unresolved authored refs with each page's links; +9 Q2 parser gaps: both DB extract paths read the line-grammar settings once, publish through replaceDerivedLinksUnlessSettingsChanged and report pages a mid-run settings change left stale; +25 GBRA-69: extract --source db reads snapshots and publishes derived links BATCH_SIZE pages at a time (readPageSnapshotsBatch in core/page-snapshot-batch.ts, replay in derived-links.ts replaceDerivedLinksBatchOrReplay); +2 GBRA-75 w7: the --source db links walk refreshes planner statistics (logic in planner-stats.ts plannerStatsForLinkDrain); +10 fix wave 13 P2.16 (#6272): both unmanaged DB link drains defer and count a page a concurrent writer moved (skipped_concurrent_write) instead of aborting the run (classifier and skip line in core/derived-links.ts); +3 fix wave 13 P2.18 (S4): extract all --source db stamps the pages both phases completed at one revision, in a finally (stampCompletedIntersection), instead of every links-phase page; +1 fix wave 13 P2.15 (#6273): --from-meetings exits non-zero while managed entity pages are still pending; +7 fix wave 13 PR2: extractFromDB split out of runExtract so runExtract stays under its function-size baseline; Wave B PR 2 (kept at the master 4e29d0904 merge): +3 PR 2 fix lane J1: extract --stale reads a segmented page block by block (extractSegmentedPageLinks, segmentedTimelineText) now that the stale list includes it; integrated at the master 4e29d0904 merge: 2635 measured; +23 fix wave 20 #6954: the fs walks and the slug-scoped extract collect lost link/timeline writes, stamp only the pages whose writes landed and throw extract_write_lost (helpers in core/extract-watermark.ts); fix wave 20 integration: lane 1 (#6808, +11) and lane 2 (#6954, +23) compose at 2677 measured; +16 #7394: --type/--since/--infer-dates on the file walk refuse with the --source db command beside the other --source db refusals, and the help says so; +5 GBRA-161 PR 5 (T-2, L-6): timeline retraction and the mention rebuild sweep are bound to the revision they read (PageRevisionConflictError import, the scanned-revision read and its FOR SHARE check); +2 GBRA-133: ExtractOpts.timeBudgetMs, forwarded to the managed stale-page pass so the cycle bounds a pass with no slug list
src/commands/jobs.ts 687 ratchet refactor wave 1 split target (W4 jobs, cut line); ceiling rationale: docs/architecture/key-files/commands-4.md entry for src/commands/jobs.ts; help lines: authorize-legacy --select (#5157 T3) and jobs cancel --select (DX-T2); submit --queue-only + PGLite work drain note (agent-first operator wave E5); +2 spend authorization: jobs submit --yes/--max-usd and jobs list --group help lines; +4 fix wave 9: supervisor stop help documents the 45 s wait, --queue and the verified-drain exit codes; +1 #7142: doctor-refresh handler registration (logic in doctor/last-known.ts); +20 #7242 jobs wait: subcommand row, import, usage lines and its --help text with the exit codes (logic in jobs/wait.ts)
src/commands/serve-http.ts 941 ratchet refactor wave 1 W4 split into serve-http-<area>.ts modules; ceiling rationale: docs/architecture/key-files/commands-5.md entry for src/commands/serve-http.ts; +2 agent operator wave A6: ServeHttpContext.noticeLedger (import + field; logic in core/notice-ledger.ts); +2 agent contract F1/F2: ServeHttpContext.readinessCache (probed readiness for initialize tails + gbrain://capabilities); +5 Lane D facts drain: serve --http starts the drain scheduler and stops it before the engine disconnect (logic in core/facts/drain-scheduler.ts); +2 agent-operator follow-up Item 3: ServeHttpOptions.adoptServer (status-only recovery hands its bound listener to listenOrRefuse; logic in serve-http-status.ts); +5 #6317: managed-sync movement watch start/stop (watch in persistence/sync-movement.ts); +1 #6589: imports loadServeConfigs (operations see the DB-plane config read once at startup; logic in mcp/serve-config.ts); +5 #6858 ask 6: on-commit HEAD watcher start/stop (watcher in core/serve-head-watcher.ts); +4 GBRA-199: serve --http --advertise parse, help line and pass-through (logic in mcp/surface.ts parseAdvertiseFlag)
diff --git a/src/commands/extract-conversation-facts.ts b/src/commands/extract-conversation-facts.ts
index e4d3c6ccbc..e2e96a2192 100644
--- a/src/commands/extract-conversation-facts.ts
+++ b/src/commands/extract-conversation-facts.ts
@@ -293,6 +293,11 @@ export interface ExtractConversationFactsCoreOpts {
* unless dryRun; bypasses the fresh-outcome skip.
*/
grammar?: string | null;
+ /**
+ * #7068: given a copy of the run's counters when the run ends in a thrown error, taken after the managed flush published
+ * the pages that finished, so a caller can report work that committed. The error is still thrown.
+ */
+ onInterrupted?: (partial: ExtractConversationFactsResult) => void;
}
export { extractExitCode, extractRunStatus };
@@ -1396,6 +1401,7 @@ export async function runExtractConversationFactsCore(
// surface. NOT a thrown failure.
return result;
}
+ opts.onInterrupted?.(structuredClone(result));
throw err;
}
diff --git a/src/core/cycle/conversation-facts-backfill.ts b/src/core/cycle/conversation-facts-backfill.ts
index 30b8e64759..168b98a69e 100644
--- a/src/core/cycle/conversation-facts-backfill.ts
+++ b/src/core/cycle/conversation-facts-backfill.ts
@@ -285,6 +285,8 @@ export async function runPhaseConversationFactsBackfill(
const onOuterAbort = () => controller.abort();
opts.signal?.addEventListener('abort', onOuterAbort, { once: true });
+ let interrupted: ExtractConversationFactsResult | null = null;
+ const committed = () => interrupted ?? zeroResult();
try {
const result = await withBudgetTracker(tracker, () =>
runExtractConversationFactsCore(engine, {
@@ -297,6 +299,7 @@ export async function runPhaseConversationFactsBackfill(
// per-source worker count. Default 1 — opt-in concurrency
// for cycle paths.
workers: cfg.workers,
+ onInterrupted: partial => { interrupted = partial; },
}, controller.signal),
);
perSourceResults[src.id] = result;
@@ -311,7 +314,7 @@ export async function runPhaseConversationFactsBackfill(
// continue with the next source.
sourcesWalltimeExhausted++;
perSourceResults[src.id] = {
- ...zeroResult(),
+ ...committed(),
walltime_exhausted: true,
error: 'walltime_exhausted',
};
@@ -319,7 +322,7 @@ export async function runPhaseConversationFactsBackfill(
// Escaped the core's own catch — same per-source posture.
sourcesBudgetExhausted++;
perSourceResults[src.id] = {
- ...zeroResult(),
+ ...committed(),
budget_exhausted: true,
budget_reason: err.reason,
budget_model: err.modelId,
@@ -338,8 +341,8 @@ export async function runPhaseConversationFactsBackfill(
} else {
// Per-source failure: record + continue with next source.
perSourceResults[src.id] = {
- ...zeroResult(),
- pages_failed: 1,
+ ...committed(),
+ pages_failed: committed().pages_failed + 1,
error: (err as Error).message,
};
}
diff --git a/test/cycle-conversation-facts-backfill.test.ts b/test/cycle-conversation-facts-backfill.test.ts
index 76dc213400..6d77b85e49 100644
--- a/test/cycle-conversation-facts-backfill.test.ts
+++ b/test/cycle-conversation-facts-backfill.test.ts
@@ -34,6 +34,13 @@ let chatDelayMs = 0;
let chatOutputTokens = 50;
let chatCalls = 0;
let chatCallsBySource: string[] = [];
+/** #7068: the chat call (1-based) that waits until its signal aborts, and a hook run as it starts waiting. */
+let stallCall: { n: number; onStall?: () => void } | null = null;
+
+function untilAborted(signal: AbortSignal | undefined): Promise<void> {
+ if (!signal || signal.aborted) return Promise.resolve();
+ return new Promise((resolve) => signal.addEventListener('abort', () => resolve(), { once: true }));
+}
const CONVO_BODY = [
'**Alice Example** (2024-03-15 9:00 AM): I just signed the offer letter for Acme Corp.',
@@ -71,6 +78,10 @@ beforeAll(async () => {
__setChatTransportForTests(async (opts): Promise<ChatResult> => {
chatCalls++;
chatCallsBySource.push(String(opts.messages[0]?.content ?? '').slice(0, 40));
+ if (stallCall?.n === chatCalls) {
+ stallCall.onStall?.();
+ await untilAborted(opts.abortSignal);
+ }
if (chatDelayMs > 0) await new Promise((r) => setTimeout(r, chatDelayMs));
if (opts.abortSignal?.aborted) {
const err = new Error('The operation was aborted.');
@@ -125,6 +136,7 @@ beforeEach(async () => {
chatOutputTokens = 50;
chatCalls = 0;
chatCallsBySource = [];
+ stallCall = null;
await engine.executeRaw(`DELETE FROM facts`);
await engine.executeRaw(`DELETE FROM op_checkpoints WHERE op = 'extract-conversation-facts'`);
await engine.executeRaw(`DELETE FROM extract_rollup_7d`);
@@ -218,3 +230,52 @@ describe('runPhaseConversationFactsBackfill per-source caps (#3627)', () => {
}
}, 120000);
});
+
+describe('runPhaseConversationFactsBackfill keeps committed work of an interrupted source (#7068)', () => {
+ async function seedTwoPageSource(id: string): Promise<void> {
+ await seedSource(id);
+ await engine.putPage(`conversations/${id}-later`, {
+ type: 'conversation', title: `Later chat in ${id}`, compiled_truth: CONVO_BODY, timeline: '', frontmatter: {},
+ }, { sourceId: id });
+ }
+ async function extractedRows(sourceId: string): Promise<number> {
+ const rows = await engine.executeRaw<{ n: number }>(
+ `SELECT count(*)::int AS n FROM facts WHERE source_id = $1 AND source = 'cli:extract-conversation-facts'`, [sourceId]);
+ return Number(rows[0].n);
+ }
+ async function completedPages(sourceId: string): Promise<number> {
+ const rows = await engine.executeRaw<{ n: number }>(
+ `SELECT count(DISTINCT source_markdown_slug)::int AS n FROM facts WHERE source_id = $1 AND fact = 'EXTRACTION_COMPLETE'`, [sourceId]);
+ return Number(rows[0].n);
+ }
+
+ test('a per-source walltime abort reports the page that finished, and the next run resumes after it', async () => {
+ await seedTwoPageSource('src-7068-walltime');
+ await engine.setConfig('cycle.conversation_facts_backfill.max_walltime_min', '0.05');
+ stallCall = { n: 2 };
+ const first = await runPhaseConversationFactsBackfill(engine);
+ const source = (first.details.per_source as Record<string, { walltime_exhausted?: boolean; pages_processed: number; facts_inserted: number }>)['src-7068-walltime'];
+ expect(source.walltime_exhausted).toBe(true);
+ expect(await extractedRows('src-7068-walltime')).toBe(1);
+ expect(await completedPages('src-7068-walltime')).toBe(1);
+ expect(source.pages_processed).toBe(1);
+ expect(source.facts_inserted).toBe(1);
+ expect(first.details.facts_inserted).toBe(1);
+
+ stallCall = null;
+ const second = await runPhaseConversationFactsBackfill(engine);
+ expect(second.details.pages_skipped_completed).toBe(1);
+ expect(second.details.pages_processed).toBe(1);
+ expect(await completedPages('src-7068-walltime')).toBe(2);
+ }, 120_000);
+
+ test('a caller abort still rejects the phase after a page committed', async () => {
+ await seedTwoPageSource('src-7068-cancel');
+ const caller = new AbortController();
+ stallCall = { n: 2, onStall: () => caller.abort() };
+ const outcome = await runPhaseConversationFactsBackfill(engine, { signal: caller.signal }).then(() => 'resolved', (err: Error) => err.name);
+ expect(outcome).toBe('AbortError');
+ expect(await extractedRows('src-7068-cancel')).toBe(1);
+ }, 120_000);
+});
+
diff --git a/test/managed-facts-writers.test.ts b/test/managed-facts-writers.test.ts
index 1b92a1efc3..62ae170b27 100644
--- a/test/managed-facts-writers.test.ts
+++ b/test/managed-facts-writers.test.ts
@@ -31,7 +31,7 @@ import { submitPageMutation } from '../src/core/persistence/page-mutations.ts';
import { disposePersistenceConsumer } from '../src/core/persistence/service.ts';
import { configureGateway, resetGateway, __setChatTransportForTests, __setEmbedTransportForTests, type ChatResult } from '../src/core/ai/gateway.ts';
import { runExtractFacts } from '../src/core/cycle/extract-facts.ts';
-import { runExtractConversationFactsCore, runExtractConversationFacts } from '../src/commands/extract-conversation-facts.ts';
+import { runExtractConversationFactsCore, runExtractConversationFacts, type ExtractConversationFactsResult } from '../src/commands/extract-conversation-facts.ts';
import { runPhaseConversationFactsBackfill } from '../src/core/cycle/conversation-facts-backfill.ts';
import { writeFactsToFence } from '../src/core/facts/fence-write.ts';
import { writeSingleFact } from '../src/core/facts/write-single.ts';
@@ -532,6 +532,40 @@ test('managed writeSingleFact reports the entity it persisted for an unresolved
});
}, 120_000);
+test('#7068: an interrupted managed extraction hands back the counts of the pages it published', async () => {
+ const later = `---\ntitle: Synthetic follow-up\ntype: conversation\n---\n${[
+ '**Bob Demo** (2024-03-16 9:00 AM): Did the move to the platform team go through?',
+ '**Alice Example** (2024-03-16 9:01 AM): Yes, I start on Monday.',
+ ].join('\n')}\n`;
+ await managed(async ({ put }) => {
+ await put('conversations/first-chat', CONVERSATION);
+ await put('conversations/second-chat', later);
+ }, async ({ engine, sourceId }) => {
+ const caller = new AbortController();
+ const replies = ['Alice Example signed an offer with Acme Corp.', 'Alice Example starts on the platform team on Monday.'];
+ let call = 0;
+ const factFor = (text: string) => [{ fact: text, kind: 'event' as const, entity_slug: null, confidence: 1,
+ notability: 'high' as const, source: 'test', visibility: 'private' as const }];
+ const interruptSecond = async () => {
+ if (++call === 2) { caller.abort(new Error('per-source walltime cap hit')); throw caller.signal.reason; }
+ return factFor(replies[0]!);
+ };
+ const handed: ExtractConversationFactsResult[] = [];
+ const thrown = await runExtractConversationFactsCore(engine, { sourceId, types: ['conversation'], sleepMs: 0,
+ extractor: interruptSecond, onInterrupted: partial => { handed.push(partial); } }, caller.signal).then(() => null, (err: Error) => err);
+ expect(thrown?.message).toBe('per-source walltime cap hit');
+ expect(handed).toHaveLength(1);
+ expect(handed[0]).toMatchObject({ pages_processed: 1, facts_inserted: 1, pages_failed: 0 });
+ expect((await facts(engine, sourceId, 'conversations/first-chat')).map(f => f.fact)).toEqual([replies[0], 'EXTRACTION_COMPLETE']);
+ expect(await facts(engine, sourceId, 'conversations/second-chat')).toEqual([]);
+
+ const resumed = await runExtractConversationFactsCore(engine, { sourceId, types: ['conversation'], sleepMs: 0,
+ extractor: async () => factFor(replies[1]!) });
+ expect(resumed).toMatchObject({ pages_skipped_completed: 1, pages_processed: 1, facts_inserted: 1 });
+ expect((await facts(engine, sourceId, 'conversations/second-chat')).map(f => f.fact)).toEqual([replies[1], 'EXTRACTION_COMPLETE']);
+ });
+}, 120_000);
+
test('managed extract-conversation-facts CLI extracts instead of refusing', async () => {
await managed(async ({ put }) => { await put('conversations/synthetic-chat', CONVERSATION); }, async ({ engine, sourceId }) => {
chatReply = JSON.stringify({ facts: [{ fact: 'Alice Example joined Acme Corp.', kind: 'event', entity: null, confidence: 1, notability: 'high' }] });